Frequent issues with .git/packed-refs

[avamsi]

I don't have a repro (I'll try to see if I can isolate something) but I'm on v0.27.0 and started running into this recently (perhaps from v0.26.0, in a couple colocated repos):

$ jj
Internal error: Unexpected error from backend
Caused by:
1: The lock for the packed-ref file could not be obtained
2: The lock for resource '[...]/.git/packed-refs' could not be obtained after 1.00s after 16 attempt(s). The lockfile at '[...]/.git/packed-refs.lock' might need manual deletion.

[zerowidth]

I ran into this error as well. After trying to run git operations a couple times, I deleted the lock file, and ended up with a corrupted/conflicted working copy and history. Unfortunately I'm not entirely sure what I did or I'd include it here.

[zerowidth]

Ok, I ran into this again today. I'd been making changes in my repo, then went to run jj commit -i. At that point it failed with the error, and continued to do so with any jj command (log, show, etc)

This could potentially be an interaction with gitstatusd used by my p10k zsh prompt plugin.

First, the error:

$ jj
Internal error: Unexpected error from backend
Caused by:
1: The lock for the packed-ref file could not be obtained
2: The lock for resource '/Users/nathan/redacted-repo/.git/packed-refs' could not be obtained after 1.00s after 15 attempt(s). The lockfile at '/Users/nathan/redacted-repo/.git/packed-refs.lock' might need manual deletion.

Checking open files:

$ lsof | grep .git | grep redacted-repo
zsh       48743 nathan  cwd       DIR               1,13       1440             9740171 /Users/nathan/redacted-repo
zsh       53128 nathan  cwd       DIR               1,13       1440             9740171 /Users/nathan/redacted-repo
gitstatus 53242 nathan  txt       REG               1,13     378176           534001790 /Users/nathan/redacted-repo/.git/objects/pack/pack-6042970ebdce3332581c0882681787b8c916fbe0.idx
gitstatus 53242 nathan  txt       REG               1,13      11068           533982427 /Users/nathan/redacted-repo/.git/objects/pack/loose-b9938f2abd8a03e066e8c88dbbcc26c1f7d1fe07.idx
gitstatus 53242 nathan  txt       REG               1,13     203792           531680877 /Users/nathan/redacted-repo/.git/objects/pack/pack-f7b14c33b796109bf376333e0b1d88406cace419.idx
gitstatus 53242 nathan  txt       REG               1,13     368180           533982437 /Users/nathan/redacted-repo/.git/objects/pack/pack-c6d0ae943db425273880f8d7806e2a06fb2f3b2d.idx
gitstatus 53242 nathan  txt       REG               1,13     361208           531680886 /Users/nathan/redacted-repo/.git/objects/pack/pack-856da67175a74a60d5340720b9a8a575369db7b4.idx
gitstatus 53242 nathan  txt       REG               1,13    2193401           531680876 /Users/nathan/redacted-repo/.git/objects/pack/pack-f7b14c33b796109bf376333e0b1d88406cace419.pack
gitstatus 53242 nathan  txt       REG               1,13    1580022           534001787 /Users/nathan/redacted-repo/.git/objects/pack/pack-6042970ebdce3332581c0882681787b8c916fbe0.pack
gitstatus 53242 nathan    4r      REG               1,13    2193401           531680876 /Users/nathan/redacted-repo/.git/objects/pack/pack-f7b14c33b796109bf376333e0b1d88406cace419.pack
gitstatus 53242 nathan    7r      REG               1,13    1580022           534001787 /Users/nathan/redacted-repo/.git/objects/pack/pack-6042970ebdce3332581c0882681787b8c916fbe0.pack
zsh       55148 nathan  cwd       DIR               1,13       1440             9740171 /Users/nathan/redacted-repo
gitstatus 55246 nathan  txt       REG               1,13     378176           534001790 /Users/nathan/redacted-repo/.git/objects/pack/pack-6042970ebdce3332581c0882681787b8c916fbe0.idx
gitstatus 55246 nathan  txt       REG               1,13     368180           533982437 /Users/nathan/redacted-repo/.git/objects/pack/pack-c6d0ae943db425273880f8d7806e2a06fb2f3b2d.idx
gitstatus 55246 nathan  txt       REG               1,13      11068           533982427 /Users/nathan/redacted-repo/.git/objects/pack/loose-b9938f2abd8a03e066e8c88dbbcc26c1f7d1fe07.idx
gitstatus 55246 nathan  txt       REG               1,13     203792           531680877 /Users/nathan/redacted-repo/.git/objects/pack/pack-f7b14c33b796109bf376333e0b1d88406cace419.idx
gitstatus 55246 nathan  txt       REG               1,13     361208           531680886 /Users/nathan/redacted-repo/.git/objects/pack/pack-856da67175a74a60d5340720b9a8a575369db7b4.idx
gitstatus 55246 nathan  txt       REG               1,13    2193401           531680876 /Users/nathan/redacted-repo/.git/objects/pack/pack-f7b14c33b796109bf376333e0b1d88406cace419.pack
gitstatus 55246 nathan  txt       REG               1,13    1580022           534001787 /Users/nathan/redacted-repo/.git/objects/pack/pack-6042970ebdce3332581c0882681787b8c916fbe0.pack
gitstatus 55246 nathan    5r      REG               1,13    2193401           531680876 /Users/nathan/redacted-repo/.git/objects/pack/pack-f7b14c33b796109bf376333e0b1d88406cace419.pack
gitstatus 55246 nathan    8r      REG               1,13    1580022           534001787 /Users/nathan/redacted-repo/.git/objects/pack/pack-6042970ebdce3332581c0882681787b8c916fbe0.pack

And you can see it's a child of my shell:

$ pstree -p 53242
-+= 00001 root /sbin/launchd
 \-+- 53160 nathan /opt/homebrew/bin/zsh -l
   \--- 53242 nathan /Users/nathan/.cache/gitstatus/gitstatusd-darwin-arm64 -G v1.5.4 -s -1 -u -1 -d -1 -c -1 -m -1 -v FATAL -t 20

The gitstatusd file has those files open, but I don't know how it would otherwise interact with them or the repo. It's possible this is unrelated! In any case, killing the processes and removing the lock file seems to have worked:

$ killall gitstatusd-darwin-arm64
gitstatus_query_p9k_:print:68: write error: broken pipe

$ jj show
# (same error)

$ rm .git/packed-refs.lock

$ jj show
Commit ID: 9c6c53b11d345171d07d0726657664f7fe394deb
Change ID: uqstpmlrputwswmznwyovtwuxsnyyvrk
...

If/when I see this again I'll try removing the lock file first, leaving those gitstatus processes in place.

EDIT: I also realized I have a git plugin in vscode that is watching this folder too, and vscode was open at the time. It didn't show up in lsof, though.

[arxanas]

Not sure if it's applicable here, but I believe one of my coworkers mentioned using auditd (on macOS?) to log all the accesses to a specific file in the background, which helped them debug their issue (unrelated to jj). Since packed-refs is at a known location, you might be able to use that and check the logs later.

[djmitche]

I'm seeing this as well. I do have a jj prompt defined, with probably irrelevant implementation below.

Details

``` ✺ djmitche@turing ~/go/src/api ⸨ql => djmitche/inf-3375* "Use Dockerfile builder …" w⸩ ⸩ type __jj_prompt __jj_prompt is a function __jj_prompt () { echo -n ' '"⸨$(__jj_log_1 @) => $(__jj_log_1 @-)"'⸩' } ✺ djmitche@turing ~/go/src/api ⸨ql => djmitche/inf-3375* "Use Dockerfile builder …" w⸩ ⸩ type __jj_log_1 __jj_log_1 is a function __jj_log_1 () { jj log -r$1 -n1 --ignore-working-copy --no-graph --color always -T ' separate(" ", bookmarks.join(" "), tags.map(|x| if( x.name().substr(0, 10).starts_with(x.name()), x.name().substr(0, 10), x.name().substr(0, 9) ++ "…") ).join(" "), surround("\"","\"", if( description.first_line().substr(0, 24).starts_with(description.first_line()), description.first_line().substr(0, 24), description.first_line().substr(0, 23) ++ "…" ) ), change_id.shortest(), if(empty, "(empty)"), if(conflict, "(conflict)"), if(divergent, "(divergent)"), if(hidden, "(hidden)"), ) ' } ```

I'm also using a complicated build system which is probably running read-only git commands during the build process, perhaps in parallel with other changes to the filesystem. But most importantly, I'm running watchman. This is on MacOS. I disabled watchman a week or so ago and this kind of error ceased, then accidentally re-enabled it yesterday and the errors are re-occurring. So I suspect some combination of all of these things is causing issues!

[supermarin]

Observed the same on two linux machines caused by syncthing syncing the repos between them.
It used to work just fine for months, then randomly occurred today. jj 0.36.0

[supermarin]

Randomly resolved itself as well 🥲