How does fugitive integrate with projectionist?

[mvanderkamp]

I have a bit of backstory here.

A few months ago I noticed that fugitive buffers were taking longer and longer to open. Turned out that the size of our repo was rapidly growing. But why was that a problem for Fugitive? I did some profiling and noticed that the vast majority of the time was being spent in one of two for-loops inside the TreeInfo function. The loops were running up to 80 thousand times per fugitive buffer. Doing a :Gdiffsplit could take up to 10 seconds:

76138              0.090380       for line in lines
76137              0.289002         let [info, filename] = split(line, "\t")
76137              0.464902         let [mode, sha, stage] = split(info, '\s\+')
76137              0.350460         let s:indexes[a:dir][1][stage][filename] = [newftime, mode, 'blob', sha, -2]
620384              1.026090         while filename =~# '/'
544247              4.434497           let filename = substitute(filename, '/[^/]*$', '', '')
544247              2.088378           let s:indexes[a:dir][1][stage][filename] = [newftime, '040000', 'tree', '', 0]
620384              0.356890         endwhile
76138              0.037660       endfor

Being naive and inexperienced with fugitive's code, I figured I'd tackle the problem right there in that function. I noticed that the "lines" being iterated came from the output of git ls-files, and that no path was being provided to the command. Yet, the culprit caller was PathInfo, which is only interested in a single path! Perfect! I'll just supply that path down into TreeInfo and provide at as an argument to git ls-files!

This actually worked. All fugitive buffers opened nice and quickly again. I left the change sitting in my local checkout for a few months, and never encountered a problem.

Cut to today, when I thought "okay maybe it's time to open a PR". I figured this was a higher bar than just whatever hacky nonsense I did for myself, and so I should really know exactly what's going on here. My first question: how often is PathInfo called, and what paths is it called on?

Turns out, the only calls were looking for the .projections.json file in various locations. A bit more digging showed that there's a suite of functions provided by fugitive that look like they're pretty much explicitly designed to be used by projectionist's nscall function, all of which share names with built-in functions. These are the functions stored in the g:io_fugitive dict, though that's not how projectionist accesses them.

I also noticed that it was possible to turn off projectionist-fugitive integration by doing let g:projectionist_ignore_fugitive = v:true though this option is undocumented, or let g:fugitive_file_api = v:false, another undocumented option. Either approach also resulted in a nice fast fugitive experience once again without having to apply my patch.

Now my questions, since this piqued my curiousity!

• Are there other plugins in the wild that use these "fugitive namespace" functions? To what benefit?
• Why does projectionist need fugitive-specific variants of some built-in functions? Is there some special integration between fugitive and projectionist?
• Is the g:projectionist_ignore_X option intentionally undocumented? I.e. is it safe for me to rely on setting this? Same question for g:fugitive_file_api.
• Would a PR to supply the path to git ls-files be a good idea? fugitive#glob also calls TreeInfo but it was less clear to me whether only providing a single path (or limited set of paths) would work there.

[tpope]

• Why does projectionist need fugitive-specific variants of some built-in functions? Is there some special integration between fugitive and projectionist?

To use Projectionist features inside Fugitive buffers. For example, you can :Gedit HEAD:file.c and then call :A to get to HEAD:file.h.

• Are there other plugins in the wild that use these "fugitive namespace" functions? To what benefit?

A few like rails.vim, rake.vim, and bundler.vim use it in basically the same way. Scriptease allows you to :source Fugitive buffers. Eunuch uses it to enable several operations on Fugitive buffers. For example, if you :Gdiffsplit and :Delete, that effectively does a git rm --cached.

This is hard to grep for so there's probably a couple of others I'm missing.

• Is the g:projectionist_ignore_X option intentionally undocumented? I.e. is it safe for me to rely on setting this? Same question for g:fugitive_file_api.

They're undocumented because I didn't want to bloat out the documentation with the multi-paragraph explanation they would need without knowing if there were actual users out there that would need them. They are safe to rely on. I would start with g:projectionist_ignore_fugitive because it's only things like hunting for .projections.json that should cause the problem.

• Would a PR to supply the path to git ls-files be a good idea? fugitive#glob also calls TreeInfo but it was less clear to me whether only providing a single path (or limited set of paths) would work there.

The result is cached at the repository level, so any fix would need to update the granularity of the cache as well. I didn't do it this way because for small repos, having one big call to ls-files performs much better than several small ones. But given the disastrous performance in the large repository case, I think it may be worth the trade-off. (Furthermore, this decision was made back when we were using system(). We now use jobs, which significantly decreases overhead, so the trade-off would be much more reasonable.)

And yeah, we would need a different approach for fugitive#glob(). Small silver lining there is we could probably use git ls-files :(glob)dir/*.whatever style pathspecs and clean up the clumsy regexp implementation. Projectionist only uses globs if you configure it to do so, so we could ignore this for now and still fix the problem for the typical use case.

Given the effort needed to diagnose the problem, I think fixing this is high value and worthy of prioritization. A good first step would be to implement this in s:PathInfo():

  if empty(path)
    let entry = [ftime, '040000', 'tree', '', -1]
  elseif commit =~# '^[0-3]$'
    " TODO
  else
    let [tree, ftime] = s:TreeInfo(dir, commit)
    let entry = get(tree, path, [])
  endif

[mvanderkamp]

Thanks for the super quick reply!

To use Projectionist features inside Fugitive buffers. For example, you can :Gedit HEAD:file.c and then call :A to get to HEAD:file.h.

That's a really cool feature! It hadn't occurred to me to even try that!

if you :Gdiffsplit and :Delete, that effectively does a git rm --cached.

Ahh that's neat.

They are safe to rely on. I would start with g:projectionist_ignore_fugitive

Perfect, thanks!

The result is cached at the repository level, so any fix would need to update the granularity of the cache as well.

I'm happy to take a crack at this unless you want to handle it yourself. You're referring to the s:indexes and s:trees caches here, right? Are you implying it will be necessary to change how these caches are structured, or would it be enough to just fill in the cache with paths as they are requested? E.g. use a condition like this in TreeInfo to check if an entry is already cached:

let stage_number = a:commit[-1:-1]
if get(index, 0, -1) < newftime || (!has_key(get(get(index, 1, {}), stage_number, {}), a:path))

[tpope]

Taking a crack at it yourself would be great. For my proposed first step, I would leave s:TreeInfo() untouched and start fresh. And I wouldn't bother adding a cache in the first prototype.

[mvanderkamp]

Ahh okay I follow now, thanks!