LEP: Changesets for unified working tree change tracking with first-class conflicts

[mjansson]

Summary

This proposal lets a single Lore instance carry several ongoing streams of work in one working tree. It introduces the changeset: a lightweight set of changes recorded as its own strictly linear line of history — a line that reduces to a single net change — layered over the branch the developer is on.

A developer can attach a changeset to materialize its changes in the working tree, detach it to park them off-disk, keep several attached at once, let an editor or tool checkpoint work into one continuously, and move a changeset as a unit — committing it onto the current branch, or switching to another to land it there.

One mechanism thereby covers a wide range of needs that today are unmet or served only by separate, partial workarounds: automatic personal backup of work, several work streams in flight at once, carrying work across branch switches, deferring a conflict to resolve in an orderly fashion rather than head-on, toggling changes on and off to test combinations, keeping long-lived debug or utility changes switchable, and picking up in-progress work on a developer's other machines.

Concretely, it folds three mechanisms Lore has today — dirty-file tracking, staging as the recording of commit intent, and the personal backup branch already used in UEFN — into a single concept, and on that foundation adds what Lore has never had: multiple parallel changesets in one working tree and first-class conflict handling.

[ahaczewski]

That's a very good proposal, and the way changesets are designed here solves so many Perforce issues, it is Unreal it can be done. Nice!

One nitpick: I think the proposal should explicitly state that binary or otherwise unmergeable files are changeset-membership atomic: a path may belong to at most one attached changeset at a time, hunk-level reassignment is unavailable, and any competing change to the same path is treated as an inter-changeset conflict requiring detach, reassign, or revert.

The other thing is I'm not entirely sure whether there already is a mechanism for classifying files as unmergeable, or how to tell it from the mergeable one.

[Stardidi]

Great read and excellent proposal! This is very much in line with the designs we had internally discussed. It's a great step in moving Lore beyond both Git and Perforce in terms of UX.

I wrote down some thoughts. Some were answered later in the proposal, but I left them in, in case it's useful for the proposal text itself.

Hopefully this wall of text is a useful feedback format. Let me know if I should format this differently.

Ground and base.

This paragraph confuses me. I think I understand the concept, but the words tripped me up (which is especially notable as the rest of the writing is excellent!). I think partially because it does not clearly explain what "base" is. I read "The base is [..] a changeset's latest", which seems incorrect :D

I am also a bit confused about the verbs. I think I'm understanding this correctly:
• dirty marks a file as changed and updates the changeset
• stage marks a file for including in a commit
• commit squashes a changeset (or all staged files) and places it onto a branch

But it seems like commit is also used for work-in-progress interactions with a changeset. Is that referring to the underlying changeset-is-really-a-branch-so-any-checkpoint-is-a-commit?

Not sharing changesets

We often find ourselves asking a colleague to 'unshelve' our changes on their machine. For example for soak testing, checking if it works in combination with their in-progress work, etc. How would this work with the current design? I would imagine this would be implemented as a higher-level feature; an additional changeset local to that developer, with metadata referring back to the original changeset and the latest sync point. Edit: I see this was addressed later. Having the link back to the original changeset would be cool to easily update from the original author. But that is an additional feature request.

Stacked reviews

Often when working on larger changes, I find myself wanting to split things up into several reviews where the later reviews are dependent on the earlier ones. With the traditional PR workflow, this is quite painful, as any feedback on the first review will require manual rebasing of all the later reviews. My holy grail for this workflow is GitButler, which allows frictionless moving of hunks between commits, reviews, and branches. How would this work for Changesets?
Edit: I see this is acknowledged in the Prior Art though StGit/Quilt/Sapling, but I don't see how changesets relate to this. As a changeset is always fully squashed before landing and only records the checkpoint history, they cannot be used for stacking changes or for organizing a series of commits. As I understand stacked branches and this proposal, changesets would fall just short of making this a well supported workflow. I think that's a shame, because it's so very close. Can expand in more details if this is an interesting topic.

No conflicts between changesets

In our own design of this same idea, we did allow for conflicts between changesets, but it imposed a strict ordering between the changesets. This would then enable stacked branches or the attaching of someone else's changeset even though it had overlap with your own. This may cause technical complexities that we didn't forsee, but it seemed useful to us.

Active changeset

I see that this is explicitly decided against, so offering this just as food-for-thought.
When working with multiple changelists in P4, everything ends up in Default and it then becomes a laboursome process to split things into distinct changes. We have internally discussed an approach with the concept of an 'active changeset' that you select before you start work. Any changes automatically go into that changeset. Any tooling can warn you if you are making edits to files/hunks that are already part of a different changeset. For example because you started on a bugfix in a separate changeset, but accidentally started editing a file that already had edits for the Feature changeset. JetBrains' Rider implements this. The problem with Rider is that the concept is local to one tool, so other tools still put files in the default changelist and you cannot rely on this. If Lore were to implement this concept, all tools would agree on what the 'active changelist' is. I could imagine this approach giving problems with parallel-agent workflows, where there is no longer one active changeset. There it would be useful for an agent to specify the active changeset for its process only.

As a related question, is it possible to assign files to a changeset when marking them as dirty? This is a common technique used in Perforce to track and group e.g. build artifacts during automated processes that run on the local machine.

This folds dirty tracking and the staged area into the changeset

It is unclear to me where hashing/content upload happens now. Does that become part of lore dirty? That would significantly increase the cost of informing lore of modifications. Or is that upload something that happens in the background? In that case we need some way of knowing what the state is of that background process.
Edit: I see that this is addressed later in "Intent-only capture and large files". I am not sure I like this size-based heuristic. It creates an inconsistent user experience, where part of a change is backed up, while another part is not, based on arbitrary criteria. I think a debounce/back-off/gradual retention approach for the changeset as a whole would make more sense to a user (it does to me, at least).

Additionally, I think users would want named snapshots. "This is where I tried the crazy refactor"; "Here it still builds"; etc.

Conflicts are first-class.

It sounds like conflicts are made 'first class' by storing the diff3 markers in the file. How does that work for binary files? We consider binary files mergable in some cases, but diff3 markers won't work for them. It would be a shame if Lore would treat binary conflicts as second-class.

Cheesy question: and what if I want to commit diff3 markers? For example for a unit test of a merge algorithm...

commit is refused until the developer syncs up to latest

Am I misunderstanding this, or does this mean --fast-forward-merge commits are no longer possible? That would be a complete showstopper for our workflows. We require users to sync to a revision that has build machine-produced artifacts.

Or is the expectation that you commit a changeset onto a temporary branch and then merge that branch into main? We would prefer to work closer to Trunk-based development, and it sounds like changesets are not compatible with that.

It's not obvious to me how committing a changeset is different from current commits. But then I'm not fully sure if the current commits can commit to non-latest.

Syncing individual files to latest

We often find ourselves syncing only a small set of files to the latest version for specific workflows. I believe Lore technically already supports this using lore sync --root-file, but I just wanted to check if there was any interaction there.

The default changeset is never discarded

Super unimportant, but I was curious: will that become a problem? Does it get reset at some point?

[mjansson]

@ahaczewski Agreed - unmergeable files should be forced to a single changeset, I will add that. The support for and details on how to mark and/or determine if files are unmergeable is orthogonal to this proposal - only that we will depend on such a feature to be added.

[mjansson]

@Stardidi thanks for the feedback!

I will clarify the terminology a bit, agree that it's a bit confusing as I'm trying to shoehorn the existing terminology into this concept. In short, "commit" is the action of squashing a changeset and commit that as a single revision on the target branch, "dirty" is the action of notifying lore that a file has changed and allow lore to record that change into the changeset (basically a revision on the changeset history), "stage" is the action of selecting a file from a changeset to be included when committing that changeset.

On sharing changesets, my idea was that this is what branches is for. If you want to collaborate with others, you create a branch for your shared work and commit your changeset(s) to that branch. The other collaborators are most likely not interested in all your incremental changes (basically auto-saves) on your changeset history lines, but the resulting state at some well known point when you're happy with the state.

Stacked reviews

Yeah, I see changesets mostly as your local organisation and incremental recording of your work as you iterate (automatic backups, persistent undo stack etc), then branches as the actual committed state for collaboration and review. So ideally stacked PRs/CRs is something we build on top of branches, and changesets is how you can easily partition, record and safely manage your work until you commit it to a branch for review.

Making it easy to attach/detach changesets to move work between branches, and easily move hunks and files between changesets is what makes local work easy to manage.

In our own design of this same idea, we did allow for conflicts between changesets, but it imposed a strict ordering between the changesets.

Interesting, maybe we could have a separate discussion around that at some point. I think it could be added as an extension to this base idea though, and not as an added complexity at this point?

Active changeset

My initial proposal floated around internally actually had the concept of an active changeset, but it was deemed an added complexity for little benefit and thus removed. I think it can also be added as an additional feature at a later stage, to reduce complexity of the initial proposal and implementation.

It is unclear to me where hashing/content upload happens now.

As default it happens when a file is dirtied and recorded in the changeset history line - this is the reconciliation with the UEFN personal backups. When you, a file watcher or an editor integration marks a file as dirty (modified) Lore will commit it on the changeset history and thus chunk/hash, store and optionally upload it.

That is why I think it is important to have good support for opt in/out of this for some file types. If you iterate on a large binary file you might not want to have automatic backups on every save. Agree on size being a blunt tool for this, but it's an easy first start. Interested in exploring the "debounce/back-off/gradual retention approach" as you mention.

Conflicts are first-class.

Exactly, lore can commit files in conflict in the changeset history. Text files will carry the conflict markers, and binary files will carry the mine/theirs/base variants - since we're content addressed, those are not actually taking up storage or have any cost of recording, it's just hash addresses.

So binary files will also be supported in this recorded conflict handling.