Skip to content

Initial KV store implementation#208

Merged
nrc merged 2 commits into
mainfrom
nrc/kv
Jun 3, 2026
Merged

Initial KV store implementation#208
nrc merged 2 commits into
mainfrom
nrc/kv

Conversation

@nrc
Copy link
Copy Markdown
Contributor

@nrc nrc commented May 27, 2026

Adds a new crate, ts_kv_store, with the first increment of a typed, in-memory KV store. So far what is implemented:

  • core storage, including typed-ness using macros, and concurrency control using a global lock. Both tabular and singleton data.
  • transactions
  • indexes on tables

I think this corresponds to the first two increments + indexes from the planning doc. Other than stuff discussed in meetings/on Slack, I don't think the implementation differs from the requirements or plan.

AI declaration: most tests are generated by Claude and reviewed by me. Everything else is written by human. I've had Claude review some of the code to help find bugs then addressed the bugs manually. I don't believe I actioned anything that wouldn't have been recommended by a human reviewer (but the code should absolutely still be reviewed by at least one human, sorry to that human in advance).

There are quire a few TODOs scattered around, I don't think any of them need fixing before landing. Some are things I intend to fix very soon, and some are longer term which I should turn into issues.

Closes #206, cc #169

Copy link
Copy Markdown
Member

@danderson danderson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling time on this first round of review. I didn't get all the way into the implementation of transactions and indexes, but I got through the schema and storage machinery.

My main thought is "wow that's a lot of type and lifetime parameters" 😂 I can see how they become necessary to make everything strongly typed, it just makes the logic hard for my tiny mind to follow. That said I think the worst of it ends up hidden way by the schema macros, so hopefully actually using a concrete store should be relatively pleasant.

I'll come back later for a second run at the remaining pieces.

Comment thread ts_kv_store/src/lib.rs Outdated
Comment thread ts_kv_store/src/lib.rs Outdated
//! into transactions which are atomic and serializable. Both singleton and tabular data can be part
//! of a transaction, and tables and transactions are orthogonal. Transactions may be read/write or
//! read-only. Transactions don't need to be explicitly committed - the effect is 'commit on drop'
//! but the implementation is that operations are individually committed.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand the implementation part of this statement. My reading of it is that it says that if dropped, the transaction applies its changes but does so in a non-transactional way, where readers can observe partial applications. Hopefully I'm wrong?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto — what happens on a panic unwind or error rethrow via ? — I assume we normally don't want that state to be committed

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread ts_kv_store/src/lib.rs Outdated
Comment thread ts_kv_store/src/lib.rs Outdated
Comment thread ts_kv_store/src/lib.rs Outdated
Comment thread ts_kv_store/src/schema.rs
Comment thread ts_kv_store/src/schema.rs
Comment thread ts_kv_store/src/schema.rs Outdated
Comment thread ts_kv_store/src/schema.rs
Comment thread ts_kv_store/src/schema.rs
Copy link
Copy Markdown
Collaborator

@npry npry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall I like the design — I may come back with more comments later but my brain is a bit fried from chasing stuff through in #209 / I assume we'll chat there first

Comment thread ts_kv_store/src/lib.rs Outdated
//! into transactions which are atomic and serializable. Both singleton and tabular data can be part
//! of a transaction, and tables and transactions are orthogonal. Transactions may be read/write or
//! read-only. Transactions don't need to be explicitly committed - the effect is 'commit on drop'
//! but the implementation is that operations are individually committed.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto — what happens on a panic unwind or error rethrow via ? — I assume we normally don't want that state to be committed

Comment thread ts_kv_store/src/lib.rs
Comment thread ts_kv_store/src/lib.rs
Comment thread ts_kv_store/src/lib.rs
Comment thread ts_kv_store/src/lib.rs
Comment thread ts_kv_store/src/schema.rs
Comment thread ts_kv_store/src/storage.rs
Comment thread ts_kv_store/src/operations.rs
Comment thread ts_kv_store/src/operations.rs
Comment thread ts_kv_store/src/operations.rs Outdated
@nrc nrc requested review from danderson and npry June 2, 2026 00:09
@nrc
Copy link
Copy Markdown
Contributor Author

nrc commented Jun 2, 2026

I've pushed with an update. It fixes all the minor comments. Other requested changes are in #216 and #217 (including #209, which I intend to review and merge after merging this one).

nrc added 2 commits June 3, 2026 08:03
Signed-off-by: Nick Cameron <nrc@ncameron.org>
Signed-off-by: Nick Cameron <nrc@ncameron.org>
Copy link
Copy Markdown
Collaborator

@npry npry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@nrc nrc merged commit 5079539 into main Jun 3, 2026
28 checks passed
@nrc nrc deleted the nrc/kv branch June 3, 2026 05:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KvStore: initial implementation

3 participants