Skip to content

moar why #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 75 additions & 0 deletions content/why/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,10 +96,85 @@ There should be a self reinforcing energy to all of this: we want tools that wor
Strategy
--------

### familiar tools

Specifically, the primary target and scope can be summarized as “must support linux-like systems” and “must pass state primarily by familiar ‘file’-like abstractions”.
(In other words -- “it should taste like containers” -- this is a proven path to successful adoption that lets others use what they know and pivot towards our tooling incrementally. Incremental is good.)
This scope may extend to more systems in the future, but will start here.

### anything not everything

The Timeless Stack tools need to support *anything* -- anything you want to put in them, with no special consideration required -- rather than supporting "everything", but needing plugins to do it. This stands in contrast to most other build tools.

### breaking down the problem

There are five major sets of problems that need to be addressed:

1. Framing and hermetic execution and fileset-slinging core
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence doesn't make a lot of sense and needs more clarity.
What is framing? I'm going to make an assumption that it means the composability of file sets

Suggested change
1. Framing and hermetic execution and fileset-slinging core
1. Fileset transport: Content addressable filesets can be discovered, fetched, and verified.
2. Framing: File sets can be composed together into a working tree.
3. Hermetic execution: Transformations to the working tree are executed with minimal side-effects. The primary side-effect should be the specified file set export.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah something like this change is getting closer... the "slinging" part in particular is a hangover from an earlier flippant draft >.<

The idea I had in mind with "Framing" is to start building up a contrast between "build tools" and "framing tools" -- maybe "framing" aka "workspace" -- generally trying to get at the idea that we want to make general purpose contains-anything systems rather than getting into the nitty-gritty of every individual thing that gets built. Urban planning vs pouring concrete, maybe; some idea like that. So composing filesets is part of it, but I'd like to make it more general as well.

(Maybe this is a lot of work to try to make one word do, so early in this intro-audience document...)

I'll cook on this in the back of my mind for a while longer and then probably take most of this diff you propose, but if you get additional ideas, also jot em :)

2. Coordination systems for connecting graphs of executions and outputs -- both for authoring graphs of intent, and then precipitating graphs of actual outcome (which can be repeated).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would calling this "pipelines" be useful here? I feel it's a good word that people should know in the context of existing CI systems already in use.

3. Extended Coordination: these systems need to work for both directly-communicating people (e.g. projects and close-knit teams -- easy) and at large (e.g. uncoordinated swarms of projects at scale on github, etc -- harder!) without compromising on the Timeless philosophy.
4. Responsible Packaging -- community standards for how to prepare software to work well with immutable systems and minimal environment presumptions
5. Last-mile applications... such as CI, new "app stores", etc -- given the platform core, building these should become near-trivial!

Some of these are easier than others, e.g. Part 1 can be done nearly off-the-shelf with existing components.
Part 2 is something that simply needs doing.
Part 3 is pretty much a leading edge research problem in distributed workflow UX, but we're doing it anyway because it needs to be done.
Part 4 is a community standard thing rather than plain code, so either easier or harder depending on your point of view.
Part 5 is actually quite open-ended, and we expect to see lots of independent projects flourish there!

The most important thing is to identify and isolate these problems;
in identifying their scope, we make them tractable.

### composable tools

Each of the layers of problems should be solved by its own tooling.

These tools should work well together, but also be usable independently.

See the [Tools Reference](/tools) pages for an index of each of the separate tools we currently develop.

### API and checkpoint driven

Each of our tools should work declaratively, be easily driven by APIs (not just CLIs and sloppy glue),
and be "checkpoint driven" -- which means the data passed between the APIs should be possible to serialize,
and -- at that instant -- inspect it, save it, repeat it, shift it to another machine, whatever.
*Code* should not be passed between processes; documents should.

"API" should mean simple common formats -- JSON piped to `stdin` and JSON streamed to `stdout` is a solid API.

See the [design/layers](/design/layers/) page for more about our API layers.

### hash all the things

One key theme we'll find recur overall in the tactics so often that it might as well be called strategy:
**we're going to hash _everything_.**
Hashes let us build content-addressable systems, which work well in decentralized systems;
hashes give us ways to dodge naming problems which would otherwise introduce coordination issues, which makes interoperability easier even for people working independently;
and hashes create quick ways to detect if data is identical at massive scales, which is critical to reproducibility detection.
Since all three of these properties are so critically useful, executing strategic design with a "hashes first" philosophy gives the Timeless Stack a distinct style -- and a distinct advantage.


Tactics
-------

Tactics are reducing strategy to action.

For us, "tactics" are "tools".

- the `rio` tool provides content addressable file warehousing, using any of a variety of mechanisms, so long as they work by hash.
- the "formulas" specification provides a way to describe (and hash) a computation we want to explore.
- the "runrecords" specificaion provides a way to describe (and hash) the *results* of a formula.
- the `repeatr` tool provides hermetic isolation for executing other tools -- following a "formula", and reporting a "runrecord"! -- using containers to deliver it.
- the "modules" specification provides a way to describe graphs of computations, which pass data between them in discrete steps -- this enables larger scale work than formulas alone.
- the `reach` tool provides a way to execute modules easily.
- the "catalog" specifications, combined with standards for "replayable modules", establishes just enough standards for name->hash mappings to make them sharable -- while leaving just enough explicitly *unanswered* to spur the evolution of a clear separation between metadata circulation and version selection problems.
- the `reach` tool also bundles features for authoring catalogs -- both updating the name->hash mappings, attaching replay instructions to those, and sharing the whole thing with others.
- in ecosystem standards, the emphasis on non-turing-complete tools for templating and manipulating catalogs and modules results in a more inspectable and analyzable ecosystem even as it increases in size and scale and number of uncoordinated authors.

You can read more about these things in the rest of the site.

Some of these tools are complete! Some of them are less complete.
All of them are prototypes, and if in experimenting with them, we find that
they don't sufficiently fit the mission, we can replace them.

Contributors to all these things are welcome!