Add an Inside Rust blog post for stage 0 std redesign

[jieyouxu]

Tip

There are intentionally a lot of WIP intermediate commits for review purposes, I will rebase once the content is ready.

Accompanying blog post for stage 0 std redesign: rust-lang/rust#119899

r? @onur-ozkan (and @Mark-Simulacrum)
cc @rust-lang/bootstrap (significant stage 0 bootstrap changes)

Rendered

[senekor]

Yes, the path key is required. The format has to be "YYYY/MM/DD/whatever-you-want".

I'll try to improve the templates so they produce better error messages in this case.

[senekor]

Oh if it's an inside-rust post, that should be in the path as well. So: "inside-rust/YYYY/MM/DD/whatever".

If the date of publication is not yet known, it's recommended to use the placeholder "9999/12/31" as the date, CI will prevent that from being published accidentally.

I can definitely improve docs/guidance/tooling/error msgs here.

[jieyouxu]

FIXME(jieyouxu): update this date when we're ready

[onur-ozkan]

Great work!

Just two suggestions in addition to my minor notes:

1. It'd be helpful to mention the implementation PR.

2. Consider showing the staging diagram to illustrate how things looked before and how they look now (the updated version is in the implementation PR).

[jieyouxu]

It'd be helpful to mention the rust-lang/rust#119899.

I have no idea how I didn't include that when I even remember telling myself not to forget that lmao

[jieyouxu]

Changes since last review (in round 2 222f4fe):

• Added a comparison diagram showing difference before/after the redesign.
• Addressed nits.
• Actually link to the stage 0 redesign PR itself.

[BoxyUwU]

Some random thoughts.

In general I feel like this post doesn't do too much to explain why the new system is a clearer/simpler mental model. And I think that does a bit of a disservice to all of the (great) explanation of what that actually means in terms of changes to building the compiler/library.

It's almost like the post frames this as an arbitrary change in what depends on what because it just so happens to make std's life simpler. When, as far as I can tell, the new system is just generally more coherent and how you would expect things to work.

E.g. generally when a rust crate is built by compiler X it also depends on the standard library built and shipped with/by X. With the previous bootstrapping setup the compiler was actually depending on some weird frankenstein std that was built by the beta compiler but wasn't shipped with it.

Similarly the inconsistency between "build the stage0 compiler" and "build the stage0 std library" was weird and has now been resolved (stage0 compiler is beta compiler, stage0 library was a weird frankenstein std but is now the beta std), but doesn't really seem to be mentioned anywhere in this post.

I think it would be a lot easier to understand what has changed if the post started with the motivation and talked about the shift in mental model to something that's generally more coherent. It's hard to fully understand what has changed without talking about how the change fits in with the wider "heres how things normally work".

Generally that's what my review comments are trying to get at I think 🤔 Small changes that would help build the right mental model for readers- but realistically it should probably be all together in one upfront paragraph for the motivation.

[jieyouxu]

I tried to do another iteration of the blog post, the motivation bits I find is still not very convincing but idk how to make it so.

Significant changes:

• Added a TL;DR
• Moved motivation way earlier and
• Dropped the exact invocation tables because I found it more noise than signal.
• Added diagrams that I hope helps to explain the difference between current vs redesigned stage 0 bootstrap sequence

Known deficiencies:

• Motivation/justification still needs work
• There still isn't concrete examples as suggested, still have to think about it.

Suggestions / feedback welcomed.

[jyn514]

this is really good work. thanks for writing it up, i can tell you spent a lot of time on it.

[jieyouxu]

Addressed feedback in (round 5):

• Added a specific example re. the trait solving feature requiring addition of a new lang item.
• Fixed various wording and links.

[jieyouxu]

FIXME(jieyouxu): squash commits when ready

[Mark-Simulacrum]

It feels a little misleading to say that this is not the case for rustc because X when that X is not changing with this change -- both lang items and intrinsics are here to stay, and they're just as tightly coupled to a compiler version.

In fact, the reality that we can make this change implies this is basically an arbitrary choice for Rust: do we prefer cfg(bootstrap) in libstd or in librustc*? (Right?)

Not sure if there's a way to rephrase this to avoid that implication...

[jieyouxu]

In fact, the reality that we can make this change implies this is basically an arbitrary choice for Rust: do we prefer cfg(bootstrap) in libstd or in librustc*? (Right?)

Yeah, this is the tradeoff that we're changing here, to prefer the less-common cfg(bootstrap) for compiler instead of the more frquent cfg(bootstrap) for std.

It's definitely not a "pure absolute positive", but a design choice that tries to mitigate the overall churn.

[jieyouxu]

I haven't changed this in (round 6), not sure how to improve this wording.

[Mark-Simulacrum]

"not yet on beta" feels a bit misleading here -- what it really wants to say is anything in the standard library that has changed unstable API and is used in the compiler.

(Out of interest, how did we perform this analysis?)

[jieyouxu]

Right.

(Out of interest, how did we perform this analysis?)

I can't claim to have done this analysis myself (though it seems reasonably accurate to me). I believe @jyn514 did the analysis originally?

I might drop

By our count, the compiler has not used a feature added that recently since before 1.61.

[jieyouxu]

I dropped (in round 6)

By our count, the compiler has not used a feature added that recently since before 1.61.

and reworded to use

anything in the standard library that has changed unstable API and is used in the compiler.

But this comment isn't technically resolved re.

(Out of interest, how did we perform this analysis?)

[jyn514]

I can't claim to have done this analysis myself (though it seems reasonably accurate to me). I believe @jyn514 did the analysis originally?

see https://jyn.dev/bootstrapping-rust-in-2023/#what-can-we-do-about-it and in particular https://github.com/jyn514/rust/tree/versions-used

[Mark-Simulacrum]

I'm not sure what the intended goal for this post is -- are we seeking to tell people what they should expect to change in their workflows? Just trying to communicate that the change is happening? Something else?

If I've understood the content correctly, we're sort of just saying "hey stuff is going to look different" but I think we're not expecting local workflows to actually meaningfully change in terms of e.g. commands to run or similar? I wonder if making that explicit at the top of the post ("Changes are coming to how things are built, but the majority of contributors shouldn't see any meaningful impact to the commands they run. Please reach out in XXX if that is not the case for you").

[jieyouxu]

I'm not sure what the intended goal for this post is -- are we seeking to tell people what they should expect to change in their workflows? Just trying to communicate that the change is happening?

Yes, the intention is to better communicate the change is happening, and help prepare contributors for the differences instead of suddenly finding out staging is different.

If I've understood the content correctly, we're sort of just saying "hey stuff is going to look different" but I think we're not expecting local workflows to actually meaningfully change in terms of e.g. commands to run or similar?

The invocations might not be too different, but the actual changes (e.g. cfg(bootstrap) and stuff) will take some getting used to.

[Mark-Simulacrum]

I think I'd encourage structuring the post to highlight the actual impact we expect to people's workflows (e.g., cfg(bootstrap), different commands to run) up front. IOW, if I link someone this post, the likelihood that they have been impacted ("help I was doing X and that doesn't work anymore") seems high -- we should have a clear guidance on that somewhere for the cases we expect to hit. Maybe that's a FAQ on the dev guide or similar though, not this post, but I think that's largely missing in the current contents.

I think having the "mental model" explanation is good -- it helps people debug why things aren't working without needing a perfect match to a workflow we call out in the FAQ -- but I think both are needed.

[jieyouxu]

I'm not quite sure how to restructure the current contents tbh. I think this blog post might be better off as the "motivation" and "high-level overview of what is being changed". For specific breakages (and workflow changes), we probably need to document them in rustc-dev-guide. We technically can amend this blog post in the future for some more backlinks if needed. Would that work for you?

[Mark-Simulacrum]

Documenting in rustc-dev-guide and linking there seems good -- maybe something like "See this page for details on how you might need to change your workflow with these changes. This blog post focuses on motivation for the change and attempts to build a mental model for how the system works, rather than deep diving on workflow changes."

[jieyouxu]

Documenting in rustc-dev-guide and linking there seems good -- maybe something like "See this page for details on how you might need to change your workflow with these changes. This blog post focuses on motivation for the change and attempts to build a mental model for how the system works, rather than deep diving on workflow changes."

I'll leave a paragraph to that effect, and backlink to https://rustc-dev-guide.rust-lang.org/building/bootstrapping/what-bootstrapping-does.html for now (this page is updated in the stage 0 redesign PR, and it's possible to flush out more concrete examples on that page or backlinked from that page if necessary).

[jieyouxu]

I added this disclaimer and linked to the What bootstrapping does page (round 7). Let me know if that seems sufficient.

[Mark-Simulacrum]

That page doesn't seem to have any workflow details? I was expecting something like a page containing the lang item example ("> Example: Implementing a trait solving feature which requires adding core lang items") amongst other sample cases.

Perhaps that's a new page? I'm not sure where that example comes from, maybe we should just link to that :)

[jieyouxu]

Perhaps that's a new page? I'm not sure where that example comes from, maybe we should just link to that :)

To clarify, this is not a page that yet exists (we'll have to write it in the redesign PR).

This example is taken from the discussion in Jack's review comment earlier.

[jieyouxu]

Tried to address some more feedback in (round 6).

[Mark-Simulacrum]

Suggested change

	We are [redesigning the initial bootstrap sequence][stage0-redesign-pr] so that the standard library will now only support one compiler version. Building the in-tree compiler will no longer involve building the in-tree standard library. Instead, the pre-built beta standard library will be used.
	We are [redesigning the bootstrap sequence for the Rust toolchain][stage0-redesign-pr]. The standard library will move from supporting being built by both the previous toolchain version and the current version to only supporting the current version. This does not change the artifacts we distribute to end users of Rust.

[Mark-Simulacrum]

Suggested change

	Building a C++ compiler toolchain like [`gcc`][gcc-bootstrap] or [`clang`][clang-bootstrap] doesn't usually involving building their stand libraries from source, but instead will link against a pre-built standard library across all bootstrap stages. Typically, this specified standard library is the system standard library.[^adapted]
	Building a C++ compiler toolchain like [`gcc`][gcc-bootstrap] or [`clang`][clang-bootstrap] doesn't usually involving building their standard libraries from source, but instead will link against a pre-built standard library across all bootstrap stages. Typically, this specified standard library is the system standard library.[^adapted]

[Mark-Simulacrum]

Suggested change

	In the context of `rustc`, this is not the case, as the compiler and standard library are tightly coupled through *intrinsics* and *lang items*. Bootstrapping `rustc` involves building the standard library from source. Building a stage `N` `rustc` requires building or acquiring the stage `N - 1` `rustc`, as well as the stage `N - 1` standard library built by the stage `N - 1` `rustc`.
	The Rust toolchain as a whole, both currently and after this change, supports being built with just two minor versions of Rust: the previous version and its own version. For example, Rust 1.85.1 requires one of 1.84.0, 1.85.0, or 1.85.1 toolchains to build its source code into the 1.85.1 distributed artifacts.
	Rust has historically always built the majority of in-tree source code against the in-tree standard library's API. This meant that the in-tree copy of the standard library had to support being built with both the previous release (e.g., 1.84) and the new release (e.g., 1.85), requiring extensive `cfg(bootstrap)` annotations to be compatible with the unstable API of the compiler (primarily changes in intrinsics and lang items).
	The choice to require std to be built with two different compiler toolchains is largely historical. We don't have specific rationale available, but expect it to have made more sense when the standard library API was rapidly evolving and the compiler was unable to use crates.io dependencies for functionality.

Maybe something like this? It's much longer... feel free to pull parts of it out or otherwise restructure.

[jieyouxu]

Yeah thanks, I'll include this better explanation tmrw. I think it's good to have some more context.