Document Docs.rs architecture for contributors

[Kixiron]

Crates.io does a pretty good job of this with their docs folder and I think that it'd be really worth it to put down the hows and whys of docs.rs so that incoming contributors can get a jump start in understanding the codebase

[jyn514]

Just writing some notes up without making it as formal as a PR:

• There's three main modules:

  1. web
  2. docbuilder
  3. db
web is the frontend server, docbuilder is what builds the crates from crates.io, db interfaces with the database. There is also a utils module that is used everywhere and should probably be split up into the relevant modules

• The docbuilder uses rustwide for sandboxing and isolation. The build environment is rustops/crates-build-env, which is based on Ubuntu 20.04. crates-build-env is also used for crater. At its heart, docbuilder runs cargo doc --lib --no-deps, but there's lots of stuff in the middle. Because rustdoc is part of the compiler, it requires building each dependency as well, which can get pretty expensive in terms of time and CPU resources.

• The web frontend uses the iron framework, which is pretty out of date and we are thinking of replacing (RFC - Migrate to asynchronous code using Hyper #747). The main routes are /about, /releases/queue, and the homepage, as well as the documentation (of course), the sources for crates, and the build logs. See src/web/routes.rs for a full list.

• The db is Postgres (currently 10.12 in prod). This stores all the metadata for the files, but not the HTML files themselves - those are stored on AWS S3 in the rust-docs-rs bucket. Access to both the production server and the bucket are currently limited to co-leads. See src/db/migrate.rs for the schema.

• Tests are done using crate::test::wrapper, which sets up mocks for most things (although s3 gives up and uses min.io instead). There are not currently tests for building documentation, but there are tests for the frontend and some parts of the database. Builds are tested manually (using cargo run build crate <name> <version> or similar) and then have to be inspected manually as well by running a local web server (cargo run start-web-server). It would be great to automate these but I'm not sure how - crater has minicrater so maybe we could do something like that?

• The current main priorities are implementing compression (Compress html documents when sending them to the user #360, Compress documentation uploaded to S3 #379). Some other nice-to-haves are Feature request: list crate features at the top of package index.html #590, Search page shows latest chronological version, not latest semver #708, iPhone scrolling is painful #595.

[Kixiron]

Some additional things:

• I think that testing could honestly get its own section, even without a minicrater type of project, as it involves database, frontend, request and "normal" testing all at the same time
• Frontend should maybe get its own section, as even if it's not currently complex, in the future it may grow to something that's more separate of a component
• General developer documentation also needs to improve somewhat, especially on the windows side of things (Which I need to do sometime)
• I think utilizing github project boards would help current and incoming developers with prioritization, I know for me personally being able to see things physically sorted helps me think through things better (If you'd like, I can do this)
• Having a physical list of things we don't like and would like improved and/or researched would also be very helpful in my opinion, as it could help take some of the guess work out of figuring out long-term goals