Add architecture guide for Machine ID (#15566)

* Add architecture guide for Machine ID

* Adjust indentation per linter rule

* Update docs/pages/machine-id/architecture.mdx

Co-authored-by: Paul Gottschling <[email protected]>

* Update docs/pages/machine-id/architecture.mdx

Co-authored-by: Paul Gottschling <[email protected]>

* Update docs/pages/machine-id/architecture.mdx

Co-authored-by: Paul Gottschling <[email protected]>

* Update docs/pages/machine-id/architecture.mdx

Co-authored-by: Paul Gottschling <[email protected]>

* Add section on joining and renewable certificates

* spag

* Add notes on bot creation, user and token.

* Clarify bot destination configuration

* Adjust indentation

* Start fleshing out notes on file permissions

* Finish filesystem permissions section

* Update docs/pages/machine-id/architecture.mdx

Co-authored-by: Tim Buckley <[email protected]>

* Address Tim's PR comments

* spelling correction

* Apply suggestions from code review

Co-authored-by: Paul Gottschling <[email protected]>

* Address a few docs structure comments

* Furth restructure architectural guide

* spag

* Avoid using the "static" terminology in reference to tokens.

* appease linter no-heading-punctuation

* Use paragraphs rather than ordered list for tbot actions

* Add notes on "destinations"

* Improve the way that the tbot section reads

* Add notes on daemon vs oneshot.

Co-authored-by: Paul Gottschling <[email protected]>
Co-authored-by: Tim Buckley <[email protected]>

docs/config.json

@@ -827,6 +827,10 @@
           "title": "Getting Started",
           "slug": "/machine-id/getting-started/"
         },
+        {
+          "title": "Architecture",
+          "slug": "/machine-id/architecture/"
+        },
         {
           "title": "Troubleshooting",
           "slug": "/machine-id/troubleshooting/"

docs/pages/machine-id/architecture.mdx

@@ -0,0 +1,178 @@
+---
+title: Machine ID Architecture
+description: How Teleport Machine ID works.
+---
+
+This section provides an overview of Teleport Machine ID's inner workings.
+
+A more in-depth specification of the workings of Machine ID can be found in
+[the Request For Discussion.](https://github.com/gravitational/teleport/blob/master/rfd/0064-bot-for-cert-renewals.md)
+
+## Bot creation
+
+In order to use the Machine ID agent (`tbot`), a bot must first be created using
+`tctl bots add`.
+
+A "bot" within Teleport is comprised of three resources, which are created by
+the Auth Service when `tctl bots add` is executed:
+
+- Bot user: this will be the user that the Machine ID agent authenticates as.
+- Bot role: the bot user is assigned the bot role, and the bot role contains
+  various permissions that the bot will need to function. For example, the
+  ability to watch the certificate authorities and the ability to 
+  [impersonate roles](#role-impersonation).
+- Token: for [onboarding](#onboarding), a token must exist that allows the
+  Machine ID agent to initially authenticate as the bot user. If an existing
+  token is not specified, then a single-use token will be created by the Auth 
+  Server.
+
+## Role Impersonation
+
+At the core of Machine ID is the concept of role impersonation.
+
+Role Impersonation allows a user to generate credentials with a set of requested
+roles. The user does not have to hold these roles, but must have been granted 
+permission to impersonate them. The impersonated credentials still include
+the username of the user that generated them, so actions can be attributed to 
+the user.
+
+These credentials can then be used to complete any action that is allowed by the
+role's configured permissions. 
+
+In the case of Machine ID, the bot user is assigned a bot role, which includes 
+permissions to impersonate the roles that the user has configured.
+
+## `tbot`
+
+`tbot` is the binary that acts as the agent for Machine ID on your machines that
+need access to resources protected by Teleport. It is typically ran in one of
+two modes. By default, it is a daemon-like long-running process. This is
+suitable for situations where your machine is long running and will need
+continous access to resources. `tbot` can also run in "oneshot" mode where it
+will fetch credentials for your machine once before exiting. This is ideal for
+short-lived environments such as CI/CD workflows.
+
+Before `tbot` can be started, at least two parts of configuration will need to
+be provided via the configuration file or as arguments provided to `tbot` when
+it is executed. This consists of:
+
+- [A join method](#onboarding) that the bot can use to prove that it should be
+  allowed to join the Teleport cluster.
+- A series of "destinations". A destination is a set of configuration that
+  specifies where a set of credentials should be output, and any options that
+  should be applied to those credentials (for example, what roles should be
+  impersonated).
+
+For more detail about the configuration options, see 
+[the reference.](./reference.mdx)
+
+On initial load, `tbot` uses the configured join method to obtain a set of 
+credentials for the bot user from the Teleport Auth Service. It can then use 
+these credentials to communicate with the Teleport Auth Service as the bot.
+
+Then on a configured regular period `tbot` begins its renewal process. It begins
+by refreshing the bot's own credentials, by renewing them or fetching a fresh 
+set of credentials depending on the configured onboarding method.
+
+For each destination provided in `tbot`'s configuration, `tbot` then uses
+impersonation to obtain credentials from the Auth Service for the roles
+specified in the destination's configuration. Once these are fetched, these are
+then persisted to the destination in various formats along with other useful
+matter such as the current certificate authority certificates.
+
+Concurrently to this, `tbot` monitors the Teleport certificate authorities to
+detect certificate rotations. When this occurs, it triggers additional renewals
+to ensure that destination's continue to have certificates that are signed by
+the latest certificate authority.
+
+## Onboarding
+
+Onboarding is the process by which `tbot` initially authenticates with the 
+Teleport Auth Service in order to receive credentials.
+
+Machine ID leverages the existing token resource within Teleport, with the
+token containing an additional `botName` field that identifies the bot user
+associated with the token. 
+
+Machine ID currently supports two methods of joining that have some key
+differences.
+
+### Ephemeral token
+
+- The name of the token is used as an opaque secret needed to join the Teleport 
+  cluster. This means it must be stored and communicated securely.
+- Once used, the token resource self-destructs. This means it can only be used 
+  to join a single bot to a Teleport cluster.
+- The certificates exchanged for the token are 
+  renewable, as we will explain in the next section.
+
+
+### Dynamic join tokens (e.g AWS IAM)
+
+- These tokens rely on an external authority that allows the bot to prove it is 
+  allowed to join the cluster. The name of the token identifies the Token
+  resource in Teleport that contains the configuration.
+- The token can be used to join as many bots as you want, and do not self
+  destruct in the same way that ephemeral tokens do.
+- The certificates exchanged for the token are not renewable. When the bot wants
+  to renew its certificates, it simply repeats the original join steps.
+
+Where possible, you should prefer to use a dynamic join token over an emphemeral
+token as this eliminates the need to handle a secret.
+
+## Renewable certificates
+
+Human Teleport users are not able to renew their credentials without
+re-authenticating. As Machine ID must work without manual human intervention,
+bots that have joined using ephemeral tokens are granted credentials that are
+marked as renewable. This means that the bot can use these credentials to
+fetch a new set of credentials with an expiry date further in the future.
+
+In order to mitigate the risk of bot user credentials being stolen, and then
+continually renewed by a malicious actor, renewable bot user certificates
+include a **generation counter**.
+
+The generation counter is stored against the user in the database and within the
+certificate. This counter is incremented each time the user renews their 
+certificate. When a bot attempts to renew, the Auth Server ensures that the 
+value within the certificate and in the database match. If they do not match,
+then the bot user is automatically locked. This means that if certificates are
+stolen, and attempted to be renewed whilst the bot is still running, the next
+renewal will render them useless.
+
+## File permissions
+
+There are two types of folder in use by `tbot`:
+
+- The bot's own files: these store credentials belonging to the `tbot` process
+  itself. As these credentials are potentially renewable, and will allow the
+  impersonation of any roles you have assigned to the bot user, they should be
+  treated as exceptionally sensitive. The bot's own files are stored by default at 
+  `/var/lib/teleport/bot/`.
+- Directory destinations: when a directory destination is configured, the bot 
+  outputs the role impersonated credentials as files in the specified directory.
+
+It is important that we ensure that these files can only be accessed by the
+fewest number of Linux processes and users on the system as is necessary.
+
+In the case of the bot's own files, it is best practice to create a Linux user
+specifically for running `tbot` and to ensure that only this user has access
+to this directory.
+
+In the case of directory destinations, the process the bot runs as requires read
+and write permissions, and processes that will need the credentials output by 
+the bot require read permissions. We recommend that you create a Linux user 
+specific to the process that needs to access these files. When using 
+`tbot init`, specify this Linux user as the "reader" to grant it access to the
+destination.
+
+In addition to basic POSIX filesystem permissions, `tbot init` also sets up
+Linux ACLs if the system supports it. This allows more granular control by
+granting individual users access.
+
+Finally, on systems that support it, `tbot` will by default attempt to prevent 
+the resolution of symbolic links when reading and writing files. This prevents a 
+class of attacks sometimes known as 
+[symlink attacks](https://capec.mitre.org/data/definitions/132.html). This 
+behaviour can be disabled using the `insecure` symlink option when configuring 
+your destination.

docs/pages/machine-id/introduction.mdx

@@ -60,6 +60,7 @@ Let's create a bot and use the machine identity to connect to a server.
 ## Getting started
 
 - [Getting started](./getting-started.mdx): Getting started with Teleport Machine ID
+- [Architecture](./architecture.mdx): A technical overview of how Machine ID works
 
 ## Guides