Released Stacks v2

Author: ribejara-te

.gitignore

@@ -0,0 +1,7 @@
+# Python
+__pycache__
+
+# uv
+.python-version
+.venv
+build

README.md

@@ -5,102 +5,57 @@
 </div>
 
 
-## What is Stacks for Terraform?
-
-**Stacks** is a code pre-processor for Terraform. It implements a **sustainable scaling pattern**, **prevents drift** and **boilerplate**, all while **plugging into your already existing Terraform pipeline**.
-
-Stacks was initially presented at [SREcon23 Americas](https://www.usenix.org/conference/srecon23americas/presentation/bejarano).
-
-***Warning:** Stacks is under heavy development, many things may change.*
-
-
-## What is a "stack"?
-
-- A **stack** is a set of Terraform resources you want to deploy one or more times.
-- Each instance of a stack is a **layer**. A stack has one or more layers, hence, the name "stacks".
-
-### Example
-
-```
-vpc/
-│
-├── base/
-│   ├── vpc.tf
-│   └── subnets.tf
-│
-├── layers/
-│   ├── production/
-│   │   └── layer.tfvars
-│   └── staging/
-│       ├── layer.tfvars
-│       └── vpn.tf
-│
-└── stack.tfvars
-```
-
-- This is an example stack called `vpc`.
-- It contains a `base` folder, containing the common Terraform configuration scoped for all layers in this stack.
-- It contains a `layers` folder with two layers, one called `production` and one called `staging`. Layer directories contain layer-specific Terraform configuration.
-- Finally, it contains an optional `stack.tfvars` file, which defines variables global to all layers in the stack. These variables can be overriden at the layer level through a layer-specific `layer.tfvars`.
-
-
-## How does Stacks work?
-
-Stacks sits between you (the Terraform user) and Terraform. It's a **code pre-processor**.
-Here's an overview of Stacks inner workings:
-
-1. It takes your stack definitions (as shown above)
-1. For each layer:
-  1. Joins the `base` code with the layer-specific code
-  1. Applies a number of transformations
-  1. Injects some extra configuration
-  1. Bundles it up for Terraform to plan/apply on it
-
-
-## How to use Stacks?
-
-First, you need to put the Stacks code somewhere close to your stack definitions.
-Here's an example (not necessarily what we recommend):
-
-```
-your-terraform-repository/
-│
-├── src/                          # the contents of the `src` directory
-│   ├── helpers.py
-│   ├── postinit.py
-│   └── preinit.py
-│
-├── environments/                 # see the `example` directory on how to set this up
-│   ├── production/
-│   │   ├── backend.tfvars
-│   │   └── environment.tfvars
-│   └── staging/
-│
-└── stacks/                       # put your stack definitions here
-    └── vpc/                      # the `vpc` stack shown above
-        ├── base/
-        │   ├── vpc.tf
-        │   └── subnets.tf
-        ├── layers/
-        │   ├── production/
-        │   │   └── layer.tfvars
-        │   └── staging/
-        │       ├── layer.tfvars
-        │       └── vpn.tf
-        └── stack.tfvars
-```
-
-You can find [another example here](example/stacks/example) with all the appropriate file contents.
-
-Then you need to run Stacks in the layer you want to apply:
-```bash
-cd stacks/vpc/layers/production
-python3 ../../../../src/preinit.py
-cd stacks.out  # where the preinit output goes
-terraform init
-python3 ../../../../../src/postinit.py
-```
-
-Now you're ready to run any further `terraform` commands in the `stacks.out` directory.
-
-***Note:** we recommend putting `stacks.out` in `.gitignore` to prevent it from being tracked by git.*
+## What is Stacks?
+
+**Stacks** is a [Terraform](https://www.terraform.io/) code pre-processor.
+Its primary goal is to minimize your total Terraform codebase without giving up on coverage. To do more with less.
+
+As a code pre-processor, Stacks receives your "input code" and returns "output code" for Terraform to consume.
+
+Stacks was originally developed and continues to be maintained by the Infrastructure SRE team at [Cisco ThousandEyes](https://www.thousandeyes.com/).
+It was initially presented and open-sourced at [SREcon23 Americas](https://www.usenix.org/conference/srecon23americas/presentation/bejarano).
+
+You can read "Terraform" and "OpenTofu" interchangeably, Stacks works with both but we've chosen to go with "Terraform" for readability.
+
+The ["I am starting from scratch" quick-start guide](<2.2. I am starting from scratch.md>) is a good introduction to Stacks and what it does.
+
+
+## Documentation
+
+1. About
+    1. [Considerations before using](<docs/1.1. Considerations before using.md>)
+    2. [Stacks vs. its alternatives](<docs/1.2. Stacks vs its alternatives.md>)
+
+2. Quick-start guide
+    1. [Installation instructions](<docs/2.1. Installation instructions.md>)
+    2. [I am starting from scratch](<docs/2.2. I am starting from scratch.md>)
+    3. [I am collaborating to an existing stack](<docs/2.3. I am collaborating to an existing stack.md>)
+    4. [I am collaborating to Stacks itself](<docs/2.4. I am collaborating to Stacks itself.md>)
+
+3. Reference
+    1. Native features
+        1. [Global Terraform code](<docs/3.1.1. Global Terraform code.md>)
+        2. [Reusable root modules](<docs/3.1.2. Reusable root modules.md>)
+        3. [Jinja templating for Terraform](<docs/3.1.3. Jinja templating for Terraform.md>)
+        4. [Jinja templating for variables](<docs/3.1.4. Jinja templating for variables.md>)
+        5. [Remote lookup functions](<docs/3.1.5. Remote lookup functions.md>)
+        6. [Inline secret encryption](<docs/3.1.6. Inline secret encryption.md>)
+        7. [Automatic variable initialization](<docs/3.1.7. Automatic variable initialization.md>)
+    2. Features you can build with Stacks
+        1. [Terraform state backend configuration](<docs/3.2.1. Terraform state backend configuration.md>)
+        2. [Terraform provider generation](<docs/3.2.2. Terraform provider generation.md>)
+        3. [Input validation](<docs/3.2.3. Input validation.md>)
+    3. Command-line interface
+        1. [`stacks render`](<docs/3.3.1. stacks render.md>)
+        2. [`stacks terraform`](<docs/3.3.2. stacks terraform.md>)
+        3. [`stacks diff`](<docs/3.3.3. stacks diff.md>)
+        4. [`stacks encrypt`](<docs/3.3.4. stacks encrypt.md>)
+        5. [`stacks decrypt`](<docs/3.3.5. stacks decrypt.md>)
+        6. [`stacks surgery list`](<docs/3.3.6. stacks surgery list.md>)
+        7. [`stacks surgery import`](<docs/3.3.7. stacks surgery import.md>)
+        8. [`stacks surgery remove`](<docs/3.3.8. stacks surgery remove.md>)
+        9. [`stacks surgery rename`](<docs/3.3.9. stacks surgery rename.md>)
+        10. [`stacks surgery move`](<docs/3.3.10. stacks surgery move.md>)
+        11. [`stacks surgery edit`](<docs/3.3.11. stacks surgery edit.md>)
+    4. [Directory structure](<docs/3.4. Directory structure.md>)
+    5. [Special variables](<docs/3.5. Special variables.md>)

docs/1.1. Considerations before using.md

@@ -0,0 +1,31 @@
+# Considerations before using
+
+## Stacks is designed for monorepos
+
+Stacks is primarily designed to be used in Terraform monorepos.
+
+We define a Terraform monorepo as a single version control repository whose code maps to two or more Terraform states.
+You can have multiple Terraform monorepos as long as they all map to more than one state each.
+
+While perfectly possible to use in "1 repository = 1 state" setups, Stacks shines brightest when "1 repository = N states".
+
+If your setup is not monorepo-like, we do not recommend you use Stacks.
+
+## Not all Terraform automation tools support pre-processors like Stacks
+
+Stacks was originally developed to run on top of [Atlantis](https://www.runatlantis.io/).
+Atlantis does support code pre-processors in the form of [pre-workflow hooks](https://www.runatlantis.io/docs/pre-workflow-hooks.html#pre-workflow-hooks).
+
+Unfortunately, not all Terraform automation tools are flexible enough for you to run a thing that will modify your checked-out code before Terraform consumes it.
+
+Same restrictions apply to any code scanning tools you may be using. You'll have to put Stacks before them, which may not be possible depending on what continuous integration platform they're running on.
+
+If your Terraform automation pipeline does not support such code pre-processors, you cannot use Stacks.
+
+## You will not be able to use code formatters like `terraform fmt`
+
+Since not all Stacks' input code is valid HCL, formatters like `terraform fmt` will not work.
+
+You can still use code formatters in output code, but since its not meant to be persisted anywhere there's little to no reason to do that either.
+
+If format enforcing through code formatters is something you're not willing to give up, you cannot use Stacks.

docs/1.2. Stacks vs its alternatives.md

@@ -0,0 +1,47 @@
+# Stacks vs. its alternatives
+
+## Stacks vs. Terraform workspaces
+
+A Stacks stack is a Terraform root module which you can deploy as many times as you have layers.
+
+Terraform CLI [workspaces](https://developer.hashicorp.com/terraform/language/state/workspaces) let you deploy the same root module as many times as you have workspaces.
+
+Both allow you to inject different input variable values on different layers or workspaces, respectively.
+
+So it would seem like both are similar, however, the primary goal of Terraform workspaces is to [enable testing changes](https://developer.hashicorp.com/terraform/cli/workspaces#use-cases) on a separate state before modifying production infrastructure, and HashiCorp [explicitly recommends against](https://developer.hashicorp.com/terraform/cli/workspaces#when-not-to-use-multiple-workspaces) using workspaces for long-lived parallel deployments of the same root module.
+
+_HCP Terraform workspaces are a different feature to Terraform CLI workspaces, and do not compare with Stacks._
+
+## Stacks vs. Terragrunt
+
+[Terragrunt](https://terragrunt.gruntwork.io/) and Stacks achieve very similar results with very different strategies.
+
+Both enforce a specific directory structure on your repository.
+Both generate output code for Terraform to consume.
+
+Terragrunt adds an [extra layer of configuration](https://terragrunt.gruntwork.io/docs/getting-started/overview/#example) on top of Terraform which lets you define what code it generates.
+Terragrunt is heavily influenced by Terraform's [specifics](https://terragrunt.gruntwork.io/docs/features/state-backend/).
+It even has special features for [AWS](https://terragrunt.gruntwork.io/docs/features/aws-authentication/).
+
+Stacks is radically simpler in that it's mainly a thin layer of [Jinja](https://jinja.palletsprojects.com/en/stable/) on top of your Terraform code.
+So much so that you can probably use Stacks for other declarative purposes like generating [Kubernetes](https://kubernetes.io/) manifests for [`kubectl`](https://kubernetes.io/docs/reference/kubectl/) to consume, for example.
+
+_Terragrunt Stacks is a Terragrunt feature that does not compare with Stacks, and while the word "stacks" is overloaded, Stacks existence precedes that of Terragrunt Stacks._
+
+## Stacks vs. CDK for Terraform
+
+Both Stacks and [CDKTF](https://developer.hashicorp.com/terraform/cdktf) can be used to achieve the same results, but again with very different approaches.
+
+Where Stacks adds a thin layer of Jinja templating on top of the HCL you already know, CDKTF replaces HCL with one of the imperative programming language it supports.
+While that can be a good thing if what you want is limitless customizability of your infrastructure set based on imperative logic, we've found that very similar results can be achieved without the complexities CDK for Terraform comes with.
+
+## Stacks vs. Pulumi
+
+Everything we said above about CDK for Terraform can be said about Pulumi, as they're basically interchangable.
+
+## Terraform Cloud Stacks
+
+Terraform Cloud Stacks is a HCP Terraform feature that enables orchestrating the deployment of multiple interdependent root modules together.
+So while the names are the same, HCP Terraform Stacks does not compare with Stacks.
+
+_And about the word "stacks" being overloaded again: Stacks was initially released on March 2023, Terraform Cloud Stacks was announced later on October that year._

docs/2.1. Installation instructions.md

@@ -0,0 +1,43 @@
+# Installation instructions
+
+## 1. Install Python
+
+Stacks is written in Python, so it'll need a working [Python](https://www.python.org/downloads/) interpreter on your machine.
+
+We recommend the version specified in [pyproject.toml](../pyproject.toml).
+
+## 2. Install `pip` or `uv`
+
+Stacks is installable with both [`pip`](https://pypi.org/project/pip/) and [`uv`](https://docs.astral.sh/uv/).
+
+Choose one and install it. If you already have `pip` installed you can skip `uv`, otherwise we recommend the latter.
+
+## 3. Install Terraform
+
+Stacks requires [Terraform](https://developer.hashicorp.com/terraform/install) (or [OpenTofu](https://opentofu.org/docs/intro/install/)) to be installed on your machine.
+
+Stacks will use the binary in the `STACKS_TERRAFORM_PATH` environment variable, which defaults to `terraform` (so it'll look up `terraform` in `$PATH` and use that).
+
+If you use OpenTofu make sure to set `STACKS_TERRAFORM_PATH` to `tofu`.
+
+If `STACKS_TERRAFORM_PATH` is not in `$PATH`, you can also set `STACKS_TERRAFORM_PATH` to the absolute path of the binary you want to use (e.g.: `STACKS_TERRAFORM_PATH=/usr/bin/terraform`).
+
+## 4. Install Stacks
+
+To install Stacks using `pip`:
+```shell
+pip3 install --break-system-packages git+https://github.com/cisco-open/stacks.git
+```
+
+To install Stacks using `uv`:
+```shell
+uv tool install git+https://github.com/cisco-open/stacks.git
+```
+
+For development, we recommend you install Stacks from source:
+```shell
+git clone [email protected]:cisco-open/stacks.git
+cd stacks/
+uv tool install --editable .
+```
+The `--editable` flag allows you to try your changes right away without reinstalling `stacks`.