Skip to content

Commit 57ecf91

Browse files
edmundmilleredmundmiller
committed
init: Add co2-reporting project
1 parent a54b424 commit 57ecf91

File tree

9 files changed

+712
-0
lines changed

9 files changed

+712
-0
lines changed

‎pulumi/co2_reports/.envrc‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Environment configuration for CO2 Reports Pulumi project
2+
# This file loads AWS credentials from 1Password
3+
4+
export OP_ACCOUNT=nf-core
5+
6+
# Load 1Password integration for direnv
7+
source_url "https://github.com/tmatilai/direnv-1password/raw/v1.0.1/1password.sh" \
8+
"sha256-4dmKkmlPBNXimznxeehplDfiV+CvJiIzg7H1Pik4oqY="
9+
10+
# Load AWS credentials from 1Password
11+
from_op AWS_ACCESS_KEY_ID="op://Dev/Pulumi-AWS-key/access key id"
12+
from_op AWS_SECRET_ACCESS_KEY="op://Dev/Pulumi-AWS-key/secret access key"
13+
14+
# AWS Configuration
15+
export AWS_REGION="eu-north-1"
16+
export AWS_DEFAULT_REGION="eu-north-1"
17+
18+
# Load 1Password service account token for Pulumi
19+
# from_op OP_SERVICE_ACCOUNT_TOKEN="op://Employee/doroenisttgrfcmzihhunyizg4/credential"
20+
21+
# Load Pulumi passphrase from 1Password
22+
from_op PULUMI_CONFIG_PASSPHRASE="op://Employee/Pulumi Passphrase/password"

‎pulumi/co2_reports/.python-version‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
3.12

‎pulumi/co2_reports/Pulumi.AWSMegatests.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
encryptionsalt: v1:Dd5GnLRGGJQ=:v1:HWX/n0HL3VxDJrWR:ouykGXCccBLXFd6kAKWW0uHByWK/qw==
2+
config:
3+
aws:region: eu-north-1
4+
github:owner: nf-core

‎pulumi/co2_reports/Pulumi.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
name: co2-reports
2+
runtime:
3+
name: python
4+
options:
5+
toolchain: uv
6+
virtualenv: .venv
7+
description: For hosting nf-core CO2 footprint reports
8+
config:
9+
pulumi:tags:
10+
value:
11+
pulumi:template: aws-python

‎pulumi/co2_reports/README.md‎

Lines changed: 192 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,192 @@
1+
# CO2 Reports - Pulumi Infrastructure
2+
3+
Infrastructure-as-Code for the nf-core CO2 footprint reports S3 bucket using Pulumi.
4+
5+
## Overview
6+
7+
This Pulumi project creates and manages AWS infrastructure for storing CO2 footprint reports generated by nf-test runs in the nf-core/modules repository.
8+
9+
**Created Resources:**
10+
11+
- 📦 S3 bucket `nf-core-co2-reports` (eu-north-1)
12+
- 👤 IAM user `nf-core-co2-reports-ci` with write access
13+
- 🔑 GitHub Actions secrets for nf-core/modules repository
14+
15+
## Quick Start
16+
17+
### Prerequisites
18+
19+
1. AWS credentials from 1Password (`Dev` vault → `Pulumi-AWS-key`)
20+
2. 1Password service account token
21+
3. uv installed (`brew install uv` or similar)
22+
23+
### Deployment
24+
25+
```bash
26+
# Set AWS credentials
27+
export AWS_ACCESS_KEY_ID="<from 1Password>"
28+
export AWS_SECRET_ACCESS_KEY="<from 1Password>"
29+
export AWS_REGION="eu-north-1"
30+
31+
# Optional: Set 1Password token (or script will prompt)
32+
export OP_SERVICE_ACCOUNT_TOKEN="<your token>"
33+
34+
# Run deployment script
35+
cd ~/src/nf-core/ops/pulumi/co2_reports
36+
./DEPLOY.sh
37+
```
38+
39+
### Manual Deployment
40+
41+
If you prefer manual control:
42+
43+
```bash
44+
cd ~/src/nf-core/ops/pulumi/co2_reports
45+
46+
# Initialize stack
47+
uv run pulumi stack init AWSMegatests
48+
49+
# Configure
50+
uv run pulumi config set aws:region eu-north-1
51+
uv run pulumi config set github:owner nf-core
52+
uv run pulumi config set pulumi-onepassword:service_account_token <TOKEN> --secret
53+
54+
# Deploy
55+
uv run pulumi preview # Preview changes
56+
uv run pulumi up # Deploy
57+
```
58+
59+
## Infrastructure Details
60+
61+
### S3 Bucket
62+
63+
- **Name**: `nf-core-co2-reports`
64+
- **Region**: `eu-north-1`
65+
- **Encryption**: AES256 server-side encryption
66+
- **Versioning**: Enabled
67+
- **Public Access**: Blocked
68+
- **Purpose**: Store CO2 footprint trace files from nf-test runs
69+
70+
### Report Organization
71+
72+
Reports are organized by:
73+
```
74+
s3://nf-core-co2-reports/
75+
└── modules/
76+
└── YYYY-MM-DD/
77+
└── branch-name/
78+
└── profile/
79+
└── shard/
80+
├── co2footprint_trace.txt
81+
└── co2footprint_trace_*.txt
82+
```
83+
84+
### IAM Permissions
85+
86+
The CI user has permissions for:
87+
- `s3:PutObject` - Upload reports
88+
- `s3:GetObject` - Download reports (for verification)
89+
- `s3:ListBucket` - List bucket contents
90+
- `s3:GetBucketLocation` - Get bucket region
91+
92+
### GitHub Secrets
93+
94+
The following secrets are automatically created in the `nf-core/modules` repository:
95+
96+
- `CO2_REPORTS_AWS_ACCESS_KEY_ID`
97+
- `CO2_REPORTS_AWS_SECRET_ACCESS_KEY`
98+
- `CO2_REPORTS_AWS_REGION`
99+
100+
## Usage in GitHub Actions
101+
102+
After deployment, update your GitHub workflow:
103+
104+
```yaml
105+
- name: Configure AWS credentials
106+
uses: aws-actions/configure-aws-credentials@v4
107+
with:
108+
aws-access-key-id: ${{ secrets.CO2_REPORTS_AWS_ACCESS_KEY_ID }}
109+
aws-secret-access-key: ${{ secrets.CO2_REPORTS_AWS_SECRET_ACCESS_KEY }}
110+
aws-region: ${{ secrets.CO2_REPORTS_AWS_REGION }}
111+
112+
- name: Upload CO2 footprint reports to S3
113+
run: |
114+
aws s3 cp co2footprint_trace.txt s3://nf-core-co2-reports/...
115+
```
116+
117+
## Management
118+
119+
### View Stack Outputs
120+
121+
```bash
122+
cd ~/src/nf-core/ops/pulumi/co2_reports
123+
uv run pulumi stack output
124+
```
125+
126+
### Update Infrastructure
127+
128+
```bash
129+
# Make changes to __main__.py
130+
# Preview changes
131+
uv run pulumi preview
132+
133+
# Apply changes
134+
uv run pulumi up
135+
```
136+
137+
### Destroy Infrastructure
138+
139+
```bash
140+
# ⚠️ WARNING: This will delete the bucket and all reports!
141+
uv run pulumi destroy
142+
```
143+
144+
## Project Structure
145+
146+
```
147+
co2_reports/
148+
├── Pulumi.yaml # Project configuration
149+
├── __main__.py # Infrastructure definition
150+
├── pyproject.toml # Python dependencies
151+
├── DEPLOY.sh # Deployment script
152+
├── README.md # This file
153+
└── .venv/ # Virtual environment
154+
```
155+
156+
## Troubleshooting
157+
158+
### Error: "operation error S3: GetObject"
159+
160+
This means Pulumi can't access the S3 backend. Ensure AWS credentials are set:
161+
162+
```bash
163+
export AWS_ACCESS_KEY_ID="<your key>"
164+
export AWS_SECRET_ACCESS_KEY="<your secret>"
165+
```
166+
167+
### Error: "No such bookmark: AWSMegatests"
168+
169+
The stack hasn't been initialized yet. Run:
170+
171+
```bash
172+
uv run pulumi stack init AWSMegatests
173+
```
174+
175+
### GitHub Secrets Not Created
176+
177+
Check that:
178+
1. The 1Password service account token is correct
179+
2. The GitHub token in 1Password has proper permissions
180+
3. The repository name is correct (`modules`, not `nf-core/modules`)
181+
182+
## Related Documentation
183+
184+
- [nf-co2footprint Plugin](https://github.com/nextflow-io/nf-co2footprint)
185+
- [GitHub Issue #9291](https://github.com/nf-core/modules/issues/9291)
186+
- [Pulumi AWS Documentation](https://www.pulumi.com/registry/packages/aws/)
187+
188+
## Support
189+
190+
For issues or questions:
191+
- Open an issue in [nf-core/modules](https://github.com/nf-core/modules/issues)
192+
- Contact the nf-core infrastructure team

‎pulumi/co2_reports/SETUP_INSTRUCTIONS.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# CO2 Reports Pulumi Setup Instructions
2+
3+
## Commands to run:
4+
5+
```bash
6+
# 1. Initialize the stack
7+
cd ~/src/nf-core/ops/pulumi/co2_reports
8+
uv run pulumi stack init AWSMegatests
9+
10+
# 2. Configure the stack
11+
uv run pulumi config set aws:region us-east-1
12+
uv run pulumi config set github:owner nf-core
13+
14+
# 3. Set the 1Password service account token (get this from 1Password)
15+
uv run pulumi config set pulumi-onepassword:service_account_token <YOUR_TOKEN> --secret
16+
17+
# 4. Preview the changes
18+
uv run pulumi preview
19+
20+
# 5. Deploy the infrastructure
21+
uv run pulumi up
22+
23+
# 6. Verify the GitHub secrets were created
24+
# Check in GitHub: https://github.com/nf-core/modules/settings/secrets/actions
25+
```
26+
27+
## What this will create:
28+
29+
1. S3 bucket: `nf-core-co2-reports`
30+
2. IAM user: `nf-core-co2-reports-ci`
31+
3. IAM policy for write access to the bucket
32+
4. GitHub Actions secrets in nf-core/modules:
33+
- CO2_REPORTS_AWS_ACCESS_KEY_ID
34+
- CO2_REPORTS_AWS_SECRET_ACCESS_KEY
35+
- CO2_REPORTS_AWS_REGION
36+
37+
## After deployment:
38+
39+
Update the GitHub workflow in nf-core/modules to use the new bucket and credentials.

0 commit comments

Comments
 (0)