From e3389e00e59ebc17fe117875d3511e9ca4c4443f Mon Sep 17 00:00:00 2001 From: Philipp Walz Date: Thu, 9 Apr 2026 15:58:42 +0200 Subject: [PATCH 1/4] Add workshop tutorial for building a Managed Service Provider on Codesphere - Introduced a comprehensive guide for creating a managed service provider using Stalwart Mail Server. - Detailed steps for setting up the environment, implementing a REST backend, and deploying on Codesphere. - Included sections on API exploration, service registration, and testing the service instance. - Provided troubleshooting tips and potential improvements for the backend implementation. --- WORKSHOP_TUTORIAL.md | 1164 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 1164 insertions(+) create mode 100644 WORKSHOP_TUTORIAL.md diff --git a/WORKSHOP_TUTORIAL.md b/WORKSHOP_TUTORIAL.md new file mode 100644 index 0000000..9d0ac0c --- /dev/null +++ b/WORKSHOP_TUTORIAL.md @@ -0,0 +1,1164 @@ +# Workshop: Building a Managed Service Provider on Codesphere + +> **Partner Days — Workshop 5: Managed Services** +> +> Duration: ~3 hours · Difficulty: Intermediate · Language: TypeScript / Node.js + +--- + +## What You'll Build + +By the end of this workshop you will have a **fully working managed service provider** registered in the Codesphere marketplace. Any Codesphere user on your instance can then open the service catalog, click "Stalwart Mailbox", fill in a few fields, and receive a complete, ready-to-use email account — with IMAP, SMTP, JMAP, webmail access, and the DNS records they need to configure their domain. + +Under the hood you will: + +1. Deploy a central **Stalwart Mail Server** instance on Codesphere. +2. Implement a **custom REST backend** (Node.js/TypeScript + Express) that wraps Stalwart's admin API and exposes it as a Codesphere Managed Service Adapter. +3. Write a **`provider.yml`** that describes your service in the Codesphere marketplace. +4. **Register** the provider via the Codesphere Public API. +5. **Book** a service instance through the Codesphere UI, verify it works, and iterate. + +``` + YOUR WORK TODAY + ┌───────────────────────────┐ + │ │ +┌─────────────┐ reconcile loop │ ┌─────────────────────┐ │ ┌──────────────────────┐ +│ Codesphere │ ◄────────────────►│ │ REST Backend │ │ ────────►│ Stalwart Mail │ +│ Platform │ HTTP REST │ │ (your code) │ │ Admin │ Server │ +│ │ │ │ POST/GET/PATCH/DEL │ │ API │ (shared instance) │ +│ • UI │ │ └─────────────────────┘ │ │ │ +│ • Public │ │ │ │ One server, many │ +│ API │ │ ┌─────────────────────┐ │ │ "logical tenants" │ +│ • Catalog │ │ │ provider.yml │ │ │ (mail accounts) │ +│ │ │ │ (your definition) │ │ └──────────────────────┘ +└─────────────┘ │ └─────────────────────┘ │ + │ │ + └───────────────────────────┘ +``` + +--- + +## Table of Contents + +- [Part 0 — Understand the Architecture](#part-0--understand-the-architecture) +- [Part 1 — Set Up Your Environment](#part-1--set-up-your-environment) +- [Part 2 — Explore the Stalwart Admin API](#part-2--explore-the-stalwart-admin-api) +- [Part 3 — Implement the REST Backend](#part-3--implement-the-rest-backend) +- [Part 4 — Deploy the Backend on Codesphere](#part-4--deploy-the-backend-on-codesphere) +- [Part 5 — Write the provider.yml](#part-5--write-the-provideryml) +- [Part 6 — Register the Provider](#part-6--register-the-provider) +- [Part 7 — Book & Test a Service Instance](#part-7--book--test-a-service-instance) +- [Part 8 — Debug & Improve](#part-8--debug--improve) +- [Part 9 — Wrap-Up: What Users Get](#part-9--wrap-up-what-users-get) +- [Appendix A — Stalwart API Quick Reference](#appendix-a--stalwart-api-quick-reference) +- [Appendix B — Gotchas & Troubleshooting](#appendix-b--gotchas--troubleshooting) +- [Appendix C — JMAP Email Sending](#appendix-c--jmap-email-sending) + +--- + +## Part 0 — Understand the Architecture + +### Why a Custom REST Backend? + +Codesphere's managed services support two backend types: + +| Type | How it works | Best for | +|------|-------------|----------| +| **Landscape-based** | Codesphere deploys a full landscape (containers, services) per booked instance | One deployment = one instance (e.g., a Mattermost server per customer) | +| **REST-based** | Codesphere calls your REST API; you decide what happens | One shared service, many logical tenants (e.g., one mail server, many mailboxes) | + +Stalwart Mail Server is a **single service** that hosts many mail accounts. We don't want to spin up a separate mail server for every user who books the service — that would be wasteful. Instead, we want to: + +- Run **one** Stalwart instance centrally. +- When a user "books" the service, create a **new mail account** (a "logical tenant") on that shared server. +- When they delete the service, remove the account. +- Expose connection details (IMAP host, SMTP host, JMAP endpoint, DNS records) back to the user. + +This is exactly the pattern a **custom REST backend** is designed for. + +### The Reconciliation Loop + +Codesphere does **not** make a single fire-and-forget API call. Instead, it runs a **reconciliation loop** that continuously ensures reality matches the desired state: + +``` +User clicks "Create" + │ + ▼ +Codesphere stores desired state (plan, config, secrets) + │ + ▼ +Reconciler calls ──► POST / (your backend creates the account) + │ + ▼ +Reconciler polls ──► GET /?id= every ~10 seconds + Your backend returns { details: { ready: true, ... } } + │ + ▼ +Codesphere shows connection details to the user +``` + +If the user changes config → `PATCH /{id}`. If they delete → `DELETE /{id}`. Your backend is the **adapter** between Codesphere's generic managed-service lifecycle and Stalwart's specific admin API. + +### The Four Endpoints You Must Implement + +| Method | Path | When Codesphere calls it | Your backend should | +|--------|------|--------------------------|---------------------| +| `POST /` | User creates a new service instance | Create a mail account on Stalwart | +| `GET /?id=` | Reconciler polls for status (~every 10s) | Return plan, config, and connection details | +| `PATCH /:id` | User updates config or secrets | Update the mail account on Stalwart | +| `DELETE /:id` | User deletes the service | Delete the mail account on Stalwart | + +--- + +## Part 1 — Set Up Your Environment + +### Prerequisites + +| What | Why | +|------|-----| +| A Codesphere account on the workshop instance | Deploy workspaces, register providers | +| `CODESPHERE_API_TOKEN` | Authenticate with the Codesphere API ([create one in user settings](https://docs.codesphere.com/api)) | +| Git + GitHub access | Clone the template repos | +| Node.js 18+ (or use the Codesphere workspace) | Run the REST backend | + +### Step 1.1 — Clone the Template Repos + +You'll work with two repositories: + +1. **`stalwart-provider`** — The managed service provider template (your REST backend + provider.yml). This is the main repo you'll be editing. +2. **`stalwart-deployment`** — A ready-made Codesphere landscape that deploys the Stalwart Mail Server itself (already set up for this workshop). + +```bash +# Clone the provider template — this is your main working repo +git clone https://github.com/codesphere-cloud/stalwart-provider.git +cd stalwart-provider +``` + +### Step 1.2 — Familiarize Yourself with the Structure + +``` +stalwart-provider/ +├── config/ +│ ├── provider.yml # ← You'll create this (service definition) +│ ├── provider.yml.example # Landscape-based example +│ └── provider.rest.yml.example # REST-based example (start from this) +├── src/ +│ └── rest-backend/ +│ ├── server.js # ← Reference implementation (your starting point) +│ ├── package.json +│ └── Dockerfile +├── docker-compose.local.yml # Local Stalwart for development +├── provider.yml # Root-level provider (used for git-based registration) +├── Makefile # validate / register / test +└── scripts/ + ├── validate.sh + ├── register.sh + └── test-provider.sh +``` + +### Step 1.3 — Verify the Stalwart Instance + +For this workshop, a central Stalwart Mail Server is already deployed and accessible. Verify you can reach it: + +```bash +# Replace with the actual workshop Stalwart URL and credentials +export STALWART_API_URL="https://" +export STALWART_ADMIN_TOKEN="admin:" + +# Test the admin API +curl -s "$STALWART_API_URL/api/principal" \ + -u "$STALWART_ADMIN_TOKEN" | python3 -m json.tool + +# Expected: {"data": {"items": [...], "total": ...}} +``` + +> **💡 Local development alternative:** If you prefer to run everything locally first, start a local Stalwart using the included Docker Compose file: +> +> ```bash +> docker compose -f docker-compose.local.yml up -d +> # Admin UI: http://localhost:1080 (admin / localdev123) +> export STALWART_API_URL="http://localhost:1080" +> export STALWART_ADMIN_TOKEN="admin:localdev123" +> ``` + +--- + +## Part 2 — Explore the Stalwart Admin API + +Before writing the adapter, let's understand the API we're wrapping. Stalwart manages everything through **principals** — entities like domains, users, groups, and mailing lists. + +### 2.1 — Create a Domain + +Every email address needs a domain. Domains are principals of type `"domain"`: + +```bash +curl -s -X POST "$STALWART_API_URL/api/principal" \ + -u "$STALWART_ADMIN_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "type": "domain", + "name": "workshop.example.com", + "description": "Workshop demo domain" + }' | python3 -m json.tool + +# Success: {"data": } +``` + +### 2.2 — Create a User (Individual Principal) + +Now create a mail account on that domain: + +```bash +curl -s -X POST "$STALWART_API_URL/api/principal" \ + -u "$STALWART_ADMIN_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "type": "individual", + "name": "alice", + "secrets": ["supersecret123"], + "emails": ["alice@workshop.example.com"], + "description": "Alice (Workshop)", + "quota": 524288000, + "roles": ["user"], + "lists": [], + "memberOf": [], + "members": [], + "enabledPermissions": [], + "disabledPermissions": [], + "urls": [], + "externalMembers": [] + }' | python3 -m json.tool + +# Success: {"data": } +``` + +> **⚠️ Key gotcha:** Stalwart returns HTTP 200 even for errors! An error looks like: +> ```json +> {"error": "notFound", "item": "workshop.example.com"} +> ``` +> You must always parse the response body and check for an `error` field. + +### 2.3 — Update a User (PATCH with Action Array) + +Stalwart uses a unique action-array format for updates — **not** a flat JSON object: + +```bash +curl -s -X PATCH "$STALWART_API_URL/api/principal/alice" \ + -u "$STALWART_ADMIN_TOKEN" \ + -H "Content-Type: application/json" \ + -d '[ + {"action": "set", "field": "description", "value": "Alice M. Doe"}, + {"action": "set", "field": "quota", "value": 1073741824} + ]' | python3 -m json.tool +``` + +> **⚠️ Key gotcha:** Use the **username string** (`alice`) in the path, not the numeric ID returned by create. + +### 2.4 — Delete a User + +```bash +curl -s -X DELETE "$STALWART_API_URL/api/principal/alice" \ + -u "$STALWART_ADMIN_TOKEN" | python3 -m json.tool +``` + +### 2.5 — Fetch DNS Records for a Domain + +Stalwart auto-generates the DNS records (MX, SPF, DKIM, DMARC) your users need to configure: + +```bash +curl -s "$STALWART_API_URL/api/dns/records/workshop.example.com" \ + -u "$STALWART_ADMIN_TOKEN" | python3 -m json.tool + +# Returns: {"data": [{"type": "MX", "name": "...", "content": "..."}, ...]} +``` + +### 2.6 — Discover JMAP Session Details + +JMAP (JSON Meta Application Protocol) is the modern email API. After creating a user, you can discover their JMAP session: + +```bash +# Get the JMAP session (provides accountId) +curl -s "$STALWART_API_URL/jmap/session" \ + -u "alice:supersecret123" | python3 -m json.tool + +# The accountId is at: .primaryAccounts["urn:ietf:params:jmap:mail"] +``` + +> **Clean up** before moving on: +> ```bash +> curl -s -X DELETE "$STALWART_API_URL/api/principal/alice" -u "$STALWART_ADMIN_TOKEN" +> curl -s -X DELETE "$STALWART_API_URL/api/principal/workshop.example.com" -u "$STALWART_ADMIN_TOKEN" +> ``` + +--- + +## Part 3 — Implement the REST Backend + +This is the core of the workshop. You will build a Node.js/Express application that implements the Codesphere Managed Service Adapter API and translates each call into the appropriate Stalwart admin API operations. + +### 3.1 — Project Setup + +```bash +cd src/rest-backend +npm install +``` + +The `package.json` already includes `express`. If you want to use TypeScript, add it now: + +```bash +npm install typescript ts-node @types/express @types/node --save-dev +npx tsc --init +``` + +### 3.2 — Architecture of Your Backend + +Your backend needs to handle this flow for each operation: + +``` +Codesphere POST / Your Backend Stalwart +───────────────── ──────────── ──────── +{id, config, secrets, plan} ──► 1. Validate input + 2. Ensure domain exists ──────────► POST /api/principal {type:"domain"} + 3. Create user ──────────► POST /api/principal {type:"individual"} + 4. Fetch DNS records ──────────► GET /api/dns/records/{domain} + 5. Fetch JMAP details ──────────► GET /jmap/session + POST /jmap/ + 6. Store mapping (id → username) + ◄── 7. Return 201 Created +``` + +### 3.3 — Key Implementation Details + +Here is what you need to implement, broken into logical pieces. The reference implementation in `server.js` is a working example — study it, then build your own version (or extend it). + +#### A. Stalwart API Helper + +Stalwart's error handling is unusual — always check the response body: + +```typescript +async function stalwartRequest(method: string, path: string, body?: any) { + const url = `${STALWART_API_URL}${path}`; + const token = STALWART_ADMIN_TOKEN; + const headers: Record = { + 'Content-Type': 'application/json', + // Auto-detect Basic vs Bearer auth + 'Authorization': token.includes(':') + ? `Basic ${Buffer.from(token).toString('base64')}` + : `Bearer ${token}`, + }; + const resp = await fetch(url, { + method, + headers, + body: body ? JSON.stringify(body) : undefined, + }); + return resp; +} + +async function parseStalwartResponse(resp: Response) { + if (!resp.ok) { + return { ok: false, error: `HTTP ${resp.status}: ${await resp.text()}` }; + } + const json = await resp.json(); + if (json.error) { + return { ok: false, error: `${json.error}: ${json.details || json.item || ''}` }; + } + return { ok: true, data: json.data }; +} +``` + +#### B. Domain Management (Idempotent) + +Domains must exist before users. Cache which domains you've already created: + +```typescript +const ensuredDomains = new Set(); + +async function ensureDomain(domain: string) { + if (ensuredDomains.has(domain)) return; + + const resp = await stalwartRequest('POST', '/api/principal', { + type: 'domain', + name: domain, + description: `Mail domain ${domain}`, + }); + const result = await parseStalwartResponse(resp); + + if (result.ok || result.error?.includes('alreadyExists')) { + ensuredDomains.add(domain); + } else { + throw new Error(`Failed to ensure domain: ${result.error}`); + } +} +``` + +#### C. POST / — Create Service + +```typescript +app.post('/', async (req, res) => { + const { id, config, secrets, plan } = req.body; + + // 1. Validate + if (!id || !isValidUUID(id)) return res.status(400).json({ error: 'Invalid id' }); + if (!config?.EMAIL_PREFIX) return res.status(400).json({ error: 'Missing EMAIL_PREFIX' }); + if (!secrets?.MAIL_PASSWORD) return res.status(400).json({ error: 'Missing MAIL_PASSWORD' }); + + const domain = config.MAIL_DOMAIN || DEFAULT_DOMAIN; + const username = config.EMAIL_PREFIX.toLowerCase(); + const email = `${username}@${domain}`; + + // 2. Ensure domain exists + await ensureDomain(domain); + + // 3. Create user on Stalwart + const resp = await stalwartRequest('POST', '/api/principal', { + type: 'individual', + name: username, + secrets: [secrets.MAIL_PASSWORD], + emails: [email], + description: config.DISPLAY_NAME || username, + quota: (config.QUOTA_MB || 0) * 1024 * 1024, + roles: ['user'], + // All array fields required even if empty: + lists: [], memberOf: [], members: [], + enabledPermissions: [], disabledPermissions: [], + urls: [], externalMembers: [], + }); + + const result = await parseStalwartResponse(resp); + if (!result.ok && !result.error?.includes('alreadyExists')) { + return res.status(502).json({ error: 'Stalwart create failed', detail: result.error }); + } + + // 4. Fetch connection details (DNS + JMAP in parallel) + const details = await buildDetails(username, email, domain, secrets.MAIL_PASSWORD); + + // 5. Store the mapping + services.set(id, { username, email, domain, plan, config, details }); + + res.status(201).end(); +}); +``` + +#### D. GET / — Status Polling + +This is called frequently by the reconciler. Return what Codesphere needs: + +```typescript +app.get('/', (req, res) => { + let ids = req.query.id; + + // No IDs? Return list of all known service IDs + if (!ids) return res.json(Array.from(services.keys())); + + if (!Array.isArray(ids)) ids = [ids]; + + const result: Record = {}; + for (const id of ids) { + const svc = services.get(id as string); + if (svc) { + result[id as string] = { + plan: svc.plan, + config: svc.config, + details: svc.details, // ← This is what the user sees in Codesphere + }; + } + } + res.json(result); +}); +``` + +#### E. PATCH /:id — Update Service + +Remember: Stalwart expects an **action array**, not a flat object: + +```typescript +app.patch('/:id', async (req, res) => { + const svc = services.get(req.params.id); + if (!svc) return res.status(404).json({ error: 'Not found' }); + + const { config, secrets } = req.body; + const actions: Array<{action: string, field: string, value: any}> = []; + + if (config?.DISPLAY_NAME) { + actions.push({ action: 'set', field: 'description', value: config.DISPLAY_NAME }); + } + if (config?.QUOTA_MB !== undefined) { + actions.push({ action: 'set', field: 'quota', value: config.QUOTA_MB * 1024 * 1024 }); + } + if (secrets?.MAIL_PASSWORD) { + actions.push({ action: 'set', field: 'secrets', value: [secrets.MAIL_PASSWORD] }); + } + + if (actions.length > 0) { + // Use USERNAME in path, not numeric ID! + const resp = await stalwartRequest('PATCH', `/api/principal/${svc.username}`, actions); + const result = await parseStalwartResponse(resp); + if (!result.ok) return res.status(502).json({ error: result.error }); + } + + res.status(204).end(); +}); +``` + +#### F. DELETE /:id — Delete Service + +```typescript +app.delete('/:id', async (req, res) => { + const svc = services.get(req.params.id); + if (!svc) return res.status(404).json({ error: 'Not found' }); + + await stalwartRequest('DELETE', `/api/principal/${svc.username}`); + services.delete(req.params.id); + + res.status(204).end(); +}); +``` + +#### G. Build Details — DNS + JMAP Discovery + +This assembles all the information the user sees in Codesphere: + +```typescript +async function buildDetails(username: string, email: string, domain: string, password: string) { + const [dnsRecords, jmapDetails] = await Promise.all([ + fetchDnsRecords(domain), + fetchJmapDetails(username, password), + ]); + + return { + email, + username, + mail_domain: domain, + imap_host: STALWART_IMAP_HOST, + imap_port: STALWART_IMAP_PORT, + smtp_host: STALWART_SMTP_HOST, + smtp_port: STALWART_SMTP_PORT, + jmap_url: STALWART_JMAP_URL, + jmap_account_id: jmapDetails.accountId || '', + jmap_identity_id: jmapDetails.identityId || '', + jmap_drafts_mailbox_id: jmapDetails.draftsMailboxId || '', + webmail_url: STALWART_WEBMAIL_URL, + dns_records: formatDnsRecords(dnsRecords), + ready: true, + }; +} +``` + +### 3.4 — Test Your Backend Locally + +Start the backend: + +```bash +cd src/rest-backend + +STALWART_API_URL=http://localhost:1080 \ +STALWART_ADMIN_TOKEN=admin:localdev123 \ +STALWART_IMAP_HOST=localhost \ +STALWART_SMTP_HOST=localhost \ +STALWART_IMAP_PORT=1993 \ +STALWART_SMTP_PORT=1587 \ +PORT=9090 \ +node server.js +``` + +Run through the full CRUD lifecycle: + +```bash +# CREATE +curl -s -w "\nHTTP %{http_code}\n" -X POST http://localhost:9090/ \ + -H "Content-Type: application/json" \ + -d '{ + "id": "550e8400-e29b-41d4-a716-446655440000", + "plan": {"id": 0, "parameters": {}}, + "config": { + "EMAIL_PREFIX": "alice", + "MAIL_DOMAIN": "workshop.example.com", + "DISPLAY_NAME": "Alice Workshop", + "QUOTA_MB": 500 + }, + "secrets": {"MAIL_PASSWORD": "supersecret123"} + }' +# Expected: HTTP 201 + +# READ (what Codesphere's reconciler does) +curl -s "http://localhost:9090/?id=550e8400-e29b-41d4-a716-446655440000" | python3 -m json.tool +# Expected: details with email, IMAP/SMTP hosts, JMAP IDs, DNS records, ready: true + +# UPDATE +curl -s -w "\nHTTP %{http_code}\n" -X PATCH \ + "http://localhost:9090/550e8400-e29b-41d4-a716-446655440000" \ + -H "Content-Type: application/json" \ + -d '{"config": {"DISPLAY_NAME": "Alice M. Workshop", "QUOTA_MB": 1000}}' +# Expected: HTTP 204 + +# DELETE +curl -s -w "\nHTTP %{http_code}\n" -X DELETE \ + "http://localhost:9090/550e8400-e29b-41d4-a716-446655440000" +# Expected: HTTP 204 +``` + +✅ **Checkpoint:** All four CRUD operations return the correct HTTP status codes and the GET response includes `ready: true` with all connection details populated. + +--- + +## Part 4 — Deploy the Backend on Codesphere + +Now let's get your REST backend running on Codesphere so it's reachable by the platform's reconciler. + +### 4.1 — Create a Workspace + +1. Log into the workshop Codesphere instance. +2. Create a new workspace from the `stalwart-provider` repository. +3. Choose the **"Always On"** plan (so the reconciler can reach it 24/7). + +### 4.2 — Configure Environment Variables + +In your workspace settings, set these environment variables: + +| Variable | Value | Description | +|----------|-------|-------------| +| `STALWART_API_URL` | `https://` | Workshop Stalwart instance | +| `STALWART_ADMIN_TOKEN` | *(provided by workshop lead)* | Admin credentials | +| `STALWART_IMAP_HOST` | `` | Public IMAP hostname | +| `STALWART_SMTP_HOST` | `` | Public SMTP hostname | +| `STALWART_IMAP_PORT` | `993` | IMAPS port | +| `STALWART_SMTP_PORT` | `587` | SMTP submission port | +| `PORT` | `3000` | Backend listen port | +| `AUTH_TOKEN` | *(generate a random string)* | Protects your backend | + +### 4.3 — Set Up the CI Pipeline + +Create a `ci.yml` in your workspace (or configure via the Codesphere UI): + +```yaml +schemaVersion: v0.2 + +prepare: + steps: + - name: Install dependencies + command: cd src/rest-backend && npm install + +run: + backend: + steps: + - command: cd src/rest-backend && node server.js + network: + ports: + - port: 3000 + isPublic: true +``` + +### 4.4 — Deploy and Verify + +Deploy the workspace, then verify the backend is reachable: + +```bash +# Use your workspace's public URL +curl -s "https:///" \ + -H "Authorization: Bearer " +# Should return [] (empty list of service IDs) +``` + +✅ **Checkpoint:** Your REST backend is live on Codesphere and responds to requests. + +--- + +## Part 5 — Write the provider.yml + +The `provider.yml` is the definition file that tells Codesphere everything about your service: what it's called, what users can configure, what secrets they need to provide, and what details they'll get back. + +### 5.1 — Create Your provider.yml + +Create `provider.yml` in the **repository root** (Codesphere reads it from there during git-based registration): + +```yaml +name: stalwart-mailbox +version: v1 +author: Workshop Team +displayName: Stalwart Mailbox +iconUrl: https://stalw.art/img/logo.svg +category: messaging +description: | + Provision individual email accounts on a shared Stalwart Mail Server. + Each service creates a new mailbox user with IMAP, SMTP, and JMAP access. + +backend: + api: + endpoint: https:// + +plans: + - id: 0 + name: starter + description: Basic mailbox with 500 MB storage + parameters: {} + - id: 1 + name: standard + description: Standard mailbox with 2 GB storage + parameters: {} + +configSchema: + type: object + properties: + EMAIL_PREFIX: + type: string + description: Local part of the email address (the part before @) + x-update-constraint: immutable + MAIL_DOMAIN: + type: string + description: Email domain (e.g. example.com). Created automatically if needed. + x-update-constraint: immutable + DISPLAY_NAME: + type: string + description: Display name shown in email clients + QUOTA_MB: + type: integer + description: Mailbox storage quota in megabytes + +secretsSchema: + type: object + properties: + MAIL_PASSWORD: + type: string + format: password + +detailsSchema: + type: object + properties: + email: + type: string + username: + type: string + mail_domain: + type: string + imap_host: + type: string + imap_port: + type: integer + smtp_host: + type: string + smtp_port: + type: integer + jmap_url: + type: string + jmap_account_id: + type: string + description: JMAP account ID for programmatic email access + jmap_identity_id: + type: string + description: JMAP identity ID for EmailSubmission + jmap_drafts_mailbox_id: + type: string + description: JMAP Drafts mailbox ID + webmail_url: + type: string + dns_records: + type: string + description: DNS records to configure for your domain (SPF, DKIM, DMARC, MX) + ready: + type: boolean +``` + +### 5.2 — Understand the Schemas + +| Schema | Purpose | Where it appears in the UI | +|--------|---------|---------------------------| +| **configSchema** | Values the user provides when creating the service. Passed to your backend in `config`. | "Configuration" section of the create dialog and service settings | +| **secretsSchema** | Sensitive values (passwords, tokens). Passed in `secrets`. Never displayed again after creation. | "Secrets" step of the create dialog | +| **detailsSchema** | Read-only values your backend returns. Shown after the service is running. | "Details" / "Overview" tab in service settings | + +### 5.3 — Validation Rules to Remember + +| Rule | Correct | Incorrect | +|------|---------|-----------| +| `name` format | `stalwart-mailbox` | `Stalwart Mailbox` | +| `version` format | `v1`, `v2` | `1.0.0`, `latest` | +| `plans[].id` | `0` (integer) | `"starter"` (string) | +| `plans[].parameters` | `{}` (object, even if empty) | *(missing)* | +| Secrets | `format: password` | Default values | + +### 5.4 — Validate + +```bash +make validate +``` + +✅ **Checkpoint:** `make validate` passes with no errors. + +--- + +## Part 6 — Register the Provider + +Now register your provider with the Codesphere marketplace so it appears in the service catalog. + +### 6.1 — Option A: Register via Swagger UI (Recommended for Workshop) + +1. Open the Codesphere Swagger UI: `https:///api/swagger-ui` +2. Authenticate with your API token. +3. Find the **POST `/managed-services/providers`** endpoint. +4. Send a request with: + +```json +{ + "gitUrl": "https://github.com//stalwart-provider", + "scope": { + "type": "team", + "teamIds": [] + } +} +``` + +Codesphere will fetch your `provider.yml` from the repository, validate it, and register the provider. + +### 6.2 — Option B: Register via curl + +```bash +export CODESPHERE_URL="https://" +export CODESPHERE_API_TOKEN="" +export CODESPHERE_TEAM_ID="" + +curl -X POST "$CODESPHERE_URL/api/managed-services/providers" \ + -H "Authorization: Bearer $CODESPHERE_API_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "gitUrl": "https://github.com//stalwart-provider", + "scope": { + "type": "team", + "teamIds": ['"$CODESPHERE_TEAM_ID"'] + } + }' +``` + +### 6.3 — Option C: Register via Make + +```bash +CODESPHERE_URL=https:// \ +CODESPHERE_API_TOKEN= \ +CODESPHERE_TEAM_ID= \ +make register +``` + +### 6.4 — Verify Registration + +After successful registration, your provider should appear in the managed services catalog: + +1. Navigate to **Managed Services** in the Codesphere UI. +2. Look for **"Stalwart Mailbox"** in the service grid. +3. Click it to see your configuration fields. + +✅ **Checkpoint:** Your provider is visible in the Codesphere service catalog. + +--- + +## Part 7 — Book & Test a Service Instance + +### 7.1 — Create a Service Instance via the UI + +1. Go to **Managed Services** → Click **"Stalwart Mailbox"** → **"Start Setup"** +2. Fill in the configuration: + - **EMAIL_PREFIX:** `yourname` + - **MAIL_DOMAIN:** `workshop.example.com` (or your assigned domain) + - **DISPLAY_NAME:** `Your Name` + - **QUOTA_MB:** `500` +3. Set secrets: + - **MAIL_PASSWORD:** Choose a strong password +4. Select the **Starter** plan. +5. Click **"Create Service"**. + +### 7.2 — Watch the Reconciliation + +Switch to the Managed Services table. You'll see your service go through these states: + +``` +Creating → Synchronized +``` + +The reconciler is calling your backend's `POST /` to create the account and then polling `GET /?id=...` to check the status. Once your backend returns `ready: true` in the details, the service transitions to `Synchronized`. + +### 7.3 — View the Connection Details + +Click the gear icon on your service to open the details view. You should see: + +| Field | Example Value | +|-------|---------------| +| **email** | `yourname@workshop.example.com` | +| **imap_host** | `mail.workshop.example.com` | +| **imap_port** | `993` | +| **smtp_host** | `mail.workshop.example.com` | +| **smtp_port** | `587` | +| **jmap_url** | `https://mail.workshop.example.com/jmap` | +| **jmap_account_id** | *(auto-discovered)* | +| **dns_records** | MX, SPF, DKIM, DMARC records | +| **ready** | `true` | + +### 7.4 — Test the Mailbox + +#### Option A: Webmail + +Open the **webmail_url** in your browser and log in with your username and password. + +#### Option B: JMAP (Programmatic — Send an Email) + +Use the JMAP IDs from the service details to send an email programmatically: + +```bash +curl -s "$STALWART_API_URL/jmap/" \ + -u 'yourname:your-password' \ + -H 'Content-Type: application/json' \ + -d '{ + "using": [ + "urn:ietf:params:jmap:core", + "urn:ietf:params:jmap:mail", + "urn:ietf:params:jmap:submission" + ], + "methodCalls": [ + ["Email/set", { + "accountId": "", + "create": { + "draft1": { + "mailboxIds": {"": true}, + "from": [{"name": "Your Name", "email": "yourname@workshop.example.com"}], + "to": [{"name": "Test", "email": "another-participant@workshop.example.com"}], + "subject": "Hello from the Workshop!", + "textBody": [{"partId": "body", "type": "text/plain"}], + "bodyValues": { + "body": { + "value": "This email was sent via JMAP through the Codesphere managed service!", + "isEncodingProblem": false + } + } + } + } + }, "c1"], + ["EmailSubmission/set", { + "accountId": "", + "create": { + "sub1": { + "identityId": "", + "emailId": "#draft1" + } + } + }, "c2"] + ] + }' | python3 -m json.tool +``` + +If both method responses contain `"created"` → the email was sent! 🎉 + +### 7.5 — Test Update and Delete + +**Update** the display name or quota via the Codesphere UI or API, and verify the change takes effect. + +**Delete** the service instance and verify the Stalwart user is removed: + +```bash +curl -s "$STALWART_API_URL/api/principal/yourname" \ + -u "$STALWART_ADMIN_TOKEN" +# Should return: {"error": "notFound", "item": "yourname"} +``` + +✅ **Checkpoint:** Full lifecycle works — create, read, update, delete. Emails can be sent. + +--- + +## Part 8 — Debug & Improve + +### Common Issues + +| Symptom | Likely Cause | Fix | +|---------|-------------|-----| +| Service stuck in "Creating" | Backend not reachable or returning errors | Check workspace logs; ensure the backend URL is correct and publicly accessible | +| `502` errors in backend logs | Stalwart API URL wrong or unreachable | Verify `STALWART_API_URL` and network connectivity | +| Domain creation fails with `notFound` | Not a bug — Stalwart returns this oddly | Ensure you're checking for `alreadyExists` in error handling | +| JMAP details empty | User just created, JMAP session needs a moment | Add a small delay or handle gracefully in GET | +| Provider registration returns 409 | Provider name+version already registered | Bump the version (`v2`, `v3`) or use a different name | +| `401` from Stalwart | Wrong admin credentials | Double-check `STALWART_ADMIN_TOKEN` format (`user:password` or bearer token) | + +### Ideas for Improvements + +Once the basic lifecycle works, consider these enhancements: + +| Improvement | Difficulty | Description | +|-------------|-----------|-------------| +| **Persistent storage** | ⭐⭐ | Replace the in-memory `Map` with a database (e.g., Codesphere's managed PostgreSQL) so state survives backend restarts | +| **Input validation** | ⭐ | Validate EMAIL_PREFIX format, domain format, password strength | +| **Auth middleware** | ⭐ | Ensure only Codesphere can call your backend (check `AUTH_TOKEN`) | +| **Error details** | ⭐ | Return more descriptive error messages that Codesphere can show to the user | +| **Health endpoint** | ⭐ | Add `GET /health` that checks Stalwart connectivity | +| **Quota enforcement** | ⭐⭐ | Map plan IDs to actual quota values (Starter=500MB, Standard=2GB, Premium=10GB) | +| **Multi-domain isolation** | ⭐⭐ | Let each team use their own domain | + +--- + +## Part 9 — Wrap-Up: What Users Get + +When everything is wired up, here's the complete experience from a Codesphere user's perspective: + +### 1. Browse the Catalog + +They open **Managed Services** and see "Stalwart Mailbox" alongside PostgreSQL, S3, and other services. + +### 2. Configure and Create + +They fill in an email prefix, domain, display name, and password. Click create. + +### 3. Get Connection Details + +Within seconds, the service shows as "Synchronized" with complete connection details: + +| What | Value | +|------|-------| +| **Email address** | `alice@theircompany.com` | +| **IMAP** | `mail.theircompany.com:993` (TLS) | +| **SMTP** | `mail.theircompany.com:587` (STARTTLS) | +| **JMAP endpoint** | `https://mail.theircompany.com/jmap` | +| **JMAP IDs** | Account ID, Identity ID, Drafts Mailbox ID — ready for API use | +| **Webmail** | `https://mail.theircompany.com/login` | +| **DNS records** | SPF, DKIM, DMARC, MX — copy-paste into DNS provider | + +### 4. Integrate + +They can: +- **Connect any email client** (Thunderbird, Apple Mail, Outlook) using IMAP/SMTP. +- **Send transactional emails** from their app via JMAP (modern, JSON-based — no SMTP library needed). +- **Configure DNS** so their emails are properly authenticated and delivered. + +### What You've Built + +You've turned a raw open-source mail server into a **self-service managed offering** on the Codesphere platform: + +``` + Before (manual) After (managed service) + ───────────────────────── ───────────────────────────────── + 1. Deploy Stalwart somehow 1. Click "Stalwart Mailbox" in catalog + 2. SSH in, create domain 2. Fill in 4 fields + 3. Create user via admin API 3. Click "Create" + 4. Figure out JMAP IDs 4. Connection details appear automatically + 5. Look up DNS record format 5. DNS records provided, ready to copy + 6. Write integration code 6. JMAP IDs ready for immediate API use + 7. Debug auth, discover IDs 7. Everything works +``` + +**That's the power of Codesphere managed services.** + +--- + +## Appendix A — Stalwart API Quick Reference + +| Operation | Method | Path | Body / Notes | +|-----------|--------|------|-------------| +| List principals | `GET` | `/api/principal?types=individual&limit=10` | | +| Create domain | `POST` | `/api/principal` | `{"type":"domain","name":"example.com"}` | +| Create user | `POST` | `/api/principal` | `{"type":"individual","name":"alice","secrets":["pass"],"emails":["alice@example.com"],...}` | +| Get user | `GET` | `/api/principal/alice` | Use username, not numeric ID | +| Update user | `PATCH` | `/api/principal/alice` | `[{"action":"set","field":"description","value":"..."}]` | +| Delete user | `DELETE` | `/api/principal/alice` | | +| DNS records | `GET` | `/api/dns/records/example.com` | Returns MX, SPF, DKIM, DMARC | +| JMAP session | `GET` | `/jmap/session` | Auth as the user, not admin | + +> **Remember:** Stalwart returns HTTP 200 for errors. Always parse `json.error`. + +--- + +## Appendix B — Gotchas & Troubleshooting + +### Stalwart API Gotchas + +| Gotcha | What happens | Solution | +|--------|-------------|----------| +| HTTP 200 for errors | `{"error":"notFound","item":"alice"}` with status 200 | Always check response body for `error` field | +| Domain must exist first | User creation fails if domain doesn't exist | Call `ensureDomain()` before creating users | +| PATCH format | Sending `{"description":"..."}` returns an error | Use `[{"action":"set","field":"description","value":"..."}]` | +| Use username not ID | `PATCH /api/principal/87` → notFound | `PATCH /api/principal/alice` | +| `alreadyExists` variants | Stalwart may return `alreadyExists` or `fieldAlreadyExists` | Check with `.includes('alreadyExists')` | +| Docker image name | `stalwartlabs/mail-server` is the old name | Use `stalwartlabs/stalwart:v0.13.2` | + +### provider.yml Gotchas + +| Gotcha | What happens | Solution | +|--------|-------------|----------| +| Plan ID is string | API rejects the provider | Use integer: `id: 0` not `id: "starter"` | +| Missing `parameters` in plan | Validation error | Always include `parameters: {}` even if empty | +| Version is semver | Registration fails | Use `v1`, `v2`, not `1.0.0` | +| Name has uppercase | Registration fails | Lowercase only: `stalwart-mailbox` | + +### Registration Gotchas + +| Gotcha | What happens | Solution | +|--------|-------------|----------| +| 401 on registration | Auth failed | Check `CODESPHERE_API_TOKEN` is valid | +| 409 on registration | Name+version already exists | Bump version or use different name | +| REST backend not in catalog | POST /providers only works for landscape-based | For REST backends, check your instance's admin config | + +--- + +## Appendix C — JMAP Email Sending + +JMAP (RFC 8620/8621) is the modern replacement for IMAP+SMTP, using JSON over HTTP. The managed service auto-discovers all the IDs you need. + +### How It Works + +A single JMAP request sends an email in two steps (both in one HTTP call): + +1. **`Email/set`** — Creates the email as a draft. +2. **`EmailSubmission/set`** — Submits it for delivery. + +### Required IDs (All from Service Details) + +| ID | Source | Purpose | +|----|--------|---------| +| `accountId` | `jmap_account_id` | Identifies your mailbox | +| `identityId` | `jmap_identity_id` | Your sender identity | +| `draftsMailboxId` | `jmap_drafts_mailbox_id` | Where to create the draft | + +### Send an Email + +```bash +curl -s "$STALWART_JMAP_URL/" \ + -u "username:password" \ + -H "Content-Type: application/json" \ + -d '{ + "using": [ + "urn:ietf:params:jmap:core", + "urn:ietf:params:jmap:mail", + "urn:ietf:params:jmap:submission" + ], + "methodCalls": [ + ["Email/set", { + "accountId": "ACCOUNT_ID", + "create": { + "draft1": { + "mailboxIds": {"DRAFTS_MAILBOX_ID": true}, + "from": [{"name": "Alice", "email": "alice@example.com"}], + "to": [{"name": "Bob", "email": "bob@example.com"}], + "subject": "Hello!", + "textBody": [{"partId": "body", "type": "text/plain"}], + "bodyValues": { + "body": { + "value": "Sent via JMAP!", + "isEncodingProblem": false + } + } + } + } + }, "c1"], + ["EmailSubmission/set", { + "accountId": "ACCOUNT_ID", + "create": { + "sub1": { + "identityId": "IDENTITY_ID", + "emailId": "#draft1" + } + } + }, "c2"] + ] + }' +``` + +> **Note:** `#draft1` is a JMAP back-reference — it automatically resolves to the email ID created by the first method call. + +### Verify Success + +If the response contains `"created"` in both method responses, the email was sent successfully. If something went wrong, look for `"notCreated"` in the response. + +--- + +*Happy building! 🚀* From 6828a52da844e0c43569b3eb0485735b0b38c6e6 Mon Sep 17 00:00:00 2001 From: Datata1 Date: Thu, 9 Apr 2026 18:31:22 +0200 Subject: [PATCH 2/4] make commands --- Makefile | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/Makefile b/Makefile index 8117f40..0ecdaa7 100644 --- a/Makefile +++ b/Makefile @@ -27,3 +27,29 @@ clean: ## Remove generated config files (keeps examples) @echo "Cleaning generated configs..." @rm -f $(PROVIDER_CONFIG) $(CI_CONFIG) @echo "Done." + +start-api-backend: ## Start the API backend for local development + STALWART_API_URL=http://localhost:1080 STALWART_ADMIN_TOKEN="admin:localdev123" STALWART_IMAP_HOST=localhost STALWART_SMTP_HOST=localhost STALWART_IMAP_PORT=1993 STALWART_SMTP_PORT=1587 PORT=9090 node server.js + +# JMAP test email configuration (override with env vars) +JMAP_URL ?= http://localhost:1080/jmap/ +JMAP_USER ?= jd +JMAP_PASS ?= jd +JMAP_ACCOUNT_ID ?= j +JMAP_IDENTITY_ID ?= b +JMAP_DRAFTS_ID ?= d +JMAP_FROM_NAME ?= JD +JMAP_FROM_EMAIL ?= jd@codesphere.com +JMAP_TO_NAME ?= Test +JMAP_TO_EMAIL ?= jd@codesphere.com +JMAP_SUBJECT ?= Test from make send-mail +JMAP_BODY ?= This email was sent via JMAP using make send-mail. + +send-mail: ## Send a test email via JMAP (override with JMAP_TO_EMAIL=... etc.) + @echo "Sending email from $(JMAP_FROM_EMAIL) to $(JMAP_TO_EMAIL)..." + @curl -s $(JMAP_URL) \ + -u '$(JMAP_USER):$(JMAP_PASS)' \ + -H 'Content-Type: application/json' \ + -d '{"using":["urn:ietf:params:jmap:core","urn:ietf:params:jmap:mail","urn:ietf:params:jmap:submission"],"methodCalls":[["Email/set",{"accountId":"$(JMAP_ACCOUNT_ID)","create":{"draft1":{"mailboxIds":{"$(JMAP_DRAFTS_ID)":true},"from":[{"name":"$(JMAP_FROM_NAME)","email":"$(JMAP_FROM_EMAIL)"}],"to":[{"name":"$(JMAP_TO_NAME)","email":"$(JMAP_TO_EMAIL)"}],"subject":"$(JMAP_SUBJECT)","textBody":[{"partId":"body","type":"text/plain"}],"bodyValues":{"body":{"value":"$(JMAP_BODY)","isEncodingProblem":false}}}}},"c1"],["EmailSubmission/set",{"accountId":"$(JMAP_ACCOUNT_ID)","create":{"sub1":{"identityId":"$(JMAP_IDENTITY_ID)","emailId":"#draft1"}}},"c2"]]}' | python3 -m json.tool + @echo "" + @echo "✅ Done. Check inbox at $(JMAP_TO_EMAIL) or open http://localhost:1080/login" \ No newline at end of file From d318dbdba5780490f2f06e6ed6b16b9efbb814da Mon Sep 17 00:00:00 2001 From: Philipp Walz Date: Fri, 10 Apr 2026 12:28:12 +0200 Subject: [PATCH 3/4] Update workshop tutorial and CI configurations for improved clarity and functionality --- .gitignore | 1 + WORKSHOP_TUTORIAL.md | 160 +++++---------------- ci.default.yml => ci.stalwart-provider.yml | 0 ci.stalwart.yml | 38 +++++ 4 files changed, 76 insertions(+), 123 deletions(-) rename ci.default.yml => ci.stalwart-provider.yml (100%) create mode 100644 ci.stalwart.yml diff --git a/.gitignore b/.gitignore index 13dfa36..53b5d97 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ .env +.claw/ node_modules/ \ No newline at end of file diff --git a/WORKSHOP_TUTORIAL.md b/WORKSHOP_TUTORIAL.md index 9d0ac0c..84da49c 100644 --- a/WORKSHOP_TUTORIAL.md +++ b/WORKSHOP_TUTORIAL.md @@ -2,7 +2,7 @@ > **Partner Days — Workshop 5: Managed Services** > -> Duration: ~3 hours · Difficulty: Intermediate · Language: TypeScript / Node.js +> Duration: ~2 hours · Difficulty: Intermediate · Language: TypeScript / Node.js --- @@ -12,10 +12,10 @@ By the end of this workshop you will have a **fully working managed service prov Under the hood you will: -1. Deploy a central **Stalwart Mail Server** instance on Codesphere. +1. Deploy a central **Stalwart Mail Server** instance on Codesphere using the included `ci.stalwart-provider.yml`. 2. Implement a **custom REST backend** (Node.js/TypeScript + Express) that wraps Stalwart's admin API and exposes it as a Codesphere Managed Service Adapter. -3. Write a **`provider.yml`** that describes your service in the Codesphere marketplace. -4. **Register** the provider via the Codesphere Public API. +3. Write a **`provider.yml`** that describes your service in the Codesphere marketplace (a reference `provider.yml` is included in the repo). +4. **Deploy** your provider backend and link it to the pre-registered marketplace entry. 5. **Book** a service instance through the Codesphere UI, verify it works, and iterate. ``` @@ -46,10 +46,9 @@ Under the hood you will: - [Part 3 — Implement the REST Backend](#part-3--implement-the-rest-backend) - [Part 4 — Deploy the Backend on Codesphere](#part-4--deploy-the-backend-on-codesphere) - [Part 5 — Write the provider.yml](#part-5--write-the-provideryml) -- [Part 6 — Register the Provider](#part-6--register-the-provider) -- [Part 7 — Book & Test a Service Instance](#part-7--book--test-a-service-instance) -- [Part 8 — Debug & Improve](#part-8--debug--improve) -- [Part 9 — Wrap-Up: What Users Get](#part-9--wrap-up-what-users-get) +- [Part 6 — Book & Test a Service Instance](#part-6--book--test-a-service-instance) +- [Part 7 — Debug & Improve](#part-7--debug--improve) +- [Part 8 — Wrap-Up: What Users Get](#part-8--wrap-up-what-users-get) - [Appendix A — Stalwart API Quick Reference](#appendix-a--stalwart-api-quick-reference) - [Appendix B — Gotchas & Troubleshooting](#appendix-b--gotchas--troubleshooting) - [Appendix C — JMAP Email Sending](#appendix-c--jmap-email-sending) @@ -121,15 +120,11 @@ If the user changes config → `PATCH /{id}`. If they delete → `DELETE /{id}`. | Git + GitHub access | Clone the template repos | | Node.js 18+ (or use the Codesphere workspace) | Run the REST backend | -### Step 1.1 — Clone the Template Repos +### Step 1.1 — Clone the Template Repo -You'll work with two repositories: - -1. **`stalwart-provider`** — The managed service provider template (your REST backend + provider.yml). This is the main repo you'll be editing. -2. **`stalwart-deployment`** — A ready-made Codesphere landscape that deploys the Stalwart Mail Server itself (already set up for this workshop). +Everything lives in a single repository — the REST backend, the `provider.yml`, and the Stalwart deployment pipeline (`ci.stalwart-provider.yml`): ```bash -# Clone the provider template — this is your main working repo git clone https://github.com/codesphere-cloud/stalwart-provider.git cd stalwart-provider ``` @@ -138,21 +133,18 @@ cd stalwart-provider ``` stalwart-provider/ -├── config/ -│ ├── provider.yml # ← You'll create this (service definition) -│ ├── provider.yml.example # Landscape-based example -│ └── provider.rest.yml.example # REST-based example (start from this) ├── src/ │ └── rest-backend/ │ ├── server.js # ← Reference implementation (your starting point) │ ├── package.json │ └── Dockerfile +├── ci.stalwart.yml # Codesphere CI pipeline for the Stalwart Mail Server +├── ci.stalwart-provider.yml # Codesphere CI pipeline for the REST provider backend +├── provider.yml # Service definition for the Codesphere marketplace ├── docker-compose.local.yml # Local Stalwart for development -├── provider.yml # Root-level provider (used for git-based registration) -├── Makefile # validate / register / test +├── Makefile # validate / test └── scripts/ ├── validate.sh - ├── register.sh └── test-provider.sh ``` @@ -623,29 +615,10 @@ In your workspace settings, set these environment variables: | `STALWART_IMAP_PORT` | `993` | IMAPS port | | `STALWART_SMTP_PORT` | `587` | SMTP submission port | | `PORT` | `3000` | Backend listen port | -| `AUTH_TOKEN` | *(generate a random string)* | Protects your backend | -### 4.3 — Set Up the CI Pipeline +### 4.3 — CI Pipeline -Create a `ci.yml` in your workspace (or configure via the Codesphere UI): - -```yaml -schemaVersion: v0.2 - -prepare: - steps: - - name: Install dependencies - command: cd src/rest-backend && npm install - -run: - backend: - steps: - - command: cd src/rest-backend && node server.js - network: - ports: - - port: 3000 - isPublic: true -``` +The repo includes a pre-configured `ci.stalwart-provider.yml` that installs dependencies and starts the backend. It already references the correct Stalwart host and reads the admin token from the Codesphere vault. Review it — no changes should be needed for the workshop. ### 4.4 — Deploy and Verify @@ -653,8 +626,7 @@ Deploy the workspace, then verify the backend is reachable: ```bash # Use your workspace's public URL -curl -s "https:///" \ - -H "Authorization: Bearer " +curl -s "https:///" # Should return [] (empty list of service IDs) ``` @@ -662,18 +634,16 @@ curl -s "https:///" \ --- -## Part 5 — Write the provider.yml +## Part 5 — The provider.yml The `provider.yml` is the definition file that tells Codesphere everything about your service: what it's called, what users can configure, what secrets they need to provide, and what details they'll get back. -### 5.1 — Create Your provider.yml - -Create `provider.yml` in the **repository root** (Codesphere reads it from there during git-based registration): +A reference `provider.yml` is already included in the repository root. Review it and adjust if needed: ```yaml name: stalwart-mailbox -version: v1 -author: Workshop Team +version: v2 +author: Codesphere displayName: Stalwart Mailbox iconUrl: https://stalw.art/img/logo.svg category: messaging @@ -683,7 +653,7 @@ description: | backend: api: - endpoint: https:// + endpoint: https://ms-provider-stalwart.csa.codesphere-demo.com plans: - id: 0 @@ -694,6 +664,10 @@ plans: name: standard description: Standard mailbox with 2 GB storage parameters: {} + - id: 2 + name: premium + description: Premium mailbox with 10 GB storage + parameters: {} configSchema: type: object @@ -785,72 +759,13 @@ make validate --- -## Part 6 — Register the Provider - -Now register your provider with the Codesphere marketplace so it appears in the service catalog. - -### 6.1 — Option A: Register via Swagger UI (Recommended for Workshop) - -1. Open the Codesphere Swagger UI: `https:///api/swagger-ui` -2. Authenticate with your API token. -3. Find the **POST `/managed-services/providers`** endpoint. -4. Send a request with: - -```json -{ - "gitUrl": "https://github.com//stalwart-provider", - "scope": { - "type": "team", - "teamIds": [] - } -} -``` - -Codesphere will fetch your `provider.yml` from the repository, validate it, and register the provider. - -### 6.2 — Option B: Register via curl - -```bash -export CODESPHERE_URL="https://" -export CODESPHERE_API_TOKEN="" -export CODESPHERE_TEAM_ID="" - -curl -X POST "$CODESPHERE_URL/api/managed-services/providers" \ - -H "Authorization: Bearer $CODESPHERE_API_TOKEN" \ - -H "Content-Type: application/json" \ - -d '{ - "gitUrl": "https://github.com//stalwart-provider", - "scope": { - "type": "team", - "teamIds": ['"$CODESPHERE_TEAM_ID"'] - } - }' -``` - -### 6.3 — Option C: Register via Make +## Part 6 — Book & Test a Service Instance -```bash -CODESPHERE_URL=https:// \ -CODESPHERE_API_TOKEN= \ -CODESPHERE_TEAM_ID= \ -make register -``` - -### 6.4 — Verify Registration - -After successful registration, your provider should appear in the managed services catalog: - -1. Navigate to **Managed Services** in the Codesphere UI. -2. Look for **"Stalwart Mailbox"** in the service grid. -3. Click it to see your configuration fields. - -✅ **Checkpoint:** Your provider is visible in the Codesphere service catalog. - ---- - -## Part 7 — Book & Test a Service Instance +> **Note:** The Stalwart Mailbox provider has already been **pre-registered** in the workshop platform instance. Custom REST-backend providers must be configured in the platform's global `config.yaml` — they cannot be registered via the Public API (which only supports landscape-based providers). For details on how providers are configured, see the [Codesphere Private Cloud Install Guide](https://docs.codesphere.com/docs/Private_Cloud/install-guide). +> +> The pre-registered provider points to `https://ms-provider-stalwart.csa.codesphere-demo.com`. Once you deploy your backend and link this custom domain to your workspace, the marketplace entry will route traffic to your implementation. -### 7.1 — Create a Service Instance via the UI +### 6.1 — Create a Service Instance via the UI 1. Go to **Managed Services** → Click **"Stalwart Mailbox"** → **"Start Setup"** 2. Fill in the configuration: @@ -863,7 +778,7 @@ After successful registration, your provider should appear in the managed servic 4. Select the **Starter** plan. 5. Click **"Create Service"**. -### 7.2 — Watch the Reconciliation +### 6.2 — Watch the Reconciliation Switch to the Managed Services table. You'll see your service go through these states: @@ -873,7 +788,7 @@ Creating → Synchronized The reconciler is calling your backend's `POST /` to create the account and then polling `GET /?id=...` to check the status. Once your backend returns `ready: true` in the details, the service transitions to `Synchronized`. -### 7.3 — View the Connection Details +### 6.3 — View the Connection Details Click the gear icon on your service to open the details view. You should see: @@ -889,7 +804,7 @@ Click the gear icon on your service to open the details view. You should see: | **dns_records** | MX, SPF, DKIM, DMARC records | | **ready** | `true` | -### 7.4 — Test the Mailbox +### 6.4 — Test the Mailbox #### Option A: Webmail @@ -943,7 +858,7 @@ curl -s "$STALWART_API_URL/jmap/" \ If both method responses contain `"created"` → the email was sent! 🎉 -### 7.5 — Test Update and Delete +### 6.5 — Test Update and Delete **Update** the display name or quota via the Codesphere UI or API, and verify the change takes effect. @@ -959,7 +874,7 @@ curl -s "$STALWART_API_URL/api/principal/yourname" \ --- -## Part 8 — Debug & Improve +## Part 7 — Debug & Improve ### Common Issues @@ -980,15 +895,14 @@ Once the basic lifecycle works, consider these enhancements: |-------------|-----------|-------------| | **Persistent storage** | ⭐⭐ | Replace the in-memory `Map` with a database (e.g., Codesphere's managed PostgreSQL) so state survives backend restarts | | **Input validation** | ⭐ | Validate EMAIL_PREFIX format, domain format, password strength | -| **Auth middleware** | ⭐ | Ensure only Codesphere can call your backend (check `AUTH_TOKEN`) | | **Error details** | ⭐ | Return more descriptive error messages that Codesphere can show to the user | | **Health endpoint** | ⭐ | Add `GET /health` that checks Stalwart connectivity | | **Quota enforcement** | ⭐⭐ | Map plan IDs to actual quota values (Starter=500MB, Standard=2GB, Premium=10GB) | -| **Multi-domain isolation** | ⭐⭐ | Let each team use their own domain | +| **Multi-domain isolation** | ⭐⭐⭐ | Currently every new instance re-uses or creates a domain without ownership checks — add validation so that a service instance creator actually belongs to the organization owning a domain | --- -## Part 9 — Wrap-Up: What Users Get +## Part 8 — Wrap-Up: What Users Get When everything is wired up, here's the complete experience from a Codesphere user's perspective: diff --git a/ci.default.yml b/ci.stalwart-provider.yml similarity index 100% rename from ci.default.yml rename to ci.stalwart-provider.yml diff --git a/ci.stalwart.yml b/ci.stalwart.yml new file mode 100644 index 0000000..34d326f --- /dev/null +++ b/ci.stalwart.yml @@ -0,0 +1,38 @@ +schemaVersion: v0.2 +prepare: + steps: [] +test: + steps: [] +run: + service: + healthEndpoint: http://localhost:8080/login + plan: 8 + replicas: 1 + network: + ports: + - port: 3000 + isPublic: false + - port: 8080 + isPublic: false + - port: 25 + isPublic: false + - port: 587 + isPublic: false + - port: 993 + isPublic: false + - port: 443 + isPublic: false + paths: + - port: 8080 + path: / + env: {} + runAsUser: 0 + runAsGroup: 0 + volumeMounts: + - name: _workspace + mountPath: /opt/stalwart-mail + workspacePath: stalwart-data + steps: + - command: whoami + - command: /bin/sh /usr/local/bin/entrypoint.sh + image: stalwartlabs/stalwart:v0.13.2 From 4fbe56e608bc3d996900a9798efdefabfa527893 Mon Sep 17 00:00:00 2001 From: Philipp Walz Date: Fri, 10 Apr 2026 12:55:42 +0200 Subject: [PATCH 4/4] Clean up - Added package.json for project metadata and dependencies. - Created server.js to handle mailbox provisioning via Stalwart API. - Defined endpoints for creating, retrieving, updating, and deleting mailbox users. - Removed outdated README.md file from rest-backend directory. --- .example.env | 2 - .github/copilot-instructions.md | 97 --- .github/instructions/CI.instructions.md | 429 ------------- .github/instructions/PROVIDER.instructions.md | 445 ------------- GUIDE_LISTMONK.md | 262 -------- HACKATHON.md | 462 -------------- JMAP_GUIDE.md | 216 ------- Makefile | 37 +- README.md | 383 +----------- STALWART_SETUP.md | 270 -------- TUTORIAL.md | 583 ------------------ WORKSHOP_TUTORIAL.md | 18 +- ci.stalwart-provider.yml | 4 +- config/provider.yml | 89 --- {config => examples}/ci.yml.example | 0 .../provider.rest.yml.example | 0 {config => examples}/provider.yml.example | 0 scripts/register.sh | 107 ---- scripts/test-provider.sh | 4 +- scripts/validate.sh | 4 +- src/.gitkeep | 0 src/{rest-backend => }/Dockerfile | 0 src/{rest-backend => }/package-lock.json | 0 src/{rest-backend => }/package.json | 0 src/rest-backend/README.md | 31 - src/{rest-backend => }/server.js | 0 26 files changed, 60 insertions(+), 3383 deletions(-) delete mode 100644 .example.env delete mode 100644 .github/copilot-instructions.md delete mode 100644 .github/instructions/CI.instructions.md delete mode 100644 .github/instructions/PROVIDER.instructions.md delete mode 100644 GUIDE_LISTMONK.md delete mode 100644 HACKATHON.md delete mode 100644 JMAP_GUIDE.md delete mode 100644 STALWART_SETUP.md delete mode 100644 TUTORIAL.md delete mode 100644 config/provider.yml rename {config => examples}/ci.yml.example (100%) rename {config => examples}/provider.rest.yml.example (100%) rename {config => examples}/provider.yml.example (100%) delete mode 100755 scripts/register.sh delete mode 100644 src/.gitkeep rename src/{rest-backend => }/Dockerfile (100%) rename src/{rest-backend => }/package-lock.json (100%) rename src/{rest-backend => }/package.json (100%) delete mode 100644 src/rest-backend/README.md rename src/{rest-backend => }/server.js (100%) diff --git a/.example.env b/.example.env deleted file mode 100644 index 0d36f94..0000000 --- a/.example.env +++ /dev/null @@ -1,2 +0,0 @@ -CS_API= -CS_TOKEN= \ No newline at end of file diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md deleted file mode 100644 index 118cc2b..0000000 --- a/.github/copilot-instructions.md +++ /dev/null @@ -1,97 +0,0 @@ -# Managed Service Provider Template — Workspace Instructions - -You are working inside a **Codesphere Managed Service Provider Template** project. Your job is to help the user create a fully functional managed service provider that can be registered and deployed on the Codesphere platform. - -## What Is a Service Provider? - -A service provider defines a reusable blueprint that others can instantiate as managed services. Providers can be backed by one of two backend types: - -### Landscape-based Providers -Transforms a Codesphere landscape into a reusable blueprint. Consists of: -- `provider.yml` — Metadata, schemas, and backend reference -- `ci.yml` — CI pipeline that defines how the landscape is prepared, tested, and run -- Source code in `src/` — Scripts, configs, and custom logic referenced by the CI pipeline - -### REST Backend Providers -Connects to a custom REST API that handles provisioning and lifecycle management externally. Consists of: -- `provider.yml` — Metadata, schemas, and REST backend URL -- Source code in `src/rest-backend/` — The REST backend implementation (or a reference to an external one) - -## Your Role - -You are a **provider scaffolding agent**. When the user describes a service they want to offer (e.g., "PostgreSQL with backups", "Redis cluster", "Mattermost"), you: - -### For Landscape-based Providers: -1. Generate `config/provider.yml` from `config/provider.yml.example` -2. Generate `config/ci.yml` from `config/ci.yml.example` -3. Create any required source files in `src/` (start scripts, health endpoints, setup scripts) -4. Ensure all configs pass `make validate` - -### For REST Backend Providers: -1. Generate `config/provider.yml` from `config/provider.rest.yml.example` -2. Create the REST backend implementation in `src/rest-backend/` (or customize the example) -3. No `ci.yml` is needed — the REST backend handles provisioning externally -4. Ensure all configs pass `make validate` - -## Critical Rules - -- **Always read** `.github/instructions/PROVIDER.instructions.md` before generating `provider.yml`. It has the exact schema. -- **Always read** `.github/instructions/CI.instructions.md` before generating `ci.yml` (landscape providers only). It has the CI schema. -- **Never invent config fields** that aren't in the schema. -- **Provider `name`** must match `^[-a-z0-9_]+$`. No uppercase, no spaces. -- **Provider `version`** must be `v1`, `v2`, etc. — NOT semver. -- **Secrets** go in `secretsSchema` with `format: password`. Never provide default values for secrets. -- **Backend type**: use exactly one of `backend.landscape` or `backend.rest`. Never both. - -### Landscape-specific Rules -- **Config values** go in `configSchema`. They become environment variables in the landscape, referenced as `${{ workspace.env['NAME'] }}` in ci.yml. -- **Secret values** are stored in the vault, referenced as `${{ vault.SECRET_NAME }}` in ci.yml. -- **ci.yml** must always start with `schemaVersion: v0.2`. -- **ci.yml** has two sections: `prepare` (build/setup) and `run` (service definitions). There is no separate test stage. -- **Services** in `run` can be Reactives (with `steps`), Managed Containers (with `baseImage` + `steps`), or Managed Services (with `provider`). -- **Managed Services** in `run` use `provider.name` and `provider.version` — these reference marketplace providers. -- **Networking**: services communicate via internal URLs `http://ws-server-[WorkspaceId]-[serviceName]:[port]`. Only expose ports publicly when necessary. -- **Filesystem**: only files in `/home/user/app/` persist. Use `mountSubPath` to isolate services. - -### REST Backend-specific Rules -- **`backend.rest.url`** must be a valid HTTPS URL (HTTP only for development). -- **`backend.rest.authTokenEnv`** references an environment variable name, not the token itself. Never hardcode auth tokens. -- **`planSchema`** defines resource plan parameters sent to the REST backend. Only relevant for REST providers. -- **No `ci.yml` needed** — the REST backend handles all provisioning externally. -- **REST API contract**: the backend must implement POST `/`, GET `/?id=...`, PATCH `/{id}`, DELETE `/{id}`. - -## Workflow - -When the user asks you to create a provider: - -1. Ask clarifying questions if the service type or backend type is ambiguous -2. Determine whether this is a **landscape-based** or **REST backend** provider -3. Read the example configs (`config/provider.yml.example` or `config/provider.rest.yml.example`) -4. Read the schema docs (`.github/instructions/PROVIDER.instructions.md`) -5. For landscape providers: read `.github/instructions/CI.instructions.md` and generate `config/ci.yml` -6. Generate `config/provider.yml` with the provider definition -7. For REST providers: create or customize the backend in `src/rest-backend/` -8. Create any supporting files in `src/` (scripts, configs, etc.) -9. Tell the user to run `make validate` to verify -10. Tell the user to run `make register` when ready - -## File Locations - -| What | Where | -|------|-------| -| Provider definition | `config/provider.yml` | -| Provider definition (REST example) | `config/provider.rest.yml.example` | -| CI pipeline (landscape only) | `config/ci.yml` | -| Provider schema docs | `.github/instructions/PROVIDER.instructions.md` | -| CI schema docs | `.github/instructions/CI.instructions.md` | -| Custom source code | `src/` | -| REST backend example | `src/rest-backend/` | -| Build/test commands | `Makefile` | - -## Environment Variables for Registration - -The user must set these before running `make register`: - -- `CODESPHERE_API_TOKEN` — API authentication token (Bearer token) -- `CODESPHERE_TEAM_ID` — Team ID (for team-scoped providers) -- `CODESPHERE_URL` — Codesphere instance URL (default: `https://codesphere.com`) diff --git a/.github/instructions/CI.instructions.md b/.github/instructions/CI.instructions.md deleted file mode 100644 index 2562b4e..0000000 --- a/.github/instructions/CI.instructions.md +++ /dev/null @@ -1,429 +0,0 @@ ---- -description: "Schema and rules for ci.yml landscape configuration. Use when creating or editing CI pipeline definitions for Codesphere landscapes." -applyTo: "**/ci.yml, **/ci.*.yml, **/ci.yml.example" ---- - -# CI Pipeline (ci.yml) Schema Reference - -The `ci.yml` is the central Infrastructure as Code (IaC) file that defines how a Codesphere landscape is built and deployed. It specifies the runtime environment, service orchestration, networking, and environment variables for your entire application landscape. - -> **CI Profiles:** Each ci.yml file represents a CI Profile. Different profiles (e.g., `ci.yml`, `ci.dev.yml`, `ci.prod.yml`) allow different configurations for different environments. The profile name referenced in `provider.yml` → `backend.landscape.ciProfile` must match. - ---- - -## 1. Top-Level Structure - -```yaml -schemaVersion: v0.2 # REQUIRED — always "v0.2" - -prepare: # OPTIONAL — build/setup stage - steps: [] - -run: # REQUIRED — service definitions - : { ... } - : { ... } -``` - -| Section | Purpose | -|---------|---------| -| `schemaVersion` | Schema version identifier. Always `v0.2`. | -| `prepare` | Installs dependencies, builds assets, prepares the shared filesystem. Runs on the Workspace's compute. | -| `run` | Defines the landscape services — Reactives, Managed Containers, and Managed Services. Each runs on its own dedicated resources in parallel. | - -> **No separate test stage.** Testing and linting commands should be integrated into the `prepare` steps. - ---- - -## 2. prepare - -The prepare stage initializes the shared filesystem before landscape services start. Steps execute sequentially on the Workspace (IDE pod) resources. - -```yaml -prepare: - steps: - - name: string # REQUIRED — human-readable step name - command: string # REQUIRED — bash command to execute -``` - -### Rules - -- Steps execute **sequentially** in order -- If any step exits non-zero, the prepare stage **fails** -- Commands run as standard bash — anything you'd run in a terminal works -- Changes to `/home/user/app/` persist on the **shared network filesystem** -- Files outside `/home/user/app/` are ephemeral and lost on restart -- The prepare stage only needs to re-run when build steps or dependencies change -- Use [Nix](https://nixos.org/) for reproducible dependency installation - -### Example - -```yaml -prepare: - steps: - - name: Download Application - command: wget -O app.zip https://example.com/releases/app-1.0.zip && unzip app.zip - - name: Install Dependencies - command: nix-env -iA nixpkgs.nodejs nixpkgs.nginx - - name: Build - command: cd app && npm install && npm run build -``` - ---- - -## 3. run - -The run section defines all services in your landscape. Each key is a **service name** and its value is the service configuration. Services start in **parallel** and have **self-healing** (automatic restart on crash). - -```yaml -run: - : # Service name (used in internal networking) - # --- Runtime type (pick one) --- - steps: [] # Codesphere Reactive (default) - baseImage: string # Managed Container (custom Docker image) - provider: {} # Managed Service (from marketplace) - - # --- Common fields --- - plan: integer # Resource tier ID - replicas: integer # Number of instances (default: 1) - env: {} # Environment variables - network: {} # Port and route configuration - healthEndpoint: string # Custom health check URL - mountSubPath: string # Restrict filesystem mount -``` - ---- - -## 4. Codesphere Reactive Services - -The default runtime. Containerized environment with shared filesystem, stateful serverless, and millisecond startup. - -```yaml -run: - my-service: - plan: 21 - replicas: 1 - mountSubPath: my-service-data - healthEndpoint: http://localhost:3000/health - steps: - - name: Start Server - command: npm start - env: - NODE_ENV: production - network: - ports: - - port: 3000 - isPublic: false - paths: - - port: 3000 - path: / -``` - -### Service Fields - -#### steps[] - -```yaml -steps: - - name: string # OPTIONAL — step name - command: string # REQUIRED — bash command to run -``` - -- Commands run **sequentially** at service startup -- The last command should be the long-running process (web server, worker, etc.) -- If the process exits, the platform automatically restarts it (self-healing) - -#### plan - -- **Type:** integer -- **Required:** yes -- **Purpose:** Resource tier ID determining CPU and memory allocation -- **Examples:** `0` (smallest), `21` (standard), higher values for more resources -- **Tip:** Use smaller plans with "off when unused" for development; larger plans for production - -#### replicas - -- **Type:** integer -- **Required:** no -- **Default:** `1` -- **Purpose:** Number of service instances for horizontal scaling -- **Note:** The Landscape Router load-balances requests across all replicas automatically - -#### mountSubPath - -- **Type:** string -- **Required:** no -- **Purpose:** Restricts the service's filesystem mount to a subdirectory of `/home/user/app/` -- **Best practice:** Use unique mount paths per service to avoid concurrent write conflicts -- **Example:** `mountSubPath: uploads` mounts only `/home/user/app/uploads` - -#### healthEndpoint - -- **Type:** string -- **Required:** no -- **Default:** `http://localhost:3000/` -- **Format:** Full URL with protocol, host, port, and path -- **Purpose:** The Landscape Router pings this endpoint to check service health -- **Examples:** - - `http://localhost:3000/health` - - `http://localhost:8080/api/status` - ---- - -## 5. Managed Containers - -Bring your own Docker image while Codesphere provides orchestration, networking, and monitoring. Same fields as Reactives plus: - -```yaml -run: - nginx-server: - baseImage: nginx:1.25-alpine # REQUIRED — Docker image - plan: 21 - runAsUser: 1000 # OPTIONAL — container user ID - runAsGroup: 1000 # OPTIONAL — container group ID - steps: - - command: nginx -g "daemon off;" - healthEndpoint: http://localhost:80/ - network: - ports: - - port: 80 - isPublic: false - paths: - - port: 80 - path: / - env: - NGINX_HOST: example.com -``` - -#### baseImage - -- **Type:** string -- **Required:** yes (this is what makes it a Managed Container instead of a Reactive) -- **Format:** Docker image reference `image:tag` -- **Best practice:** Pin a specific tag, never use `latest` -- **Examples:** `nginx:1.25-alpine`, `node:20-slim`, `postgres:16-alpine` - -#### runAsUser / runAsGroup - -- **Type:** integer -- **Required:** no -- **Purpose:** Set the UID/GID the container process runs as - ---- - -## 6. Managed Services - -Pre-configured services from the Codesphere marketplace (databases, caches, message queues). Defined using the `provider` field instead of `steps`. - -```yaml -run: - primary-db: - provider: - name: postgres # REQUIRED — provider name from marketplace - version: v1 # REQUIRED — provider version - plan: - id: 0 # REQUIRED — resource plan ID -``` - -### Provider Fields - -| Field | Type | Required | Description | -|-------|------|----------|-------------| -| `provider.name` | string | yes | Provider name as registered in the marketplace | -| `provider.version` | string | yes | Provider version (e.g., `v1`, `v2`) | -| `plan.id` | integer | yes | Resource plan ID for the managed service | - -### Managed Service Hostname - -Managed services get a deterministic internal hostname: - -``` -ms-${providerName}-${providerVersion}-${teamId}-${serviceName} -``` - -All characters are lowercased and non-alphanumeric characters (except `-`) are replaced with `-`. - -**Example:** For provider `postgres` version `v1`, team ID `42`, service name `primary-db`: -→ `ms-postgres-v1-42-primary-db` - -### Lifecycle - -- **Created** when the landscape is deployed -- **Updated** when ci.yml config changes and landscape is re-synced -- **Deleted** when the landscape is deleted (prevents orphan resources) -- Renaming a managed service in ci.yml recreates it — **this can cause data loss** - ---- - -## 7. Network Configuration - -### network.ports[] - -```yaml -network: - ports: - - port: integer # REQUIRED — container port number - isPublic: boolean # OPTIONAL — default: false -``` - -- `isPublic: false` (recommended) — port is only accessible within the private landscape network -- `isPublic: true` — port gets a direct public URL (not recommended for most services) - -### network.paths[] - -```yaml -network: - paths: - - port: integer # REQUIRED — which port to route to - path: string # REQUIRED — URL path prefix -``` - -- Configures the **Landscape Router** to map incoming HTTP requests by path prefix -- Multiple services can each claim different path prefixes -- The router load-balances across all replicas of a service automatically - -**Example multi-service routing:** - -```yaml -run: - frontend: - network: - paths: - - port: 3000 - path: / - backend: - network: - paths: - - port: 8080 - path: /api - websocket: - network: - paths: - - port: 9000 - path: /ws -``` - -### Public URLs - -Services are accessed through: -- **Dev Domain:** `https://[workspace-id]-[port].[datacenter-id].[instance-url]/*` -- **Custom Domain:** Configured in Domain Settings - -### Private Networking - -Services within a landscape communicate via internal URLs: - -``` -http://ws-server-[WorkspaceId]-[serviceName]:[port] -``` - -These URLs are **only resolvable within the landscape's private network** — not from the public internet or a user's browser. - ---- - -## 8. Environment Variables - -```yaml -env: - KEY: value # Plain text value - SECRET: ${{ vault.secretName }} # From vault (encrypted) - CONFIG: ${{ workspace.env['KEY'] }} # From workspace config - WS_ID: ${{ workspace.id }} # Workspace ID - TEAM: ${{ team.id }} # Team ID -``` - -### Template Syntax - -| Template | Description | -|----------|-------------| -| `${{ vault.NAME }}` | Secret from the Codesphere vault (encrypted, injected at runtime) | -| `${{ workspace.env['KEY'] }}` | Global workspace environment variable (from `configSchema` values) | -| `${{ workspace.id }}` | Resolves to the Workspace ID | -| `${{ team.id }}` | Resolves to the Team ID | - -### Rules - -- **Secrets** must use `${{ vault.NAME }}` — never hardcode sensitive values -- Vault secrets are stored encrypted and only injected at runtime -- `workspace.env['KEY']` maps to values from the provider's `configSchema` -- `vault.NAME` maps to values from the provider's `secretsSchema` - ---- - -## 9. Filesystem - -- **Persistent:** Files in `/home/user/app/` are stored on the shared network filesystem -- **Ephemeral:** Files outside `/home/user/app/` exist only on the local pod and are lost on restart -- **Shared:** All Reactive services (including the Workspace) share the same network filesystem -- **Storage:** Total storage is defined at the Workspace level -- **Best practice:** Run each service in its own directory (via `mountSubPath`) to avoid concurrent write conflicts - ---- - -## 10. Complete Example (Landscape with Reactive + Managed Service) - -```yaml -schemaVersion: v0.2 - -prepare: - steps: - - name: Download Application - command: wget -O app.zip https://download.example.com/releases/app-1.0.zip - - name: Install Dependencies - command: nix-env -iA nixpkgs.php83 nixpkgs.nginx - -run: - # Codesphere Reactive — the main application - webapp: - plan: 21 - replicas: 2 - healthEndpoint: http://localhost:3000/health - steps: - - name: Start Application - command: | - php-fpm -y ./config/php-fpm.conf - nginx -c $(pwd)/config/nginx.conf - env: - DB_HOST: ms-postgres-v1-42-primary-db - DB_PASSWORD: ${{ vault.DB_PASSWORD }} - SITE_NAME: ${{ workspace.env['SITE_NAME'] }} - network: - ports: - - port: 3000 - isPublic: false - paths: - - port: 3000 - path: / - - # Managed Service — PostgreSQL database - primary-db: - provider: - name: postgres - version: v1 - plan: - id: 0 -``` - ---- - -## 11. Validation Rules - -These rules are checked by `make validate`: - -1. `schemaVersion` must be `v0.2` -2. `run` section must be present with at least one service -3. Each service must have either `steps`, `baseImage` + `steps`, or `provider` -4. Service names should be lowercase, alphanumeric with hyphens -5. Template variables must use valid syntax: `${{ vault.X }}`, `${{ workspace.env['X'] }}` - -### Common Mistakes - -| Mistake | Fix | -|---------|-----| -| Missing `schemaVersion` | Add `schemaVersion: v0.2` at the top | -| Using a separate `test` stage | Integrate tests into `prepare` steps | -| Writing files outside `/home/user/app/` | Only `/home/user/app/` is persistent | -| Hardcoding secrets in `env` | Use `${{ vault.SECRET_NAME }}` | -| Using `isPublic: true` for internal services | Keep internal services private, use `paths` for routing | -| Concurrent writes to same files from multiple services | Use `mountSubPath` to isolate service filesystems | -| Missing `plan` on a service | Every Reactive/Container service needs a `plan` | -| Using `latest` tag for `baseImage` | Pin a specific version tag | \ No newline at end of file diff --git a/.github/instructions/PROVIDER.instructions.md b/.github/instructions/PROVIDER.instructions.md deleted file mode 100644 index ed4da8c..0000000 --- a/.github/instructions/PROVIDER.instructions.md +++ /dev/null @@ -1,445 +0,0 @@ ---- -description: "Schema and validation rules for provider.yml configuration. Use when creating or editing landscape or REST backend provider definitions." -applyTo: "**/provider.yml, **/provider.yml.example, **/provider.rest.yml.example" ---- - -# Provider Definition Schema Reference - -A service provider defines a reusable blueprint that others can instantiate as managed services on the Codesphere platform. Providers can be backed by either a **Codesphere landscape** or a **custom REST backend**. The `provider.yml` file at the repository root defines all metadata and configuration schemas. - -> **Important:** Publishing providers requires cluster admin permissions. Team admins can request providers scoped to specific teams. - ---- - -## 1. provider.yml Top-Level Structure - -```yaml -name: string # REQUIRED — unique provider identifier -version: string # REQUIRED — version in format v[0-9]+ (e.g., v1) -author: string # REQUIRED — organization or individual -displayName: string # REQUIRED — human-readable name for Marketplace UI -iconUrl: string # OPTIONAL — URL to provider icon -category: string # REQUIRED — grouping category -description: string # REQUIRED — markdown-formatted description - -backend: # REQUIRED — deployment backend configuration (pick one) - landscape: # Option A: Landscape-based backend - gitUrl: string # REQUIRED — git repo URL containing the landscape - ciProfile: string # REQUIRED — CI profile name from ci.yml - rest: # Option B: Custom REST backend - url: string # REQUIRED — base URL of the REST backend - authTokenEnv: string # OPTIONAL — env var name containing the auth token - -configSchema: object # OPTIONAL — JSON Schema for user-configurable options -secretsSchema: object # OPTIONAL — JSON Schema for secret values -detailsSchema: object # OPTIONAL — JSON Schema for runtime details -planSchema: object # OPTIONAL — JSON Schema for plan parameters (REST backends) -``` - ---- - -## 2. Field Reference - -### name - -- **Type:** string -- **Required:** yes -- **Pattern:** `^[-a-z0-9_]+$` -- **Constraints:** Lowercase, alphanumeric, hyphens, and underscores only -- **Examples:** `mattermost`, `postgresql`, `redis-cluster` -- **Invalid:** `Mattermost`, `my service`, `My_Provider` - -### version - -- **Type:** string -- **Required:** yes -- **Format:** `v[0-9]+` — e.g., `v1`, `v2`, `v10` -- **Invalid:** `1.0.0`, `latest`, `v1.0` -- **Note:** This is NOT semver. It's a simple integer version prefixed with `v`. - -### author - -- **Type:** string -- **Required:** yes -- **Purpose:** Organization or individual shown in the Marketplace - -### displayName - -- **Type:** string -- **Required:** yes -- **Purpose:** Human-readable name shown in the Marketplace UI -- **Examples:** `Mattermost`, `PostgreSQL 16`, `Redis Cluster` - -### iconUrl - -- **Type:** string -- **Required:** no -- **Format:** Absolute URL or relative path to an icon image - -### category - -- **Type:** string -- **Required:** yes -- **Purpose:** Grouping in the Marketplace UI -- **Common values:** `databases`, `messaging`, `monitoring`, `collaboration`, `storage`, `networking` - -### description - -- **Type:** string (multiline) -- **Required:** yes -- **Format:** Markdown-formatted. Use `|` for multiline in YAML. -- **Example:** - ```yaml - description: | - Open-source team messaging and collaboration platform. - Supports channels, direct messaging, and file sharing. - ``` - ---- - -## 3. backend.landscape - -Defines where the landscape configuration lives and which CI profile to use. - -```yaml -backend: - landscape: - gitUrl: string # REQUIRED — git repository URL - ciProfile: string # REQUIRED — CI profile name from ci.yml -``` - -### gitUrl - -- **Type:** string -- **Required:** yes -- **Format:** Valid Git URL (HTTPS or SSH) -- **Constraint:** The repository must contain a valid Codesphere landscape with a `ci.yml` file -- **Note:** Git provider permissions must be configured in your Codesphere account -- **Examples:** - - `https://github.com/your-org/mattermost-landscape` - - `git@github.com:your-org/redis-landscape.git` - -### ciProfile - -- **Type:** string -- **Required:** yes -- **Purpose:** References a profile defined in the landscape's `ci.yml` -- **Examples:** `production`, `default`, `staging` - ---- - -## 3b. backend.rest (REST Backend Providers) - -Defines a custom REST backend that implements the Codesphere Managed Service Adapter API. Use this instead of `backend.landscape` when your service is provisioned by an external REST API rather than a Codesphere landscape. - -```yaml -backend: - rest: - url: string # REQUIRED — base URL of the REST backend - authTokenEnv: string # OPTIONAL — env var name holding the Bearer token -``` - -> **Important:** `backend.landscape` and `backend.rest` are mutually exclusive. Use exactly one. - -### url - -- **Type:** string -- **Required:** yes -- **Format:** Valid HTTPS URL (HTTP allowed for development only) -- **Purpose:** Base URL where Codesphere sends lifecycle requests -- **Constraint:** Must implement the Managed Service Adapter API (see below) -- **Examples:** - - `https://my-backend.example.com/postgres` - - `https://internal-api.corp.net/services/redis` - -### authTokenEnv - -- **Type:** string -- **Required:** no -- **Purpose:** Name of the environment variable containing the Bearer token sent in the `Authorization` header -- **Default:** If omitted, no authentication header is sent -- **Example:** `BACKEND_AUTH_TOKEN` -- **Note:** The actual token value should be set as an environment variable during registration, never hardcoded in `provider.yml` - -### REST Backend API Contract - -Your backend must implement these four endpoints relative to the `url`: - -#### Create Service — `POST /` - -Called when a new service is requested. - -| Field | Description | -|-------|-------------| -| **Request Body** | `{ "id": "uuid", "plan": { "parameters": {...} }, "config": {...}, "secrets": {...} }` | -| **Response** | `201 Created` (empty body) | - -#### Get Status — `GET /?id=...` - -Polled periodically to sync service status. IDs are passed as repeatable query parameters. - -| Field | Description | -|-------|-------------| -| **Query Params** | `id` (repeatable, UUID). If omitted, return all known service IDs. | -| **Response** | `200 OK` with a map of ID → status objects containing `plan`, `config`, and `details` | - -#### Update Service — `PATCH /{id}` - -Called when configuration, plan, or secrets change, or when drift is detected. - -| Field | Description | -|-------|-------------| -| **Path Param** | `id` (UUID) | -| **Request Body** | Partial object with only changed fields | -| **Response** | `204 No Content` | - -#### Delete Service — `DELETE /{id}` - -Called when a user deletes the service. - -| Field | Description | -|-------|-------------| -| **Path Param** | `id` (UUID) | -| **Response** | `204 No Content` | - -### Security Considerations - -- **Network Isolation:** The backend should ideally only be accessible from the Codesphere control plane -- **Authentication:** Use `authTokenEnv` to configure Bearer token authentication -- **Input Validation:** Strictly validate all incoming `config`, `plan`, and `secrets` parameters -- **TLS:** Always use HTTPS in production - ---- - -## 4. configSchema - -Defines user-configurable options using [JSON Schema](https://json-schema.org/). These values are passed to the landscape as **environment variables**. - -```yaml -configSchema: - type: object - properties: - SITE_NAME: - type: string - description: Display name for your instance - MAX_USERS: - type: integer - description: Maximum number of users allowed - x-update-constraint: increase-only - DB_ENGINE: - type: string - description: Database engine version - enum: ['17.6', '16.10', '15.14'] - x-update-constraint: immutable -``` - -### Rules for configSchema - -- Must be `type: object` at the top level -- Each property becomes an environment variable in the landscape -- Property names should be `UPPER_SNAKE_CASE` (they map to env vars) -- Use `description` for each property — shown in the Codesphere UI config section -- Values are referenced in ci.yml as: `${{ workspace.env['PROPERTY_NAME'] }}` -- Supported JSON Schema types: `string`, `integer`, `number`, `boolean` -- Supported formats: `int32`, `int64`, `float`, `double`, `byte`, `binary`, `date`, `date-time`, `password`, `uri`, `hostname` -- Use `enum` to constrain allowed values - -### x-update-constraint Extension - -Restricts how properties can change after initial creation: - -| Constraint | Behavior | Applies To | -|------------|----------|------------| -| `increase-only` | New value must be >= current value | Numeric fields only | -| `immutable` | Cannot be changed once set | Any field | - -> Update constraints are only enforced when updating an existing service. During initial creation, all values are accepted. - ---- - -## 5. secretsSchema - -Defines secret values (passwords, tokens, API keys) using JSON Schema. Secrets are stored in the landscape's **vault**. - -```yaml -secretsSchema: - type: object - properties: - ADMIN_PASSWORD: - type: string - format: password - API_KEY: - type: string - format: password -``` - -### Rules for secretsSchema - -- Must be `type: object` at the top level -- Use `format: password` for password fields -- **Never provide default values** for secrets — they must always be user-provided -- Secret values are injected into the landscape's vault -- Referenced in ci.yml as: `${{ vault.SECRET_NAME }}` -- Secret names should be `UPPER_SNAKE_CASE` - ---- - -## 6. detailsSchema - -Defines runtime details exposed after provisioning. These are read-only values that describe the running service. - -```yaml -detailsSchema: - type: object - properties: - hostname: - type: string - port: - type: integer - status: - type: object - properties: - state: - type: string - uptime: - type: number - x-endpoint: "https://{{hostname}}:{{port}}/status" -``` - -### Rules for detailsSchema - -- Must be `type: object` at the top level -- Properties describe information visible to the user after the service is running -- Common properties: `hostname`, `port`, `connectionString`, `dashboardUrl` - -### x-endpoint Extension - -Allows fetching runtime details dynamically from the running service: - -- **Format:** URL string with `{{property}}` interpolation using other details fields -- **Method:** Only `GET` requests are supported -- **Response:** Must return JSON matching the property's schema definition -- **Example:** `x-endpoint: "https://{{hostname}}:{{port}}/status"` - ---- - -## 7. planSchema (REST Backend Providers) - -Defines the resource plan parameters for REST backend providers. This schema describes what resource options (CPU, memory, storage, etc.) users can select when provisioning a service. Plan parameters are sent to the REST backend in the `plan.parameters` field. - -```yaml -planSchema: - type: object - properties: - storage: - type: integer - description: Storage size in MB - cpu: - type: integer - description: CPU allocation in tenths - memory: - type: integer - description: Memory allocation in MB -``` - -### Rules for planSchema - -- Must be `type: object` at the top level -- Only relevant for REST backend providers (landscape providers use `plan` IDs instead) -- Properties describe resource parameters sent in `plan.parameters` to the backend -- Use `description` for each property — shown in the Codesphere UI -- Use `x-update-constraint: increase-only` for resources that can only scale up -- Supported JSON Schema types: `string`, `integer`, `number`, `boolean` - ---- - -## 7. Publishing / Registration - -Providers are published via the Codesphere Public API. Two methods are available: - -### Method 1: Using Git URL (Recommended) - -Provide the Git repository URL. Codesphere fetches and validates `provider.yml` automatically. - -```bash -curl -X POST "https://codesphere.com/api/managed-services/providers" \ - -H "Authorization: Bearer $CODESPHERE_API_TOKEN" \ - -H "Content-Type: application/json" \ - -d '{ - "gitUrl": "https://github.com/your-org/your-landscape", - "scope": { - "type": "global" - } - }' -``` - -### Method 2: Using Full Specification - -Send the complete provider definition in the request payload (used by `make register`). - -### Provider Scopes - -| Scope | Description | -|-------|-------------| -| `global` | Available to all teams. Requires cluster admin permissions. | -| `team` | Available only to specified teams. Provide `teamIds` array. | - -```json -{ - "scope": { "type": "team", "teamIds": [123, 456] } -} -``` - -> **Note:** Replace `codesphere.com` with your instance URL if using a self-hosted deployment. - ---- - -## 8. Validation Rules - -These rules are checked by `make validate`: - -### Required Fields - -1. `name` must match `^[-a-z0-9_]+$` -2. `version` must match `^v[0-9]+$` -3. `displayName`, `author`, `category`, `description` must be non-empty -4. Exactly one backend type must be specified: `backend.landscape` or `backend.rest` - -#### Landscape Backend (`backend.landscape`) - -5. `backend.landscape.gitUrl` must be a valid URL -6. `backend.landscape.ciProfile` must be non-empty - -#### REST Backend (`backend.rest`) - -5. `backend.rest.url` must be a valid URL -6. `backend.rest.authTokenEnv` if present must be a valid env var name - -### Schema Validation - -1. `configSchema`, `secretsSchema`, `detailsSchema`, `planSchema` must have `type: object` at top level if present -2. Secret properties should use `format: password` -3. Properties with `x-update-constraint` must use valid constraint values -4. `planSchema` is only valid for REST backend providers - -### Common Mistakes - -| Mistake | Fix | -|---------|-----| -| Using semver version (`1.0.0`) | Use `v1`, `v2`, etc. | -| Missing backend section | Provide either `backend.landscape` or `backend.rest` | -| Specifying both `backend.landscape` and `backend.rest` | Use exactly one backend type | -| Default values in secretsSchema | Never set defaults for secrets | -| Hardcoding auth tokens in `backend.rest` | Use `authTokenEnv` to reference an env var | -| `x-update-constraint: increase-only` on a string | Only use on numeric fields | -| Uppercase characters in `name` | Use lowercase with hyphens/underscores only | -| Missing `ciProfile` reference (landscape) | Must match a profile in your ci.yml | -| Using HTTP URL for REST backend in production | Always use HTTPS | - ---- - -## 9. Complete Example - -See `config/provider.yml.example` for a full working example of a landscape-based provider (Mattermost collaboration platform). - -See `config/provider.rest.yml.example` for a full working example of a REST backend provider (PostgreSQL database). diff --git a/GUIDE_LISTMONK.md b/GUIDE_LISTMONK.md deleted file mode 100644 index c19fa72..0000000 --- a/GUIDE_LISTMONK.md +++ /dev/null @@ -1,262 +0,0 @@ -# Listmonk als Landscape-based Managed Service auf Codesphere - -## Idee - -**[Listmonk](https://listmonk.app)** ist ein leichtgewichtiger, Open-Source Newsletter- und Mailing-List-Manager (Go, single binary). Er eignet sich perfekt, weil: - -- **Einfach**: Single binary, konfigurierbar via Environment Variables -- **Braucht SMTP**: Nutzt den gerade gebauten **Stalwart Mailbox** Service zum Versenden -- **Braucht PostgreSQL**: Codesphere hat bereits `postgres/v1` als Managed Service -- **Zeigt Komposition**: Ein Landscape-Service, der zwei Managed Services konsumiert - -### Architektur - -``` -┌─────────────────────────────────────────────────────┐ -│ Codesphere Landscape (listmonk Provider) │ -│ │ -│ ┌──────────────┐ │ -│ │ listmonk │ Reactive Service (Go binary) │ -│ │ Port 9000 │──────────────────────┐ │ -│ └──────┬───────┘ │ │ -│ │ SQL SMTP (Port 587) │ -│ ▼ ▼ │ -│ ┌──────────────┐ ┌──────────────────┐ │ -│ │ db │ │ mailbox │ │ -│ │ postgres/v1 │ │ stalwart-mailbox │ │ -│ │ Managed Svc │ │ Managed Service │ │ -│ └──────────────┘ └──────────────────┘ │ -└─────────────────────────────────────────────────────┘ -``` - ---- - -## Schritt-für-Schritt Plan - -### 1. Neues Repo anlegen - -```bash -mkdir listmonk-provider && cd listmonk-provider -# ms-template als Basis kopieren oder neues Repo mit der Struktur: -# config/provider.yml -# config/ci.yml -# src/ (optional, für custom scripts) -``` - -### 2. `config/provider.yml` erstellen - -```yaml -name: listmonk -version: v1 -author: Codesphere -displayName: Listmonk Newsletter -iconUrl: https://listmonk.app/static/images/logo.svg -category: messaging -description: | - Self-hosted newsletter and mailing list manager. - Powered by Listmonk with PostgreSQL storage and Stalwart SMTP delivery. - -backend: - landscape: - gitUrl: https://github.com/codesphere-cloud/listmonk-provider - ciProfile: default - -plans: - - id: 0 - name: starter - displayName: Starter - description: Small instance for up to 1,000 subscribers - parameters: {} - -configSchema: - type: object - properties: - ADMIN_USER: - type: string - description: Admin username for the Listmonk web UI - SITE_NAME: - type: string - description: Name shown in emails and the web UI - -secretsSchema: - type: object - properties: - ADMIN_PASSWORD: - type: string - format: password - SMTP_PASSWORD: - type: string - format: password - description: Password of the Stalwart mailbox used for sending - -detailsSchema: - type: object - properties: - url: - type: string - admin_user: - type: string - ready: - type: boolean -``` - -### 3. `config/ci.yml` erstellen - -```yaml -schemaVersion: v0.2 - -prepare: - steps: - - name: Download Listmonk - command: | - LISTMONK_VERSION=4.1.0 - wget -q "https://github.com/knadh/listmonk/releases/download/v${LISTMONK_VERSION}/listmonk_${LISTMONK_VERSION}_linux_amd64.tar.gz" - tar xzf "listmonk_${LISTMONK_VERSION}_linux_amd64.tar.gz" - chmod +x listmonk - - - name: Create config - command: | - cat > config.toml <<'EOF' - [app] - address = "0.0.0.0:9000" - admin_username = "${LISTMONK_ADMIN_USER}" - admin_password = "${LISTMONK_ADMIN_PASSWORD}" - - [db] - host = "${DB_HOST}" - port = 5432 - user = "${DB_USER}" - password = "${DB_PASSWORD}" - database = "${DB_NAME}" - ssl_mode = "disable" - EOF - -run: - # ── Managed Services ──────────────────────────────── - db: - provider: - name: postgres - version: v1 - plan: - id: 0 - - mailbox: - provider: - name: stalwart-mailbox - version: v1 - plan: - id: 0 - - # ── Listmonk Application ──────────────────────────── - app: - plan: 0 - replicas: 1 - mountSubPath: listmonk - healthEndpoint: http://localhost:9000/health - steps: - - name: Run DB migrations - command: ./listmonk --install --idempotent --config config.toml - - - name: Configure SMTP via API - command: | - # Wait for listmonk to accept connections - sleep 3 - # Add Stalwart as SMTP server via Listmonk API - curl -s -u "${LISTMONK_ADMIN_USER}:${LISTMONK_ADMIN_PASSWORD}" \ - -X PUT http://localhost:9000/api/settings \ - -H 'Content-Type: application/json' \ - -d '{ - "smtp": [{ - "enabled": true, - "host": "'${SMTP_HOST}'", - "port": 587, - "auth_protocol": "login", - "username": "'${SMTP_USER}'", - "password": "'${SMTP_PASSWORD}'", - "tls_type": "STARTTLS", - "max_conns": 5 - }] - }' || true - - - name: Start Listmonk - command: ./listmonk --config config.toml - - env: - # PostgreSQL — kommt vom Managed Service "db" - # Der Hostname folgt dem Schema: ms-postgres-v1--db - DB_HOST: ms-postgres-v1-${{ team.id }}-db - DB_PORT: "5432" - DB_USER: listmonk - DB_PASSWORD: ${{ vault.DB_PASSWORD }} - DB_NAME: listmonk - - # Listmonk Admin - LISTMONK_ADMIN_USER: ${{ workspace.env['ADMIN_USER'] }} - LISTMONK_ADMIN_PASSWORD: ${{ vault.ADMIN_PASSWORD }} - - # SMTP — kommt vom Managed Service "mailbox" (Stalwart) - # Der SMTP Host muss der echte Stalwart-Host sein, nicht der interne MS-Hostname - SMTP_HOST: ${{ workspace.env['SMTP_HOST'] }} - SMTP_USER: ${{ workspace.env['SMTP_USER'] }} - SMTP_PASSWORD: ${{ vault.SMTP_PASSWORD }} - - network: - ports: - - port: 9000 - isPublic: false - paths: - - port: 9000 - path: / -``` - -### 4. Offene Fragen / Entscheidungen - -| Thema | Optionen | Empfehlung | -|-------|----------|------------| -| **SMTP-Credentials** | Statisch in configSchema oder automatisch aus Stalwart MS Details? | Erstmal statisch über `configSchema` + `secretsSchema` — der User gibt die Mailbox-Credentials an, die er über den Stalwart-Service erstellt hat | -| **PostgreSQL Setup** | Eigene DB oder vom postgres/v1 MS automatisch? | `postgres/v1` als Managed Service im Landscape — Codesphere erstellt die DB automatisch | -| **Listmonk Config** | config.toml oder Environment Variables? | Listmonk unterstützt beides. `config.toml` ist expliziter und wird im prepare-Step generiert | -| **Listmonk Version** | Welche? | v4.1.0 (aktuell stabil). Im prepare-Step als Variable, einfach aktualisierbar | -| **TLS für SMTP** | STARTTLS oder Implicit TLS? | STARTTLS auf Port 587 (Standard für Submission) | - -### 5. Ablauf aus User-Sicht - -1. User geht ins Codesphere Marketplace -2. Erstellt zuerst einen **Stalwart Mailbox** Service → bekommt Email + SMTP-Credentials -3. Erstellt dann einen **Listmonk** Service: - - Gibt `ADMIN_USER`, `SITE_NAME` an - - Gibt `ADMIN_PASSWORD`, `SMTP_PASSWORD` (= Mailbox-Passwort) als Secrets an -4. Codesphere provisioniert automatisch PostgreSQL + startet Listmonk -5. User greift über die Landscape-URL auf das Listmonk Web-UI zu -6. Listmonk versendet Emails über den Stalwart SMTP-Server - -### 6. Implementierungs-Reihenfolge - -| Schritt | Was | Geschätzter Aufwand | -|---------|-----|---------------------| -| **A** | Repo erstellen, `provider.yml` + `ci.yml` schreiben | Gering | -| **B** | Lokal testen: Listmonk binary + lokale PostgreSQL + lokalen Stalwart | Mittel | -| **C** | `make validate` bestehen | Gering | -| **D** | Provider registrieren (`make register` oder gitUrl API) | Gering | -| **E** | End-to-End Test: Landscape deployen, Newsletter versenden | Mittel | -| **F** | Feinschliff: Health-Endpoint, Bounce-Handling, Templates | Optional | - -### 7. Alternative Projekte (falls Listmonk nicht passt) - -| Projekt | Sprache | Beschreibung | Komplexität | -|---------|---------|--------------|-------------| -| **[Listmonk](https://listmonk.app)** | Go | Newsletter/Mailing-Listen | ⭐ Niedrig | -| **[Mailtrain](https://mailtrain.org)** | Node.js | Newsletter-Manager (mehr Features, komplexer) | ⭐⭐ Mittel | -| **[Postal](https://docs.postalserver.io)** | Ruby | Mail Delivery Platform (MTA-Level) | ⭐⭐⭐ Hoch | -| **[Mautic](https://mautic.org)** | PHP | Marketing Automation + Email | ⭐⭐⭐ Hoch | -| **[Chatwoot](https://chatwoot.com)** | Ruby | Kundenkommunikation mit Email-Kanal | ⭐⭐⭐ Hoch | - -**Empfehlung: Listmonk** — einfachstes Setup, single binary, perfekter Fit. - ---- - -## Nächste Schritte - -1. Bestätige ob Listmonk die richtige Wahl ist -2. Ich erstelle das komplette Repo mit `provider.yml`, `ci.yml` und Hilfsscripts -3. Wir testen lokal, dann registrieren wir den Provider auf Codesphere diff --git a/HACKATHON.md b/HACKATHON.md deleted file mode 100644 index 9f7b2fa..0000000 --- a/HACKATHON.md +++ /dev/null @@ -1,462 +0,0 @@ -# 📬 Stalwart Mail Provider — Hackathon Guide - -Welcome! This guide will help you understand, run, and extend the **Stalwart Mailbox Provider** — a Codesphere managed service that provisions email accounts on demand. - ---- - -## What Does This Project Do? - -It lets anyone create a **fully functional email account** (with IMAP, SMTP, JMAP, and webmail) by clicking a button in Codesphere. Under the hood, a Node.js REST backend talks to [Stalwart Mail Server](https://stalw.art/) to create users, fetch DNS records, and discover JMAP session details — all automatically. - ---- - -## Architecture Overview - -```mermaid -graph TB - subgraph USER["👤 Hackathon Participant"] - CS_UI["Codesphere Dashboard"] - EMAIL_CLIENT["Email Client
(Thunderbird, Apple Mail, Outlook)"] - JMAP_APP["Your App
(JMAP API calls)"] - end - - subgraph CODESPHERE["☁️ Codesphere Platform"] - RECONCILER["Reconciliation Loop
polls every ~10s"] - end - - subgraph BACKEND["🔌 REST Backend (Node.js / Express)"] - ADAPTER["Managed Service
Adapter API
POST / GET / PATCH / DELETE"] - end - - subgraph STALWART["📬 Stalwart Mail Server"] - ADMIN_API["Admin API
/api/principal
/api/dns/records"] - JMAP_EP["JMAP Endpoint
/jmap/session
/jmap/"] - IMAP["IMAP Server
:993 (TLS)"] - SMTP["SMTP Server
:587 (STARTTLS)"] - WEBMAIL["Webmail UI
/login"] - end - - CS_UI -- "provision mailbox" --> RECONCILER - RECONCILER -- "HTTP REST" --> ADAPTER - ADAPTER -- "Create/Update/Delete users" --> ADMIN_API - ADAPTER -- "Fetch JMAP IDs" --> JMAP_EP - ADAPTER -- "Fetch DNS records" --> ADMIN_API - - EMAIL_CLIENT -- "IMAP (read mail)" --> IMAP - EMAIL_CLIENT -- "SMTP (send mail)" --> SMTP - JMAP_APP -- "JMAP (send/read)" --> JMAP_EP - CS_UI -- "open webmail" --> WEBMAIL - - style USER fill:#e8f4fd,stroke:#2196F3,color:#000 - style CODESPHERE fill:#e8f5e9,stroke:#4CAF50,color:#000 - style BACKEND fill:#fff3e0,stroke:#FF9800,color:#000 - style STALWART fill:#fce4ec,stroke:#E91E63,color:#000 -``` - -**How the pieces connect:** - -| Component | Role | Tech | -|-----------|------|------| -| **Codesphere** | Orchestrates provisioning, shows UI to users | Platform | -| **REST Backend** | Translates Codesphere API calls → Stalwart API calls | Node.js + Express | -| **Stalwart** | Stores mailboxes, handles email protocols | Rust mail server | - ---- - -## Quickstart Flowchart - -```mermaid -flowchart TB - START(["🏁 Start Here"]) - - CLONE["1. Clone the repo
git clone stalwart-provider"] - DOCKER["2. Start Stalwart
docker compose -f docker-compose.local.yml up -d"] - BACKEND["3. Start REST Backend
cd src/rest-backend && npm install && node server.js"] - CREATE["4. Create a mailbox
curl -X POST localhost:9090/"] - GET["5. Get connection details
curl localhost:9090/?id=..."] - - CHOOSE{How will you
send email?} - - JMAP_PATH["Use JMAP API
accountId + identityId
+ draftsMailboxId
from details response"] - SMTP_PATH["Use SMTP
smtp_host + smtp_port
+ username + password"] - IMAP_PATH["Use IMAP
imap_host + imap_port
+ username + password"] - WEBMAIL_PATH["Use Webmail
Open webmail_url
in browser"] - - DONE(["🎉 Sending & Receiving!"]) - - START --> CLONE --> DOCKER --> BACKEND --> CREATE --> GET --> CHOOSE - CHOOSE -- "Programmatic
(recommended)" --> JMAP_PATH --> DONE - CHOOSE -- "Traditional" --> SMTP_PATH --> DONE - CHOOSE -- "Read mail" --> IMAP_PATH --> DONE - CHOOSE -- "Quick test" --> WEBMAIL_PATH --> DONE - - style START fill:#4CAF50,stroke:#2E7D32,color:#fff - style DONE fill:#4CAF50,stroke:#2E7D32,color:#fff - style CHOOSE fill:#FFF9C4,stroke:#F9A825,color:#000 - style JMAP_PATH fill:#E8F5E9,stroke:#43A047,color:#000 - style SMTP_PATH fill:#E3F2FD,stroke:#1E88E5,color:#000 - style IMAP_PATH fill:#F3E5F5,stroke:#8E24AA,color:#000 - style WEBMAIL_PATH fill:#FFF3E0,stroke:#EF6C00,color:#000 -``` - ---- - -## Step-by-Step Setup - -### Prerequisites - -- Docker and Docker Compose -- Node.js 18+ -- `curl` (for testing) - -### 1. Start Stalwart Mail Server - -```bash -docker compose -f docker-compose.local.yml up -d -``` - -This gives you a local Stalwart instance: - -```mermaid -graph LR - subgraph DOCKER["🐳 Docker Compose (Local Dev)"] - direction TB - SW["stalwartlabs/stalwart:v0.13.2"] - BE["REST Backend
node server.js"] - end - - subgraph PORTS["🔗 Exposed Ports"] - direction TB - P1080["1080 → HTTP Admin + JMAP + Webmail"] - P1587["1587 → SMTP Submission"] - P1993["1993 → IMAPS"] - P9090["9090 → REST Backend API"] - end - - subgraph STORAGE["💾 Persistence"] - VOL["stalwart-data
/opt/stalwart-mail"] - end - - SW --> P1080 - SW --> P1587 - SW --> P1993 - BE --> P9090 - BE -- "STALWART_API_URL" --> SW - SW --> VOL - - style DOCKER fill:#e3f2fd,stroke:#1565C0,color:#000 - style PORTS fill:#f3e5f5,stroke:#7B1FA2,color:#000 - style STORAGE fill:#fff8e1,stroke:#F57F17,color:#000 -``` - -| Port | Service | What it's for | -|------|---------|---------------| -| `1080` | HTTP | Admin UI, Webmail, JMAP endpoint | -| `1587` | SMTP | Send emails (submission port) | -| `1993` | IMAPS | Read emails (TLS) | -| `9090` | REST Backend | Codesphere adapter API | - -> **Admin login:** `admin` / `localdev123` at http://localhost:1080 - -### 2. Start the REST Backend - -```bash -cd src/rest-backend -npm install - -STALWART_API_URL=http://localhost:1080 \ -STALWART_ADMIN_TOKEN="admin:localdev123" \ -STALWART_IMAP_HOST=localhost \ -STALWART_SMTP_HOST=localhost \ -STALWART_IMAP_PORT=1993 \ -STALWART_SMTP_PORT=1587 \ -PORT=9090 \ -node server.js -``` - -### 3. Create a Mailbox - -```bash -curl -s -X POST http://localhost:9090/ \ - -H 'Content-Type: application/json' \ - -d '{ - "id": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", - "plan": {"id": 0, "parameters": {}}, - "config": { - "EMAIL_PREFIX": "alice", - "MAIL_DOMAIN": "example.com", - "DISPLAY_NAME": "Alice" - }, - "secrets": {"MAIL_PASSWORD": "supersecret123"} - }' -``` - -### 4. Get Connection Details - -```bash -curl -s http://localhost:9090/?id=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee | python3 -m json.tool -``` - -You'll get back everything needed to connect: - -```json -{ - "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee": { - "plan": { "id": 0, "parameters": {} }, - "config": { "EMAIL_PREFIX": "alice", "MAIL_DOMAIN": "example.com", "DISPLAY_NAME": "Alice" }, - "details": { - "email": "alice@example.com", - "username": "alice", - "mail_domain": "example.com", - "imap_host": "localhost", - "imap_port": 1993, - "smtp_host": "localhost", - "smtp_port": 1587, - "jmap_url": "http://localhost:1080/jmap", - "jmap_account_id": "h", - "jmap_identity_id": "b", - "jmap_drafts_mailbox_id": "d", - "webmail_url": "http://localhost:1080/login", - "dns_records": "MX example.com. 10 ...\nTXT example.com. v=spf1 ...", - "ready": true - } - } -} -``` - ---- - -## How Provisioning Works - -```mermaid -sequenceDiagram - actor User as 👤 Participant - participant CS as ☁️ Codesphere - participant BE as 🔌 REST Backend - participant ST as 📬 Stalwart - - Note over User,ST: 1️⃣ Provision a Mailbox - User->>CS: Create managed service
(EMAIL_PREFIX, MAIL_DOMAIN, PASSWORD) - CS->>BE: POST / {id, config, secrets, plan} - BE->>ST: POST /api/principal {type: "domain", name: "example.com"} - ST-->>BE: 200 OK (or alreadyExists) - BE->>ST: POST /api/principal {type: "individual", name: "alice", ...} - ST-->>BE: 200 OK {data: principalId} - - Note over BE,ST: Fetch connection details in parallel - par DNS Records - BE->>ST: GET /api/dns/records/example.com - ST-->>BE: [{type: "MX", ...}, {type: "TXT", ...}] - and JMAP Discovery - BE->>ST: GET /jmap/session (as alice) - ST-->>BE: {primaryAccounts: {mail: "accountId"}} - BE->>ST: POST /jmap/ Identity/get + Mailbox/get - ST-->>BE: {identityId, draftsMailboxId} - end - - BE-->>CS: 201 Created - CS-->>User: ✅ Mailbox ready! - - Note over User,ST: 2️⃣ Codesphere polls details - CS->>BE: GET /?id= - BE-->>CS: {plan, config, details: {email, imap_host, jmap_account_id, dns_records, ...}} - CS-->>User: Show connection details -``` - ---- - -## REST Adapter API Reference - -```mermaid -graph TB - subgraph ADAPTER_API["🔌 REST Backend — Adapter API Contract"] - direction TB - POST["POST /
Create mailbox user
→ 201 Created"] - GET["GET /?id=uuid
Get service status + details
→ 200 JSON"] - PATCH["PATCH /:id
Update password, display name, quota
→ 204 No Content"] - DELETE["DELETE /:id
Remove mailbox user
→ 204 No Content"] - end - - subgraph REQUEST["📥 POST Request Body"] - REQ_ID["id: UUID"] - REQ_PLAN["plan: {id: 0, parameters: {}}"] - REQ_CONFIG["config: {
EMAIL_PREFIX: 'alice',
MAIL_DOMAIN: 'example.com',
DISPLAY_NAME: 'Alice'
}"] - REQ_SECRETS["secrets: {MAIL_PASSWORD: '***'}"] - end - - subgraph RESPONSE["📤 GET Response (details)"] - RES_EMAIL["email: alice@example.com"] - RES_IMAP["imap_host / imap_port"] - RES_SMTP["smtp_host / smtp_port"] - RES_JMAP["jmap_url / jmap_account_id
jmap_identity_id
jmap_drafts_mailbox_id"] - RES_DNS["dns_records: SPF, DKIM, DMARC, MX"] - RES_READY["ready: true"] - end - - POST --- REQUEST - GET --- RESPONSE - - style ADAPTER_API fill:#FFF3E0,stroke:#E65100,color:#000 - style REQUEST fill:#E8F5E9,stroke:#2E7D32,color:#000 - style RESPONSE fill:#E3F2FD,stroke:#1565C0,color:#000 -``` - -| Endpoint | Method | Purpose | Response | -|----------|--------|---------|----------| -| `/` | `POST` | Create a new mailbox | `201` (empty body) | -| `/?id=` | `GET` | Get mailbox status & connection details | `200` JSON | -| `/:id` | `PATCH` | Update display name, quota, or password | `204` | -| `/:id` | `DELETE` | Delete mailbox and Stalwart user | `204` | - -### Config Fields - -| Field | Type | Mutable | Description | -|-------|------|---------|-------------| -| `EMAIL_PREFIX` | string | ❌ immutable | Local part of the email (before `@`) | -| `MAIL_DOMAIN` | string | ❌ immutable | Email domain (auto-created in Stalwart) | -| `DISPLAY_NAME` | string | ✅ | Friendly name shown in email clients | -| `QUOTA_MB` | integer | ✅ | Storage quota in MB (0 = unlimited) | - -### Secret Fields - -| Field | Type | Description | -|-------|------|-------------| -| `MAIL_PASSWORD` | password | Mailbox login password | - ---- - -## Sending Email via JMAP - -JMAP is the modern replacement for SMTP. The managed service returns everything you need — no discovery step required. - -```mermaid -sequenceDiagram - actor App as 🚀 Your App - participant ST as 📬 Stalwart JMAP - - Note over App,ST: Send an email via JMAP (using details from managed service) - - App->>ST: POST /jmap/
Email/set (create draft in Drafts mailbox) - Note right of App: accountId, draftsMailboxId
from service details - ST-->>App: {created: {draft1: {id: "emailId"}}} - - App->>ST: POST /jmap/
EmailSubmission/set (send it) - Note right of App: identityId from
service details - ST-->>App: {created: {sub1: {id: "..."}}} - ST->>ST: Deliver via SMTP - - Note over App,ST: ✅ Email sent! No SMTP config needed. -``` - -### Copy-Paste Example - -Replace the values from your `GET` response: - -```bash -curl -s http://localhost:1080/jmap/ \ - -u 'alice:supersecret123' \ - -H 'Content-Type: application/json' \ - -d '{ - "using": [ - "urn:ietf:params:jmap:core", - "urn:ietf:params:jmap:mail", - "urn:ietf:params:jmap:submission" - ], - "methodCalls": [ - ["Email/set", { - "accountId": "", - "create": { - "draft1": { - "mailboxIds": {"": true}, - "from": [{"name": "Alice", "email": "alice@example.com"}], - "to": [{"name": "Bob", "email": "bob@example.com"}], - "subject": "Hello from the hackathon!", - "textBody": [{"partId": "body", "type": "text/plain"}], - "bodyValues": { - "body": { - "value": "This email was sent programmatically via JMAP!", - "isEncodingProblem": false - } - } - } - } - }, "c1"], - ["EmailSubmission/set", { - "accountId": "", - "create": { - "sub1": { - "identityId": "", - "emailId": "#draft1" - } - } - }, "c2"] - ] - }' -``` - -> **Tip:** The `#draft1` reference automatically resolves to the email ID created in the first method call. - ---- - -## File Structure - -``` -stalwart-provider/ -├── config/ -│ └── provider.yml ← Provider definition (schemas, plans, backend URL) -├── src/ -│ └── rest-backend/ -│ ├── server.js ← REST backend (the main code!) -│ └── package.json -├── docker-compose.local.yml ← Local Stalwart for development -├── scripts/ -│ ├── validate.sh ← Validates provider.yml -│ └── register.sh ← Registers provider with Codesphere -├── Makefile ← make validate / make register -├── STALWART_SETUP.md ← Production deployment guide -└── HACKATHON.md ← You are here! -``` - ---- - -## Ideas for Extending - -Here are some things you could build on top of this: - -| Idea | Difficulty | Description | -|------|-----------|-------------| -| **Email-sending microservice** | ⭐ Easy | Build an app that provisions a mailbox and sends transactional emails via JMAP | -| **Newsletter platform** | ⭐⭐ Medium | Create a service that sends bulk emails using JMAP batch operations | -| **Multi-tenant SaaS email** | ⭐⭐ Medium | Let each customer bring their own domain, auto-configure DNS | -| **Email webhook bridge** | ⭐⭐ Medium | Poll JMAP for new emails and forward them to a webhook | -| **Persistent storage** | ⭐⭐ Medium | Replace the in-memory `Map()` with a database (SQLite, PostgreSQL) | -| **Mailing list manager** | ⭐⭐⭐ Hard | Use Stalwart's `list` principal type to manage mailing lists | - ---- - -## Troubleshooting - -| Problem | Cause | Fix | -|---------|-------|-----| -| `Connection refused` on port 1080 | Stalwart not running | `docker compose -f docker-compose.local.yml up -d` | -| `401 Unauthorized` from Stalwart | Wrong admin password | Check `STALWART_ADMIN_TOKEN` matches `admin:localdev123` | -| Domain creation fails | Stalwart IP-blocked you | `docker compose -f docker-compose.local.yml down -v && docker compose -f docker-compose.local.yml up -d` (wipes data!) | -| JMAP details are empty | User just created, needs a moment | JMAP session may take a second to initialize. Retry the GET. | -| Port 9090 in use | Old backend still running | `lsof -ti:9090 \| xargs kill` | -| Emails to Gmail rejected | Missing DNS/TLS (local only) | This is expected locally. See `STALWART_SETUP.md` for production DNS config. | - ---- - -## Environment Variable Reference - -| Variable | Required | Default | Description | -|----------|----------|---------|-------------| -| `STALWART_API_URL` | ✅ | — | Stalwart admin API base URL | -| `STALWART_ADMIN_TOKEN` | ✅ | — | `user:password` for Basic Auth or Bearer token | -| `STALWART_IMAP_HOST` | ✅ | — | Public IMAP hostname | -| `STALWART_SMTP_HOST` | ✅ | — | Public SMTP hostname | -| `STALWART_IMAP_PORT` | — | `993` | IMAP port | -| `STALWART_SMTP_PORT` | — | `587` | SMTP submission port | -| `STALWART_MAIL_DOMAIN` | — | — | Default domain (if not set per-service) | -| `PORT` | — | `8080` | Backend listen port | -| `AUTH_TOKEN` | — | — | Bearer token for securing the backend | diff --git a/JMAP_GUIDE.md b/JMAP_GUIDE.md deleted file mode 100644 index f132cf0..0000000 --- a/JMAP_GUIDE.md +++ /dev/null @@ -1,216 +0,0 @@ -# Sending Email via JMAP — Explained - -This guide breaks down how to send an email using the JMAP API and how to verify it worked. - ---- - -## The Command - -```bash -curl -s http://localhost:1080/jmap/ \ - -u 'jd:jd' \ - -H 'Content-Type: application/json' \ - -d '{ ... }' -``` - -| Part | What it does | -|------|--------------| -| `http://localhost:1080/jmap/` | The JMAP endpoint on your Stalwart server | -| `-u 'jd:jd'` | Authenticates as user `jd` with password `jd` (Basic Auth) | -| `-H 'Content-Type: application/json'` | Tells the server we're sending JSON | -| `-d '{ ... }'` | The JMAP request body (see below) | - ---- - -## The Request Body — Step by Step - -A single JMAP request contains **two method calls** that execute in order: - -### Step 1: `Email/set` — Create the email draft - -```json -["Email/set", { - "accountId": "j", - "create": { - "draft1": { - "mailboxIds": {"d": true}, - "from": [{"name": "JD", "email": "jd@codesphere.com"}], - "to": [{"name": "Recipient", "email": "recipient@example.com"}], - "subject": "Hello from Codesphere!", - "textBody": [{"partId": "body", "type": "text/plain"}], - "bodyValues": { - "body": { - "value": "This email was sent via JMAP from a Codesphere managed service!", - "isEncodingProblem": false - } - } - } - } -}, "c1"] -``` - -| Field | Value | Meaning | -|-------|-------|---------| -| `accountId` | `"j"` | Your JMAP account ID (from service details) | -| `"draft1"` | — | A temporary client-side label for this email (you pick the name) | -| `mailboxIds` | `{"d": true}` | Put the email in the Drafts mailbox (`"d"` = your drafts mailbox ID) | -| `from` | `jd@codesphere.com` | The sender address | -| `to` | `recipient@example.com` | The recipient | -| `subject` | `"Hello from Codesphere!"` | Email subject line | -| `bodyValues.body.value` | `"This email was sent..."` | The plain text body | - -> This only **creates** the email object. It doesn't send it yet. - -### Step 2: `EmailSubmission/set` — Actually send it - -```json -["EmailSubmission/set", { - "accountId": "j", - "create": { - "sub1": { - "identityId": "b", - "emailId": "#draft1" - } - } -}, "c2"] -``` - -| Field | Value | Meaning | -|-------|-------|---------| -| `identityId` | `"b"` | Your sender identity ID (from service details) | -| `emailId` | `"#draft1"` | A back-reference — resolves to the email ID created in step 1 | - -> The `#draft1` syntax is a JMAP **creation reference**. It automatically gets replaced with the actual email ID from the `Email/set` response. This is how the two steps are linked in a single request. - ---- - -## The Response — What It Means - -```json -{ - "methodResponses": [ - ["Email/set", { - "accountId": "j", - "oldState": "saa", - "newState": "sam", - "created": { - "draft1": { - "id": "eaaaaab", - "threadId": "b", - "blobId": "cagjaxd0mob...", - "size": 383 - } - } - }, "c1"], - ["EmailSubmission/set", { - "accountId": "j", - "newState": "saq", - "created": { - "sub1": { - "id": "b" - } - } - }, "c2"] - ], - "sessionState": "3e25b2a0" -} -``` - -### Reading the response - -| Response Part | What it tells you | -|---------------|-------------------| -| `Email/set → created.draft1` | The email was successfully created. It got ID `eaaaaab`. | -| `EmailSubmission/set → created.sub1` | The email was successfully **submitted for delivery**. | -| No `"notCreated"` key | Nothing went wrong. If something fails, errors appear here. | - -**If you see `"created"` in both responses → the email was sent.** - ---- - -## How to Verify the Email Was Sent - -### 1. Check the Sent mailbox via JMAP - -Query for emails in your Sent folder: - -```bash -curl -s http://localhost:1080/jmap/ \ - -u 'jd:jd' \ - -H 'Content-Type: application/json' \ - -d '{ - "using": ["urn:ietf:params:jmap:core", "urn:ietf:params:jmap:mail"], - "methodCalls": [ - ["Mailbox/get", { - "accountId": "j", - "properties": ["name", "role", "totalEmails"] - }, "m1"] - ] - }' | python3 -m json.tool -``` - -Look for the mailbox with `"role": "sent"` — its `totalEmails` count tells you how many emails have been sent. - -### 2. Fetch the actual sent email - -Once you know the Sent mailbox ID (from the response above), list emails in it: - -```bash -curl -s http://localhost:1080/jmap/ \ - -u 'jd:jd' \ - -H 'Content-Type: application/json' \ - -d '{ - "using": ["urn:ietf:params:jmap:core", "urn:ietf:params:jmap:mail"], - "methodCalls": [ - ["Email/query", { - "accountId": "j", - "filter": {"subject": "Hello from Codesphere!"}, - "sort": [{"property": "receivedAt", "isAscending": false}], - "limit": 5 - }, "q1"], - ["Email/get", { - "accountId": "j", - "#ids": {"resultOf": "q1", "name": "Email/query", "path": "/ids"}, - "properties": ["subject", "from", "to", "sentAt", "preview"] - }, "g1"] - ] - }' | python3 -m json.tool -``` - -This searches for the email by subject and returns its details. You should see your sent email with the subject, sender, recipient, and a preview of the body. - -### 3. Check via Webmail - -Open http://localhost:1080/login in your browser, log in as `jd` / `jd`, and check the **Sent** folder. - -### 4. Check Stalwart server logs - -```bash -docker logs stalwart-mail 2>&1 | grep -i "queue\|deliver\|sent" | tail -20 -``` - -This shows Stalwart's delivery queue activity. You'll see entries for outbound delivery attempts. - -> **Note:** In local development, delivery to external addresses (like `recipient@example.com`) will fail because there's no real DNS or TLS. The email is still _sent_ from Stalwart's perspective — it just can't be _delivered_ to the outside world. To test end-to-end, send between two mailboxes on the same server (e.g. `jd@codesphere.com` → `alice@codesphere.com`). - ---- - -## Quick Reference - -| Value | Where it comes from | What it is | -|-------|-------------------|------------| -| `accountId: "j"` | `GET` service details → `jmap_account_id` | Your JMAP account | -| `identityId: "b"` | `GET` service details → `jmap_identity_id` | Your sender identity | -| `mailboxIds: {"d": true}` | `GET` service details → `jmap_drafts_mailbox_id` | Drafts folder | -| `#draft1` | You choose this label | Back-reference to email created in same request | - ---- - -## Error Cases - -| Symptom | Cause | Fix | -|---------|-------|-----| -| `401 Unauthorized` | Wrong username or password | Check `-u 'user:password'` | -| `"notCreated"` in Email/set | Invalid mailbox ID or account ID | Verify IDs from service details | -| `"notCreated"` in EmailSubmission/set | Invalid identity ID | Verify `identityId` from service details | -| Email created but not in Sent | Submission failed silently | Check for errors in the `EmailSubmission/set` response | diff --git a/Makefile b/Makefile index 0ecdaa7..155aaa3 100644 --- a/Makefile +++ b/Makefile @@ -1,35 +1,32 @@ -.PHONY: help validate register test clean +.PHONY: help validate test clean start-api-backend send-mail SHELL := /bin/bash -# Configuration -PROVIDER_CONFIG := config/provider.yml -CI_CONFIG := config/ci.yml SCRIPTS_DIR := scripts help: ## Show available commands @echo "" - @echo "Codesphere Landscape Provider — Available Commands" - @echo "==================================================" - @grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf " \033[36m%-15s\033[0m %s\n", $$1, $$2}' + @echo "Stalwart Managed Service Provider — Available Commands" + @echo "======================================================" + @grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf " \033[36m%-20s\033[0m %s\n", $$1, $$2}' @echo "" -validate: ## Validate provider.yml and ci.yml +validate: ## Validate provider.yml @bash $(SCRIPTS_DIR)/validate.sh -register: validate ## Register the provider with Codesphere (validates first) - @bash $(SCRIPTS_DIR)/register.sh - -test: validate ## Deploy a test instance and run smoke tests +test: validate ## Run smoke tests (validates first) @bash $(SCRIPTS_DIR)/test-provider.sh -clean: ## Remove generated config files (keeps examples) - @echo "Cleaning generated configs..." - @rm -f $(PROVIDER_CONFIG) $(CI_CONFIG) - @echo "Done." - -start-api-backend: ## Start the API backend for local development - STALWART_API_URL=http://localhost:1080 STALWART_ADMIN_TOKEN="admin:localdev123" STALWART_IMAP_HOST=localhost STALWART_SMTP_HOST=localhost STALWART_IMAP_PORT=1993 STALWART_SMTP_PORT=1587 PORT=9090 node server.js +start-api-backend: ## Start the REST backend for local development + cd src && \ + STALWART_API_URL=http://localhost:1080 \ + STALWART_ADMIN_TOKEN="admin:localdev123" \ + STALWART_IMAP_HOST=localhost \ + STALWART_SMTP_HOST=localhost \ + STALWART_IMAP_PORT=1993 \ + STALWART_SMTP_PORT=1587 \ + PORT=9090 \ + node server.js # JMAP test email configuration (override with env vars) JMAP_URL ?= http://localhost:1080/jmap/ @@ -52,4 +49,4 @@ send-mail: ## Send a test email via JMAP (override with JMAP_TO_EMAIL=... etc.) -H 'Content-Type: application/json' \ -d '{"using":["urn:ietf:params:jmap:core","urn:ietf:params:jmap:mail","urn:ietf:params:jmap:submission"],"methodCalls":[["Email/set",{"accountId":"$(JMAP_ACCOUNT_ID)","create":{"draft1":{"mailboxIds":{"$(JMAP_DRAFTS_ID)":true},"from":[{"name":"$(JMAP_FROM_NAME)","email":"$(JMAP_FROM_EMAIL)"}],"to":[{"name":"$(JMAP_TO_NAME)","email":"$(JMAP_TO_EMAIL)"}],"subject":"$(JMAP_SUBJECT)","textBody":[{"partId":"body","type":"text/plain"}],"bodyValues":{"body":{"value":"$(JMAP_BODY)","isEncodingProblem":false}}}}},"c1"],["EmailSubmission/set",{"accountId":"$(JMAP_ACCOUNT_ID)","create":{"sub1":{"identityId":"$(JMAP_IDENTITY_ID)","emailId":"#draft1"}}},"c2"]]}' | python3 -m json.tool @echo "" - @echo "✅ Done. Check inbox at $(JMAP_TO_EMAIL) or open http://localhost:1080/login" \ No newline at end of file + @echo "Done. Check inbox at $(JMAP_TO_EMAIL) or open http://localhost:1080/login" diff --git a/README.md b/README.md index 1dae8cc..3ac88ea 100644 --- a/README.md +++ b/README.md @@ -1,372 +1,45 @@ -# Codesphere Managed Service Provider Template +# Stalwart Managed Service Provider -A template repository for creating **managed service providers** on the Codesphere platform. Supports both **landscape-based** and **REST backend** providers. Clone this repo, describe what service you want to offer, and let the AI agent scaffold the entire provider for you — then register it with a single `make` command. +A Codesphere managed service provider that wraps [Stalwart Mail Server](https://stalw.art/) as a self-service email offering. Users can provision individual mailboxes (with IMAP, SMTP, JMAP, and webmail access) through the Codesphere marketplace. ---- +## How It Works -## What Is a Service Provider? +A single shared Stalwart instance hosts many mail accounts. When a user books the service through Codesphere, the REST backend creates a new mail account ("logical tenant") on that shared server. When they delete the service, the account is removed. -A **service provider** defines a reusable blueprint that others can instantiate as managed services on the Codesphere platform. Providers can be backed by one of two types: - -### Landscape-based Providers - -Transforms a Codesphere landscape into a reusable blueprint. Codesphere handles provisioning internally through the landscape's CI pipeline. - -- **Metadata** — name, version, display name, category, description, icon -- **Backend** — Git repository URL and CI profile reference -- **CI Pipeline** — `ci.yml` defining prepare, build, and run stages -- **Configuration schemas** — `configSchema`, `secretsSchema`, `detailsSchema` - -### REST Backend Providers - -Connects to a custom REST API that handles provisioning and lifecycle management externally. Your backend implements the Codesphere Managed Service Adapter API (POST, GET, PATCH, DELETE). - -- **Metadata** — name, version, display name, category, description, icon -- **Backend** — REST endpoint URL and authentication -- **Configuration schemas** — `configSchema`, `secretsSchema`, `detailsSchema`, `planSchema` -- **No `ci.yml` needed** — the REST backend handles all provisioning - -When registered, your provider appears in the Codesphere Marketplace and can be deployed by any team with access. - ---- - -## Quick Start - -### Prerequisites - -| Requirement | Why | -|-------------|-----| -| [Codesphere account](https://codesphere.com) | To register and deploy providers | -| `CODESPHERE_API_TOKEN` | API Bearer token for authentication (set as env var) | -| `CODESPHERE_TEAM_ID` | Your team ID — for team-scoped providers (set as env var) | -| `CODESPHERE_URL` | Codesphere instance URL (default: `https://codesphere.com`) | -| `make` | Build automation | -| `yq` (optional) | YAML validation in scripts | -| Git provider configured | Git permissions in your Codesphere account settings | - -### 1. Clone this template - -```bash -git clone https://github.com/codesphere-cloud/ms-landscape-template.git my-provider -cd my-provider ``` - -### 2. Tell the agent what you want - -Open the project in VS Code with GitHub Copilot enabled, then prompt: - -> "I want to create a **landscape provider** for PostgreSQL 16 with automated backups and a health check endpoint." - -Or for a REST backend: - -> "I want to create a **REST backend provider** for a custom database service with provisioning via my existing API." - -The agent will: -- Read the instructions in `.github/copilot-instructions.md` -- Follow the detailed schema in `.github/instructions/PROVIDER.instructions.md` -- For landscape providers: generate `config/ci.yml` using `.github/instructions/CI.instructions.md` -- For REST providers: scaffold the backend in `src/rest-backend/` -- Generate `config/provider.yml` from the appropriate example - -### 3. Validate locally - -```bash -make validate +Codesphere Platform ◄──► REST Backend (this repo) ──► Stalwart Mail Server + (reconcile loop) POST/GET/PATCH/DELETE (shared instance) ``` -This checks your `provider.yml` and `ci.yml` against the expected schema and catches common mistakes before registration. - -### 4. Register the provider - -```bash -make register -``` - -This calls the Codesphere API to register your provider with your team. - -### 5. Test the provider - -```bash -make test -``` - -Deploys a test workspace with your provider and runs smoke tests. - ---- - -## Project Structure - -``` -ms-landscape-template/ -├── README.md # This file -├── Makefile # validate, register, test commands -│ -├── .github/ -│ ├── copilot-instructions.md # Agent persona & workflow (auto-loaded) -│ └── instructions/ -│ ├── PROVIDER.instructions.md # Provider definition schema & rules -│ └── CI.instructions.md # CI pipeline schema & rules -│ -├── config/ -│ ├── provider.yml.example # Example landscape-based provider -│ ├── provider.rest.yml.example # Example REST backend provider -│ └── ci.yml.example # Example CI pipeline config (landscape only) -│ -├── scripts/ -│ ├── validate.sh # Validate config files locally -│ ├── register.sh # Register provider via API -│ └── test-provider.sh # Smoke-test a deployed provider -│ -└── src/ # Provider source code (agent-generated) - ├── .gitkeep - └── rest-backend/ # Example REST backend implementation - ├── README.md - ├── package.json - └── server.js -``` - -### Key Files - -| File | Purpose | -|------|---------| -| `config/provider.yml` | Provider definition: metadata, backend, config/secrets/details schemas | -| `config/ci.yml` | CI pipeline: prepare and run stages for the landscape (landscape providers only) | -| `config/provider.rest.yml.example` | Example REST backend provider definition | -| `src/rest-backend/` | Example REST backend implementing the Managed Service Adapter API | -| `.github/copilot-instructions.md` | Tells the AI agent *what this project is* and *how to work in it* | -| `.github/instructions/PROVIDER.instructions.md` | Detailed schema reference for `provider.yml` (both types) | -| `.github/instructions/CI.instructions.md` | Detailed schema reference for `ci.yml` (landscape only) | - ---- +## Repository Structure -## Configuration Reference - -### provider.yml (Landscape Backend) - -The provider definition file for landscape-based providers: - -```yaml -name: mattermost # Unique name (lowercase, hyphens, underscores) -version: v1 # Version: v1, v2, etc. (NOT semver) -author: Your Team -displayName: Mattermost # Human-readable name for the Marketplace -category: collaboration # e.g., databases, messaging, monitoring -description: | # Markdown description - Open-source team messaging and collaboration platform. - -backend: - landscape: - gitUrl: https://github.com/your-org/mattermost-landscape - ciProfile: production # CI profile name from ci.yml - -configSchema: # User-configurable options → env vars - type: object - properties: - SITE_NAME: - type: string - description: Display name for your instance - MAX_USERS: - type: integer - description: Maximum number of users - x-update-constraint: increase-only - -secretsSchema: # Secrets → stored in vault - type: object - properties: - ADMIN_PASSWORD: - type: string - format: password - -detailsSchema: # Runtime details exposed after provisioning - type: object - properties: - hostname: - type: string - port: - type: integer ``` - -**Key concepts:** -- Config values are referenced in ci.yml as `${{ workspace.env['NAME'] }}` -- Secrets are referenced in ci.yml as `${{ vault.SECRET_NAME }}` -- Use `x-update-constraint: increase-only` or `immutable` to restrict post-creation updates -- Use `x-endpoint` in detailsSchema to fetch live data from the running service - -### provider.yml (REST Backend) - -The provider definition file for REST backend providers: - -```yaml -name: custom-postgres # Unique name (lowercase, hyphens, underscores) -version: v1 # Version: v1, v2, etc. (NOT semver) -author: Your Team -displayName: Custom PostgreSQL -category: databases -description: | - Custom PostgreSQL provider backed by an external REST API. - -backend: - rest: - url: https://my-backend.example.com/postgres - authTokenEnv: BACKEND_AUTH_TOKEN # Env var name (not the token!) - -configSchema: # User-configurable options - type: object - properties: - DATABASE_NAME: - type: string - description: Name of the database to create - VERSION: - type: string - description: PostgreSQL version - enum: ['16.10', '15.14'] - x-update-constraint: immutable - -secretsSchema: # Secrets sent to the REST backend - type: object - properties: - SUPERUSER_PASSWORD: - type: string - format: password - -detailsSchema: # Runtime details returned by the backend - type: object - properties: - hostname: - type: string - port: - type: integer - ready: - type: boolean - -planSchema: # Resource plan parameters - type: object - properties: - storage: - type: integer - description: Storage size in MB - x-update-constraint: increase-only - cpu: - type: integer - description: CPU allocation in tenths +├── src/ # REST backend (Node.js/Express) +│ ├── server.js # Managed service adapter implementation +│ └── package.json +├── ci.stalwart-provider.yml # CI pipeline for the REST backend +├── ci.stalwart.yml # CI pipeline for the Stalwart Mail Server +├── provider.yml # Marketplace service definition +├── docker-compose.local.yml # Local Stalwart for development +├── examples/ # Generic provider.yml / ci.yml examples +├── Makefile # validate, test, start-api-backend, send-mail +└── WORKSHOP_TUTORIAL.md # Step-by-step workshop guide ``` -**Key concepts:** -- No `ci.yml` is needed — the REST backend handles provisioning -- `planSchema` defines resource parameters sent in `plan.parameters` to the backend -- The REST backend must implement: `POST /`, `GET /?id=...`, `PATCH /{id}`, `DELETE /{id}` -- Auth token is referenced by env var name, never hardcoded - -### ci.yml (Landscape Providers Only) - -The CI pipeline definition. Defines how to prepare the environment and orchestrate landscape services: - -```yaml -schemaVersion: v0.2 +## Quick Start -prepare: # Build stage — runs on Workspace compute - steps: - - name: Install Dependencies - command: nix-env -iA nixpkgs.nodejs +```bash +# Local development with Docker +docker compose -f docker-compose.local.yml up -d +make start-api-backend -run: # Landscape services — run in parallel - webapp: # Codesphere Reactive - plan: 21 - steps: - - command: npm start - env: - DB_HOST: ms-postgres-v1-42-primary-db - SECRET: ${{ vault.MY_SECRET }} - network: - ports: - - port: 3000 - isPublic: false - paths: - - port: 3000 - path: / +# Validate provider.yml +make validate - primary-db: # Managed Service from marketplace - provider: - name: postgres - version: v1 - plan: - id: 0 +# Send a test email via JMAP +make send-mail JMAP_TO_EMAIL=someone@example.com ``` -**Key concepts:** -- `prepare` installs deps and builds assets on the shared filesystem (`/home/user/app/`) -- `run` defines services: Reactives (`steps`), Managed Containers (`baseImage`), or Managed Services (`provider`) -- Services communicate via private networking: `http://ws-server-[WorkspaceId]-[serviceName]:[port]` -- Environment templates: `${{ vault.NAME }}`, `${{ workspace.env['KEY'] }}`, `${{ workspace.id }}`, `${{ team.id }}` - -> Full ci.yml schema documented in `.github/instructions/CI.instructions.md` - ---- - -## Makefile Commands - -| Command | Description | -|---------|-------------| -| `make validate` | Validate `provider.yml` (and `ci.yml` for landscape providers) | -| `make register` | Register the provider with Codesphere (requires `CODESPHERE_API_TOKEN`) | -| `make test` | Deploy a test instance and run smoke tests | -| `make clean` | Remove generated files | -| `make help` | Show all available commands | - ---- - -## Development Workflow - -### Using the AI Agent - -This template is designed to be **agent-first**. The recommended workflow: - -1. **Describe your service** — Tell the agent what managed service you want to create, and whether it should be landscape-based or REST backend -2. **Review generated configs** — The agent creates `provider.yml` (and `ci.yml` for landscape providers) -3. **Add custom logic** — For landscape: setup scripts in `src/`. For REST: implement your backend in `src/rest-backend/` -4. **Validate** — Run `make validate` to catch issues -5. **Register** — Run `make register` to publish to Codesphere -6. **Test** — Run `make test` to verify end-to-end - -### Manual Workflow — Landscape Provider - -1. Copy `config/provider.yml.example` → `config/provider.yml` -2. Copy `config/ci.yml.example` → `config/ci.yml` -3. Edit both files following `.github/instructions/PROVIDER.instructions.md` and `.github/instructions/CI.instructions.md` -4. Add any source code to `src/` -5. Run `make validate && make register` - -### Manual Workflow — REST Backend Provider - -1. Copy `config/provider.rest.yml.example` → `config/provider.yml` -2. Edit the file following `.github/instructions/PROVIDER.instructions.md` -3. Implement or customize the REST backend in `src/rest-backend/` (see the example) -4. Deploy the backend and update `backend.rest.url` in `provider.yml` -5. Run `make validate && make register` - ---- - -## Troubleshooting - -| Problem | Solution | -|---------|----------| -| `make validate` fails with schema errors | Check your YAML against the schema in `PROVIDER.instructions.md` | -| `make register` returns 401 | Verify `CODESPHERE_API_TOKEN` is set and valid | -| `make register` returns 409 | Provider name/version already registered — bump version or use a different name | -| Agent doesn't follow the template | Ensure `.github/copilot-instructions.md` exists and VS Code is using the workspace | -| `version` validation fails | Use `v1`, `v2` format — NOT semver like `1.0.0` | - ---- - -## Contributing - -1. Fork this repository -2. Create a feature branch -3. Make your changes -4. Submit a pull request - ---- - -## License +## Workshop -MIT +See [WORKSHOP_TUTORIAL.md](WORKSHOP_TUTORIAL.md) for the full hands-on guide. diff --git a/STALWART_SETUP.md b/STALWART_SETUP.md deleted file mode 100644 index 19a10c0..0000000 --- a/STALWART_SETUP.md +++ /dev/null @@ -1,270 +0,0 @@ -# Stalwart Mail Server — Docker Setup Guide - -This guide walks you through deploying Stalwart Mail Server with Docker and configuring it to work with the Codesphere managed service provider backend. - ---- - -## Prerequisites - -- A server (VPS or dedicated) with a **public IP address** -- Docker and Docker Compose installed -- A **domain name** you control (e.g. `example.com`) -- DNS access to create MX, A, SPF, DKIM, and DMARC records - ---- - -## 1. DNS Configuration - -Before starting Stalwart, set up these DNS records for your domain (replace `203.0.113.10` with your server IP and `example.com` with your domain): - -| Type | Name | Value | TTL | -|-------|--------------------------|------------------------------------|------| -| A | `mail.example.com` | `203.0.113.10` | 3600 | -| MX | `example.com` | `10 mail.example.com` | 3600 | -| TXT | `example.com` | `v=spf1 a mx ip4:203.0.113.10 -all` | 3600 | -| TXT | `_dmarc.example.com` | `v=DMARC1; p=reject; rua=mailto:postmaster@example.com` | 3600 | - -> DKIM will be set up after Stalwart generates its keys (see step 4). - ---- - -## 2. Docker Compose Setup - -Create a project directory and the following files: - -```bash -mkdir -p stalwart-mail && cd stalwart-mail -``` - -### `docker-compose.yml` - -```yaml -services: - stalwart: - image: stalwartlabs/mail-server:latest - container_name: stalwart-mail - restart: unless-stopped - ports: - - "25:25" # SMTP - - "465:465" # SMTP/TLS (implicit) - - "587:587" # SMTP/STARTTLS (submission) - - "993:993" # IMAPS - - "4190:4190" # ManageSieve - - "443:443" # HTTPS (webmail + JMAP + admin) - - "8080:8080" # HTTP (redirect or API) - volumes: - - ./data:/opt/stalwart-mail - environment: - # The admin password is set on first run only. - # Change it via the web admin UI afterwards. - - STALWART_ADMIN_PASSWORD=changeme-on-first-login -``` - -### Start the server - -```bash -docker compose up -d -``` - -Stalwart will initialize its data directory on first boot. Give it a minute, then verify: - -```bash -docker logs stalwart-mail -``` - ---- - -## 3. Initial Admin Setup - -1. Open `https://mail.example.com` in your browser -2. Log in with: - - **Username:** `admin` - - **Password:** the value of `STALWART_ADMIN_PASSWORD` from docker-compose.yml -3. **Change the admin password immediately** via Settings → Account - -### Generate an API Token - -The REST backend needs an API token to manage accounts: - -1. In the Stalwart web admin, go to **Settings → API Keys** (or **Management → API Access**) -2. Create a new API token with **full account management** permissions -3. Copy the token — you'll need it for the backend configuration - -> If your Stalwart version uses Basic Auth for the management API instead of bearer tokens, you can base64-encode `admin:yourpassword` and use that as the token value instead. The backend sends it as a Bearer token. - ---- - -## 4. Configure TLS - -Stalwart supports automatic TLS via ACME (Let's Encrypt). In the Stalwart admin UI: - -1. Go to **Settings → TLS / ACME** -2. Enable ACME with Let's Encrypt -3. Set the domain to `mail.example.com` -4. Stalwart will automatically obtain and renew certificates - -Alternatively, mount your own certificates: - -```yaml -# In docker-compose.yml volumes: -volumes: - - ./data:/opt/stalwart-mail - - /etc/letsencrypt/live/mail.example.com:/opt/stalwart-mail/certs:ro -``` - ---- - -## 5. DKIM Setup - -After Stalwart starts, it generates DKIM keys. Retrieve your DKIM DNS record: - -1. In the admin UI, go to **Settings → Domain → DKIM** -2. Copy the DNS TXT record value -3. Add it to your DNS: - -| Type | Name | Value | -|------|-------------------------------------|------------------------------------| -| TXT | `._domainkey.example.com` | *(the DKIM public key from admin)* | - ---- - -## 6. Configure & Start the REST Backend - -The REST backend bridges Codesphere and Stalwart. It needs the following environment variables: - -### Required Environment Variables - -| Variable | Description | Example | -|-----------------------|----------------------------------------------------------|-----------------------------------| -| `STALWART_API_URL` | Stalwart admin API base URL | `https://mail.example.com` | -| `STALWART_ADMIN_TOKEN`| API token from step 3 | `your-api-token-here` | -| `STALWART_MAIL_DOMAIN`| Email domain for created accounts | `example.com` | -| `STALWART_IMAP_HOST` | Public IMAP hostname | `mail.example.com` | -| `STALWART_SMTP_HOST` | Public SMTP hostname | `mail.example.com` | -| `AUTH_TOKEN` | Bearer token for Codesphere → backend auth | `a-strong-random-secret` | - -### Optional Environment Variables - -| Variable | Default | Description | -|-----------------------|------------------------------------|-------------------------------------| -| `STALWART_IMAP_PORT` | `993` | Public IMAP port | -| `STALWART_SMTP_PORT` | `587` | Public SMTP submission port | -| `STALWART_JMAP_URL` | `${STALWART_API_URL}/jmap` | Public JMAP endpoint | -| `STALWART_WEBMAIL_URL` | `${STALWART_API_URL}/login` | Public webmail URL | -| `PORT` | `8080` | Backend listen port | - -### Run with Docker - -You can add the REST backend as a second service in the same docker-compose.yml or run it separately. Here's a standalone example: - -```yaml -# docker-compose.backend.yml -services: - stalwart-backend: - build: . - container_name: stalwart-backend - restart: unless-stopped - ports: - - "9090:8080" - environment: - - STALWART_API_URL=https://mail.example.com - - STALWART_ADMIN_TOKEN=your-api-token-here - - STALWART_MAIL_DOMAIN=example.com - - STALWART_IMAP_HOST=mail.example.com - - STALWART_SMTP_HOST=mail.example.com - - AUTH_TOKEN=a-strong-random-secret -``` - -Create a `Dockerfile` in `src/rest-backend/`: - -```dockerfile -FROM node:20-alpine -WORKDIR /app -COPY package.json ./ -RUN npm install --production -COPY server.js ./ -EXPOSE 8080 -USER node -CMD ["node", "server.js"] -``` - -Build and run: - -```bash -cd src/rest-backend -docker build -t stalwart-backend . -docker compose -f docker-compose.backend.yml up -d -``` - -Or run directly with Node.js: - -```bash -cd src/rest-backend -npm install -STALWART_API_URL=https://mail.example.com \ -STALWART_ADMIN_TOKEN=your-api-token \ -STALWART_MAIL_DOMAIN=example.com \ -STALWART_IMAP_HOST=mail.example.com \ -STALWART_SMTP_HOST=mail.example.com \ -AUTH_TOKEN=your-secret \ -npm start -``` - ---- - -## 7. Register the Provider on Codesphere - -Once the backend is running and reachable, update `config/provider.yml`: - -```yaml -backend: - rest: - url: https://your-stalwart-backend.example.com # ← your backend URL - authTokenEnv: BACKEND_AUTH_TOKEN -``` - -Then validate and register: - -```bash -# Validate the provider definition -make validate - -# Set required env vars -export CODESPHERE_API_TOKEN="your-codesphere-token" -export CODESPHERE_TEAM_ID="your-team-id" -export BACKEND_AUTH_TOKEN="a-strong-random-secret" # must match AUTH_TOKEN on the backend - -# Register -make register -``` - ---- - -## 8. What Users Get - -When a user provisions a Stalwart mailbox through Codesphere, they receive: - -| Field | Example | -|---------------|--------------------------------------| -| **Email** | `alice@example.com` | -| **Username** | `alice` | -| **IMAP Host** | `mail.example.com` | -| **IMAP Port** | `993` (TLS) | -| **SMTP Host** | `mail.example.com` | -| **SMTP Port** | `587` (STARTTLS) | -| **JMAP URL** | `https://mail.example.com/jmap` | -| **Webmail** | `https://mail.example.com/login` | - -They can use any standard email client (Thunderbird, Apple Mail, Outlook) with these settings. - ---- - -## Security Checklist - -- [ ] Changed default admin password -- [ ] TLS enabled (ACME or manual certificates) -- [ ] Firewall allows only necessary ports (25, 465, 587, 993, 443) -- [ ] REST backend behind HTTPS (use a reverse proxy like Caddy or nginx) -- [ ] `AUTH_TOKEN` set on both backend and Codesphere provider -- [ ] DKIM, SPF, and DMARC DNS records configured -- [ ] Stalwart admin API not directly exposed to the public internet (only backend can reach it) diff --git a/TUTORIAL.md b/TUTORIAL.md deleted file mode 100644 index cc2bd8f..0000000 --- a/TUTORIAL.md +++ /dev/null @@ -1,583 +0,0 @@ -# Tutorial: Building a Custom REST Backend Provider for Codesphere - -This tutorial walks you through creating a **REST backend managed service provider** for Codesphere — from scratch to a fully working local setup. We'll build a **Stalwart Mailbox** provider that provisions email accounts on a [Stalwart Mail Server](https://stalw.art/) instance. - -By the end, you'll have: -- A Stalwart Mail Server running locally in Docker -- A REST backend that creates/updates/deletes mailbox accounts via Stalwart's admin API -- A `provider.yml` registered with your local Codesphere dev instance -- Multi-domain support with automatic DNS record retrieval -- JMAP auto-discovery so consumers can send email programmatically - ---- - -## Table of Contents - -- [1. Architecture Overview](#1-architecture-overview) -- [2. Prerequisites](#2-prerequisites) -- [3. Step 1 — Start Stalwart Mail Server](#3-step-1--start-stalwart-mail-server) -- [4. Step 2 — Write the provider.yml](#4-step-2--write-the-provideryml) -- [5. Step 3 — Implement the REST Backend](#5-step-3--implement-the-rest-backend) -- [6. Step 4 — Test Locally](#6-step-4--test-locally) -- [7. Step 5 — Register with Codesphere](#7-step-5--register-with-codesphere) -- [8. Gotchas & Lessons Learned](#8-gotchas--lessons-learned) -- [9. Production Deployment](#9-production-deployment) - ---- - -## 1. Architecture Overview - -``` -┌─────────────────────┐ ┌─────────────────────┐ ┌──────────────────────────┐ -│ Codesphere UI │ │ REST Backend │ │ Stalwart Mail Server │ -│ (marketplace) │────────▶│ (Node.js/Express) │────────▶│ (Docker v0.13.2) │ -│ │ HTTP │ localhost:9090 │ HTTP │ localhost:1080 │ -│ POST / GET / │ │ │ │ │ -│ PATCH / DELETE │ │ Adapter layer: │ │ /api/principal (CRUD) │ -│ │◀────────│ translates CS API │◀────────│ /api/dns/records (DNS) │ -│ Shows details: │ JSON │ → Stalwart API │ JSON │ /jmap/session (JMAP) │ -│ email, IMAP, SMTP, │ │ │ │ /jmap/ (send/receive) │ -│ JMAP IDs, DNS │ │ Auto-discovers: │ │ │ -│ │ │ • JMAP account IDs │ │ IMAP :1993, SMTP :1587 │ -│ │ │ • DNS records │ │ │ -└─────────────────────┘ └─────────────────────┘ └──────────────────────────┘ -``` - -**How it works:** -1. A user requests a new Stalwart Mailbox in the Codesphere UI -2. Codesphere calls `POST /` on your REST backend with config + secrets -3. Your backend auto-creates the mail domain, then creates the user via Stalwart's admin API -4. The backend fetches DNS records and JMAP session details (account ID, identity ID, drafts mailbox ID) in parallel -5. Codesphere polls `GET /?id=...` to get connection details (IMAP, SMTP, JMAP IDs, DNS records, etc.) -6. The user sees everything they need to connect in the Codesphere dashboard - ---- - -## 2. Prerequisites - -- **Docker** (or Colima on macOS) for running Stalwart -- **Node.js 18+** for the REST backend -- A **Codesphere dev instance** (for registration/testing) -- `yq` — `brew install yq` (used by the validate script) - ---- - -## 3. Step 1 — Start Stalwart Mail Server - -Create a `docker-compose.local.yml`: - -```yaml -services: - stalwart: - image: stalwartlabs/stalwart:v0.13.2 - container_name: stalwart-mail - restart: unless-stopped - ports: - - "1080:8080" # HTTP (admin UI + JMAP + webmail) - - "1025:25" # SMTP - - "1587:587" # SMTP submission - - "1993:993" # IMAPS - volumes: - - stalwart-data:/opt/stalwart-mail - environment: - - STALWART_ADMIN_PASSWORD=localdev123 - -volumes: - stalwart-data: -``` - -Start it: - -```bash -docker compose -f docker-compose.local.yml up -d -``` - -> **Note:** Use `stalwartlabs/stalwart:v0.13.2` — not `stalwartlabs/mail-server` (old image name) and not `latest` (doesn't exist). The HTTP admin port is `8080` inside the container, mapped to `1080` locally. - -Verify it's running: - -```bash -# Check admin API responds -curl -s http://localhost:1080/api/principal \ - -u admin:localdev123 -# → {"data":{"items":[],"total":0}} -``` - -The admin UI is at http://localhost:1080 (login: `admin` / `localdev123`). - ---- - -## 4. Step 2 — Write the provider.yml - -Create `config/provider.yml`: - -```yaml -name: stalwart-mailbox -version: v2 -author: Codesphere -displayName: Stalwart Mailbox -iconUrl: https://stalw.art/img/logo.svg -category: messaging -description: | - Provision individual email accounts on a shared Stalwart Mail Server instance. - Each service creates a new mailbox user with IMAP, SMTP, and JMAP access. - Backed by a REST API that communicates with the Stalwart admin API. - -backend: - api: - endpoint: http://localhost:9090 - -plans: - - id: 0 - name: starter - description: Basic mailbox with 500 MB storage - parameters: {} - - id: 1 - name: standard - description: Standard mailbox with 2 GB storage - parameters: {} - - id: 2 - name: premium - description: Premium mailbox with 10 GB storage - parameters: {} - -configSchema: - type: object - properties: - EMAIL_PREFIX: - type: string - description: Local part of the email address (the part before @) - x-update-constraint: immutable - MAIL_DOMAIN: - type: string - description: Email domain (e.g. example.com). Each domain is created automatically. - x-update-constraint: immutable - DISPLAY_NAME: - type: string - description: Display name shown in email clients - QUOTA_MB: - type: integer - description: Mailbox storage quota in megabytes - -secretsSchema: - type: object - properties: - MAIL_PASSWORD: - type: string - format: password - -detailsSchema: - type: object - properties: - email: - type: string - username: - type: string - mail_domain: - type: string - imap_host: - type: string - imap_port: - type: integer - smtp_host: - type: string - smtp_port: - type: integer - jmap_url: - type: string - jmap_account_id: - type: string - description: JMAP account ID for API calls - jmap_identity_id: - type: string - description: JMAP identity ID for EmailSubmission - jmap_drafts_mailbox_id: - type: string - description: JMAP mailbox ID for Drafts folder - webmail_url: - type: string - dns_records: - type: string - description: DNS records to add to your domain for email delivery (SPF, DMARC, MX, DKIM) - ready: - type: boolean -``` - -### Key rules for provider.yml - -| Field | Rule | -|-------|------| -| `name` | Lowercase, hyphens/underscores only. Pattern: `^[-a-z0-9_]+$` | -| `version` | Must be `v1`, `v2`, etc. — NOT semver | -| `plans[].id` | Must be a non-negative **integer** (0, 1, 2...), not a string | -| `plans[].name` | Required string identifier | -| `plans[].parameters` | Required object (can be empty `{}`) | -| `secretsSchema` | Use `format: password`. Never set default values | -| `backend` | Use `backend.api.endpoint` for REST backends | - ---- - -## 5. Step 3 — Implement the REST Backend - -Your backend must implement 4 endpoints — the **Codesphere Managed Service Adapter API**: - -| Method | Path | Purpose | -|--------|------|---------| -| `POST /` | Create a new service | -| `GET /?id=...` | Get status (polled periodically) | -| `PATCH /:id` | Update config/secrets | -| `DELETE /:id` | Delete the service | - -### Project setup - -```bash -mkdir -p src/rest-backend && cd src/rest-backend -npm init -y -npm install express -``` - -### server.js — The complete implementation - -The backend translates between Codesphere's adapter API and Stalwart's admin API. Here are the critical parts: - -#### Stalwart API quirks you need to know - -**1. Error responses use HTTP 200** - -Stalwart returns `HTTP 200` for errors, with the error in the JSON body: -```json -{"error": "notFound", "item": "alice"} -``` - -You CANNOT just check `response.ok` — you must parse the body: - -```js -async function parseStalwartResponse(response) { - if (!response.ok) { - const text = await response.text(); - return { ok: false, error: `HTTP ${response.status}: ${text}` }; - } - const json = await response.json(); - if (json.error) { - return { ok: false, error: `${json.error}: ${json.details || json.item || ''}` }; - } - return { ok: true, data: json.data }; -} -``` - -**2. Domains must exist before creating users** - -Creating a user with email `alice@example.com` requires the domain `example.com` to already exist as a Stalwart principal. Otherwise you get `{"error": "notFound", "item": "example.com"}` (with HTTP 200!). - -The backend supports **multiple domains** — each service specifies its own `MAIL_DOMAIN`. Domains are auto-created and cached: - -```js -const ensuredDomains = new Set(); -async function ensureDomain(domain) { - if (ensuredDomains.has(domain)) return; - const resp = await stalwartRequest('POST', '/api/principal', { - type: 'domain', - name: domain, - description: `Mail domain ${domain}`, - }); - const result = await parseStalwartResponse(resp); - if (result.ok || (result.error && (result.error.includes('alreadyExists') || result.error.includes('AlreadyExists')))) { - ensuredDomains.add(domain); - } -} -``` - -**3. PATCH uses an action-array format** - -Stalwart's update endpoint does NOT accept a flat object. It expects an array of action objects: - -```json -[ - {"action": "set", "field": "description", "value": "Alice M. Doe"}, - {"action": "set", "field": "quota", "value": 1048576000}, - {"action": "set", "field": "secrets", "value": ["newpassword"]} -] -``` - -Other actions include `addItem` and `removeItem` for array fields like `emails`. - -**4. Use username (not numeric ID) for PATCH/DELETE paths** - -`POST /api/principal` returns `{"data": 87}` (a numeric principal ID), but `PATCH` and `DELETE` expect the **username string** in the path: - -``` -PATCH /api/principal/alice ✅ works -PATCH /api/principal/87 ❌ "notFound" -``` - -**5. Auth can be Basic or Bearer** - -For local dev with username:password, use Basic auth. For API tokens, use Bearer. The backend auto-detects: - -```js -headers: { - 'Authorization': STALWART_ADMIN_TOKEN.includes(':') - ? `Basic ${Buffer.from(STALWART_ADMIN_TOKEN).toString('base64')}` - : `Bearer ${STALWART_ADMIN_TOKEN}`, -} -``` - -**6. Create principal body** - -Stalwart expects all array fields even if empty: - -```js -{ - type: 'individual', - name: username, - secrets: [password], - emails: [email], - description: displayName, - quota: quotaMB * 1024 * 1024, // bytes - roles: ['user'], - lists: [], - memberOf: [], - members: [], - enabledPermissions: [], - disabledPermissions: [], - urls: [], - externalMembers: [], -} -``` - -### Full source - -See `src/rest-backend/server.js` in this repository for the complete implementation, which also includes: -- **`fetchDnsRecords(domain)`** — retrieves required DNS records (SPF, DKIM, DMARC, MX) from Stalwart's `/api/dns/records/{domain}` endpoint -- **`fetchJmapDetails(username, password)`** — auto-discovers JMAP session details (account ID, identity ID, drafts mailbox ID) so consumers can send email via JMAP without manual discovery -- **`buildDetails()`** — assembles all connection details including DNS and JMAP data, fetched in parallel - ---- - -## 6. Step 4 — Test Locally - -### Start the backend - -```bash -cd src/rest-backend && npm install - -STALWART_API_URL=http://localhost:1080 \ -STALWART_ADMIN_TOKEN=admin:localdev123 \ -STALWART_IMAP_HOST=localhost \ -STALWART_SMTP_HOST=localhost \ -STALWART_IMAP_PORT=1993 \ -STALWART_SMTP_PORT=1587 \ -PORT=9090 \ -node server.js -``` - -### Run the CRUD tests - -**Create:** -```bash -curl -s -w "\nHTTP %{http_code}\n" -X POST http://localhost:9090/ \ - -H "Content-Type: application/json" \ - -d '{ - "id": "550e8400-e29b-41d4-a716-446655440000", - "plan": {"id": 0, "parameters": {}}, - "config": { - "EMAIL_PREFIX": "alice", - "MAIL_DOMAIN": "example.com", - "DISPLAY_NAME": "Alice Doe", - "QUOTA_MB": 500 - }, - "secrets": { "MAIL_PASSWORD": "supersecret123" } - }' -# → HTTP 201 -``` - -**Read:** -```bash -curl -s http://localhost:9090/?id=550e8400-e29b-41d4-a716-446655440000 | python3 -m json.tool -# → Shows email, IMAP/SMTP details, JMAP IDs, DNS records, ready: true -``` - -**Verify in Stalwart directly:** -```bash -curl -s http://localhost:1080/api/principal?types=individual\&limit=10 \ - -u admin:localdev123 | python3 -m json.tool -# → Shows alice with quota 524288000 (500 MB), role "user" -``` - -**Update:** -```bash -curl -s -w "\nHTTP %{http_code}\n" -X PATCH \ - http://localhost:9090/550e8400-e29b-41d4-a716-446655440000 \ - -H "Content-Type: application/json" \ - -d '{ "config": { "DISPLAY_NAME": "Alice M. Doe", "QUOTA_MB": 1000 } }' -# → HTTP 204 -``` - -**Delete:** -```bash -curl -s -w "\nHTTP %{http_code}\n" -X DELETE \ - http://localhost:9090/550e8400-e29b-41d4-a716-446655440000 -# → HTTP 204 -``` - -### Test JMAP email sending - -After creating a mailbox, use the JMAP IDs from the GET response to send an email: - -```bash -curl -s http://localhost:1080/jmap/ \ - -u 'alice:supersecret123' \ - -H 'Content-Type: application/json' \ - -d '{ - "using": [ - "urn:ietf:params:jmap:core", - "urn:ietf:params:jmap:mail", - "urn:ietf:params:jmap:submission" - ], - "methodCalls": [ - ["Email/set", { - "accountId": "", - "create": { - "draft1": { - "mailboxIds": {"": true}, - "from": [{"name": "Alice", "email": "alice@example.com"}], - "to": [{"name": "Bob", "email": "bob@example.com"}], - "subject": "Hello!", - "textBody": [{"partId": "body", "type": "text/plain"}], - "bodyValues": {"body": {"value": "Sent via JMAP!", "isEncodingProblem": false}} - } - } - }, "c1"], - ["EmailSubmission/set", { - "accountId": "", - "create": { - "sub1": { - "identityId": "", - "emailId": "#draft1" - } - } - }, "c2"] - ] - }' | python3 -m json.tool -``` - -If both method responses contain `"created"`, the email was sent. See `JMAP_GUIDE.md` for a detailed walkthrough. - ---- - -## 7. Step 5 — Register with Codesphere - -REST backend providers are registered through the **platform configuration** (`MANAGED_SERVICE_PROVIDERS` environment variable on the Codesphere instance). - -### For Codesphere Private Cloud / Dev Instance - -Add your provider to the `MANAGED_SERVICE_PROVIDERS` environment variable as a JSON entry. The provider definition (including schemas, plans, and backend endpoint) is read from the `provider.yml` and applied to the platform config. - -> **Important:** The `POST /api/managed-services/providers` API endpoint only works for **landscape-based** providers (via `gitUrl`). REST backend providers must be added to `MANAGED_SERVICE_PROVIDERS`. - -### Using `make register` (landscape providers) - -For landscape-based providers, you can use the git-based registration: - -```bash -CODESPHERE_URL=http://localhost:8080 \ -CODESPHERE_API_TOKEN=your-token \ -CODESPHERE_TEAM_ID=your-team-id \ -make register -``` - -This clones the repo and reads `provider.yml`. - -This clones the repo and reads `provider.yml` from the **repo root** (not from `config/`). - ---- - -## 8. Gotchas & Lessons Learned - -These are real issues encountered during development — save yourself the debugging time: - -### Stalwart API - -| Gotcha | Detail | -|--------|--------| -| **HTTP 200 for errors** | Stalwart returns 200 with `{"error": "notFound"}`. Always parse the response body for an `error` field. | -| **Domain must exist first** | Creating `alice@example.com` fails if the `example.com` domain principal doesn't exist. Create it with `type: "domain"` first. | -| **`fieldAlreadyExists` vs `alreadyExists`** | Stalwart can return either error string for duplicate principals. Check for both with `includes()`. | -| **PATCH is an action array** | Send `[{"action":"set","field":"...","value":"..."}]`, not a flat object. | -| **Use username, not ID** | `PATCH/DELETE /api/principal/{name}` — use the string username, not the numeric ID from create. | -| **Docker image name changed** | Use `stalwartlabs/stalwart:v0.13.2` — not the old `stalwartlabs/mail-server`. | -| **IP blocking** | Too many failed TLS handshakes can trigger Stalwart's brute-force protection. Fix by wiping the volume: `docker compose down -v`. | - -### provider.yml - -| Gotcha | Detail | -|--------|--------| -| **Plan IDs are integers** | `id: 0`, not `id: "starter"`. The API rejects string IDs. | -| **Plans need `name`** | Each plan needs both `id` (integer) and `name` (string). | -| **Plans need `parameters`** | Even if empty, `parameters: {}` is required. | -| **Use `backend.api.endpoint`** | For REST backends, use `backend.api.endpoint`, not `backend.rest.url`. | - -### Registration - -| Gotcha | Detail | -|--------|--------| -| **REST ≠ gitUrl registration** | The `POST /providers` API only supports landscape backends. REST backends go in `MANAGED_SERVICE_PROVIDERS` env var. | -| **Global scope needs admin** | Creating global providers requires cluster admin permissions. Use team scope for testing. | -| **Repo must be accessible** | If using gitUrl (landscape providers), the repo must be public or Codesphere needs GitHub credentials. | - ---- - -## 9. Production Deployment - -For production, you'll need: - -1. **A real domain** with DNS records (MX, SPF, DKIM, DMARC) — the backend auto-retrieves these per domain via `/api/dns/records/{domain}` -2. **TLS certificates** (Stalwart supports ACME/Let's Encrypt) -3. **The REST backend behind HTTPS** (use a reverse proxy like Caddy or nginx) -4. **Persistent storage** for the backend state (replace the in-memory `Map` with a database) -5. **Auth tokens** set on both the backend (`AUTH_TOKEN`) and in Codesphere -6. **Port 25 outbound access** — many cloud providers block this; consider an SMTP relay (Amazon SES, Mailgun) if needed - -See `STALWART_SETUP.md` in this repository for the full production Docker setup guide. - -### What end users receive - -When a user provisions a Stalwart Mailbox through Codesphere, they get: - -| Field | Example | -|-------|---------| -| **Email** | `alice@example.com` | -| **Username** | `alice` | -| **Mail Domain** | `example.com` | -| **IMAP Host** | `mail.example.com:993` (TLS) | -| **SMTP Host** | `mail.example.com:587` (STARTTLS) | -| **JMAP URL** | `https://mail.example.com/jmap` | -| **JMAP Account ID** | Auto-discovered, ready for API use | -| **JMAP Identity ID** | Auto-discovered, for EmailSubmission | -| **JMAP Drafts Mailbox ID** | Auto-discovered, for creating drafts | -| **Webmail** | `https://mail.example.com/login` | -| **DNS Records** | SPF, DKIM, DMARC, MX records for the domain | - -They can plug the IMAP/SMTP into any email client (Thunderbird, Apple Mail, Outlook), or use the JMAP IDs to send email programmatically. See `JMAP_GUIDE.md` for a step-by-step JMAP walkthrough. - ---- - -## Quick Reference — File Layout - -``` -stalwart-provider/ -├── config/ -│ └── provider.yml # Provider definition (schemas, plans, backend URL) -├── src/ -│ └── rest-backend/ -│ ├── server.js # REST backend implementation -│ └── package.json -├── docker-compose.local.yml # Local Stalwart instance (v0.13.2) -├── STALWART_SETUP.md # Production deployment guide -├── HACKATHON.md # Hackathon quick-start with architecture diagrams -├── JMAP_GUIDE.md # JMAP email sending guide with examples -├── Makefile # validate / register / test commands -└── scripts/ - ├── validate.sh - └── register.sh -``` diff --git a/WORKSHOP_TUTORIAL.md b/WORKSHOP_TUTORIAL.md index 84da49c..adbe87d 100644 --- a/WORKSHOP_TUTORIAL.md +++ b/WORKSHOP_TUTORIAL.md @@ -134,15 +134,15 @@ cd stalwart-provider ``` stalwart-provider/ ├── src/ -│ └── rest-backend/ -│ ├── server.js # ← Reference implementation (your starting point) -│ ├── package.json -│ └── Dockerfile -├── ci.stalwart.yml # Codesphere CI pipeline for the Stalwart Mail Server -├── ci.stalwart-provider.yml # Codesphere CI pipeline for the REST provider backend +│ ├── server.js # ← Reference implementation (your starting point) +│ ├── package.json +│ └── Dockerfile +├── ci.stalwart.yml # CI pipeline for the Stalwart Mail Server deployment +├── ci.stalwart-provider.yml # CI pipeline for the REST provider backend ├── provider.yml # Service definition for the Codesphere marketplace ├── docker-compose.local.yml # Local Stalwart for development -├── Makefile # validate / test +├── examples/ # Generic provider.yml / ci.yml examples (for reference) +├── Makefile # validate / test / start-api-backend / send-mail └── scripts/ ├── validate.sh └── test-provider.sh @@ -291,7 +291,7 @@ This is the core of the workshop. You will build a Node.js/Express application t ### 3.1 — Project Setup ```bash -cd src/rest-backend +cd src npm install ``` @@ -540,7 +540,7 @@ async function buildDetails(username: string, email: string, domain: string, pas Start the backend: ```bash -cd src/rest-backend +cd src STALWART_API_URL=http://localhost:1080 \ STALWART_ADMIN_TOKEN=admin:localdev123 \ diff --git a/ci.stalwart-provider.yml b/ci.stalwart-provider.yml index 945bb29..98ca850 100644 --- a/ci.stalwart-provider.yml +++ b/ci.stalwart-provider.yml @@ -1,7 +1,7 @@ schemaVersion: v0.2 prepare: steps: - - command: cd src/rest-backend && npm i + - command: cd src && npm i test: steps: [] run: @@ -22,4 +22,4 @@ run: STALWART_SMTP_PORT: 587 PORT: 3000 steps: - - command: cd src/rest-backend && node server.js + - command: cd src && node server.js diff --git a/config/provider.yml b/config/provider.yml deleted file mode 100644 index 548d840..0000000 --- a/config/provider.yml +++ /dev/null @@ -1,89 +0,0 @@ -name: stalwart-mailbox -version: v2 -author: Codesphere -displayName: Stalwart Mailbox -iconUrl: https://stalw.art/img/logo.svg -category: messaging -description: | - Provision individual email accounts on a shared Stalwart Mail Server instance. - Each service creates a new mailbox user with IMAP, SMTP, and JMAP access. - Backed by a REST API that communicates with the Stalwart admin API. - -backend: - api: - endpoint: http://localhost:9090 - -plans: - - id: 0 - name: starter - description: Basic mailbox with 500 MB storage - parameters: {} - - id: 1 - name: standard - description: Standard mailbox with 2 GB storage - parameters: {} - - id: 2 - name: premium - description: Premium mailbox with 10 GB storage - parameters: {} - -configSchema: - type: object - properties: - EMAIL_PREFIX: - type: string - description: Local part of the email address (the part before @) - x-update-constraint: immutable - MAIL_DOMAIN: - type: string - description: Email domain (e.g. example.com). Each domain is created automatically. - x-update-constraint: immutable - DISPLAY_NAME: - type: string - description: Display name shown in email clients - QUOTA_MB: - type: integer - description: Mailbox storage quota in megabytes - -secretsSchema: - type: object - properties: - MAIL_PASSWORD: - type: string - format: password - -detailsSchema: - type: object - properties: - email: - type: string - username: - type: string - mail_domain: - type: string - imap_host: - type: string - imap_port: - type: integer - smtp_host: - type: string - smtp_port: - type: integer - jmap_url: - type: string - jmap_account_id: - type: string - description: JMAP account ID for API calls - jmap_identity_id: - type: string - description: JMAP identity ID for EmailSubmission - jmap_drafts_mailbox_id: - type: string - description: JMAP mailbox ID for Drafts folder - webmail_url: - type: string - dns_records: - type: string - description: DNS records to add to your domain for email delivery (SPF, DMARC, MX, DKIM) - ready: - type: boolean diff --git a/config/ci.yml.example b/examples/ci.yml.example similarity index 100% rename from config/ci.yml.example rename to examples/ci.yml.example diff --git a/config/provider.rest.yml.example b/examples/provider.rest.yml.example similarity index 100% rename from config/provider.rest.yml.example rename to examples/provider.rest.yml.example diff --git a/config/provider.yml.example b/examples/provider.yml.example similarity index 100% rename from config/provider.yml.example rename to examples/provider.yml.example diff --git a/scripts/register.sh b/scripts/register.sh deleted file mode 100755 index 5a6fedd..0000000 --- a/scripts/register.sh +++ /dev/null @@ -1,107 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -PROVIDER_CONFIG="config/provider.yml" - -echo "=== Registering Provider with Codesphere ===" -echo "" - -# ── Check required env vars ──────────────────────────────────────── -if [[ -z "${CODESPHERE_API_TOKEN:-}" ]]; then - echo "ERROR: CODESPHERE_API_TOKEN is not set." - echo "" - echo " export CODESPHERE_API_TOKEN=your-token-here" - echo "" - exit 1 -fi - -# ── Determine API base URL ───────────────────────────────────────── -CODESPHERE_URL="${CODESPHERE_URL:-https://codesphere.com}" -API_ENDPOINT="${CODESPHERE_URL}/api/managed-services/providers" - -# ── Extract provider metadata ────────────────────────────────────── -if ! command -v yq &>/dev/null; then - echo "ERROR: yq is required for registration." - echo " Install: brew install yq (macOS) or apt-get install yq (Linux)" - exit 1 -fi - -PROVIDER_NAME=$(yq eval '.name' "$PROVIDER_CONFIG") -PROVIDER_VERSION=$(yq eval '.version' "$PROVIDER_CONFIG") - -# ── Detect backend type ──────────────────────────────────────────── -HAS_LANDSCAPE=$(yq eval '.backend.landscape.gitUrl // "absent"' "$PROVIDER_CONFIG" 2>/dev/null) -HAS_REST=$(yq eval '.backend.rest.url // "absent"' "$PROVIDER_CONFIG" 2>/dev/null) - -if [[ "$HAS_LANDSCAPE" != "absent" ]]; then - BACKEND_TYPE="landscape" - GIT_URL="$HAS_LANDSCAPE" -elif [[ "$HAS_REST" != "absent" ]]; then - BACKEND_TYPE="rest" - REST_URL="$HAS_REST" -else - echo "ERROR: No backend configured. Specify backend.landscape or backend.rest in $PROVIDER_CONFIG" - exit 1 -fi - -# ── Determine Git URL (required for all backend types) ───────────── -if [[ -z "${GIT_URL:-}" ]]; then - GIT_URL=$(git remote get-url origin 2>/dev/null || true) - if [[ -z "$GIT_URL" ]]; then - echo "ERROR: Could not determine Git URL. Set a git remote or use a landscape backend." - exit 1 - fi -fi - -echo "Provider: $PROVIDER_NAME $PROVIDER_VERSION" -echo "Backend: $BACKEND_TYPE" -echo "Git URL: $GIT_URL" -if [[ "$BACKEND_TYPE" == "rest" ]]; then - echo "REST URL: $REST_URL" -fi -echo "API: $API_ENDPOINT" - -# ── Determine scope ──────────────────────────────────────────────── -if [[ -n "${CODESPHERE_TEAM_ID:-}" ]]; then - SCOPE_JSON='{"type": "team", "teamIds": ['"$CODESPHERE_TEAM_ID"']}' - echo "Scope: team ($CODESPHERE_TEAM_ID)" -else - SCOPE_JSON='{"type": "global"}' - echo "Scope: global" -fi -echo "" - -# ── Register provider ────────────────────────────────────────────── -echo "Registering provider..." - -# ── Register provider (always via gitUrl) ────────────────────────── -RESPONSE=$(curl -s -w "\n%{http_code}" \ - -X POST "$API_ENDPOINT" \ - -H "Authorization: Bearer $CODESPHERE_API_TOKEN" \ - -H "Content-Type: application/json" \ - -d '{"gitUrl": "'"$GIT_URL"'", "scope": '"$SCOPE_JSON"'}') - -HTTP_CODE=$(echo "$RESPONSE" | tail -n1) -BODY=$(echo "$RESPONSE" | sed '$d') - -case "$HTTP_CODE" in - 200|201) - echo "SUCCESS: Provider '$PROVIDER_NAME' $PROVIDER_VERSION registered." - echo "$BODY" | python3 -m json.tool 2>/dev/null || echo "$BODY" - ;; - 401) - echo "ERROR: Authentication failed (HTTP 401)" - echo " Check your CODESPHERE_API_TOKEN" - exit 1 - ;; - 409) - echo "ERROR: Provider already exists (HTTP 409)" - echo " Bump the version in $PROVIDER_CONFIG or use a different name" - exit 1 - ;; - *) - echo "ERROR: Registration failed (HTTP $HTTP_CODE)" - echo "$BODY" - exit 1 - ;; -esac diff --git a/scripts/test-provider.sh b/scripts/test-provider.sh index 0d47292..77cacfa 100755 --- a/scripts/test-provider.sh +++ b/scripts/test-provider.sh @@ -1,8 +1,8 @@ #!/usr/bin/env bash set -euo pipefail -PROVIDER_CONFIG="config/provider.yml" -CI_CONFIG="config/ci.yml" +PROVIDER_CONFIG="provider.yml" +CI_CONFIG="ci.stalwart-provider.yml" echo "=== Testing Provider ===" echo "" diff --git a/scripts/validate.sh b/scripts/validate.sh index 842829b..5c8750f 100755 --- a/scripts/validate.sh +++ b/scripts/validate.sh @@ -1,8 +1,8 @@ #!/usr/bin/env bash set -euo pipefail -PROVIDER_CONFIG="config/provider.yml" -CI_CONFIG="config/ci.yml" +PROVIDER_CONFIG="provider.yml" +CI_CONFIG="ci.stalwart-provider.yml" ERRORS=0 BACKEND_TYPE="" diff --git a/src/.gitkeep b/src/.gitkeep deleted file mode 100644 index e69de29..0000000 diff --git a/src/rest-backend/Dockerfile b/src/Dockerfile similarity index 100% rename from src/rest-backend/Dockerfile rename to src/Dockerfile diff --git a/src/rest-backend/package-lock.json b/src/package-lock.json similarity index 100% rename from src/rest-backend/package-lock.json rename to src/package-lock.json diff --git a/src/rest-backend/package.json b/src/package.json similarity index 100% rename from src/rest-backend/package.json rename to src/package.json diff --git a/src/rest-backend/README.md b/src/rest-backend/README.md deleted file mode 100644 index 6d4d4f8..0000000 --- a/src/rest-backend/README.md +++ /dev/null @@ -1,31 +0,0 @@ -# Example REST Backend - -A minimal Node.js implementation of the [Codesphere Managed Service Adapter API](../../.github/instructions/PROVIDER.instructions.md). - -This backend provides the four required endpoints: - -| Method | Path | Description | -|--------|------|-------------| -| `POST` | `/` | Create a new service | -| `GET` | `/?id=...` | Get status of services (or list all IDs) | -| `PATCH` | `/:id` | Update an existing service | -| `DELETE` | `/:id` | Delete a service | - -## Quick Start - -```bash -cd src/rest-backend -npm install -npm start -``` - -## Configuration - -| Env Var | Default | Description | -|---------|---------|-------------| -| `PORT` | `8080` | Port to listen on | -| `AUTH_TOKEN` | _(none)_ | Bearer token for authentication (recommended) | - -## Customization - -This example uses an in-memory store. Replace the `TODO` comments in `server.js` with your actual infrastructure provisioning logic (e.g., cloud API calls, Kubernetes operations, database management). diff --git a/src/rest-backend/server.js b/src/server.js similarity index 100% rename from src/rest-backend/server.js rename to src/server.js