Jobs snapshot cleaner

README.md

@@ -83,6 +83,7 @@ Table of Contents:
 | **[Serverless Jobs Hello World](jobs/terraform-hello-world/README.md)** <br/> An example of building a container image and running it as a Serverless Job using Terraform. | N/A      | [Terraform]-[Console]       |
 | **[Serverless MLOps](jobs/ml-ops/README.md)** <br/> An example of running a Serverless Machine Leaning workflow.                                                           | Python   | [Terraform]-[Console]-[CLI] |
 | **[Auto Snapshot Instances](jobs/instances-snapshot/README.md)** <br/> Use Serverless Jobs to create snapshots of your instances                                           | Go       | [Console]                   |
+| **[Old Snapshot Cleaner](jobs/instances-snapshot-cleaner/README.md)** <br/> Use Serverless Jobs to clean old instances snapshots                                           | Go       | [Console]                   |
 
 ### 💬 Messaging and Queueing
 

jobs/instances-snapshot-cleaner/Dockerfile

@@ -0,0 +1,16 @@
+# Using apline/golang image
+FROM golang:1.23-alpine
+
+# Set destination for COPY
+WORKDIR /app
+
+# Copy required files
+COPY go.mod ./
+COPY go.sum ./
+COPY *.go ./
+
+# Build the executable
+RUN go build -o /jobs-snapshot-cleaner
+
+# Run the executable
+CMD [ "/jobs-snapshot-cleaner" ]

jobs/instances-snapshot-cleaner/README.md

@@ -0,0 +1,83 @@
+# Serverless Jobs for cleaning old snapshots
+
+This project shows how it's possible to automate tasks using Serverless Jobs.
+
+This simple example shows how to clean up snapshots after X days, it's useful to avoid a growing list of snapshots.
+
+# Set-up
+
+## Requirements
+
+- Scaleway Account
+- Docker daemon running to build the image
+- Container registry namespace created, for this example we assume that your namespace name is `jobs-snapshot-cleaner`: [doc here](https://www.scaleway.com/en/docs/containers/container-registry/how-to/create-namespace/)
+
+## Step 1 : Build and push to Container registry
+
+Serverless Jobs, like Serverless Containers (which are suited for HTTP applications), works
+with containers. So first, use your terminal reach this folder and run the following commands:
+
+```shell
+# First command is to login to container registry, you can find it in Scaleway console
+docker login rg.fr-par.scw.cloud/jobs-snapshot-cleaner -u nologin --password-stdin <<< "$SCW_SECRET_KEY"
+
+# Here we build the image to push
+docker build -t rg.fr-par.scw.cloud/jobs-snapshot-cleaner/jobs-snapshot-cleaner:v1 .
+
+## TIP: for Apple Silicon or other ARM processors, please use the following command as Serverless Jobs supports amd64 architecture
+# docker buildx build --platform linux/amd64 -t rg.fr-par.scw.cloud/jobs-snapshot-cleaner/jobs-snapshot-cleaner:v1 .
+
+# Push the image online to be used on Serverless Jobs
+docker push rg.fr-par.scw.cloud/jobs-snapshot-cleaner/jobs-snapshot-cleaner:v1
+```
+> [!TIP]
+> As we do not expose a web server and we do not require features such as auto-scaling, Serverless Jobs are perfect for this use case.
+
+To check if everyting is ok, on the Scaleway Console you can verify if your tag is present in Container Registry.
+
+## Step 2: Creating the Job Definition
+
+On Scaleway Console on the following link you can create a new Job Definition: https://console.scaleway.com/serverless-jobs/jobs/create?region=fr-par
+
+1. On Container image, select the image you created in the step before.
+1. You can set the image name to something clear like `jobs-snapshot-cleaner` too.
+1. For the region you can select the one you prefer :)
+1. Regarding the resources you can keep the default values, this job is fast and do not require specific compute power or memory.
+1. To schedule your job for example every two days at 2am, you can set the cron to `0 2 */2 * *`.
+1. Important: advanced option, you need to set the following environment variables:
+
+> [!TIP]
+> For sensitive data like `SCW_ACCESS_KEY` and `SCW_SECRET_KEY` we recommend to inject them via Secret Manager, [more info here](https://www.scaleway.com/en/docs/serverless/jobs/how-to/reference-secret-in-job/).
+
+- `SCW_DELETE_AFTER_DAYS`: number of days after the snapshots will be deleted
+- `SCW_PROJECT_ID`: project you want to clean up
+- `SCW_ZONE`: you need to give the ZONE of your snapshot you want to clean, like `fr-par-2`
+- `SCW_ACCESS_KEY`: your access key
+- `SCW_SECRET_KEY`: your secret key
+- `SCW_DEFAULT_ORGANIZATION_ID`: your organzation ID
+
+* Then click "create job"
+
+## Step 3: Run the job
+
+On your created Job Definition, just click the button "Run Job" and within seconds it should be successful.
+
+## Troubleshooting
+
+If your Job Run state goes in error, you can use the "Logs" tab in Scaleway Console to get more informations about the error.
+
+# Possible improvements
+
+You can exercice by adding the following features:
+
+- Add tags to exclude
+- Add alerts if a Job goes in error
+- Use Secret Manager instead of job environment variables
+- Support multiple zones dans projects
+
+# Additional content
+
+- [Jobs Documentation](https://www.scaleway.com/en/docs/serverless/jobs/how-to/create-job-from-scaleway-registry/)
+- [Other methods to deploy Jobs](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/deploy-job/)
+- [Secret key / access key doc](https://www.scaleway.com/en/docs/identity-and-access-management/iam/how-to/create-api-keys/)
+- [CRON schedule help](https://www.scaleway.com/en/docs/serverless/jobs/reference-content/cron-schedules/)

jobs/instances-snapshot-cleaner/go.mod

@@ -0,0 +1,7 @@
+module github.com/scaleway/serverless-examples/jobs/instances-snapshot
+
+go 1.23
+
+require github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32
+
+require gopkg.in/yaml.v2 v2.4.0 // indirect

jobs/instances-snapshot-cleaner/go.sum

@@ -0,0 +1,8 @@
+github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI=
+github.com/dnaeon/go-vcr v1.2.0/go.mod h1:R4UdLID7HZT3taECzJs4YgbbH6PIGXB6W/sc5OLb6RQ=
+github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32 h1:4+LP7qmsLSGbmc66m1s5dKRMBwztRppfxFKlYqYte/c=
+github.com/scaleway/scaleway-sdk-go v1.0.0-beta.32/go.mod h1:kzh+BSAvpoyHHdHBCDhmSWtBc1NbLMZ2lWHqnBoxFks=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
+gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=

jobs/instances-snapshot-cleaner/main.go

@@ -0,0 +1,130 @@
+package main
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"os"
+	"strconv"
+	"time"
+
+	"github.com/scaleway/scaleway-sdk-go/api/instance/v1"
+	"github.com/scaleway/scaleway-sdk-go/scw"
+)
+
+const (
+	envOrgID     = "SCW_DEFAULT_ORGANIZATION_ID"
+	envAccessKey = "SCW_ACCESS_KEY"
+	envSecretKey = "SCW_SECRET_KEY"
+	envProjectID = "SCW_PROJECT_ID"
+	envZone      = "SCW_ZONE"
+
+	// envDeleteAfter name of env variable to deleter older images.
+	envDeleteAfter = "SCW_DELETE_AFTER_DAYS"
+
+	// defaultDaysDeleteAfter is the default days value for older images to be deleted.
+	defaultDaysDeleteAfter = int(90)
+)
+
+func main() {
+	fmt.Println("cleaning instances snapshots...")
+
+	// Create a Scaleway client with credentials from environment variables.
+	client, err := scw.NewClient(
+		// Get your organization ID at https://console.scaleway.com/organization/settings
+		scw.WithDefaultOrganizationID(os.Getenv(envOrgID)),
+
+		// Get your credentials at https://console.scaleway.com/iam/api-keys
+		scw.WithAuth(os.Getenv(envAccessKey), os.Getenv(envSecretKey)),
+
+		// Get more about our availability
+		// zones at https://www.scaleway.com/en/docs/console/my-account/reference-content/products-availability/
+		scw.WithDefaultRegion(scw.RegionFrPar),
+	)
+	if err != nil {
+		panic(err)
+	}
+
+	// Create SDK objects for Scaleway Instance product
+	instanceAPI := instance.NewAPI(client)
+
+	deleteAfterDays := defaultDaysDeleteAfter
+
+	deleteAfterDaysVar := os.Getenv(envDeleteAfter)
+
+	if deleteAfterDaysVar != "" {
+		deleteAfterDays, err = strconv.Atoi(deleteAfterDaysVar)
+		if err != nil {
+			panic(err)
+		}
+	}
+
+	if err := cleanSnapshots(deleteAfterDays, instanceAPI); err != nil {
+		var precondErr *scw.PreconditionFailedError
+
+		if errors.As(err, &precondErr) {
+			fmt.Println("\nExtracted Error Details:")
+			fmt.Println("Precondition:", precondErr.Precondition)
+			fmt.Println("Help Message:", precondErr.HelpMessage)
+
+			// Decode RawBody (if available)
+			if len(precondErr.RawBody) > 0 {
+				var parsedBody map[string]interface{}
+				if json.Unmarshal(precondErr.RawBody, &parsedBody) == nil {
+					fmt.Println("RawBody (Decoded):", parsedBody)
+				} else {
+					fmt.Println("RawBody (Raw):", string(precondErr.RawBody))
+				}
+			}
+		}
+		panic(err)
+	}
+}
+
+// cleanSnapshots when called will clean snapshots in the project (if specified)
+// that are older than the number of days.
+func cleanSnapshots(days int, instanceAPI *instance.API) error {
+	// Get the list of all snapshots
+	snapshotsList, err := instanceAPI.ListSnapshots(&instance.ListSnapshotsRequest{
+		Zone:    scw.Zone(os.Getenv(envZone)),
+		Project: scw.StringPtr(os.Getenv(envProjectID)),
+	},
+		scw.WithAllPages())
+	if err != nil {
+		return fmt.Errorf("error while listing snapshots %w", err)
+	}
+
+	const hoursPerDay = 24
+
+	currentTime := time.Now()
+
+	// For each snapshot, check conditions
+	for _, snapshot := range snapshotsList.Snapshots {
+		// Check if snapshot is in ready state and if it's older than the number of days definied.
+		if snapshot.State == instance.SnapshotStateAvailable && (currentTime.Sub(*snapshot.CreationDate).Hours()/hoursPerDay) > float64(days) {
+			fmt.Printf("\nDeleting snapshot <%s>:%s created at: %s\n", snapshot.ID, snapshot.Name, snapshot.CreationDate.Format(time.RFC3339))
+
+			// Delete snapshot found.
+			err := instanceAPI.DeleteSnapshot(&instance.DeleteSnapshotRequest{
+				SnapshotID: snapshot.ID,
+				Zone:       snapshot.Zone,
+			})
+			if err != nil {
+				return fmt.Errorf("error while deleting snapshot: %w", err)
+			}
+		}
+	}
+
+	return nil
+}
+
+// Check for mandatory variables before starting to work.
+func init() {
+	mandatoryVariables := [...]string{envOrgID, envAccessKey, envSecretKey, envZone, envProjectID}
+
+	for idx := range mandatoryVariables {
+		if os.Getenv(mandatoryVariables[idx]) == "" {
+			panic("missing environment variable " + mandatoryVariables[idx])
+		}
+	}
+}