| title | html_title | description |
|---|---|---|
Certificate Renewal |
Certificate renewal options |
Learn how to renew certificates and set up automated renewals |
By default, step-ca issues short-lived certificates that expire after 24
hours. Short-lived certificates are excellent security hygiene because they
offer regular key rotation and passive revocation
Short-lived certificates create a problem though: For any long-lived workloads,
you will need to renew your certificates each day before they expire. This
section will show three approaches to renew certs with step.
step-ca instance using the steps in Getting Started. As an
alternative, you can use our hosted CA, Smallstep Certificate Manager.
First, let's initialize a new PKI and start step-ca. We'll write a password
out to password.txt so we don't have to enter it repeatedly.
$ echo "p4ssword" > password.txt
$ step ca init --name "Speedy" --provisioner admin \
--dns localhost --address ":443" \
--password-file password.txt \
--provisioner-password-file password.txt
$ step-ca $(step path)/config/ca.json --password-file password.txtNow let's generate a single-use bootstrap token and use it to obtain a certificate:
{`$ TOKEN=$(step ca token --password-file password.txt foo.local) ✔ Key ID: w1OUFng_fCqWygHHpc9Ak8m_HGmE0TEasYIfahLoZUg (admin)`}In a production environment, you might use something like Kubernetes or Chef to generate this token and give it to a host or client that needs a certificate.
Now we can generate a keypair locally, and use our bootstrap token to obtain a certificate for foo.local from step-ca:
$ step ca certificate foo.local foo.crt foo.key --token $TOKEN
✔ CA: https://localhost
✔ Certificate: foo.crt
✔ Private Key: foo.key
$ step certificate inspect --short foo.crt
X.509v3 TLS Certificate (ECDSA P-256) [Serial: 2599...1204]
Subject: foo.local
Issuer: Speedy Intermediate CA
Provisioner: admin [ID: w1OU...oZUg]
Valid from: 2019-05-01T21:06:25Z
to: 2019-05-02T21:06:25ZBy default, step-ca issues certificates valid for 24 hours. This is suitably
short for many scenarios. If it's not right for you, you can adjust the
defaultTLSCertDuration per provisioner
or pass the --not-after flag to the step ca certificate to adjust the
lifetime of an individual certificate. Very short lifetimes (eg. five minutes)
are better from a security perspective, but this can be difficult in
practice.
With short-lived certificates, your services and hosts will need to renew their certificates regularly, by extending their lifetimes before they expire.
You can do this manually with the following command:
{`$ step ca renew --force foo.crt foo.key Your certificate has been saved in foo.crt \n $ step certificate inspect --short foo.crt X.509v3 TLS Certificate (ECDSA P-256) [Serial: 1664...3445] Subject: foo.local Issuer: Speedy Intermediate CA Provisioner: admin [ID: w1OU...oZUg] Valid from: 2019-05-01T21:15:16Z to: 2019-05-02T21:15:16Z `}Note the change in the validity period relative to the original certificate above.
What good are short-lived certificates if we can't renew them automatically?
Here are three options for setting up automated renewal of certificates using step ca renew:
-
Renewal using
systemdtimers. This is the preferred approach.
This approach runs a periodic systemd timer for each certificate you want to keep current. The timer will run a one-shot systemd service every few minutes. The one-shot service checks the certificate and renews it if more than ⅔ of its lifetime has elapsed. Upon renewal, the service can try to reload or restart a service using the certificate files, if it exists. Custom post-renewal commands can be configured as well.
We will leverage systemd service templates to simplify configuration of certificate renewal for many target services.
In /etc/systemd/system, we'll start by creating template files [email protected], and [email protected].
Service templates accept a single argument after the @, called the service unit argument.
For example, for [email protected], the service unit argument is postgresql.
In the template, %i represents the service unit argument.
Let's review the service template first, then we'll look at the timer template.
Create a service unit template file:
$ sudo touch /etc/systemd/system/[email protected]
Add the following configuration:
[Unit]
Description=Certificate renewer for %I
After=network-online.target
Documentation=https://smallstep.com/docs/step-ca/certificate-authority-server-production
StartLimitIntervalSec=0
; PartOf=cert-renewer.target
[Service]
Type=oneshot
User=root
Environment=STEPPATH=/etc/step-ca \
CERT_LOCATION=/etc/step/certs/%i.crt \
KEY_LOCATION=/etc/step/certs/%i.key
; ExecCondition checks if the certificate is ready for renewal,
; based on the exit status of the command.
; (In systemd <242, you can use ExecStartPre= here.)
ExecCondition=/usr/bin/step certificate needs-renewal ${CERT_LOCATION}
; ExecStart renews the certificate, if ExecStartPre was successful.
ExecStart=/usr/bin/step ca renew --force ${CERT_LOCATION} ${KEY_LOCATION}
; Try to reload or restart the systemd service that relies on this cert-renewer
; If the relying service doesn't exist, forge ahead.
; (In systemd <229, use `reload-or-try-restart` instead of `try-reload-or-restart`)
ExecStartPost=/usr/bin/env sh -c "! systemctl --quiet is-active %i.service || systemctl try-reload-or-restart %i"
[Install]
WantedBy=multi-user.target(This file is maintained on GitHub)
With this template file in place, we can now ask systemd to start any cert-renewer@*.service.
For example, if you have a systemd service called postgresql.service,
with certificate and key files located in /etc/step/certs/postgresql.crt and /etc/step/certs/postgresql.key,
you can manually run systemctl start [email protected]
and systemd will immediately check that certificate's readiness for renewal, and potentially renew it.
If the certificate is successfully renewed, the postgresql service will be reloaded or restarted.
You'll often need to customize your service unit for a given service.
For example, you may need to:
-
Deploy the certificate and key to a service
-
Reload or restart additional dependent services
-
Combine the certificate and key files into a bundled
.pemor a PKCS#12.p12file, as needed by some services -
Ping a health check service or perform another action after a specific certificate is renewed
Instead of modifying the service template, we'll use service template overrides here. The overrides live in the drop-in configuration directory for the service being customized.
Here's an example override for a Lighttpd service that uses Docker Compose and is not managed by systemd:
/etc/systemd/system/[email protected]/override.conf:
[Service]
; `Environment=` overrides are applied per environment variable. This line does not
; affect any other variables set in the service template.
Environment=CERT_LOCATION=/etc/docker/compose/lighttpd/certs/example.com.crt \
KEY_LOCATION=/etc/docker/compose/lighttpd/certs/example.com.key
WorkingDirectory=/etc/docker/compose/lighttpd
; Restart lighttpd docker containers after the certificate is successfully renewed.
ExecStartPost=/usr/local/bin/docker-compose restartExecStartPost lines in your override.conf will run in addition to those in the service template. Furthermore, adding an empty ExecStartPost= override will disable the template's ExecStartPost line(s).
Here's a more complex example that calls the Grafana Data source HTTP API to refresh a client certificate stored in Grafana's configuration database.
/etc/systemd/system/[email protected]/override.conf:
[Service]
ExecStartPost=/usr/bin/env bash -c 'jq -n \
--rawfile ca_cert $STEPPATH/certs/root_ca.crt \
--rawfile client_cert $CERT_LOCATION \
--rawfile client_key $KEY_LOCATION \
-f /etc/systemd/system/[email protected]/datasource.jq \
| curl -s -X PUT \
-H @/etc/systemd/system/[email protected]/api_headers \
-d @- \
--cacert $STEPPATH/certs/root_ca.crt \
https://grafana:3000/api/datasources/1 > /dev/null'
ExecStartPost=curl -s -m 10 --retry 5 https://hc-ping.com/a66...fbba2When the certificate is successfully renewed:
- The
ExecStartPostin the service template will attempt to reload or restartgrafana-loki-datasource.service—which will do nothing, because no service with that name exists. - The
ExecStartPostin the override configuration will construct JSON and pass it to curl, updating the certificate and key in Grafana. - If all goes well, the final
ExecStartPostin the override configuration will ping a health check service at Healthchecks that expects to hear from this unit daily.
The final piece of the puzzle is the renewal timer. Timers and services go hand in hand in systemd:
A [email protected] will always trigger a corresponding [email protected].
Therefore, instead of enabling the cert-renewer@*.service service units directly, we'll enable timer units that will periodically trigger each service unit.
Create a timer unit template file:
$ sudo touch /etc/systemd/system/[email protected]
Add the following configuration to the file:
[Unit]
Description=Timer for certificate renewal of %I
Documentation=https://smallstep.com/docs/step-ca/certificate-authority-server-production
; PartOf=cert-renewer.target
[Timer]
Persistent=true
; Run the timer unit every 15 minutes.
OnCalendar=*:1/15
; Always run the timer on time.
AccuracySec=1us
; Add jitter to prevent a "thundering hurd" of simultaneous certificate renewals.
RandomizedDelaySec=5m
[Install]
WantedBy=timers.target(This file is maintained on GitHub)
Timers using this template will run every 5-10 minutes, with a randomized delay on each timer. The randomized delay helps with thundering herd problems that can occur when many virtual machines that were provisioned together try to renew their certificates at the same time.
With this template in place,
let's start timers for our postgresql and grafana-server renewer services:
$ systemctl enable --now [email protected]
Created symlink /etc/systemd/system/multi-user.target.wants/[email protected] → /etc/systemd/system/[email protected].
$ systemctl enable --now [email protected]
Created symlink /etc/systemd/system/multi-user.target.wants/[email protected] → /etc/systemd/system/[email protected].
$ systemctl list-timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Wed 2020-12-16 17:06:23 PST 8min left n/a n/a [email protected] [email protected]
Wed 2020-12-16 17:04:12 PST 6min left n/a n/a [email protected] [email protected]
...
Your periodic timers are now running and will run on system startup. You can override your timer units too, but you probably won't need to.
Another way to automate renewal is with the step renewal daemon.
With this method, step ca renew operates as a daemon that will keep your certificates up-to-date:
$ step ca renew --daemon foo.crt foo.key
INFO: 2019/06/20 12:36:54 first renewal in 14h46m57s
INFO: 2019/06/21 03:14:23 certificate renewed, next in 15h17m31s
INFO: 2019/06/21 18:31:00 certificate renewed, next in 14h33m17s
ERROR: 2019/06/22 11:04:39 error renewing certificate: client
POST https://localhost/renew failed: Post https://localhost/renew: dial tcp [::1]:443: connect: connection refused
INFO: 2019/06/22 11:05:00 certificate renewed, next in 14h33m17s
When daemonized, step ca renew will attempt a renewal when the certificate's lifetime is approximately two-thirds elapsed.
So, for a certificate with a 24 hour lifetime, it will attempt a renewal after about 16 hours.
There is some random jitter built into the daemon's schedule to prevent a large number of renewals from being sent to the CA simultaneously (eg. by a multitude of virtual machines that were provisioned at the same time and have identical certificate expiration dates).
If the CA is unreachable, renewals are retried every minute.
You can trigger renewal anytime by sending a SIGHUP signal to the step ca renew process ID.
You can add step ca renew --daemon as a systemd service that runs on startup and restarts as needed.
Here's an example of setting up everything via systemd:
$ cat <<EOF | sudo tee /etc/systemd/system/step.service > /dev/null
[Unit]
Description=Step TLS Renewer for Foo service
After=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
Restart=always
RestartSec=1
User=step
ExecStart=/usr/bin/step ca renew --daemon /home/step/foo.crt /home/step/foo.key
[Install]
WantedBy=multi-user.target
EOF
$ systemctl daemon-reload
Be sure the User has write access to the certificate and key you're renewing.
Rescan the systemd unit files:
$ sudo systemctl daemon-reloadEnable and start the service:
sudo systemctl enable --now stepMany services that depend on certificates will only read the certificate files on startup. So when you renew a certificate, a server process that depends on it may not detect that it has changed.
It's common for services (eg. nginx) to respond to a SIGHUP signal by reloading configuration files and certificates.
To address this, step ca renew can send a SIGHUP to your service after each renewal.
Here's an example for nginx:
$ step ca renew --daemon --exec "kill -HUP $NGINX_PID" foo.crt foo.key
INFO: 2019/05/01 14:22:18 first renewal in 15h50m43s
With cron-based renewal, you can have step ca renew run at a regular cadence (eg. every five minutes) and renew a certificate only when it's approaching its expiration date.
You can request that certificate renewal only occur if the certificate is approaching its expiry using the --expires-in <duration> flag.
The <duration> is a time interval like 4h or 30m.
Renewal will only occur if the expiry is within <duration> of the current time.
For example:
$ step ca renew --force --expires-in 4h foo.crt foo.key
certificate not renewed: expires in 23h58m44s
$ step ca renew --force --expires-in 24h foo.crt foo.key
Your certificate has been saved in foo.crt.
With --expires-in, we add a random jitter to <duration> (between 0 and <duration>/20). This helps with thundering herd problems if many virtual machines that were provisioned together try to renew their certificates at the same time.
Here's an example of setting up renewal via cron on a Debian-based system:
$ cat <<EOF | sudo tee /etc/cron.d/step-ca-renew
# Check for possible certificate renewal every five minutes
*/5 * * * * step step ca renew --force --expires-in 4h /home/step/foo.crt /home/step/foo.key
EOF
Be sure the user (in this example, step) has write access to the certificate and key you're renewing.
step ca renew allows a certificate owner to extend the lifetime of a certificate before it expires.
Unfortunately, it also lets an attacker with the private key do the same thing.
To prevent this, you need to explicitly tell step-ca to revoke a retired certificate.
See the certificate revocation section for details.
You can configure a provisioner to allow certificates to be renewed after they expire. This is not the default behavior, but it's useful if you have devices with intermittent network connectivity. The device will be able to renew its certificate the next time it has network access, provided the certificate has not been revoked.
To configure renewal after expiry for a provisioner, pass the --allow-renewal-after-expiry flag to the step ca provisioner management commands. See managing provisioners.