Allowing rendering a template even when reading an optional secret fails

[busser]

When reading multiple secrets from Vault in the same template, if reading one of those secrets fails then the template's execution fails and the templated file is never created/updated. It would be nice to have a function (eg: optionalSecret) that allows the template's execution to continue even if reading one of the secrets fails.

For instance, let's say we have three dynamic secrets in Vault. Each secret provides credentials for a different database, and each database runs in a separate failure domain. We read those secrets with the Vault Agent, which uses the Consul template library:

{{- with secret "database_alpha/creds/read-only" }}
alpha:
  username: {{ .Data.username }}
  password: {{ .Data.password }}
{{- end }}
{{- with secret "database_bravo/creds/read-only" }}
bravo:
  username: {{ .Data.username }}
  password: {{ .Data.password }}
{{- end }}
{{- with secret "database_charlie/creds/read-only" }}
charlie:
  username: {{ .Data.username }}
  password: {{ .Data.password }}
{{- end }}

Executing the template calls the secret function three times, each sending a request to Vault. If all databases are available then the template executes successfully and the Vault Agent creates/updates the template file. However if one of the databases is unavailable then Vault will respond to the corresponding request with a 500 error. The call to secret will return an error and the template's execution will fail. The Vault Agent will not create/update the templated file.

In our use case, we want to tolerate a single database failing, so that our services can still connect to the other two databases. Would you consider adding a new function to the Consul template library that allows for finer error handling?

For example, we could have a optionalSecret function that acts like the secret function except that it does not return an error if reading the secret fails. Instead, the function could return a nil value. The template from above would look something like this:

{{- with optionalSecret "database_alpha/creds/read-only" }}
alpha:
  username: {{ .Data.username }}
  password: {{ .Data.password }}
{{- end }}
{{- with optionalSecret "database_bravo/creds/read-only" }}
bravo:
  username: {{ .Data.username }}
  password: {{ .Data.password }}
{{- end }}
{{- with optionalSecret "database_charlie/creds/read-only" }}
charlie:
  username: {{ .Data.username }}
  password: {{ .Data.password }}
{{- end }}

This would also open the door to finer error handling:

{{- with optionalSecret "database_alpha/creds/read-only" }}
alpha:
  username: {{ .Data.username }}
  password: {{ .Data.password }}
{{- else }}
alpha:
  error: failed to get credentials, check app dashboard for details
{{- end }}

In principle, this is related to the secretOrDefault function described in issue #942. That issue describes a use case with non-existent secrets, which is a little different from our use case where the secrets exist but reading them fails.

One could work around this by writing each secret to a separate file: an unavailable database would only prevent a single file from being created/updated. However that doesn't match our use case: the list of secrets to fetch is obtained dynamically with the secrets function, so we need to use a single template.

What do you think about adding the optionalSecret function described above, or something similar?