-
Notifications
You must be signed in to change notification settings - Fork 2k
docs: updated info on pg and redis with Infisical self hosting #5852
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
akhilmhdh
wants to merge
3
commits into
main
Choose a base branch
from
docs/self-host-infra
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| --- | ||
| title: "PostgreSQL" | ||
| description: "Understand the role PostgreSQL plays in Infisical, what data it stores, and how to operate it safely." | ||
| --- | ||
|
|
||
| PostgreSQL is the **authoritative datastore** for all persistent data in Infisical. Every piece of long-lived state — secrets, users, configurations, audit history — is stored in PostgreSQL. It is the single source of truth for the entire platform. | ||
|
|
||
| ## What it stores | ||
|
|
||
| | Data | Description | | ||
| |------|-------------| | ||
| | Secrets | All secrets and their full version history | | ||
| | Identities & Users | User accounts, machine identities, and their credentials | | ||
| | Projects & Organizations | Project configurations, organization settings, and memberships | | ||
| | Access Control | Roles, permissions, and access policies | | ||
| | Audit Logs | Records of all actions taken within the platform | | ||
| | Integrations | Configuration for third-party integrations and sync targets | | ||
| | Certificates & PKI | Certificate authorities, certificates, and PKI configurations | | ||
| | Encryption Keys | Encrypted project keys used for secret encryption | | ||
|
|
||
| ## Supported modes | ||
|
|
||
| | Mode | Supported | | ||
| |------|-----------| | ||
| | Standalone (Single Instance) | Yes | | ||
| | Read Replicas (Streaming Replication) | Yes | | ||
|
|
||
| For multi-region deployments with read replicas, see the [replication guide](/self-hosting/guides/replication). | ||
|
|
||
| ## Configuration | ||
|
|
||
| Configure PostgreSQL connectivity by setting the following environment variables on your Infisical instance. | ||
|
|
||
| <Info> | ||
| The database user must be granted all privileges on the Infisical database, including the ability to create schemas, tables, indexes, and perform all CRUD operations. | ||
| </Info> | ||
|
|
||
| <Tabs> | ||
| <Tab title="Primary"> | ||
| <ParamField query="DB_CONNECTION_URI" type="string" default="" required> | ||
| Postgres database connection string. | ||
| </ParamField> | ||
|
|
||
| <ParamField query="DB_ROOT_CERT" type="string" default="" optional> | ||
| Configure the SSL certificate for securing a Postgres connection by first encoding it in base64. | ||
| Use the following command to encode your certificate: `echo "<certificate>" | base64` | ||
|
|
||
| Many cloud providers provide a CA certificate for their data regions that you can use to secure your connection with SSL. | ||
|
|
||
| <AccordionGroup> | ||
| <Accordion title="AWS RDS"> | ||
| If you're hosting your database on AWS RDS, you can use their publicly available CA certificate as the database root certificate. | ||
|
|
||
| You can find all the available CA certificates for AWS RDS on the official [AWS RDS documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html). | ||
|
|
||
| As an example, if your RDS cluster is hosted in `us-east-1` _(US East, N. Virginia)_, you can use the following root certificate: https://truststore.pki.rds.amazonaws.com/us-east-1/us-east-1-bundle.pem. | ||
|
|
||
| Remember to base64 encode the certificate before setting it as the `DB_ROOT_CERT` environment variable. `cat /path/to/certificate.pem | base64`. | ||
|
|
||
| ```bash | ||
| DB_ROOT_CERT=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1 # .... (base64 encoded certificate) | ||
| DB_CONNECTION_URI=<rds-endpoint>?sslmode=verify-ca # or verify-full depending on your security policies | ||
| ``` | ||
| </Accordion> | ||
| </AccordionGroup> | ||
| </ParamField> | ||
| </Tab> | ||
| <Tab title="Read Replica"> | ||
| <ParamField query="DB_READ_REPLICAS" type="string" default="" optional> | ||
| Postgres database read replica connection strings. It accepts a JSON string. | ||
| ``` | ||
| DB_READ_REPLICAS=[{"DB_CONNECTION_URI":""}] | ||
| ``` | ||
| <Expandable title="Format"> | ||
| <ParamField query="DB_CONNECTION_URI" type="string" default="" required> | ||
| Postgres read replica connection string. | ||
| </ParamField> | ||
| <ParamField query="DB_ROOT_CERT" type="string" default="" optional> | ||
| Configure the SSL certificate for securing a Postgres replica connection by first encoding it in base64. | ||
| Use the following command to encode your certificate: `echo "<certificate>" | base64` | ||
|
|
||
| If not provided, it will use the primary SSL certificate. | ||
| </ParamField> | ||
| </Expandable> | ||
| </ParamField> | ||
| </Tab> | ||
| </Tabs> | ||
|
|
||
| ## Impact of data loss | ||
|
|
||
| PostgreSQL data loss is **critical and unrecoverable without a backup**. Because PostgreSQL is the sole source of truth for all persistent state, losing this data means losing: | ||
|
|
||
| - All secrets and their version history | ||
| - User accounts, identities, and credentials | ||
| - Project and organization configurations | ||
| - Audit logs and compliance records | ||
| - Integration settings and encryption keys | ||
|
|
||
| There is no way to reconstruct this data from other components. Redis caches are ephemeral and do not contain the full dataset. | ||
|
|
||
| <Warning> | ||
| Always maintain regular PostgreSQL backups and test your restore process. Without a working backup, PostgreSQL data loss is **permanent**. | ||
| </Warning> | ||
|
|
||
| ## Recommendations | ||
|
|
||
| - **Use a managed service** (e.g., AWS RDS, Google Cloud SQL, Azure Database for PostgreSQL) for automated backups, failover, and maintenance. | ||
| - **Enable point-in-time recovery (PITR)** for granular restore capabilities. | ||
| - **Test restores regularly**. A backup that has never been tested is not a backup. | ||
| - **High availability**: Configure streaming replication with automatic failover to minimize downtime. | ||
| - **Version**: Infisical supports PostgreSQL versions 14+, with version 16 being the most extensively tested. | ||
|
|
||
| For hardware sizing, see [hardware requirements](/self-hosting/configuration/requirements). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,141 @@ | ||
| --- | ||
| title: "Redis" | ||
| description: "Understand the role Redis plays in Infisical, why persistence is required, and how to operate it safely." | ||
| --- | ||
|
|
||
| Redis serves as both a **caching layer** and a **persistent job queue** in Infisical. Unlike typical cache-only use cases, Infisical relies on Redis to durably store background job data, making persistence a hard requirement. | ||
|
|
||
| <Warning> | ||
| Redis **must be configured with persistence enabled** (AOF, RDB, or both). Infisical uses Redis-backed queues for background processing. A non-persistent Redis instance that restarts will lose all pending and in-progress jobs. | ||
| </Warning> | ||
|
|
||
| ## What it stores | ||
|
|
||
| | Data | Description | | ||
| |------|-------------| | ||
| | Cache | Frequently accessed data and configurations, reducing load on PostgreSQL | | ||
| | Background Job Queues | Jobs for secret syncs to third-party integrations, secret rotations, scheduled tasks, webhook deliveries, and audit log processing | | ||
| | Job State | Metadata for in-progress, completed, and failed jobs including retry information | | ||
| | Session Data | Active user session tokens and ephemeral authentication state | | ||
|
|
||
| ## Supported modes | ||
|
|
||
| | Mode | Supported | | ||
| |------|-----------| | ||
| | Standalone (Single Node) | Yes | | ||
| | Sentinel (High Availability) | Yes | | ||
| | Cluster | Yes | | ||
| | Active-Active Geo-Replication | No | | ||
|
|
||
| <Warning> | ||
| Redis **active-active geo-replication** is not supported. If you need multi-region deployments, refer to Infisical's [replication architecture](/self-hosting/guides/replication) which uses independent Redis instances per region instead. | ||
| </Warning> | ||
|
|
||
| ## Configuration | ||
|
|
||
| Configure Redis connectivity by setting the following environment variables on your Infisical instance. | ||
|
|
||
| <Tabs> | ||
| <Tab title="Redis Standalone"> | ||
| <ParamField query="REDIS_URL" type="string" default="none" required> | ||
| Redis connection string. For SSL/TLS connections, use the `rediss://` protocol (note the double 's'). | ||
|
|
||
| Examples: | ||
| - Without SSL: `redis://localhost:6379` | ||
| - With SSL: `rediss://localhost:6379` | ||
| - With authentication: `redis://:password@localhost:6379` | ||
| - With SSL and authentication: `rediss://:password@localhost:6379` | ||
| </ParamField> | ||
| </Tab> | ||
| <Tab title="Redis Sentinel"> | ||
| <ParamField query="REDIS_SENTINEL_HOSTS" type="string" default="none" required> | ||
| Comma-separated list of Sentinel host:port pairs. ``` | ||
| 192.168.65.254:26379,192.168.65.254:26380 ``` | ||
| </ParamField> | ||
| <ParamField query="REDIS_SENTINEL_MASTER_NAME" type="string" default="mymaster"> | ||
| The name of the Redis master set monitored by Sentinel | ||
| </ParamField> | ||
| <ParamField query="REDIS_SENTINEL_ENABLE_TLS" type="bool" default="false"> | ||
| Whether to use TLS/SSL for Redis Sentinel connection | ||
| </ParamField> | ||
| <ParamField query="REDIS_SENTINEL_USERNAME" type="string" default="none"> | ||
| Authentication username for Redis Sentinel | ||
| </ParamField> | ||
| <ParamField query="REDIS_SENTINEL_PASSWORD" type="string" default="none"> | ||
| Authentication password for Redis Sentinel | ||
| </ParamField> | ||
| <ParamField query="REDIS_USERNAME" type="string" default="none"> | ||
| Authentication username for Redis Node | ||
| </ParamField> | ||
| <ParamField query="REDIS_PASSWORD" type="string" default="none"> | ||
| Authentication password for Redis Node | ||
| </ParamField> | ||
| </Tab> | ||
| <Tab title="Redis Cluster"> | ||
| <ParamField query="REDIS_CLUSTER_HOSTS" type="string" default="none" required> | ||
| Comma-separated list of Redis Cluster host:port pairs. ``` | ||
| 192.168.65.254:26379,192.168.65.254:26380 ``` | ||
| </ParamField> | ||
| <ParamField query="REDIS_CLUSTER_ENABLE_TLS" type="boolean" default="false"> | ||
| Enable Redis TLS encryption on connection. | ||
| </ParamField> | ||
| <ParamField query="REDIS_CLUSTER_AWS_ELASTICACHE_DNS_LOOKUP_MODE" type="boolean" default="false"> | ||
| Enable this if you are using AWS encrypt on transit for Elasticache cluster. For more information refer [here](https://github.com/redis/ioredis?tab=readme-ov-file#special-note-aws-elasticache-clusters-with-tls). | ||
| </ParamField> | ||
| <ParamField query="REDIS_USERNAME" type="string" default="none"> | ||
| Authentication username for Redis Node | ||
| </ParamField> | ||
| <ParamField query="REDIS_PASSWORD" type="string" default="none"> | ||
| Authentication password for Redis Node | ||
| </ParamField> | ||
| </Tab> | ||
| <Tab title="Redis Read Replica"> | ||
| <ParamField query="REDIS_READ_REPLICAS" type="string" default="none" optional> | ||
| Comma-separated list of Redis read replicas host:port pairs. ``` | ||
| 192.168.65.254:26379,192.168.65.254:26380 ``` | ||
| </ParamField> | ||
|
|
||
| The parameters like username, password, TLS, and Redis type of the primary instance will be inherited. | ||
| </Tab> | ||
| </Tabs> | ||
|
|
||
| ### SSL/TLS | ||
|
|
||
| To connect to Redis with SSL/TLS, use the `rediss://` protocol (note the double 's') in your connection string. | ||
|
|
||
| If your Redis server uses a certificate signed by a private CA or a self-signed certificate, set the `NODE_EXTRA_CA_CERTS` environment variable to the path of your CA certificate file: | ||
|
|
||
| ```bash | ||
| REDIS_URL=rediss://your-redis-host:6379 | ||
| NODE_EXTRA_CA_CERTS=/path/to/ca.crt | ||
| ``` | ||
|
|
||
| For Redis Sentinel or Cluster mode, use the `REDIS_SENTINEL_ENABLE_TLS` or `REDIS_CLUSTER_ENABLE_TLS` environment variables respectively. | ||
|
|
||
| ## Impact of data loss | ||
|
|
||
| Redis data loss is **disruptive but recoverable**. Unlike PostgreSQL, Redis does not hold the source of truth — secrets and configurations remain safe in PostgreSQL. However, the impact depends on what was in Redis at the time: | ||
|
|
||
| - **Cache loss**: No lasting impact. The cache repopulates automatically as read requests come in. You may see a temporary spike in PostgreSQL load until the cache warms up. | ||
| - **Queue loss**: Pending background jobs are lost. In-flight operations such as secret syncs to third-party integrations, pending secret rotations, or scheduled webhook deliveries will not complete. These operations will need to be re-triggered manually or will recover on the next scheduled cycle. | ||
|
|
||
| ## Why persistence is required | ||
|
|
||
| Infisical uses Redis to manage background job queues. These queues handle operations that must eventually complete: | ||
|
|
||
| - Syncing secrets to external providers (AWS Secrets Manager, GCP Secret Manager, etc.) | ||
| - Rotating credentials on a schedule | ||
| - Delivering webhooks | ||
| - Processing audit logs | ||
|
|
||
| If Redis is running without persistence and restarts, all queued and in-progress jobs disappear silently. There is no mechanism to replay these jobs from PostgreSQL. This can result in secrets being out of sync with third-party integrations, missed rotations, or lost webhook deliveries — with no indication that anything went wrong. | ||
|
|
||
| ## Recommendations | ||
|
|
||
| - **Enable persistence** using AOF (`appendonly yes`) with `appendfsync everysec` for a good balance between durability and performance. RDB snapshots can be used as an additional safety net. | ||
| - **Set eviction policy to `noeviction`** to prevent Redis from silently dropping queue data under memory pressure. | ||
| - **Use Redis 6.2+**. Infisical supports Redis versions 6.x and 7.x, with 6.2 as the minimum recommended version. | ||
| - **High availability**: Use Redis Sentinel or a managed Redis service with automatic failover. | ||
| - **Monitor memory usage**. Because `noeviction` is required, Redis will reject new writes if it runs out of memory rather than evicting data. Ensure sufficient memory is provisioned for your workload. | ||
|
|
||
| For hardware sizing, see [hardware requirements](/self-hosting/configuration/requirements). | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.