Everything is YAML in one directory (default ./config, override with the
CONFIG_DIR environment variable), laid out the way AWX config as code is:
config.yaml at the root holds the server settings, and every other *.yaml /
*.yml file, at any depth, holds items under a typed top-level key.
Only config.yaml is mandatory; a directory with nothing else is a valid,
empty inventory.
All cross-references are validated at startup — an unknown source, connector,
credential or project anywhere in the config aborts boot with a list of every
error found. The same validation runs standalone with unified-api --check-config
(see CLI) — for CI on a config repository.
- The directory
- config.yaml (required)
- sources
- connectors
- views
- credentials
- enrichers
- endpoints
- projects
- api_keys
- Environment variables
Configuration is read at startup, and can also be written and reloaded over HTTP where that is enabled — see Configuration API.
/etc/unified-api/
├── config.yaml # required, at the root, under this name
├── sources/ # folders are for people: any names, any depth, or none
│ ├── acme/
│ │ ├── src-d42-acme.yaml
│ │ └── src-ssh-acme.yaml
│ └── src-netbox.yaml
├── connectors/
│ ├── d42.yaml
│ └── ssh-facts.yaml
├── views/
│ └── vw-facts-all.yaml
├── credentials/
│ └── cred-d42.yaml
├── projects/
│ └── prj-inventory.yaml
└── api_keys/
└── key-awx.yaml
| Top-level key | Holds |
|---|---|
sources |
Each inventory instance: its connector, credentials, config, schedule and TTLs |
connectors |
How to talk to a kind of system: the script (or SSH command), its project, its arguments |
views |
Read-only composites over several sources, routed by declared ownership |
endpoints |
Outputs combining cached datasets through a transformer |
enrichers |
Post-processors over data already in the cache |
projects |
Git repositories holding scripts and static inventories |
credentials |
Where each credential's secrets are read from (never the values) |
api_keys |
API keys by name and role (the values come from env vars) |
Every item file looks the same: a key, and a list of items under it.
# sources/acme/src-d42-acme.yaml
sources:
- name: src-d42-acme # identity: URLs, references, cache keys
description: "Device42 Acme" # label, for people
connector: d42
credentials: [cred-d42]
sync_interval_seconds: 300
ttl_seconds: 3600The rules, and why they are what they are:
- The key says what an item is, never the folder. A file in
connectors/holdingsources:is read as a source. Folders exist for people — by team, by estate, by datacenter — and may nest as deep as they like, or not exist at all. Tying meaning to a folder would make moving a file a semantic change. - One item per file is the convention,
<section>/<name>.yaml, and the one this documentation uses. It is not a rule: a file may hold several items, of several types. A review of "add a tenant" is then one new file, not a hunk in the middle of a 2,000-line map. nameis the identity, as in AWX: what other items reference, what URLs (/api/v1/sources/{id}) and cache keys use. Letters, digits,-,_and., not starting with a dot — it appears in URLs and in file names. The values are the same ids 0.34 used as map keys, so nothing that reads the API changes.descriptionis optional, for people. Every API response field callednameanswers the description, falling back to the identity when there is none — so responses read exactly as they did before item files.- A
namedefined twice is an error naming both files. Which one won would otherwise depend on the order files happen to be read in. - References drop the
_id: a source names itsconnectorandcredentials, an enricher itstarget,sourceandproject, an endpoint itssourcesandproject, a project itscredentials(a list, as on a source, holding at most one: a clone authenticates once; the singularcredential:still loads, deprecated). - Hidden files and folders are skipped (editor swap files,
.git, the config API's staging folders). Symlinks are followed: a Kubernetes ConfigMap mount is nothing but links into a hidden..datafolder, and skipping them would load an empty configuration. - A path may only use letters, digits,
-,_and.in each segment, none starting with a dot, ending in.yamlor.yml— the same rule the config API applies to what it is asked to write, so a file the API cannot name cannot be part of the configuration either.
Parsing is strict. Every way a file can be wrong fails startup naming the file, and every error is reported at once, so a push with three typos in three files names all three:
| Mistake | Error |
|---|---|
| An unknown top-level key | sources/a.yaml: unknown key 'sourcse' (expected one of: sources, views, ...) |
| A 0.34 file | the same, ending — this is the 0.34 layout (items keyed by id), which this version does not read: write it as items, or convert it with unified-api 0.36's --migrate-config |
| An unknown key inside an item | sources/a.yaml: source 'src-a': unknown field `ttl_second`, expected one of ... |
An item without name |
sources/a.yaml: sources item 2 needs a 'name', its identity |
| A name defined twice | sources/b.yaml: source 'src-a' is already defined in sources/a.yaml |
A typo'd key failing loudly is the point: silently applying the default for
ttl_second: is how an inventory ends up an hour stale with nobody knowing why.
The free-form config: maps are the exception — they are arbitrary by design,
and their scalars are taken as text, exactly as written (concurrency: 20
arrives as "20", version: 1.10 as "1.10", account: 0x1F as "0x1F"), since
they become environment variables and script settings. Every other text field
reads an unquoted scalar the same way: script_args: ["--days", 7] is
["--days", "7"].
A configuration in an older layout is refused, never converted: the 0.34
layout (eight fixed files, each a map keyed by id), a 0.35-form ssh connector
(see connectors) and
output_format: native each fail the load with a message saying what to write
instead. A 0.34 directory can still be converted to item files with unified-api
0.36's --migrate-config, run on it before upgrading past 0.36.
The one file that must exist, holding the server, cache, projects and secrets settings.
It is always at the root, under exactly this name, and holds no items.
server:
host: "0.0.0.0"
port: 8182
# Optional. Empty/absent (default) = no CORS headers at all — right for
# server-to-server consumers. List browser origins to enable, "*" for any.
cors_allowed_origins: []
# Optional, default false. false = /readyz is green once ANY source has
# synced (a pod serving part of the inventory beats one serving nothing).
# true = every configured source must have synced first — for deployments
# where a partial inventory is worse than none.
readyz_require_all_sources: false
# Optional, default false. false keeps /metrics public alongside the health
# probes, which is what Prometheus scrape configs expect. true requires an
# API key — for a shared network, since the exposition labels every source
# id and host count.
metrics_require_auth: false
# Optional, default 15. How long a read with ?refresh=true may wait before it
# gives up and serves the cached data. Sized for a consumer waiting on a page,
# not for a scheduled sync. See docs/on-demand-refresh.md.
refresh_timeout_seconds: 15
# Optional, default 8. Concurrent on-demand refreshes, process-wide. The TTL
# bounds repeat requests for the same host; this bounds requests for many
# different hosts arriving together.
refresh_max_concurrent: 8
# Optional, default 20. How long shutdown waits for background tasks (syncs,
# enrichers, project pulls, the snapshot task) to finish in-flight work
# before writing the final cache snapshot anyway. Size it to fit inside the
# pod's terminationGracePeriodSeconds with room for the snapshot write.
shutdown_grace_seconds: 20
# Optional, default 2097152 (2 MiB — what was always enforced, implicitly).
# Largest request body accepted on any route; an oversized one answers 413
# naming this key. Size it for the biggest configuration push (a whole
# directory is one body) or host-vars write you expect.
max_body_bytes: 2097152
# Optional. This instance's address, as people and other instances reach it
# (the ingress, not the port above). A central sends it with the federation
# map, so each edge's Topology page links back to it. Reloadable.
url: "https://unified-api-central.example.com"
# Optional. Absent = purely in-memory cache (restarts start empty).
# With the block, the cache is snapshotted to `path` every `interval_seconds`
# (default 60) and on graceful shutdown, and reloaded at boot. See docs/caching.md.
cache:
persistence:
path: "/var/lib/unified-api/cache.json"
interval_seconds: 60
# Optional. Where git projects are cloned (one subdirectory per project name).
projects:
dir: "/var/lib/unified-api/projects" # default "projects"
# Optional, OFF by default. Lets an admin key read, write and reload this
# directory over HTTP, so a pipeline can push a change instead of publishing
# an artifact the instance has to pull. Turning it on means an admin key can
# rewrite every file the loader reads, the api_keys items included — the same
# authority as editing the mounted directory. See docs/config-api.md.
config_api:
enabled: falseOne item per inventory instance: which system, reached how (its
connector), as whom (credentials), with which settings (config), how often,
and how long the data stays fresh. The name is the source id used in URLs.
# sources/src-section9.yaml
sources:
- name: src-section9
description: "Section 9 Inventory"
connector: "sample-inventory" # a connector item, or a built-in (below)
credentials: ["cred-section9-api"] # credential items
sync_mode: "replace" # "replace" (default) or "merge"
sync_interval_seconds: 60 # background sync; 0/absent = manual only
# schedule: "30 2 * * *" # OR a cron cadence (UTC) — pick one
ttl_seconds: 3600 # dataset-level TTL (required)
# timeout_seconds: 900 # override the connector's bound, this source only
host_sync_updates_group_vars: false # may a host-scoped sync take the vars of
# the groups it joins? (default false)
allow_on_demand_refresh: false # may a READ trigger a sync? (default false)
config: # free-form strings passed to the connector
scenario: "large"| Field | Notes |
|---|---|
connector |
Required. The name of a connector item, or a built-in: remote (another unified-api, see federation) or static_inventory (an Ansible YAML inventory in a project checkout). See connectors |
credentials |
Credential items resolved before each sync and handed to the connector |
project, path |
static_inventory sources only: the project whose checkout holds the inventory, and the file inside it. Refused on every other source, where they would look like they select a script and would not |
timeout_seconds |
Overrides the connector's timeout_seconds for this source only — the bigger tenant of the same system. Absent = the connector's (default 300). A timed-out sync fails with a clear error instead of hanging its scheduler task or HTTP request |
host_sync_updates_group_vars |
Whether a host-scoped sync also takes the vars of the groups it places those hosts in, and only those groups. Off by default: a granular call answers about a host, not about what a group means for its other members |
sync_mode |
How a full sync lands in the cache: replace swaps the dataset, merge patches it |
ttl_seconds |
See caching for the freshness model |
ttl_overrides |
Optional per-host and per-group TTLs on this source: {hosts: {<host>: <seconds>}, groups: {<group>: <seconds>}}. The same can be said from the host's or the group's side, in a host or group item |
allow_on_demand_refresh |
Whether GET /dataset?host=X&refresh=true may gather the named hosts. Off by default: a read that can cause SSH into a datacenter is a capability. Turning it on makes ttl_seconds load bearing — it becomes the threshold at which a read pays for a gather — so review it first. See on-demand refresh |
schedule |
Cron cadence instead of an interval: standard 5-field cron, optional leading seconds field, evaluated in UTC. Validated at startup; mutually exclusive with a non-zero sync_interval_seconds. Cron sources get no startup jitter (their times are deliberate) but back off on failure like interval sources, by letting occurrences pass |
hosts_from_source |
SSH sources only: dynamic host list from another source's cached dataset (source + optional match_pattern.groups/hosts + connect_via); mutually exclusive with config.hosts — see connectors |
advertise_scope |
Explicit ownership claim served on GET /sources/{id}/scope (groups + hosts; must name at least one). Optional — an SSH source already advertises its hosts_from_source pattern; this overrides the derivation. The consumer side of federation reads it so view ownership stops being a duplicated truth |
config |
Arbitrary key: value strings the connector receives (SOURCE_CONFIG for a script). The SSH connector reads hosts, port, concurrency, ssh_connect_timeout_seconds (per-host, default 30) and ssh_known_hosts (host key verification — unset accepts any key, with a warning per sync) from here (0.35's gather_mode and fact_path are refused: the connector decides what runs); a remote source reads url and source — see connectors |
What a source does differently for one host, or for every host in one of
its groups, lives in a host or group item rather than in the source: one
file per host (hosts/<host>.yaml) or group (groups/<group>.yaml), named
by the host or group as the sources' datasets spell it. Each entry under
sources names a source and what that source does for it.
# hosts/motoko.section9.net.yaml
hosts:
- name: motoko.section9.net
description: "Major's workstation" # optional label
sources:
src-section9:
ttl_seconds: 300 # fresh for 5 min here; the source says 1 h
# groups/production.yaml
groups:
- name: production
sources:
src-section9:
ttl_seconds: 900 # every host in the group, joiners included| Field | Notes |
|---|---|
sources.<id>.ttl_seconds |
This source's TTL for the host, or for every host in the group of that source's dataset: the refresh gate and what its freshness is reported against. A host's own TTL beats a group's, which beats the source's ttl_seconds |
- Per source, not for the host everywhere. A TTL describes one source's copy of the host: a host in Device42, VMware and the SSH facts may only need its facts kept fresh.
- Checked against the configuration, not the data. A source id that does not exist is refused, and a view's id is refused naming the member to use instead (a view keeps no copy of its own). A host or group no source has today is not an error: hosts come and go with the systems the sources read.
- Beside a source's own
ttl_overrides. Both work, and both land in the same place. Write it where it reads best: on the source when it is about that source's hosts, in an item when it is about one host across several sources. The same host or group set for the same source in both places is refused, naming both, so no precedence rule has to be known. - Folded in at load, as a source's connector is: a reload applies a changed item, and reports it as a change to the sources it names.
What a source used to say about its script now lives on a connector, so ten Device42 tenants are ten sources and one connector instead of ten copies of the same script path and arguments that drift apart one edit at a time. The split follows one question — does it describe the script, or this instance of the system?
| Field | Lives on | Why |
|---|---|---|
type (script / ssh) |
connector | The protocol |
project, path |
connector | Where the script is |
script_args, host_args |
connector | How the script is called |
timeout_seconds |
connector, overridable per source | A property of the script, except for the tenant ten times bigger |
credentials, config |
source | Which instance, as whom |
schedule, sync_interval_seconds, ttl_seconds |
source | How often and how fresh, per instance (per host or group: their own items) |
sync_mode, host_sync_updates_group_vars |
source | Cache policy of this instance's entry, not a property of the script |
hosts_from_source, allow_on_demand_refresh, advertise_scope |
source | Per instance |
Two connectors need no item: the protocol is fixed, and the only thing to configure — where — is per source.
sources:
# Another unified-api instance (federation)
- name: src-edge-dc2
description: "Datacenter 2"
connector: remote
credentials: ["cred-edge-dc2"] # a token credential = the remote API key
sync_interval_seconds: 120
ttl_seconds: 600
config:
url: "https://unified-api-dc2.example.com"
source: "src-ssh" # the source id ON THE REMOTE instance
# An Ansible YAML inventory in a project checkout
- name: src-linux
description: "Linux static inventory"
connector: static_inventory
project: "prj-inventories" # the checkout
path: "linux/inventory.yaml" # the file inside it
sync_interval_seconds: 300
ttl_seconds: 600remoterequiresconfig.urlandconfig.source, and refusesproject/path. See federation.static_inventoryrequiresprojectandpathon the source itself. See connectors.
project + path mean the same thing everywhere they appear — a file inside
that project's checkout — whether it is a connector's script or a static
source's inventory.
How to talk to a kind of system. Full treatment, including the script contract, in connectors.
# connectors/d42.yaml
connectors:
- name: d42
description: "Device42 inventory script"
type: "script" # script (default) | ssh
project: "prj-inventory" # the checkout `path` resolves in
path: "dynamic/d42/inventory.py"
script_args: ["--list"] # CLI args, verbatim (no shell)
host_args: ["--host-with-groups", "{target}"] # replace script_args for a host sync
timeout_seconds: 300 # default 300; a source may override it
# connectors/ssh-facts.yaml
connectors:
- name: ssh-facts
description: "SSH fact gathering"
type: "ssh"
location: "host" # path is on each host (or a script here,
path: "/etc/ansible/facts.d/*.fact" # with project / location: instance)| Field | Notes |
|---|---|
type |
script (default) runs a local process; ssh connects to the fleet natively |
project |
The project whose checkout holds the script. Required for a script connector with a relative path; on ssh, the project of a script streamed to each host (refused with location: host) |
location |
ssh only: host (path is on each host: files run or read as facts) or instance (a script here, streamed to each host; the default with a project). See connectors |
path |
A file, always. script: the one to run, relative to the project checkout or absolute. ssh: a script here, or a file or pattern of files on each host |
script |
ssh only, instead of path: the script written in the item, streamed to each host as a file from here would be |
script_args |
CLI arguments, quoted. ["--list"] makes standard Ansible dynamic inventory scripts work unmodified. On ssh, passed to the streamed script or to every executable a host path matches |
host_args |
Script only. CLI arguments for a sync that names hosts, replacing script_args for that run, {target} substituted by the requested hostnames (comma-joined). Empty = call the script as a full sync would and filter the answer. See connectors |
timeout_seconds |
Bound on a sync through this connector (default 300); a source's own timeout_seconds overrides it |
Settings that look active and are not are refused rather than ignored:
host_args or output_format on an ssh connector, output_format other than
ansible on a script connector, project with
location: host, location or script on a script connector, path and
script together, a relative script path
without a project, and the names remote and static_inventory, which belong
to the built-ins.
Read-only composites over several sources: one id that routes a per-host read to whichever member owns that host. Views answer on the source routes, share the source id space (a collision fails startup) and hold no cache entry of their own. Full treatment in views.
# views/vw-facts-all.yaml
views:
- name: vw-facts-all
description: "Facts, both datacenters"
members: # ordered: the first claim wins
- source: "src-ssh-dc1" # where the facts come from
owns:
source: "src-d42" # inventory the pattern resolves against
groups: ["datacenter_dc1"]
- source: "src-edge-dc2" # a remote (federated) member works too
owns:
source: "src-d42"
groups: ["datacenter_dc2"]
hosts: ["appliance01.dc2.example"] # claimed literally as well
# ttl_seconds: 30 # optional; absent = inherit the owner's| Field | Meaning |
|---|---|
members |
Ordered list. Overlap resolves by declared order — the first member that claims a host wins it |
members[].source |
Source id this member serves data from; must exist and must not be another view |
members[].owns.source |
Source whose cached dataset the groups below resolve against |
members[].owns.groups |
Groups of that source whose members this member owns |
members[].owns.hosts |
Hosts claimed literally, whether or not the inventory knows them — a host built this morning is exactly the one somebody needs to reach |
members[].owns.advertised |
Route by what the member SOURCE claims to own (its advertise_scope locally, the edge's GET /scope for remote members, refreshed per sync with the last-known claim surviving an unreachable edge). groups/hosts become the fallback; with neither claim nor fallback the member claims nothing — see views |
ttl_seconds |
The view's own freshness policy, which is also the gate for refresh=true. Absent = each host inherits its owning member's TTL. A member's host and group items win either way |
Ownership is declared, not inferred from what a member has cached: facts sources often sync daily on purpose, and some hosts (appliances that take no SSH) never enter any cache at all — both are exactly the hosts on-demand refresh exists for. See views → ownership.
Strict parsing matters most here. Ownership is the routing table, and grups:
instead of groups: would otherwise parse as an empty pattern, which claims
everything.
A view is granted to an API key exactly like a source, by its id under
sources:; the key needs no access to the members.
Credentials never contain secrets — they describe where to read them; this is the field reference, and the full resolution story (backends, the cache and rotation, native Vault, failure semantics) is its own page.
# credentials/cred-section9-api.yaml
credentials:
- name: cred-section9-api
description: "Section 9 API"
type: "username_password" # username_password | token | ssh_key
env_vars: # our key -> the full env var name
username: "SECTION9_USERNAME"
password: "SECTION9_PASSWORD"
- name: cred-ssh-infra
description: "SSH Infrastructure"
type: "ssh_key"
env_vars:
username: "INFRA_SSH_USERNAME"
file_keys: # our key -> file path; passed as <key>_path
ssh_key: "/run/secrets/infra-ssh/id_rsa"
- name: cred-d42-vault
description: "Device42 (from Vault)"
type: "username_password"
vault: "cred-vault" # the type: vault credential to read from
vault_path: "infra/device42" # KV v2 path under that Vault's mount
secret_keys: # our key -> field in the Vault secret
username: "user" # (omit secret_keys to take every field)
password: "pass"Resolution order per credential: vault_path (native Vault, see below), or
env_vars (environment variables, each named in full) / secret_file (a JSON
file, with secret_keys naming its fields), plus file_keys entries which
are passed through as <key>_path values. A credential that fails to resolve
fails the sync with a clear error — it is never silently skipped.
The older env form, env_prefix + secret_keys (SECTION9 + {username: USERNAME} reads SECTION9_USERNAME), is still read but deprecated: startup
and --check-config print the env_vars to write instead, and a later release
refuses it. Setting both forms on one credential is refused, and so is
env_vars beside a vault_path or a secret_file: a credential reads from
one place.
Two rules keep the variables a script receives unambiguous (see
connectors): a source's credentials must not be delivered
under one id (cred-vc-aa1 and cred_vc_aa1 both become
CREDENTIAL_VC_AA1_*), and a source on the built-in ssh or remote
connector lists one credential — those connectors log in with one, reading its
bare username and key, or token.
A Vault is a credential too, of type: vault, as AWX models a HashiCorp
Vault lookup: the credentials that read from it name it with vault: and
say where with vault_path. Several Vaults can coexist, each with its own
login and cache.
# credentials/cred-vault.yaml — the connection to a Vault
credentials:
- name: cred-vault
description: Vault
type: vault
address: https://vault.example.com:8200
mount: secret # KV v2 mount (default "secret")
env_vars:
token: VAULT_TOKEN # token auth: re-read per resolution,
# so rotating the token needs no restart
# kubernetes_role: unified-api # OR Kubernetes auth: the service-account
# jwt_path: /var/run/secrets/kubernetes.io/serviceaccount/token
# # JWT is exchanged for a client token,
# # cached and renewed at 80% of its lease
timeout_seconds: 10 # bound on every request (default 10)
cache_ttl_seconds: 60 # how long a value read here is reused
# (default 60; 0 = every resolution)
# credentials/cred-d42-vault.yaml — a login read from it
credentials:
- name: cred-d42-vault
type: username_password
vault: cred-vault
vault_path: infra/device42
secret_keys: # our key -> field in the Vault secret
username: user # (omit secret_keys to take every field)
password: passtype: vault field |
Notes |
|---|---|
address |
Required. http:// or https:// |
mount |
The KV v2 mount vault_path values live under (default secret) |
env_vars.token / kubernetes_role |
How to log in: exactly one. jwt_path goes with kubernetes_role |
timeout_seconds |
Bound on every request (default 10) |
cache_ttl_seconds |
How long a value read from this Vault is reused (default 60; 0 = every resolution): what keeps the sync schedule from becoming a request storm, and also how long a rotation takes to show |
The adapter speaks KV v2 only and reads <address>/v1/<mount>/data/<vault_path>.
Credentials without a vault_path keep resolving from env/files, uncached, so
adoption is per credential — move them one at a time. A Vault credential is
never handed to a source, project or LDAP bind: naming one there is refused,
since its token is not a login.
The secrets: block in config.yaml (cache_ttl_seconds, and a single
vault: with token_env) still loads but is deprecated: startup and
--check-config say what to write instead, and a later release refuses it.
While it is set, its cache_ttl_seconds still caches every credential, as it
did (in place of each Vault's own cache, never on top of it), and a
vault_path with no vault: reads from its vault:, cached for 60 seconds
when cache_ttl_seconds is absent, as 0.36 did by default.
Post-processors over data already in the cache. Two modes:
Script-based — runs a script that receives the target dataset on stdin and returns a partial dataset (modified/removed hosts):
# enrichers/enrich-resolve-ssh.yaml
enrichers:
- name: enrich-resolve-ssh
description: "Resolve SSH reachability"
target: "src-section9" # dataset being enriched
project: "prj-connectors" # optional: resolve script_path in this checkout
script_path: "enrichers/resolve.py"
script_args: [] # optional CLI args for the script
sync_interval_seconds: 300 # scheduled run; 0/absent = manual only
# schedule: "45 2 * * *" # OR a cron cadence (UTC) — pick one
timeout_seconds: 300 # abort a run that takes longer (default 300)
config: {}Declarative merge — copies specified hostvars fields from one source into another, matched by hostname. No script needed:
# enrichers/enrich-business-unit.yaml
enrichers:
- name: enrich-business-unit
description: "Add business_unit from D42"
target: "src-ssh-dc1" # dataset being enriched
source: "src-d42" # dataset to copy fields from
fields: ["business_unit"] # top-level hostvars keys to copy
sync_interval_seconds: 3600An enricher is not a connector — it gathers nothing — so it keeps its own
script_path rather than naming a connector.
Output endpoints combine one or more cached datasets through a transformer.
Each endpoint sets exactly one of output (a builtin, in-process
transformer) or script_path (an external script).
# endpoints/ep-ansible-full.yaml — builtin transformer, no process spawn
endpoints:
- name: ep-ansible-full
description: "Full Ansible Inventory"
sources: ["src-section9", "src-infra"]
output: ansible # builtins: ansible, json, csv
config:
filter: "datacenter=section9" # host, group, filter, fields, exclude_fields (see docs/endpoints.md)
# endpoints/ep-ansible-managed.yaml — a constructed inventory: merge everything,
# return only the hosts one source has. The others still enrich them and still
# group them. Optional; a limit must name one of the endpoint's own sources.
endpoints:
- name: ep-ansible-managed
description: "Managed Hosts Only"
sources: ["src-section9", "src-infra"]
output: ansible
limit:
by_hosts_from_inventory: "src-section9"
# endpoints/ep-dns-zone.yaml — script transformer, for a bespoke format
endpoints:
- name: ep-dns-zone
description: "DNS zone file"
sources: ["src-infra"]
project: "prj-connectors" # optional: resolve script_path in a checkout
script_path: "outputs/dns_zone.py"
script_args: [] # optional CLI args for the script
timeout_seconds: 300 # script only: abort a transform that takes longer (default 300)Git repositories that hold connector/enricher/transformer scripts and static
inventories — the field reference; the full lifecycle (boot behavior, script
resolution, virtualenvs, health) is its own page. At boot the
app clones each project (shallow, one directory per project name under the
projects.dir from config.yaml, default ./projects) and re-pulls it every
sync_interval_seconds (fetch + hard reset: local drift is discarded). The
boot clones run in the background, after the listener is up: they run
concurrently and each is bounded by the project's timeout_seconds, so an
unreachable git remote delays that project's scripts — never /healthz or a
Kubernetes startup probe. The sync schedulers start once the clones have had
their bounded chance.
# projects/prj-connectors-infra.yaml
projects:
- name: prj-connectors-infra
description: "Infrastructure Connectors"
git_url: "https://github.com/OpusProjects/connectors-infra.git"
branch: "main" # default "main"
credentials: ["cred-github-token"] # optional, for private repos (one)
sync_interval_seconds: 1800 # 0/absent = no periodic re-pull
# schedule: "15 2 * * *" # OR a cron cadence (UTC) — pick one
timeout_seconds: 300 # abort a clone/pull that runs longer (default 300)
python_venv: false # build a venv from requirements.txt; see below
sync_on_boot: true # default true; see belowThree sync styles compose from those two knobs:
| Style | Config | Behavior |
|---|---|---|
| Automatic | sync_interval_seconds: N or a cron schedule |
update at boot + every N seconds / at the cron's UTC times |
| Boot only | interval 0/absent | clone/update at boot, then frozen |
| Manual / pipeline-driven | sync_on_boot: false, interval 0/absent |
an existing checkout is used as-is (no network at startup); updates only via POST /api/v1/projects/{id}/sync |
With sync_on_boot: false a missing checkout is still cloned at boot (no
scripts, nothing to run), so first bring-up needs no special casing. Keep
projects.dir on a persistent volume for the manual style — the checkout IS
the durable state, there is nothing else to save at shutdown. A pipeline in
the scripts repo can push and then call the sync endpoint (admin key) to roll
new scripts without restarting; script files are re-read on every execution,
so the update takes effect on the next run.
How script paths resolve: a relative path whose file exists inside the
checkout runs from there (<projects.dir>/<project>/<path>); otherwise the
configured path is kept as-is (image-baked and mounted scripts keep working,
with a warning when the checkout exists but the file doesn't). Resolution
happens at every execution, not once at boot, so a script that first
appears after startup — a slow clone, a pipeline's first push — is used on the
very next run without a restart. The project comes from the connector (its
project + path), from a static source (its own project + path), or from
an enricher's or endpoint's optional project (for their script_path). SSH
connectors are never resolved — their path is a remote command. Scripts must
carry the executable bit in git (git update-index --chmod=+x).
With python_venv: true, a requirements.txt in the checkout gets a real
virtualenv: built after the clone, refreshed after any pull that changed
requirements.txt (unchanged pulls cost two file reads, not a pip run), and
kept OUTSIDE the checkout (<projects.dir>/.venvs/<project>) so the hard
reset cannot wipe it. When this project's scripts run, the venv's bin/ is
prepended to their PATH — a #!/usr/bin/env python3 shebang resolves to the
venv's interpreter, pip-installed imports work, and non-Python scripts in the
same project are untouched. A failing install (a typo'd package, an
unreachable index) fails the project sync, visible in its sync_health
and bounded by timeout_seconds — not a per-source import error at the next
sync with the cause nowhere.
Private repos: a token credential authenticates over https (the token is
passed to git through the environment, never on the command line); an
ssh_key credential sets GIT_SSH_COMMAND with the key file. A failed
clone/pull logs an error and does not stop the boot — affected sources fail
at sync time and the periodic re-pull retries.
API keys with per-consumer permissions. The secret is NEVER here — env_vars.key
names the environment variable that holds it (the same shape as a credential's
env_vars; key is its only field), so these files can live in git and
rotation stays an external process (swap the env var, restart). The older
env: <VAR> is still read but deprecated, like a credential's env_prefix.
# api_keys/consumers.yaml
api_keys:
- name: key-awx
description: "AWX"
env_vars:
key: "UNIFIED_API_KEY_AWX"
role: "admin" # everything
- name: key-forms
description: "AnsibleForms"
env_vars:
key: "UNIFIED_API_KEY_FORMS"
# role defaults to "restricted": only what is listed below
sources: ["src-ssh-linux"] # may list/read/sync these sources (or views)
endpoints: ["ep-ansible-full"] # may list/run these endpointsA declared key whose env var is missing or empty fails startup (a typo must
not silently lock a consumer out at request time). Restricted keys referencing
unknown source/endpoint ids also fail startup; a sources: entry may name a
view, which grants the view without granting its members. No
api_keys items and no UNIFIED_API_KEY = open API (with a loud warning). See
API → Authentication for the exact route semantics.
Everything the process reads from the environment, including where each credential looks for its secret.
| Variable | Effect |
|---|---|
CONFIG_DIR |
Config directory (default config) |
UNIFIED_API_KEY |
Legacy single admin key: when set, it authenticates like an admin api_keys item (see API); health probes and Swagger stay public |
per api_keys item |
Each key definition names the env var holding its secret |
RUST_LOG |
Log filter, e.g. debug or unified_api=debug (default info) |
| per credential | Secret values, each variable named by a credential's env_vars |