Skip to content

Latest commit

 

History

History
765 lines (658 loc) · 39.1 KB

configuration.md

File metadata and controls

765 lines (658 loc) · 39.1 KB

Configuration

Everything is YAML in one directory (default ./config, override with the CONFIG_DIR environment variable), laid out the way AWX config as code is: config.yaml at the root holds the server settings, and every other *.yaml / *.yml file, at any depth, holds items under a typed top-level key. Only config.yaml is mandatory; a directory with nothing else is a valid, empty inventory.

All cross-references are validated at startup — an unknown source, connector, credential or project anywhere in the config aborts boot with a list of every error found. The same validation runs standalone with unified-api --check-config (see CLI) — for CI on a config repository.

Configuration is read at startup, and can also be written and reloaded over HTTP where that is enabled — see Configuration API.


The directory

/etc/unified-api/
├── config.yaml                        # required, at the root, under this name
├── sources/                           # folders are for people: any names, any depth, or none
│   ├── acme/
│   │   ├── src-d42-acme.yaml
│   │   └── src-ssh-acme.yaml
│   └── src-netbox.yaml
├── connectors/
│   ├── d42.yaml
│   └── ssh-facts.yaml
├── views/
│   └── vw-facts-all.yaml
├── credentials/
│   └── cred-d42.yaml
├── projects/
│   └── prj-inventory.yaml
└── api_keys/
    └── key-awx.yaml
Top-level key Holds
sources Each inventory instance: its connector, credentials, config, schedule and TTLs
connectors How to talk to a kind of system: the script (or SSH command), its project, its arguments
views Read-only composites over several sources, routed by declared ownership
endpoints Outputs combining cached datasets through a transformer
enrichers Post-processors over data already in the cache
projects Git repositories holding scripts and static inventories
credentials Where each credential's secrets are read from (never the values)
api_keys API keys by name and role (the values come from env vars)

Every item file looks the same: a key, and a list of items under it.

# sources/acme/src-d42-acme.yaml
sources:
  - name: src-d42-acme              # identity: URLs, references, cache keys
    description: "Device42 Acme"    # label, for people
    connector: d42
    credentials: [cred-d42]
    sync_interval_seconds: 300
    ttl_seconds: 3600

The rules, and why they are what they are:

  • The key says what an item is, never the folder. A file in connectors/ holding sources: is read as a source. Folders exist for people — by team, by estate, by datacenter — and may nest as deep as they like, or not exist at all. Tying meaning to a folder would make moving a file a semantic change.
  • One item per file is the convention, <section>/<name>.yaml, and the one this documentation uses. It is not a rule: a file may hold several items, of several types. A review of "add a tenant" is then one new file, not a hunk in the middle of a 2,000-line map.
  • name is the identity, as in AWX: what other items reference, what URLs (/api/v1/sources/{id}) and cache keys use. Letters, digits, -, _ and ., not starting with a dot — it appears in URLs and in file names. The values are the same ids 0.34 used as map keys, so nothing that reads the API changes.
  • description is optional, for people. Every API response field called name answers the description, falling back to the identity when there is none — so responses read exactly as they did before item files.
  • A name defined twice is an error naming both files. Which one won would otherwise depend on the order files happen to be read in.
  • References drop the _id: a source names its connector and credentials, an enricher its target, source and project, an endpoint its sources and project, a project its credentials (a list, as on a source, holding at most one: a clone authenticates once; the singular credential: still loads, deprecated).
  • Hidden files and folders are skipped (editor swap files, .git, the config API's staging folders). Symlinks are followed: a Kubernetes ConfigMap mount is nothing but links into a hidden ..data folder, and skipping them would load an empty configuration.
  • A path may only use letters, digits, -, _ and . in each segment, none starting with a dot, ending in .yaml or .yml — the same rule the config API applies to what it is asked to write, so a file the API cannot name cannot be part of the configuration either.

Parsing is strict. Every way a file can be wrong fails startup naming the file, and every error is reported at once, so a push with three typos in three files names all three:

Mistake Error
An unknown top-level key sources/a.yaml: unknown key 'sourcse' (expected one of: sources, views, ...)
A 0.34 file the same, ending — this is the 0.34 layout (items keyed by id), which this version does not read: write it as items, or convert it with unified-api 0.36's --migrate-config
An unknown key inside an item sources/a.yaml: source 'src-a': unknown field `ttl_second`, expected one of ...
An item without name sources/a.yaml: sources item 2 needs a 'name', its identity
A name defined twice sources/b.yaml: source 'src-a' is already defined in sources/a.yaml

A typo'd key failing loudly is the point: silently applying the default for ttl_second: is how an inventory ends up an hour stale with nobody knowing why. The free-form config: maps are the exception — they are arbitrary by design, and their scalars are taken as text, exactly as written (concurrency: 20 arrives as "20", version: 1.10 as "1.10", account: 0x1F as "0x1F"), since they become environment variables and script settings. Every other text field reads an unquoted scalar the same way: script_args: ["--days", 7] is ["--days", "7"].

A configuration in an older layout is refused, never converted: the 0.34 layout (eight fixed files, each a map keyed by id), a 0.35-form ssh connector (see connectors) and output_format: native each fail the load with a message saying what to write instead. A 0.34 directory can still be converted to item files with unified-api 0.36's --migrate-config, run on it before upgrading past 0.36.


config.yaml (required)

The one file that must exist, holding the server, cache, projects and secrets settings. It is always at the root, under exactly this name, and holds no items.

server:
  host: "0.0.0.0"
  port: 8182
  # Optional. Empty/absent (default) = no CORS headers at all — right for
  # server-to-server consumers. List browser origins to enable, "*" for any.
  cors_allowed_origins: []
  # Optional, default false. false = /readyz is green once ANY source has
  # synced (a pod serving part of the inventory beats one serving nothing).
  # true = every configured source must have synced first — for deployments
  # where a partial inventory is worse than none.
  readyz_require_all_sources: false
  # Optional, default false. false keeps /metrics public alongside the health
  # probes, which is what Prometheus scrape configs expect. true requires an
  # API key — for a shared network, since the exposition labels every source
  # id and host count.
  metrics_require_auth: false
  # Optional, default 15. How long a read with ?refresh=true may wait before it
  # gives up and serves the cached data. Sized for a consumer waiting on a page,
  # not for a scheduled sync. See docs/on-demand-refresh.md.
  refresh_timeout_seconds: 15
  # Optional, default 8. Concurrent on-demand refreshes, process-wide. The TTL
  # bounds repeat requests for the same host; this bounds requests for many
  # different hosts arriving together.
  refresh_max_concurrent: 8
  # Optional, default 20. How long shutdown waits for background tasks (syncs,
  # enrichers, project pulls, the snapshot task) to finish in-flight work
  # before writing the final cache snapshot anyway. Size it to fit inside the
  # pod's terminationGracePeriodSeconds with room for the snapshot write.
  shutdown_grace_seconds: 20
  # Optional, default 2097152 (2 MiB — what was always enforced, implicitly).
  # Largest request body accepted on any route; an oversized one answers 413
  # naming this key. Size it for the biggest configuration push (a whole
  # directory is one body) or host-vars write you expect.
  max_body_bytes: 2097152
  # Optional. This instance's address, as people and other instances reach it
  # (the ingress, not the port above). A central sends it with the federation
  # map, so each edge's Topology page links back to it. Reloadable.
  url: "https://unified-api-central.example.com"

# Optional. Absent = purely in-memory cache (restarts start empty).
# With the block, the cache is snapshotted to `path` every `interval_seconds`
# (default 60) and on graceful shutdown, and reloaded at boot. See docs/caching.md.
cache:
  persistence:
    path: "/var/lib/unified-api/cache.json"
    interval_seconds: 60

# Optional. Where git projects are cloned (one subdirectory per project name).
projects:
  dir: "/var/lib/unified-api/projects"   # default "projects"

# Optional, OFF by default. Lets an admin key read, write and reload this
# directory over HTTP, so a pipeline can push a change instead of publishing
# an artifact the instance has to pull. Turning it on means an admin key can
# rewrite every file the loader reads, the api_keys items included — the same
# authority as editing the mounted directory. See docs/config-api.md.
config_api:
  enabled: false

sources

One item per inventory instance: which system, reached how (its connector), as whom (credentials), with which settings (config), how often, and how long the data stays fresh. The name is the source id used in URLs.

# sources/src-section9.yaml
sources:
  - name: src-section9
    description: "Section 9 Inventory"
    connector: "sample-inventory"        # a connector item, or a built-in (below)
    credentials: ["cred-section9-api"]   # credential items
    sync_mode: "replace"                 # "replace" (default) or "merge"
    sync_interval_seconds: 60            # background sync; 0/absent = manual only
    # schedule: "30 2 * * *"             # OR a cron cadence (UTC) — pick one
    ttl_seconds: 3600                    # dataset-level TTL (required)
    # timeout_seconds: 900               # override the connector's bound, this source only
    host_sync_updates_group_vars: false  # may a host-scoped sync take the vars of
                                         #   the groups it joins? (default false)
    allow_on_demand_refresh: false       # may a READ trigger a sync? (default false)
    config:                              # free-form strings passed to the connector
      scenario: "large"
Field Notes
connector Required. The name of a connector item, or a built-in: remote (another unified-api, see federation) or static_inventory (an Ansible YAML inventory in a project checkout). See connectors
credentials Credential items resolved before each sync and handed to the connector
project, path static_inventory sources only: the project whose checkout holds the inventory, and the file inside it. Refused on every other source, where they would look like they select a script and would not
timeout_seconds Overrides the connector's timeout_seconds for this source only — the bigger tenant of the same system. Absent = the connector's (default 300). A timed-out sync fails with a clear error instead of hanging its scheduler task or HTTP request
host_sync_updates_group_vars Whether a host-scoped sync also takes the vars of the groups it places those hosts in, and only those groups. Off by default: a granular call answers about a host, not about what a group means for its other members
sync_mode How a full sync lands in the cache: replace swaps the dataset, merge patches it
ttl_seconds See caching for the freshness model
ttl_overrides Optional per-host and per-group TTLs on this source: {hosts: {<host>: <seconds>}, groups: {<group>: <seconds>}}. The same can be said from the host's or the group's side, in a host or group item
allow_on_demand_refresh Whether GET /dataset?host=X&refresh=true may gather the named hosts. Off by default: a read that can cause SSH into a datacenter is a capability. Turning it on makes ttl_seconds load bearing — it becomes the threshold at which a read pays for a gather — so review it first. See on-demand refresh
schedule Cron cadence instead of an interval: standard 5-field cron, optional leading seconds field, evaluated in UTC. Validated at startup; mutually exclusive with a non-zero sync_interval_seconds. Cron sources get no startup jitter (their times are deliberate) but back off on failure like interval sources, by letting occurrences pass
hosts_from_source SSH sources only: dynamic host list from another source's cached dataset (source + optional match_pattern.groups/hosts + connect_via); mutually exclusive with config.hosts — see connectors
advertise_scope Explicit ownership claim served on GET /sources/{id}/scope (groups + hosts; must name at least one). Optional — an SSH source already advertises its hosts_from_source pattern; this overrides the derivation. The consumer side of federation reads it so view ownership stops being a duplicated truth
config Arbitrary key: value strings the connector receives (SOURCE_CONFIG for a script). The SSH connector reads hosts, port, concurrency, ssh_connect_timeout_seconds (per-host, default 30) and ssh_known_hosts (host key verification — unset accepts any key, with a warning per sync) from here (0.35's gather_mode and fact_path are refused: the connector decides what runs); a remote source reads url and source — see connectors

Hosts and groups

What a source does differently for one host, or for every host in one of its groups, lives in a host or group item rather than in the source: one file per host (hosts/<host>.yaml) or group (groups/<group>.yaml), named by the host or group as the sources' datasets spell it. Each entry under sources names a source and what that source does for it.

# hosts/motoko.section9.net.yaml
hosts:
  - name: motoko.section9.net
    description: "Major's workstation"   # optional label
    sources:
      src-section9:
        ttl_seconds: 300                 # fresh for 5 min here; the source says 1 h

# groups/production.yaml
groups:
  - name: production
    sources:
      src-section9:
        ttl_seconds: 900                 # every host in the group, joiners included
Field Notes
sources.<id>.ttl_seconds This source's TTL for the host, or for every host in the group of that source's dataset: the refresh gate and what its freshness is reported against. A host's own TTL beats a group's, which beats the source's ttl_seconds
  • Per source, not for the host everywhere. A TTL describes one source's copy of the host: a host in Device42, VMware and the SSH facts may only need its facts kept fresh.
  • Checked against the configuration, not the data. A source id that does not exist is refused, and a view's id is refused naming the member to use instead (a view keeps no copy of its own). A host or group no source has today is not an error: hosts come and go with the systems the sources read.
  • Beside a source's own ttl_overrides. Both work, and both land in the same place. Write it where it reads best: on the source when it is about that source's hosts, in an item when it is about one host across several sources. The same host or group set for the same source in both places is refused, naming both, so no precedence rule has to be known.
  • Folded in at load, as a source's connector is: a reload applies a changed item, and reports it as a change to the sources it names.

Source or connector?

What a source used to say about its script now lives on a connector, so ten Device42 tenants are ten sources and one connector instead of ten copies of the same script path and arguments that drift apart one edit at a time. The split follows one question — does it describe the script, or this instance of the system?

Field Lives on Why
type (script / ssh) connector The protocol
project, path connector Where the script is
script_args, host_args connector How the script is called
timeout_seconds connector, overridable per source A property of the script, except for the tenant ten times bigger
credentials, config source Which instance, as whom
schedule, sync_interval_seconds, ttl_seconds source How often and how fresh, per instance (per host or group: their own items)
sync_mode, host_sync_updates_group_vars source Cache policy of this instance's entry, not a property of the script
hosts_from_source, allow_on_demand_refresh, advertise_scope source Per instance

Built-in connectors

Two connectors need no item: the protocol is fixed, and the only thing to configure — where — is per source.

sources:
  # Another unified-api instance (federation)
  - name: src-edge-dc2
    description: "Datacenter 2"
    connector: remote
    credentials: ["cred-edge-dc2"]     # a token credential = the remote API key
    sync_interval_seconds: 120
    ttl_seconds: 600
    config:
      url: "https://unified-api-dc2.example.com"
      source: "src-ssh"                # the source id ON THE REMOTE instance

  # An Ansible YAML inventory in a project checkout
  - name: src-linux
    description: "Linux static inventory"
    connector: static_inventory
    project: "prj-inventories"         # the checkout
    path: "linux/inventory.yaml"       # the file inside it
    sync_interval_seconds: 300
    ttl_seconds: 600
  • remote requires config.url and config.source, and refuses project / path. See federation.
  • static_inventory requires project and path on the source itself. See connectors.

project + path mean the same thing everywhere they appear — a file inside that project's checkout — whether it is a connector's script or a static source's inventory.


connectors

How to talk to a kind of system. Full treatment, including the script contract, in connectors.

# connectors/d42.yaml
connectors:
  - name: d42
    description: "Device42 inventory script"
    type: "script"                      # script (default) | ssh
    project: "prj-inventory"            # the checkout `path` resolves in
    path: "dynamic/d42/inventory.py"
    script_args: ["--list"]             # CLI args, verbatim (no shell)
    host_args: ["--host-with-groups", "{target}"]  # replace script_args for a host sync
    timeout_seconds: 300                # default 300; a source may override it

# connectors/ssh-facts.yaml
connectors:
  - name: ssh-facts
    description: "SSH fact gathering"
    type: "ssh"
    location: "host"                    # path is on each host (or a script here,
    path: "/etc/ansible/facts.d/*.fact" #   with project / location: instance)
Field Notes
type script (default) runs a local process; ssh connects to the fleet natively
project The project whose checkout holds the script. Required for a script connector with a relative path; on ssh, the project of a script streamed to each host (refused with location: host)
location ssh only: host (path is on each host: files run or read as facts) or instance (a script here, streamed to each host; the default with a project). See connectors
path A file, always. script: the one to run, relative to the project checkout or absolute. ssh: a script here, or a file or pattern of files on each host
script ssh only, instead of path: the script written in the item, streamed to each host as a file from here would be
script_args CLI arguments, quoted. ["--list"] makes standard Ansible dynamic inventory scripts work unmodified. On ssh, passed to the streamed script or to every executable a host path matches
host_args Script only. CLI arguments for a sync that names hosts, replacing script_args for that run, {target} substituted by the requested hostnames (comma-joined). Empty = call the script as a full sync would and filter the answer. See connectors
timeout_seconds Bound on a sync through this connector (default 300); a source's own timeout_seconds overrides it

Settings that look active and are not are refused rather than ignored: host_args or output_format on an ssh connector, output_format other than ansible on a script connector, project with location: host, location or script on a script connector, path and script together, a relative script path without a project, and the names remote and static_inventory, which belong to the built-ins.


views

Read-only composites over several sources: one id that routes a per-host read to whichever member owns that host. Views answer on the source routes, share the source id space (a collision fails startup) and hold no cache entry of their own. Full treatment in views.

# views/vw-facts-all.yaml
views:
  - name: vw-facts-all
    description: "Facts, both datacenters"
    members:                            # ordered: the first claim wins
      - source: "src-ssh-dc1"           # where the facts come from
        owns:
          source: "src-d42"             # inventory the pattern resolves against
          groups: ["datacenter_dc1"]
      - source: "src-edge-dc2"          # a remote (federated) member works too
        owns:
          source: "src-d42"
          groups: ["datacenter_dc2"]
          hosts: ["appliance01.dc2.example"]   # claimed literally as well
    # ttl_seconds: 30                   # optional; absent = inherit the owner's
Field Meaning
members Ordered list. Overlap resolves by declared order — the first member that claims a host wins it
members[].source Source id this member serves data from; must exist and must not be another view
members[].owns.source Source whose cached dataset the groups below resolve against
members[].owns.groups Groups of that source whose members this member owns
members[].owns.hosts Hosts claimed literally, whether or not the inventory knows them — a host built this morning is exactly the one somebody needs to reach
members[].owns.advertised Route by what the member SOURCE claims to own (its advertise_scope locally, the edge's GET /scope for remote members, refreshed per sync with the last-known claim surviving an unreachable edge). groups/hosts become the fallback; with neither claim nor fallback the member claims nothing — see views
ttl_seconds The view's own freshness policy, which is also the gate for refresh=true. Absent = each host inherits its owning member's TTL. A member's host and group items win either way

Ownership is declared, not inferred from what a member has cached: facts sources often sync daily on purpose, and some hosts (appliances that take no SSH) never enter any cache at all — both are exactly the hosts on-demand refresh exists for. See views → ownership.

Strict parsing matters most here. Ownership is the routing table, and grups: instead of groups: would otherwise parse as an empty pattern, which claims everything.

A view is granted to an API key exactly like a source, by its id under sources:; the key needs no access to the members.


credentials

Credentials never contain secrets — they describe where to read them; this is the field reference, and the full resolution story (backends, the cache and rotation, native Vault, failure semantics) is its own page.

# credentials/cred-section9-api.yaml
credentials:
  - name: cred-section9-api
    description: "Section 9 API"
    type: "username_password"    # username_password | token | ssh_key
    env_vars:                    # our key -> the full env var name
      username: "SECTION9_USERNAME"
      password: "SECTION9_PASSWORD"

  - name: cred-ssh-infra
    description: "SSH Infrastructure"
    type: "ssh_key"
    env_vars:
      username: "INFRA_SSH_USERNAME"
    file_keys:                   # our key -> file path; passed as <key>_path
      ssh_key: "/run/secrets/infra-ssh/id_rsa"

  - name: cred-d42-vault
    description: "Device42 (from Vault)"
    type: "username_password"
    vault: "cred-vault"           # the type: vault credential to read from
    vault_path: "infra/device42"  # KV v2 path under that Vault's mount
    secret_keys:                  # our key -> field in the Vault secret
      username: "user"            # (omit secret_keys to take every field)
      password: "pass"

Resolution order per credential: vault_path (native Vault, see below), or env_vars (environment variables, each named in full) / secret_file (a JSON file, with secret_keys naming its fields), plus file_keys entries which are passed through as <key>_path values. A credential that fails to resolve fails the sync with a clear error — it is never silently skipped.

The older env form, env_prefix + secret_keys (SECTION9 + {username: USERNAME} reads SECTION9_USERNAME), is still read but deprecated: startup and --check-config print the env_vars to write instead, and a later release refuses it. Setting both forms on one credential is refused, and so is env_vars beside a vault_path or a secret_file: a credential reads from one place.

Two rules keep the variables a script receives unambiguous (see connectors): a source's credentials must not be delivered under one id (cred-vc-aa1 and cred_vc_aa1 both become CREDENTIAL_VC_AA1_*), and a source on the built-in ssh or remote connector lists one credential — those connectors log in with one, reading its bare username and key, or token.

A Vault is a credential too, of type: vault, as AWX models a HashiCorp Vault lookup: the credentials that read from it name it with vault: and say where with vault_path. Several Vaults can coexist, each with its own login and cache.

# credentials/cred-vault.yaml — the connection to a Vault
credentials:
  - name: cred-vault
    description: Vault
    type: vault
    address: https://vault.example.com:8200
    mount: secret                  # KV v2 mount (default "secret")
    env_vars:
      token: VAULT_TOKEN           # token auth: re-read per resolution,
                                   # so rotating the token needs no restart
    # kubernetes_role: unified-api # OR Kubernetes auth: the service-account
    # jwt_path: /var/run/secrets/kubernetes.io/serviceaccount/token
    #                              # JWT is exchanged for a client token,
    #                              # cached and renewed at 80% of its lease
    timeout_seconds: 10            # bound on every request (default 10)
    cache_ttl_seconds: 60          # how long a value read here is reused
                                   # (default 60; 0 = every resolution)

# credentials/cred-d42-vault.yaml — a login read from it
credentials:
  - name: cred-d42-vault
    type: username_password
    vault: cred-vault
    vault_path: infra/device42
    secret_keys:                   # our key -> field in the Vault secret
      username: user               # (omit secret_keys to take every field)
      password: pass
type: vault field Notes
address Required. http:// or https://
mount The KV v2 mount vault_path values live under (default secret)
env_vars.token / kubernetes_role How to log in: exactly one. jwt_path goes with kubernetes_role
timeout_seconds Bound on every request (default 10)
cache_ttl_seconds How long a value read from this Vault is reused (default 60; 0 = every resolution): what keeps the sync schedule from becoming a request storm, and also how long a rotation takes to show

The adapter speaks KV v2 only and reads <address>/v1/<mount>/data/<vault_path>. Credentials without a vault_path keep resolving from env/files, uncached, so adoption is per credential — move them one at a time. A Vault credential is never handed to a source, project or LDAP bind: naming one there is refused, since its token is not a login.

The secrets: block in config.yaml (cache_ttl_seconds, and a single vault: with token_env) still loads but is deprecated: startup and --check-config say what to write instead, and a later release refuses it. While it is set, its cache_ttl_seconds still caches every credential, as it did (in place of each Vault's own cache, never on top of it), and a vault_path with no vault: reads from its vault:, cached for 60 seconds when cache_ttl_seconds is absent, as 0.36 did by default.


enrichers

Post-processors over data already in the cache. Two modes:

Script-based — runs a script that receives the target dataset on stdin and returns a partial dataset (modified/removed hosts):

# enrichers/enrich-resolve-ssh.yaml
enrichers:
  - name: enrich-resolve-ssh
    description: "Resolve SSH reachability"
    target: "src-section9"           # dataset being enriched
    project: "prj-connectors"        # optional: resolve script_path in this checkout
    script_path: "enrichers/resolve.py"
    script_args: []                  # optional CLI args for the script
    sync_interval_seconds: 300       # scheduled run; 0/absent = manual only
    # schedule: "45 2 * * *"         # OR a cron cadence (UTC) — pick one
    timeout_seconds: 300             # abort a run that takes longer (default 300)
    config: {}

Declarative merge — copies specified hostvars fields from one source into another, matched by hostname. No script needed:

# enrichers/enrich-business-unit.yaml
enrichers:
  - name: enrich-business-unit
    description: "Add business_unit from D42"
    target: "src-ssh-dc1"           # dataset being enriched
    source: "src-d42"               # dataset to copy fields from
    fields: ["business_unit"]       # top-level hostvars keys to copy
    sync_interval_seconds: 3600

An enricher is not a connector — it gathers nothing — so it keeps its own script_path rather than naming a connector.


endpoints

Output endpoints combine one or more cached datasets through a transformer. Each endpoint sets exactly one of output (a builtin, in-process transformer) or script_path (an external script).

# endpoints/ep-ansible-full.yaml — builtin transformer, no process spawn
endpoints:
  - name: ep-ansible-full
    description: "Full Ansible Inventory"
    sources: ["src-section9", "src-infra"]
    output: ansible                   # builtins: ansible, json, csv
    config:
      filter: "datacenter=section9"   # host, group, filter, fields, exclude_fields (see docs/endpoints.md)

# endpoints/ep-ansible-managed.yaml — a constructed inventory: merge everything,
# return only the hosts one source has. The others still enrich them and still
# group them. Optional; a limit must name one of the endpoint's own sources.
endpoints:
  - name: ep-ansible-managed
    description: "Managed Hosts Only"
    sources: ["src-section9", "src-infra"]
    output: ansible
    limit:
      by_hosts_from_inventory: "src-section9"

# endpoints/ep-dns-zone.yaml — script transformer, for a bespoke format
endpoints:
  - name: ep-dns-zone
    description: "DNS zone file"
    sources: ["src-infra"]
    project: "prj-connectors"         # optional: resolve script_path in a checkout
    script_path: "outputs/dns_zone.py"
    script_args: []                   # optional CLI args for the script
    timeout_seconds: 300              # script only: abort a transform that takes longer (default 300)

projects

Git repositories that hold connector/enricher/transformer scripts and static inventories — the field reference; the full lifecycle (boot behavior, script resolution, virtualenvs, health) is its own page. At boot the app clones each project (shallow, one directory per project name under the projects.dir from config.yaml, default ./projects) and re-pulls it every sync_interval_seconds (fetch + hard reset: local drift is discarded). The boot clones run in the background, after the listener is up: they run concurrently and each is bounded by the project's timeout_seconds, so an unreachable git remote delays that project's scripts — never /healthz or a Kubernetes startup probe. The sync schedulers start once the clones have had their bounded chance.

# projects/prj-connectors-infra.yaml
projects:
  - name: prj-connectors-infra
    description: "Infrastructure Connectors"
    git_url: "https://github.com/OpusProjects/connectors-infra.git"
    branch: "main"                  # default "main"
    credentials: ["cred-github-token"] # optional, for private repos (one)
    sync_interval_seconds: 1800     # 0/absent = no periodic re-pull
    # schedule: "15 2 * * *"        # OR a cron cadence (UTC) — pick one
    timeout_seconds: 300            # abort a clone/pull that runs longer (default 300)
    python_venv: false              # build a venv from requirements.txt; see below
    sync_on_boot: true              # default true; see below

Three sync styles compose from those two knobs:

Style Config Behavior
Automatic sync_interval_seconds: N or a cron schedule update at boot + every N seconds / at the cron's UTC times
Boot only interval 0/absent clone/update at boot, then frozen
Manual / pipeline-driven sync_on_boot: false, interval 0/absent an existing checkout is used as-is (no network at startup); updates only via POST /api/v1/projects/{id}/sync

With sync_on_boot: false a missing checkout is still cloned at boot (no scripts, nothing to run), so first bring-up needs no special casing. Keep projects.dir on a persistent volume for the manual style — the checkout IS the durable state, there is nothing else to save at shutdown. A pipeline in the scripts repo can push and then call the sync endpoint (admin key) to roll new scripts without restarting; script files are re-read on every execution, so the update takes effect on the next run.

How script paths resolve: a relative path whose file exists inside the checkout runs from there (<projects.dir>/<project>/<path>); otherwise the configured path is kept as-is (image-baked and mounted scripts keep working, with a warning when the checkout exists but the file doesn't). Resolution happens at every execution, not once at boot, so a script that first appears after startup — a slow clone, a pipeline's first push — is used on the very next run without a restart. The project comes from the connector (its project + path), from a static source (its own project + path), or from an enricher's or endpoint's optional project (for their script_path). SSH connectors are never resolved — their path is a remote command. Scripts must carry the executable bit in git (git update-index --chmod=+x).

With python_venv: true, a requirements.txt in the checkout gets a real virtualenv: built after the clone, refreshed after any pull that changed requirements.txt (unchanged pulls cost two file reads, not a pip run), and kept OUTSIDE the checkout (<projects.dir>/.venvs/<project>) so the hard reset cannot wipe it. When this project's scripts run, the venv's bin/ is prepended to their PATH — a #!/usr/bin/env python3 shebang resolves to the venv's interpreter, pip-installed imports work, and non-Python scripts in the same project are untouched. A failing install (a typo'd package, an unreachable index) fails the project sync, visible in its sync_health and bounded by timeout_seconds — not a per-source import error at the next sync with the cause nowhere.

Private repos: a token credential authenticates over https (the token is passed to git through the environment, never on the command line); an ssh_key credential sets GIT_SSH_COMMAND with the key file. A failed clone/pull logs an error and does not stop the boot — affected sources fail at sync time and the periodic re-pull retries.


api_keys

API keys with per-consumer permissions. The secret is NEVER here — env_vars.key names the environment variable that holds it (the same shape as a credential's env_vars; key is its only field), so these files can live in git and rotation stays an external process (swap the env var, restart). The older env: <VAR> is still read but deprecated, like a credential's env_prefix.

# api_keys/consumers.yaml
api_keys:
  - name: key-awx
    description: "AWX"
    env_vars:
      key: "UNIFIED_API_KEY_AWX"
    role: "admin"                    # everything

  - name: key-forms
    description: "AnsibleForms"
    env_vars:
      key: "UNIFIED_API_KEY_FORMS"
    # role defaults to "restricted": only what is listed below
    sources: ["src-ssh-linux"]       # may list/read/sync these sources (or views)
    endpoints: ["ep-ansible-full"]   # may list/run these endpoints

A declared key whose env var is missing or empty fails startup (a typo must not silently lock a consumer out at request time). Restricted keys referencing unknown source/endpoint ids also fail startup; a sources: entry may name a view, which grants the view without granting its members. No api_keys items and no UNIFIED_API_KEY = open API (with a loud warning). See API → Authentication for the exact route semantics.


Environment variables

Everything the process reads from the environment, including where each credential looks for its secret.

Variable Effect
CONFIG_DIR Config directory (default config)
UNIFIED_API_KEY Legacy single admin key: when set, it authenticates like an admin api_keys item (see API); health probes and Swagger stay public
per api_keys item Each key definition names the env var holding its secret
RUST_LOG Log filter, e.g. debug or unified_api=debug (default info)
per credential Secret values, each variable named by a credential's env_vars