Skip to content

v1.1.0 - Stripe objects and product subscriptions
Compare
Choose a tag to compare
@jwhitlock jwhitlock released this 06 Dec 20:26
· 534 commits to main since this release
v1.1.0
This tag was signed with the committer’s verified signature.
jwhitlock John Whitlock
GPG key ID: 082C735D154FB750
Verified
Learn about vigilant mode.
cde9694
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

This release adds the ability to ingest Stripe objects from the Firefox Accounts (FxA) Firestore cache via a PubSub queue. These are processed to determine the product subscriptions for a contact, and these are synced to a relational table in Acoustic. The product subscriptions are not exposed on the contact in the API.

API

  • A new endpoint, POST /stripe, takes Stripe objects and adds them to the CTMS database. The supported objects are customer, subscription, and invoice. This endpoint takes CTMS OAuth2 credentials.
  • A new endpoint, POST /stripe_from_pubsub, takes PubSub push requests with a Stripe object, or dictionary of keys to Stripe objects, as payload. This endpoint checks the Javascript Web Token (JWT) authentication header, and verifies the claimed audience and email. The endpoint also takes a client "secret" as a URL parameter. This endpoint returns 202 for content issues to prevent PubSub from submitting again.
  • Loading contacts now loads the related Stripe data, and converts them to products. This will increase the number of database requests to read or update a contact.

Acoustic Sync Service

  • A contact's product subscriptions are synced to a new Acoustic relational table. This includes placeholder columns for future subscription data.
  • The Acoustic sync service does not sleep if it processed a full batch of contacts, to speed up processing a backlog of contacts.
  • Added a timeout to Acoustic requests, with a default of 5.0 seconds. If the timeout is reached, syncing fails for that contact and it is retried later.

Deployments

  • The database includes new tables for Stripe data, added by migrations: stripe_customer, stripe_price, stripe_invoice, stripe_invoice_line_item, stripe_subscription, and stripe_subscription_item. The primary key is the stripe_id column. The tables refer to each other - stripe_subscription.stripe_customer_id refers to a stripe_customer.stripe_id - but foreign keys are not used because the data may come in an unexpected order from FxA.
  • The API __heartbeat__ endpoint now includes details of the Acoustic sync backlog. Optional settings sets maximum levels for the backlog and the retry backlog, to make the heartbeat fail. The default is no maximum.
  • The API process now reads the background process settings from environment variables as well. Some are reported in the __heartbeat__ endpoint.
  • The background sync service can optionally write the current time to a file, at startup and once per loop. This can be checked by a new process ctms/bin/healthcheck_sync.py as Kubernetes startup and liveness check.
  • Environment Variables:
    • Added CTMS_PUBSUB_AUDIENCE and CTMS_PUBSUB_EMAIL, to validate the JWT claim for POST /stripe_from_pubsub.
    • Added CTMS_PUBSUB_CLIENT, checked against the query string parameter in POST /stripe_from_pubsub?pubsub_client=<client_id>.
    • Added CTMS_ACOUSTIC_PRODUCT_SUBSCRIPTIONS_ID, required in the background process, for the product relational table ID.
    • Added optional CTMS_ACOUSTIC_MAX_BACKLOG and CTMS_ACOUSTIC_MAX_RETRY_BACKLOG. If set, __healthcheck__ will fail if the backlog or the retry backlog exceeds these limits.
    • Added optional CTMS_BACKGROUND_HEALTHCHECK_PATH and CTMS_BACKGROUND_HEALTHCHECK_AGE_S. If the path is set, the background process will write the current timestamp. If both are set, ctms/bin/healthcheck_sync.py will read the timestamp file and exit with a failing code if it is older than the age in seconds.
    • Added optional CTMS_ACOUSTIC_TIMEOUT_S, to set the timeout for requests to Acoustic. The default is 5.0 seconds.
  • Metrics updates:
    • The new counter ctms_pending_acoustic_sync_total is incremented when an Acoustic sync is scheduled, from an existing endpoint like POST /ctms or PATCH /ctms/<email_id>, as well as the new Stripe ingest endpoints.
    • The ctms_background_acoustic_requests_duration and ctms_background_acoustic_sync_loopsmetrics now include tag table, to identify the table synced (main for the main contact table, newsletter and product for the relational tables).
    • The new counter ctms_background_acoustic_sync_loops increments when a sync loop completes processing a batch of contacts and before sleeping (if requested). This can be used to detect if the sync process is stuck.
    • The new gauge ctms_background_acoustic_sync_age_s gives the age of the sync request for the last synced item that was not re-queued for retrying. This can be used to determine the impact of Acoustic API slowdowns or large backlogs.
  • Log updates:
    • The background process now emits structured logs, and the log lines have been reduced.
    • The background process emits on INFO message at startup, "Setting up sync_service.", with the sync_feature_flag in context.
    • The background process emits one INFO message per loop, "sync_service cycle complete". The log context includes:
      • How many contacts were synced, and and the count by sync status.
      • "trivial": true if no contact were synced.
      • The duration of the loop, and the planned sleep duration.
    • The background process emits one DEBUG message per contact ("Successfully sync'd contact to acoustic..." or "Failure for contact in sync to acoustic..."). The log context includes:
      • The email_id.
      • The email address, if a contact's email matches the +trace_me_mozilla_ pattern.
      • The names of skipped columns, except for known columns, such as update_timestamp, which are silently skipped.
      • If the fxa_created date was successfully parsed into a datetime, or what went wrong.
      • The slugs of any skipped newsletters.
      • The status and duration of Acoustic sync requests.
      • The count of rows for the newsletter and product relational tables.
    • The new Stripe endpoints log the payload if the Stripe object has an email that matches the +trace_me_mozilla_ pattern

Other

  • Added adminer to the development database as postgres-admin, to allow viewing the database.
  • Added new script ctms/bin/ingest_stripe_data.py that can import one or more Stripe objects from a JSON file.
  • Updated to Python 3.9.7. The accepted range is 3.7.x to 3.10.x (raised from 3.9.x).
  • The PostgreSQL client psycopg2 is now built from source rather than installed as a wheel, meaning that libpq5 is shipped in the deployment object, and development libraries are needed when building on a local developer's machine. This allows an arm64 build for Apple Silicon.
  • Updated several dependencies, such as fastapi 0.65.3, alembic 1.7.5, google-cloud-core 2.2.1, psycopg2 2.9.2, and uvicorn 0.15.0.
  • Updated several development tools, such as black 21.10b0, bandit 1.7.1, mypy 0.910, pylint 2.12.1, and black 21.11.b1.
  • Switched pre-commit to the Poetry environment, to avoid out-of-date dependencies.
  • Moved documentation from guides/ to docs/, and refreshed and reworded documentation.
  • Added docs/adrs for Architectural Decision Records, with ADR for Stripe syncing.
  • Moved scripts/lint.sh to docker/lint.sh and scripts/test.sh to docker/test.sh. Removed some unused scripts.
  • Removed auto-documentation stubs and documentation deploy to Github pages.
  • Set CODEOWNERS from a team to the current development staff .
Assets 2
Loading