PostHog SDK Test Harness

Language-agnostic test harness for validating PostHog SDK compliance.

What is This?

This test harness validates that PostHog SDKs correctly implement the PostHog API by:

1. Running a mock PostHog server
2. Exercising your SDK through a simple HTTP adapter
3. Verifying behavior matches the contract defined in CONTRACT.yaml

Quick Start

Using Docker (Recommended)

# Run tests against your SDK adapter
docker run --rm \
  --network host \
  ghcr.io/posthog/sdk-test-harness:latest \
  run --adapter-url http://localhost:8080

Using in CI/CD

Add to your SDK's .github/workflows/:

jobs:
  sdk-compliance:
    uses: PostHog/posthog-sdk-test-harness/.github/workflows/test-sdk-action.yml@v1
    with:
      adapter-dockerfile: "tests/adapter/Dockerfile"
      adapter-context: "."
      sdk-type: "server"        # or "client"
      suite: "capture"          # optional; comma/space/newline separated values are supported
      continue-on-error: true    # set false when compliance should block CI

The action will run tests, generate reports, and comment on PRs with results.

Choosing sdk-type

sdk-type describes the SDK's capture wire format, not where the SDK runs. Do not choose it based on frontend/backend, mobile/server, or browser/native labels alone.

• Use client for SDKs that send client-style capture payloads to /e/, with event data such as token and distinct_id carried in the event/properties payload.
• Use server for SDKs that send server-style capture payloads to /batch, with a body shaped like { "api_key": "...", "batch": [...] }.

For example, a mobile SDK that posts { "api_key": "...", "batch": [...] } to /batch should use sdk-type: "server" for harness filtering.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         Test Harness                            │
│  Reads CONTRACT.yaml and executes tests                         │
└─────────────────────────────────────────────────────────────────┘
              │                              │
              │ HTTP                         │ HTTP
              ▼                              ▼
┌─────────────────────────┐    ┌─────────────────────────────────┐
│   Mock PostHog Server   │    │       SDK Adapter               │
│  Simulates API          │◄───│  Wraps your SDK                 │
│  Records requests       │    │  Exposes REST API               │
└─────────────────────────┘    └─────────────────────────────────┘

Creating an SDK Adapter

Your adapter is a simple HTTP service that wraps your SDK. It needs these endpoints:

Required Endpoints

GET  /health    - Return SDK name/version and capabilities
POST /init      - Initialize SDK with config
POST /capture   - Capture an event
POST /get_feature_flag - Evaluate a feature flag
POST /flush     - Flush pending events
GET  /state     - Return internal state
POST /reset     - Reset SDK state

If your SDK can evaluate flags locally, make /get_feature_flag honor force_remote=true so the harness can verify remote /flags payloads without depending on adapter defaults. distinct_id is always a top-level adapter parameter.

Example (Python)

from flask import Flask, request, jsonify
import posthog

app = Flask(__name__)
client = None

@app.route("/init", methods=["POST"])
def init():
    global client
    data = request.json
    client = posthog.Client(
        api_key=data["api_key"],
        host=data["host"]
    )
    return jsonify({"success": True})

@app.route("/capture", methods=["POST"])
def capture():
    data = request.json
    uuid = client.capture(
        distinct_id=data["distinct_id"],
        event=data["event"]
    )
    return jsonify({"success": True, "uuid": uuid})

# ... implement other endpoints

See ADAPTER_GUIDE.md for complete implementation details and examples/minimal_adapter/ for a full example.

Tests

Tests are defined in CONTRACT.yaml and organized into suites. Which suites run depends on the capabilities your adapter declares in /health:

Suite	Requires	Protocol
capture	capture_v0	POST /batch
capture_v1	capture_v1	POST /i/v1/analytics/events
capture_ai	capture_ai_v0	POST /i/v0/ai/batch/

Some individual tests have additional requirements (e.g., encoding_gzip, encoding_zstd).

Capabilities

Your adapter declares capabilities in its /health response:

{
  "sdk_name": "posthog-python",
  "sdk_version": "3.0.0",
  "adapter_version": "1.0.0",
  "capabilities": ["capture_v0", "capture_v1", "capture_ai_v0", "encoding_gzip"]
}

The harness skips any suite or test whose requires field isn't satisfied by the adapter's capabilities. If capabilities is omitted, only tests with no requires field will run.

See CONTRACT.yaml for the complete test specification.

CLI Usage

# Run all tests
uv run posthog-test-harness run --adapter-url http://localhost:8080

# Run specific suite
uv run posthog-test-harness run --adapter-url http://localhost:8080 --suite capture

# Generate report
uv run posthog-test-harness run --adapter-url http://localhost:8080 --report report.md

# Run tests in parallel (requires adapter support)
uv run posthog-test-harness run --adapter-url http://localhost:8080 --concurrency 4

# JSON output
uv run posthog-test-harness run --adapter-url http://localhost:8080 --output json

# Run mock server standalone
uv run posthog-test-harness mock-server --port 8081

# Check adapter health
uv run posthog-test-harness health --adapter-url http://localhost:8080

Adding New Tests

Tests are defined in CONTRACT.yaml. To add a test:

test_suites:
  capture:
    categories:
      my_new_category:
        tests:
          - name: my_new_test
            steps:
              - action: init
              - action: capture
                params:
                  distinct_id: user1
                  event: test_event
              - action: assert_event_has_field
                params:
                  field: uuid

No Python code needed! See EXTENDING.md for details on adding custom actions.

Local Development

# Clone and install
git clone https://github.com/PostHog/posthog-sdk-test-harness.git
cd posthog-sdk-test-harness
bin/install

# Test the example adapter
bin/test

# Format code
bin/fmt

# Run tests
uv run pytest

Contributing

When making changes to the test harness, add a Sampo changeset describing the release before opening your PR:

sampo add

This prompts for the bump type and a release note, and writes a file under .sampo/changesets/. Commit that file with your PR.

• patch — bug fixes, documentation, internal refactors
• minor — new tests, new actions (backwards compatible)
• major — breaking changes to CONTRACT.yaml or the adapter interface

CI will fail (Changeset hygiene check) if you change releasable code without including a changeset.

The actual version bump, changelog entry, tag, and Docker image publish happen in a single gated workflow after a maintainer approves the release in Slack. See RELEASING.md for the full flow.

Versioning

Docker images are published with semantic versioning:

• latest — most recently approved release
• 1 — latest v1.x.x release (only published once the major is non-zero)
• 1.0 — latest v1.0.x release
• 1.0.0 — specific version

All tags only move when a maintainer approves a release through the gated workflow — there is no automatic publish on every push to main.

Pin to a specific version in your CI for stability:

test-harness-version: "1.0"  # Recommended: pin to major.minor

Documentation

• ADAPTER_GUIDE.md - Complete guide to implementing adapters
• EXTENDING.md - How to add new tests and actions
• CONTRACT.yaml - Main contract (references modular contracts)
• Feature Flag Rules v2 - Versioned config, definitions, response and event schemas, fixtures, and evaluation corpus
• contracts/ - Modular contract definitions:
  • adapter_actions.yaml - Actions that call the adapter
  • test_actions.yaml - Test harness actions (assertions, etc.)
  • capture_tests.yaml - Capture V0 test suite
  • capture_analytics_v1_tests.yaml - Capture V1 test suite
  • capture_ai_tests.yaml - Dedicated AI capture endpoint test suite
• examples/minimal_adapter/ - Working example

License

MIT - see LICENSE

Opt-in local feature flag evaluation

Adapters may explicitly advertise feature_flags_local_evaluation_v1 to enable local-rule compliance tests for both property matching versions 1 and 2. No existing adapter or default health response opts in automatically; remote fixtures and requests remain unchanged. See the optional adapter protocol for privileged definitions loading, bounded readiness/reload and local-only result requirements.