Kafka protobuf backend refactor

.dockerignore

@@ -0,0 +1,85 @@
+# Docker ignore file for cryptofeed container builds
+# Excludes unnecessary files to reduce build context size
+
+# Version control
+.git
+.gitignore
+.gitattributes
+
+# Documentation
+docs/
+*.md
+!README.md
+!INSTALL.md
+!CHANGES.md
+
+# Tests
+tests/
+.pytest_cache/
+.coverage
+.coveragerc
+htmlcov/
+
+# Python artifacts
+__pycache__/
+*.pyc
+*.py[cod]
+*$py.class
+*.so
+*.egg
+*.egg-info/
+dist/
+build/
+.eggs/
+
+# Virtual environments
+.venv/
+venv/
+ENV/
+env/
+
+# IDE and editor files
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+.DS_Store
+
+# CI/CD
+.github/
+.gitlab-ci.yml
+.travis.yml
+
+# Benchmarks and profiling
+.benchmarks/
+*.prof
+
+# Configuration examples (use ConfigMaps instead)
+config.yaml
+config_example.yaml
+.env
+.env.*
+
+# Kiro/spec development files
+.kiro/
+.claude/
+.codex/
+.factory/
+
+# Quality and formatting
+.flake8
+.pyscn.toml
+.pre-commit-config.yaml
+.mypy_cache/
+.ruff_cache/
+
+# Editor logs
+.editor.log
+
+# Python version file
+.python-version
+
+# Build artifacts
+*.whl
+*.tar.gz

.env.example

@@ -0,0 +1,125 @@
+# =============================================================================
+# Cryptofeed Docker Compose Environment Configuration
+# =============================================================================
+# This file documents all environment variables used by docker-compose.yml
+#
+# Setup Instructions:
+#   1. Copy this file to .env: cp .env.example .env
+#   2. Fill in your exchange API keys and credentials
+#   3. Configure proxy settings if needed
+#   4. Run docker-compose up
+#
+# IMPORTANT:
+#   - Never commit .env file to version control (it's in .gitignore)
+#   - Keep API keys and secrets secure
+#   - Use read-only API keys when possible
+# =============================================================================
+
+# -----------------------------------------------------------------------------
+# Build Configuration
+# -----------------------------------------------------------------------------
+# Docker image version and metadata (optional, defaults to 'dev')
+VERSION=dev
+BUILD_TIMESTAMP=
+GIT_COMMIT_SHA=
+GIT_BRANCH=main
+
+# -----------------------------------------------------------------------------
+# Kafka Configuration
+# -----------------------------------------------------------------------------
+# Partition strategy: composite (default), symbol, exchange, round-robin
+KAFKA_PARTITION_STRATEGY=composite
+
+# -----------------------------------------------------------------------------
+# Exchange API Keys - Binance
+# -----------------------------------------------------------------------------
+# Binance API credentials (https://www.binance.com/en/my/settings/api-management)
+# Required permissions: Read-only market data (no trading permissions needed)
+BINANCE_API_KEY=
+BINANCE_API_SECRET=
+
+# -----------------------------------------------------------------------------
+# Exchange API Keys - Coinbase
+# -----------------------------------------------------------------------------
+# Coinbase Pro API credentials (https://pro.coinbase.com/profile/api)
+# Required permissions: View (read-only)
+COINBASE_API_KEY=
+COINBASE_API_SECRET=
+COINBASE_API_PASSPHRASE=
+
+# -----------------------------------------------------------------------------
+# Exchange API Keys - Kraken
+# -----------------------------------------------------------------------------
+# Kraken API credentials (https://www.kraken.com/u/security/api)
+# Required permissions: Query Funds, Query Open Orders & Trades, Query Closed Orders & Trades
+KRAKEN_API_KEY=
+KRAKEN_API_SECRET=
+
+# -----------------------------------------------------------------------------
+# Exchange API Keys - Bybit
+# -----------------------------------------------------------------------------
+# Bybit API credentials (https://www.bybit.com/app/user/api-management)
+# Required permissions: Read-only
+BYBIT_API_KEY=
+BYBIT_API_SECRET=
+
+# -----------------------------------------------------------------------------
+# Exchange API Keys - OKX
+# -----------------------------------------------------------------------------
+# OKX API credentials (https://www.okx.com/account/my-api)
+# Required permissions: Read-only
+OKX_API_KEY=
+OKX_API_SECRET=
+OKX_API_PASSPHRASE=
+
+# -----------------------------------------------------------------------------
+# Exchange API Keys - Additional Exchanges
+# -----------------------------------------------------------------------------
+# Add additional exchange credentials as needed following the pattern:
+# EXCHANGE_NAME_API_KEY=
+# EXCHANGE_NAME_API_SECRET=
+# EXCHANGE_NAME_API_PASSPHRASE=  # (if required by exchange)
+
+# Examples:
+# BITFINEX_API_KEY=
+# BITFINEX_API_SECRET=
+# BITMEX_API_KEY=
+# BITMEX_API_SECRET=
+# DERIBIT_API_KEY=
+# DERIBIT_API_SECRET=
+# GEMINI_API_KEY=
+# GEMINI_API_SECRET=
+# KUCOIN_API_KEY=
+# KUCOIN_API_SECRET=
+# KUCOIN_API_PASSPHRASE=
+
+# -----------------------------------------------------------------------------
+# Proxy Configuration
+# -----------------------------------------------------------------------------
+# HTTP proxy URL (optional, for geo-restricted exchanges)
+# Format: http://username:password@proxy-host:port
+# Example: http://user:pass@proxy.example.com:8080
+PROXY_HTTP=
+
+# SOCKS5 proxy URL (optional, for geo-restricted exchanges)
+# Format: socks5://username:password@proxy-host:port
+# Example: socks5://user:pass@proxy.example.com:1080
+PROXY_SOCKS5=
+
+# Proxy config path inside container (do not change unless you remount)
+PROXY_CONFIG_PATH=/config/proxy.yaml
+
+# -----------------------------------------------------------------------------
+# Logging Configuration
+# -----------------------------------------------------------------------------
+# Log level: DEBUG, INFO, WARNING, ERROR, CRITICAL
+LOG_LEVEL=INFO
+
+# -----------------------------------------------------------------------------
+# Development and Testing
+# -----------------------------------------------------------------------------
+# Set to 'true' to enable additional debugging features
+DEBUG=false
+
+# Enable mock exchange data for testing (bypasses real exchange connections)
+ENABLE_MOCK_DATA=false

.github/workflows/pr-size-check.yml

@@ -0,0 +1,60 @@
+name: pr-size-check
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, ready_for_review]
+
+jobs:
+  enforce-size-limits:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Evaluate PR size
+        uses: actions/github-script@v7
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          script: |
+            const fileLimit = 100;
+            const additionLimit = 5000;
+
+            const pr = context.payload.pull_request;
+            if (!pr) {
+              core.setFailed('No pull_request payload found.');
+              return;
+            }
+
+            const files = await github.paginate(github.rest.pulls.listFiles, {
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              pull_number: pr.number,
+              per_page: 100,
+            });
+
+            const fileCount = files.length;
+            const additions = files.reduce((sum, f) => sum + (f.additions || 0), 0);
+            const docOnly = files.every(f => f.filename.startsWith('docs/'));
+
+            core.info(`Files changed: ${fileCount}`);
+            core.info(`Total additions: ${additions}`);
+            core.info(`Doc-only PR: ${docOnly}`);
+
+            if (docOnly) {
+              core.info('Docs-only PR detected: size limits skipped.');
+              return;
+            }
+
+            const violations = [];
+            if (fileCount > fileLimit) violations.push(`file count ${fileCount} > ${fileLimit}`);
+            if (additions > additionLimit) violations.push(`additions ${additions} > ${additionLimit}`);
+
+            if (violations.length) {
+              core.setFailed(`PR size policy violated: ${violations.join('; ')}`);
+            }
+
+      - name: Summary
+        if: always()
+        run: |
+          echo "PR size check complete" \
+            && echo "Files: ${{ steps.evaluate.outputs.fileCount || 'n/a' }}" || true

.github/workflows/tests.yml

@@ -8,7 +8,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ['3.11', '3.12']
+        python-version: ['3.10', '3.11', '3.12', '3.13']
     env:
       BUF_TOKEN: ${{ secrets.BUF_TOKEN }}
 
@@ -37,18 +37,39 @@ jobs:
         if: env.BUF_TOKEN != ''
         run: |
           buf breaking proto --against 'buf.build/tommyk/crypto-market-data:main'
+      - name: Check for breaking changes (v2beta1)
+        if: env.BUF_TOKEN != '' && env.BUF_V2BETA1_AGAINST != ''
+        env:
+          BUF_V2BETA1_AGAINST: ${{ env.BUF_V2BETA1_AGAINST }}
+        run: |
+          echo "Using against ref: ${BUF_V2BETA1_AGAINST}"
+          buf breaking proto/cryptofeed/normalized/v2beta1 --against "${BUF_V2BETA1_AGAINST}"
+      - name: Skip v2beta1 breaking (no target set)
+        if: env.BUF_TOKEN != '' && env.BUF_V2BETA1_AGAINST == ''
+        run: |
+          echo "Skipping v2beta1 breaking check (set BUF_V2BETA1_AGAINST to enable)"
       - name: Generate Protobuf bindings
         run: |
           buf generate --template buf.gen.yaml proto
           echo "✓ Protobuf bindings generated"
-      - name: Run schema regression tests
-        if: matrix.python-version == '3.12'
+      - name: Run schema regression tests (min version)
+        if: matrix.python-version == '3.10'
         run: |
           python tools/schema_regression.py \
             --events docs/schemas/examples/events/trades.jsonl \
             --output reports/parity-ci.json \
             --tolerance 1e-8 \
             --verbose
+      - name: Run schema regression tests (latest)
+        if: matrix.python-version == '3.13'
+        run: |
+          python tools/schema_regression.py \
+            --events docs/schemas/examples/events/trades.jsonl \
+            --output reports/parity-ci-latest.json \
+            --tolerance 1e-8 \
+            --verbose
       - name: Test with pytest
         run: |
           pytest tests -v --tb=short
+          # Guard: legacy imports + protobuf on generic KafkaCallback
+          pytest tests/unit/kafka/test_import_guards.py -q

.kiro/settings/rules/tasks-parallel-analysis.md

@@ -0,0 +1,34 @@
+# Parallel Task Analysis Rules
+
+## Purpose
+Provide a consistent way to identify implementation tasks that can be safely executed in parallel while generating `tasks.md`.
+
+## When to Consider Tasks Parallel
+Only mark a task as parallel-capable when **all** of the following are true:
+
+1. **No data dependency** on pending tasks.
+2. **No conflicting files or shared mutable resources** are touched.
+3. **No prerequisite review/approval** from another task is required beforehand.
+4. **Environment/setup work** needed by this task is already satisfied or covered within the task itself.
+
+## Marking Convention
+- Append `(P)` immediately after the numeric identifier for each qualifying task.
+  - Example: `- [ ] 2.1 (P) Build background worker for emails`
+- Apply `(P)` to both major tasks and sub-tasks when appropriate.
+- If sequential execution is requested (e.g. via `--sequential` flag), omit `(P)` markers entirely.
+- Keep `(P)` **outside** of checkbox brackets to avoid confusion with completion state.
+
+## Grouping & Ordering Guidelines
+- Group parallel tasks under the same parent whenever the work belongs to the same theme.
+- List obvious prerequisites or caveats in the detail bullets (e.g., "Requires schema migration from 1.2").
+- When two tasks look similar but are not parallel-safe, call out the blocking dependency explicitly.
+- Skip marking container-only major tasks (those without their own actionable detail bullets) with `(P)`—evaluate parallel execution at the sub-task level instead.
+
+## Quality Checklist
+Before marking a task with `(P)`, ensure you have:
+
+- Verified that running this task concurrently will not create merge or deployment conflicts.
+- Captured any shared state expectations in the detail bullets.
+- Confirmed that the implementation can be tested independently.
+
+If any check fails, **do not** mark the task with `(P)` and explain the dependency in the task details.