Geyser | Kafka | gRPC Account Update Source

[thlorenz]

Architecture Specification: Solana Account Streaming via Geyser → Kafka → ksqlDB → Rust gRPC

1. Overview

This document specifies a production-ready architecture for real-time and historical-aware Solana account update streaming. A Geyser plugin ingests account updates directly from the validator into Kafka. ksqlDB materializes latest account state as a queryable table. A custom Rust gRPC service seeds each client subscription with the latest state snapshot, then streams live updates. The design covers schema and partitioning, delivery semantics and ordering, observability, performance, and alternatives considered.

2. Architecture Diagram

flowchart TD
  subgraph Solana
    V[Validator] --> GP[Geyser Plugin accountsdb-plugin-kafka]
  end

  GP -->|keyed messages . pubkey| K[Kafka Topic: account-updates]

  subgraph Streaming_DB
    K -->|STREAM -> TABLE materialization| KSQL[ksqlDB]
    KSQL --> DB[RocksDB State Store]
  end

  subgraph Service
    S[gRPC Service . Rust]
    MC[In-Memory Cache . Moka]
  end

  C[Clients] -->|subscribe . pubkeys| S
  S -->|1 . pull query . latest by pubkey| KSQL
  S -->|2 . cache lookup/store| MC
  K -->|3 . consumer group . filtered stream| S
  S -->|4 . initial snapshot + live updates| C

Loading

Account Update Flow

1. Ingestion — The Geyser plugin captures every account write from the validator's AccountsDB and produces a message to the account-updates Kafka topic, keyed by pubkey.
2. Materialization — ksqlDB consumes the stream and maintains a materialized TABLE keyed by account pubkey, always reflecting the latest state.
3. Client Subscription —
   • Client sends a SubscribeAccounts request with one or more pubkeys.
   • The gRPC service checks its local Moka cache for each pubkey.
   • On cache miss, it executes a ksqlDB pull query to fetch the latest state and populates the cache.
   • The initial snapshot is sent to the client, followed by a continuous stream of live updates filtered from Kafka.

3. Components

Geyser Plugin (solana-accountsdb-plugin-kafka)

• Streams account updates from the validator's AccountsDB directly into Kafka.
• Emits one message per account write, including metadata (slot, write_version, etc.).
• Runs in-process with the validator; Kafka buffering isolates validation from downstream processing.
• High-performance, low-latency ingestion with message keying by account address for ordering.
• Docs: solana-accountsdb-plugin-kafka

Kafka (Broker + Schema Registry)

• Durable, scalable event log; partitions provide parallelism and per-key ordering.
• Schema Registry manages Protobuf/Avro schemas and evolution.
• Compaction and retention control history vs. latest-state trade-offs (see §5).
• Docs: Apache Kafka, Confluent Schema Registry

ksqlDB

• Defines a STREAM over the Kafka topic and materializes a TABLE keyed by account pubkey (LATEST_BY_OFFSET semantics).
• Pull queries provide low-latency point lookups for initial state snapshots.
• Push queries available for internal change-stream needs if required.
• Docs: ksqlDB, Pull Queries

Rust gRPC Service

• Built with tonic (gRPC) and rust-rdkafka (Kafka consumer).
• API: SubscribeAccounts(stream SubscribeRequest) returns (stream AccountUpdate) — client sends pubkeys to add/remove interest; server responds with initial snapshots followed by live updates.
• Internal modules:
  • api — Protobuf definitions and tonic server.
  • kafka — Consumer setup, partition assignment, message deserialization.
  • router — Subscription registry, per-pubkey fan-out with backpressure.
  • snapshot — ksqlDB pull-query client, response parsing, retries, cache integration.
  • cache — TTL-based last-state store keyed by pubkey (Moka, 2–5 min configurable).
• Crates: tonic, tokio, rust-rdkafka, prost, moka, reqwest (ksqlDB REST), opentelemetry.
• Docs: tonic, rust-rdkafka

4. Data Schema and Serialization

• Format: Protobuf (recommended) or Avro; register schemas in Schema Registry for evolution and compatibility.

• Account Update Schema:

  message AccountUpdate {
    string pubkey = 1;
    uint64 slot = 2;
    uint64 write_version = 3;
    bytes  data = 4;
    uint64 lamports = 5;
    string owner = 6;
    bool   executable = 7;
    uint64 rent_epoch = 8;
    // optional extensions
    string tx_signature = 9;
    bool   is_deletion = 10;   // tombstone for account closes
  }

• Key: pubkey (32-byte binary preferred for efficiency; include base58 in value for readability).

• Headers (lightweight routing/ops): cluster, validator_id, commitment, compression, content-type, schema-id.

• Compression: Enable lz4 or zstd at the producer level.

• Mapping: Use the same Protobuf definition across Kafka payloads, ksqlDB table columns (via Schema Registry), and gRPC response streams to minimize serialization overhead.

5. Kafka Topic Configuration and Retention

• Topic: account-updates
• Partitions: 64–128 (tune based on peak throughput and consumer parallelism; validate with load tests).
• Partitioning Strategy: Keyed by pubkey — guarantees all updates for a single account go to the same partition, preserving order.
• Compaction: cleanup.policy=compact to keep at least the last update per key, preventing unbounded growth while acting as a recovery source.
  • Tune min.cleanable.dirty.ratio and segment.ms appropriately.
• Retention: Set retention.ms to a reasonable window (e.g., 7–30 days) in addition to compaction.
• Optional split-topic strategy:
  • account-updates — append-only, time-retained, for analytics/historical replay.
  • account-latest — compacted, latest-by-key, fed by stream/table materialization.

6. ksqlDB Materialization

• Source stream: Create STREAM over account-updates with KEY=pubkey and the chosen value format.
• Latest-state table: Create TABLE keyed by pubkey (implicit LATEST_BY_OFFSET semantics).
  • Pull query example: SELECT * FROM account_state_table WHERE pubkey = '...';
• Configuration:
  • Enable exactly-once processing where available: processing.guarantee=exactly_once_v2.
  • Tune ksql.streams.num.stream.threads, cache sizes, RocksDB settings for lookup performance.
  • Ensure key format compatibility between topic and ksqlDB (KAFKA/JSON/PROTOBUF).

7. Delivery Semantics and Ordering

• Producer: Idempotent (enable.idempotence=true), acks=all, appropriate retries with backoff — exactly-once within a partition from the producer's perspective.
• Kafka ordering: Key-based partitioning preserves per-pubkey order. Cross-key ordering is neither guaranteed nor required.
• Consumer offsets: Committed after messages are enqueued to client fan-out, yielding at-least-once delivery. Duplicates are tolerated via idempotency key (pubkey, slot, write_version).
• ksqlDB: Exactly-once v2 where available to avoid duplicates in materialized tables during failures.
• gRPC flow control: Respect backpressure via tonic's streaming send readiness. Apply bounded per-client queues; on sustained pressure, choose policy: slow/linger, drop latest/oldest, or disconnect.

8. Performance Considerations and Bottlenecks

Area	Concern	Mitigation
Validator impact	Geyser plugin runs in-process; slow brokers or serialization could stall the validator	Efficient serialization, lz4/zstd compression, tuned linger.ms/batch.size
ksqlDB pull query latency	High volume of initial subscriptions stresses ksqlDB	gRPC service Moka cache; target P50 < 10 ms intra-cluster
Kafka consumer throughput	Deserializing all updates at the gRPC service	Efficient Rust Protobuf libraries (prost); pre-filtering where possible
Hot-account fan-out	Popular accounts concentrate load on a single partition/instance	Per-client bounded buffers, shared message references (avoid per-subscriber copies)
Message size	Large program-data accounts inflate payloads	Producer-level compression; external blob storage for extreme cases
End-to-end lag	Cumulative latency across pipeline	Track validator→Kafka, broker E2E, consumer lag, gRPC send queue; set SLO-based alerts

Performance Goals:

• Sub-millisecond ingestion latency from validator to Kafka.
• < 10 ms latency for initial state lookups via ksqlDB.
• Scalability to millions of unique accounts and thousands of concurrent gRPC clients.

9. Scale-out and Failure Handling

• Horizontal scaling: gRPC service instances are stateless; join a shared Kafka consumer group to split partitions. Key-partitioning ensures per-key ordering is preserved end-to-end.
• Kafka broker partitions unavailable: Producer/consumer retries; service may momentarily stall; clients retain connection and receive updates when lag catches up.
• ksqlDB unavailable: Service serves only live updates; initial snapshot falls back to cache (may be stale) or returns a transient error prompting client retry.
• Service instance crash: Clients reconnect to another instance; consumer group rebalances; router rebuilds live interest from active connections; cache warms on demand.

10. Observability and Monitoring

• Metrics:
  • Producer: records/s, batch size, compression ratio, retries, error rates.
  • Kafka: broker throughput, ISR health, partition under-replication, end-to-end lag.
  • ksqlDB: pull query latency, RocksDB hit rate, processing threads.
  • Service: subscribe rate, active streams, per-client queue depth, send latency, dropped messages, cache hit rate, ksqlDB query rate.
• Tracing: Propagate trace context from producer; include pubkey as attribute. Trace spans across gRPC calls and Kafka consumer processing (OpenTelemetry).
• Logging: Structured logs with pubkey, slot, partition, offset, consumer group, client id. Sample at high volume.
• Dashboards/Alerts: SLOs for initial snapshot latency and live update lag. Alerts on consumer lag, ksqlDB P95 pull latency, error spikes.

11. Capacity Planning

• Estimate peak updates/s and average payload size to derive producer throughput and broker disk/network needs.
• Target per-partition throughput < 10–20 MB/s; consumer CPU within budget.
• Size ksqlDB memory and RocksDB to keep hot working set in cache; provision NVMe for state store.
• Partition count: start at 64–128; validate with load tests.

12. Alternatives Considered

yellowstone-grpc-kafka (gRPC → Kafka bridge)

Not preferred. Adds an extra hop and an additional service to manage vs. writing directly from the Geyser plugin.

Feature	Geyser Plugin (Preferred)	yellowstone-grpc-kafka
Integration	Native (in-process)	External (gRPC)
Simplicity	High (1 component)	Medium (2 components)
Reliability	Very high	High (dependent on gRPC availability)
Backpressure	Kafka native producer handles retries/buffering	Split across gRPC and Kafka
Operational overhead	Standard plugin deployment	Requires securing additional gRPC service on validator nodes

Seeding initial state without ksqlDB

Fewer components but requires replay/scan or bespoke compacted-topic management; ksqlDB provides mature materialization and low-latency pull queries out of the box.

Kafka Streams inside the service vs. external ksqlDB

Tighter control and possibly lower latency, but increases service complexity. ksqlDB centralizes materialization and enables reuse across multiple services.

Alternative brokers (e.g., Redpanda)

Could reduce ops complexity while keeping Kafka API compatibility; still compatible with ksqlDB only when running separately. Keep standard Kafka unless a strong operational reason emerges.

13. Open Questions / Next Steps

• Confirm exact schema fields and encodings expected from the chosen Geyser plugin distribution.
• Decide whether to maintain a dedicated compacted account-latest topic alongside the source stream.
• Define SLOs for initial snapshot latency and live-stream freshness; feed into partition count and sizing.
• Prototype a minimal pipeline, load-test with synthetic account updates, and validate ksqlDB pull query latencies under load.

14. Config Knobs (Reference)

Category	Parameters
Kafka	topic name(s), bootstrap servers, group id, TLS, linger.ms, batch.size, acks, compression.type, max.poll.records, fetch.max.bytes
ksqlDB	URL, timeouts, processing.guarantee, num.stream.threads, RocksDB cache size
gRPC Service	listen address, TLS, backpressure limits (per-client queue size), cache TTL/capacity

[thlorenz]

Architecture Specification: On‑Demand Solana Account Streaming via Geyser → Kafka → ksqlDB → gRPC

The revised version of the above after our discussion in our sync

1. Overview

This document specifies a production‑ready architecture for real‑time and historical‑aware Solana account update streaming using an on‑demand tracking model. Instead of streaming all validator account updates into Kafka, the Geyser plugin only tracks and ingests accounts that clients explicitly subscribe to. The gRPC service orchestrates this tracking and relies solely on ksqlDB as the authoritative state store for initial snapshots — no gRPC‑level in‑memory cache.

Key design decisions:

• On‑demand tracking: Geyser starts with zero accounts tracked and only streams updates for requested accounts, avoiding global data overload.
• ksqlDB as sole cache: The gRPC service has no local Moka/in‑memory cache. ksqlDB materializes the latest state of every ever‑tracked account and serves initial snapshots via pull queries.
• Unsubscribe does not stop tracking: Once an account is tracked it remains tracked indefinitely (Geyser → Kafka → ksqlDB), avoiding distributed reference counting. The gRPC service simply stops relaying updates to the unsubscribed client.
• Initial state sourcing: For accounts already in ksqlDB, the last seen state is returned immediately. For unknown accounts, Geyser seeds Kafka via getMultipleAccounts on the same validator node.

2. Architecture Diagram

sequenceDiagram
    participant C as Client
    participant S as gRPC Service (Rust)
    participant KSQL as ksqlDB
    participant K as Kafka (account‑updates)
    participant GP as Geyser Plugin
    participant V as Validator (AccountsDB + RPC)

    Note over GP: Initially tracks no accounts

    C->>S: Subscribe(pubkeys)
    S->>S: Add pubkeys to client filter

    rect rgba(60, 100, 160, 0.3)
    Note right of S: 2.a — Initial state (known accounts)
    S->>KSQL: Pull query (pubkeys)
    KSQL-->>S: Last seen state (subset found)
    S->>C: Stream initial updates (from ksqlDB)
    end

    rect rgba(60, 140, 80, 0.3)
    Note right of S: 2.b — Request tracking (unknown accounts)
    S->>GP: Control: start tracking (missing pubkeys)
    GP->>GP: Add pubkeys to track list
    GP->>V: RPC getMultipleAccounts(missing pubkeys)
    V-->>GP: Current account state
    GP->>K: Produce initial state snapshots
    K->>KSQL: STREAM → TABLE materialization
    K->>S: Consumer receives initial snapshots
    S->>C: Stream initial updates (from Kafka)
    end

    rect rgba(180, 130, 50, 0.3)
    Note right of K: 3 — Continuous live updates
    V->>GP: AccountsDB write (tracked pubkey)
    GP->>K: Produce update
    K->>KSQL: TABLE update (latest by key)
    K->>S: Consumer receives update
    S->>C: Relay update (matches client filter)
    end

    rect rgba(160, 60, 60, 0.3)
    Note right of C: 4 — Unsubscribe
    C->>S: Unsubscribe(pubkeys)
    S->>S: Remove pubkeys from client filter
    Note over GP: Continues tracking — no stop signal
    Note over KSQL: Keeps latest state for future reuse
    end

Loading

3. End‑to‑End Flow

1. Silent start — The Geyser plugin starts tracking zero accounts and produces no Kafka messages.
2. Client subscribes — A client sends a SubscribeAccounts request with one or more pubkeys. The gRPC service adds these pubkeys to the client's filter so future Kafka updates are relayed.
   • 2.a. ksqlDB lookup — The service executes a pull query against ksqlDB for each pubkey. Accounts found are streamed to the client immediately as initial updates.
   • 2.b. Geyser tracking request — For pubkeys not found in ksqlDB, the service sends a control request to the Geyser plugin to start tracking them. The Geyser plugin:
     1. Adds the pubkeys to its internal track list.
     2. Calls getMultipleAccounts against the validator RPC on the same node to fetch the current state.
     3. Produces those states as initial update messages into the account-updates Kafka topic.
     4. Begins publishing ongoing AccountsDB writes for these pubkeys to Kafka.
   • The gRPC service consumes these initial snapshots from Kafka and streams them to the client, followed by all subsequent live updates.
3. Live streaming — The client receives a continuous stream of updates: initial state (from ksqlDB or Geyser seed) followed by real‑time AccountsDB writes, filtered to the client's subscribed pubkeys.
4. Unsubscribe — The gRPC service removes the pubkey from the client's filter and stops relaying updates. The Geyser plugin continues tracking the account to avoid cross‑service coordination complexity. Kafka and ksqlDB continue receiving and materializing updates for future reuse.

Result: All accounts ever subscribed to remain tracked end‑to‑end (Geyser → Kafka → ksqlDB), but only updates for currently‑subscribed pubkeys are relayed to clients. This is vastly smaller than the entire Solana account space.

4. Components and Responsibilities

Geyser Plugin (accountsdb‑plugin‑kafka)

• Runs in‑process with the validator; Kafka buffering isolates validation from downstream processing.
• Emits account updates only for tracked accounts into Kafka, keyed by pubkey for per‑key ordering.
• Exposes a control endpoint for "start tracking" requests from the gRPC service (transport TBD — dedicated gRPC port or control Kafka topic).
• Seeds Kafka with initial state for newly tracked accounts by calling getMultipleAccounts against the same‑node validator RPC to minimize time‑to‑first‑snapshot.
• Must persist its track list across validator restarts (see §11).
• Docs: Solana Geyser Plugins, Plugin Reference, getMultipleAccounts

Kafka (Broker + Schema Registry)

• Durable, scalable event log. Messages keyed by pubkey preserve per‑account ordering.
• Single topic account-updates (compacted + retained) carries initial and live updates for tracked accounts only.
• Schema Registry manages Protobuf/Avro schemas and evolution.
• Volume is limited to tracked accounts — dramatically lower than global ingestion.
• Docs: Apache Kafka, Confluent Schema Registry

ksqlDB

• Consumes account-updates and materializes a TABLE keyed by pubkey (LATEST_BY_OFFSET semantics) — always reflecting the latest state of every ever‑tracked account.
• Serves pull queries for low‑latency initial snapshot lookups. This replaces the gRPC service's former in‑memory cache.
• Docs: ksqlDB, Pull Queries

gRPC Service (Rust)

• Built with tonic (gRPC) and rust-rdkafka (Kafka consumer).
• API: SubscribeAccounts(stream SubscribeRequest) returns (stream AccountUpdate) — bidi stream; clients add/remove pubkeys from their subscription.
• Stateless (no shared in‑memory cache): relies on ksqlDB for initial state and Kafka for live updates.
• May keep per‑subscription last‑sent cursor (slot, write_version) for de‑duplication and idempotency.
• Internal modules:
  • api — Protobuf definitions and tonic server.
  • kafka — Consumer setup, partition assignment, message deserialization.
  • router — Subscription registry, per‑pubkey fan‑out with backpressure.
  • snapshot — ksqlDB pull‑query client, response parsing, retries.
  • control — Geyser tracking request client (retries, idempotency).
• Crates: tonic, tokio, rust-rdkafka, prost, reqwest (ksqlDB REST), opentelemetry.
• Docs: tonic, rust-rdkafka

5. Data Model and Serialization

Account Update Schema (Protobuf)

message AccountUpdate {
  string pubkey       = 1;
  uint64 slot         = 2;
  uint64 write_version = 3;
  bytes  data         = 4;
  uint64 lamports     = 5;
  string owner        = 6;
  bool   executable   = 7;
  uint64 rent_epoch   = 8;
  string tx_signature = 9;   // optional
  bool   is_deletion  = 10;  // tombstone for account closes
}

• Key: pubkey (32‑byte binary preferred for efficiency; include base58 in value for readability).
• Headers: cluster, validator_id, commitment, compression, content-type, schema-id.
• Compression: lz4 or zstd at the producer level.
• Mapping: Use the same Protobuf definition across Kafka payloads, ksqlDB table columns (via Schema Registry), and gRPC response streams to minimize serialization overhead.

Kafka Topic Configuration

• Topic: account-updates
• Partitions: 64–128 (tune via load tests for expected tracked‑account set and hot‑key skew).
• Partitioning: Keyed by pubkey — guarantees per‑account ordering within a partition.
• Compaction: cleanup.policy=compact — keeps at least the last update per key; prevents unbounded growth while acting as a recovery source. Tune min.cleanable.dirty.ratio and segment.ms.
• Retention: retention.ms set to a reasonable window (e.g. 7–30 days) in addition to compaction.

ksqlDB Materialization

• Source: CREATE STREAM account_updates ... over account-updates with KEY = pubkey.
• Table: CREATE TABLE account_state AS SELECT ... FROM account_updates GROUP BY pubkey; (implicit LATEST_BY_OFFSET).
• Pull query: SELECT * FROM account_state WHERE pubkey = '...';
• Enable processing.guarantee=exactly_once_v2 where available.
• Tune ksql.streams.num.stream.threads, cache sizes, and RocksDB settings for lookup performance.

6. Delivery Semantics, Ordering, and Idempotency

• Producer (Geyser): Idempotent (enable.idempotence=true), acks=all, keyed by pubkey for per‑account order.
• Kafka ordering: Key‑based partitioning preserves per‑pubkey order. Cross‑key ordering is neither guaranteed nor required.
• Consumer (gRPC): At‑least‑once with commit after enqueue to per‑client fan‑out. De‑duplicate via idempotency key (pubkey, slot, write_version).
• ksqlDB: Exactly‑once v2 where available to avoid duplicates in materialized tables during failures.
• Race safety: Initial states from ksqlDB may be followed by getMultipleAccounts snapshots arriving via Kafka. De‑duplicate using write_version; higher slot/write_version wins.
• gRPC flow control: Respect backpressure via tonic's streaming send readiness. Apply bounded per‑client queues; on sustained pressure, choose policy: slow/linger, drop latest/oldest, or disconnect.

7. Performance Considerations and Bottlenecks

Area	Concern	Mitigation
Validator impact	Geyser runs in‑process; slow brokers or serialization could stall the validator	Efficient serialization, lz4/zstd compression, tuned linger.ms/batch.size; reduced impact since only tracked accounts are serialized
Cold‑start latency	First‑ever subscription for an account includes control call + getMultipleAccounts + Kafka round‑trip	Batch getMultipleAccounts calls; provide immediate feedback to client that tracking has started; accept higher latency for first request
ksqlDB query load	Burst subscriptions stress pull queries	Connection pooling in gRPC service; tune RocksDB block cache; small read timeouts + retries
Kafka consumer throughput	Deserializing all tracked‑account updates	Efficient Rust Protobuf (prost); volume limited to tracked accounts, not global state
Hot‑account fan‑out	Popular accounts concentrate load on a single partition/instance	Per‑client bounded buffers, shared message references (avoid per‑subscriber copies)
Cumulative tracking growth	Tracked set grows monotonically (never stop tracking)	Still orders of magnitude smaller than full Solana state; use Kafka compaction + ksqlDB TTLs if necessary; consider archival exports for very old/unused keys
Control plane reliability	If tracking request fails, account is never tracked	Idempotent retries in the control channel; persist control ops logs for auditing
RPC rate limits	getMultipleAccounts calls from Geyser could hit limits	Use same‑node RPC (internal validator access); batch calls; concurrency limits
End‑to‑end lag	Cumulative latency across pipeline stages	Track validator→Kafka, broker E2E, consumer lag, gRPC send queue; set SLO‑based alerts

Performance Goals:

• Sub‑millisecond ingestion latency from validator to Kafka (for tracked accounts).
• < 10 ms latency for initial state lookups via ksqlDB pull queries.
• Scalability to hundreds of thousands of tracked accounts and thousands of concurrent gRPC clients.

8. Scale‑out and Failure Handling

• Horizontal scaling: gRPC service instances are stateless; join a shared Kafka consumer group to split partitions. Key‑partitioning ensures per‑key ordering end‑to‑end.
• Kafka broker unavailable: Producer/consumer retries; service may momentarily stall; clients retain connection and receive updates when lag catches up.
• ksqlDB unavailable: Service cannot serve initial snapshots; returns a transient error prompting client retry. Live streaming continues.
• Geyser control endpoint unavailable: Tracking requests are retried with backoff; idempotent design ensures no duplicate tracking on recovery.
• Service instance crash: Clients reconnect to another instance; consumer group rebalances; router rebuilds live interest from active connections.
• Validator restart: Geyser plugin must restore its track list from persistent storage and resume tracking (see §11).

9. Security and Access

• Network policies: Restrict Geyser control endpoint to gRPC service; TLS mutual auth where possible.
• Kafka ACLs: Produce from Geyser only; consume from gRPC service only.
• ksqlDB auth: Scoped to pull queries from gRPC service.
• gRPC: TLS for client connections; authentication and authorization as required.

10. Observability and Monitoring

• Metrics:
  • Producer (Geyser): records/s, batch size, compression ratio, retries, error rates, tracked account count.
  • Kafka: broker throughput, ISR health, partition under‑replication, end‑to‑end lag.
  • ksqlDB: pull query latency (p50/p99), RocksDB hit rate, processing threads.
  • gRPC service: subscribe/unsubscribe rate, active streams, per‑client queue depth, send latency, dropped messages, ksqlDB query rate, control request rate/success.
• Tracing: Propagate trace context from gRPC request → control call → Geyser produce → Kafka consume → client send. Include pubkey as attribute (OpenTelemetry).
• Logging: Structured logs with pubkey, slot, partition, offset, consumer group, client id. Sample at high volume.
• Alerting: SLOs for initial snapshot latency and live update freshness. Alerts on consumer lag, ksqlDB p95 pull latency, staleness (no update for hot pubkey), control errors, error spikes.

11. Open Questions / Next Steps

• Control plane transport: Define the Geyser control API surface — dedicated gRPC port on Geyser, or a "control" Kafka topic? Include auth, idempotency, and batching semantics.
• Track list persistence: How does the Geyser plugin persist its track list across validator restarts? Options: local file, Kafka compacted topic, or external store.
• Schema finalization: Confirm exact Protobuf/Avro fields, encodings, and Schema Registry compatibility rules for the chosen Geyser plugin distribution.
• ksqlDB cleanup: Define TTL or archival policy for accounts not updated in a long period.
• SLO definition: Define SLOs for initial snapshot latency, steady‑state update freshness, and max accepted consumer lag; feed into partition count and infrastructure sizing.
• Load testing: Prototype a minimal pipeline, load‑test with synthetic account updates, and validate ksqlDB pull query latencies under burst subscribe patterns.
• Compacted topic: Decide whether to maintain a dedicated compacted account-latest topic alongside the source stream, or rely on compaction on the single account-updates topic.

12. Alternatives Considered

Alternative	Why Rejected
Track all accounts (original approach)	Volume of updates for the entire Solana account space overwhelms Kafka/ksqlDB without massive horizontal scaling; 99%+ write amplification for updates never consumed by any client.
In‑memory cache in gRPC service (Moka)	Shared state between gRPC instances is hard to keep consistent; adds memory pressure and cache coherence complexity. ksqlDB centralizes "latest state" logic and simplifies the Rust service to near‑stateless.
Stop tracking on unsubscribe	Requires distributed reference counting across gRPC service instances to know if any client still needs the account. Adds coordination complexity and risks thrashing when subscription churn is high.
yellowstone‑grpc‑kafka (gRPC → Kafka bridge)	Adds an extra hop and an additional service vs. writing directly from the Geyser plugin in‑process.
Direct JSON‑RPC to clients (no Kafka/ksqlDB)	Hard to scale fan‑out and replay; lacks durable buffering; rejected for reliability.
Redis/DB as cache instead of ksqlDB	Adds new infrastructure and custom compaction/materialization logic; ksqlDB is purpose‑built for latest‑by‑key materialization over Kafka streams.
Kafka Streams inside the service vs. external ksqlDB	Tighter control but increases service complexity. ksqlDB centralizes materialization and enables reuse across multiple services.

13. Config Reference

Category	Parameters
Kafka	topic name, bootstrap servers, group id, TLS, acks=all, enable.idempotence=true, compression.type=zstd, linger.ms, batch.size, max.poll.records, fetch.max.bytes, partitions 64–128 (load‑test), cleanup.policy=compact, retention.ms
ksqlDB	URL, timeouts, processing.guarantee=exactly_once_v2, num.stream.threads, RocksDB block cache size, pull query timeouts
gRPC Service	listen address, TLS, per‑client queue limits, consumer max.poll.records, fetch.max.bytes, ksqlDB connection pool size, control request retry policy
Geyser Plugin	Kafka bootstrap servers, topic name, serialization format, compression, control endpoint address/port, track list persistence path

[bmuddha]

A bit overly verbose, with almost as many questions as design decision. But good starting point.

Here's list of concerns/opinions:

Geyser seeds Kafka via getMultipleAccounts on the same validator node.

might introduce a few problems:

1. The validator will already be strained by plugin, hitting it with gMA might be problematic, if the frequency and account count is high.
2. The synchronization between the kafka stream updates and fetch will be a classical what overwrites what problem.

A client sends a SubscribeAccounts
We need a different approach to triton, where the request doesn't replace the filter, but adds to the set of tracked pubkeys, so the client doesn't need to keep the entire accounts and send it every time.

The gRPC service consumes these initial snapshots from Kafka and streams them to the client, followed by all subsequent live updates.

Needs clarification, how exactly that will be done?

The Geyser plugin continues tracking the account to avoid cross‑service coordination complexity

This will require periodic reboots of the validator to reset the accumulated accounts list, since people absolutely will use us as proxy (even with proxies), but probably we can keep track of few million accounts, but that needs some research to confirm that it won't slow down the plugin/validator too much.

Messages keyed by pubkey preserve per‑account ordering

If kafka compaction is used (and it most likely will be), then some sort of overwrite priority needs to be defined to keep only the latest state.

Serves pull queries for low‑latency initial snapshot lookups. This replaces the gRPC service's former in‑memory cache.

IF direct queries are used, we need a way to make sure we get the data atomically, i.e. not some accounts being read at one slot and others at another.

message AccountUpdate {
  string pubkey       = 1; better to use custom struct of 4 u64s, and type cast for efficiency, otherwice it's quite cumbersome to encode and decode (which can never fail) to and fro.
  uint64 slot         = 2;
  uint64 write_version = 3;
  bytes  data         = 4;
  uint64 lamports     = 5;
  string owner        = 6;
  bool   executable   = 7;
  uint64 rent_epoch   = 8;
  string tx_signature = 9;  not sure why we need that
  bool   is_deletion  = 10;  slot == 0, is an indicator
}

Use the same Protobuf definition across Kafka payloads, ksqlDB table columns (via Schema Registry), and gRPC response streams to minimize serialization overhead.

deserialization will be required on all stages for filtering and querying purposes. protobuf is a very cumbersome format to work with. For filtering purposes I would consider adding keys to the header data, so the routing can be done without parsing

Partitions: 64–128 (tune via load tests for expected tracked‑account set and hot‑key skew)

sounds like AI hallucination, there's no need for such number of partitions or partitioning in general in this case, even with no account filtering 4 partitions would more than suffice.

ksqlDB Materialization

have you researched whether ksqlDB can automatically parse the incoming protobuf stream?