Jaeger Ingestor Scaling

[RichardWarburton]

I'm looking into deploying Jaeger and I want understand what my scalability options are. The deployment configuration we're looking at is using kafka as a buffer between a collector and an ingestor which then writes into an ES database. The documentation talks a lot about scaling collectors as stateless services but doesn't explain how to scale ingestor instances.

Normally with kafka you partition topics up and use that to shard out data but I don't see any way to configure ingestors to pick a particular kafka partition, not any way to tell different ingestors how to shard out sub-sets of the data that they are ingesting from kafka.

Can you explain what the right strategy here is for scaling ingestors beyond a single instance and how to achieve HA with them?

One thing I considered was creating a separate topic for each ingestor and partitioning up the incoming traffic of traces across different kafka topics by configuring a subset of collectors to publish on different kafka topics. Is that what people do?

[yurishkuro]

You can just run multiple ingesters against the same Kafka topic and they will balance the partitions across them. Just make sure your Kafka topic has more partitions than ingesters, otherwise some of them will be idle.

[yurishkuro]

Do you have a place in mind in the documentation where this would be useful?

[RichardWarburton]

Hi Yuri, That sounds great, thanks. I would say that the performance tuning guide is the best place as that's where the discussion on scaling collectors goes. Am I right in thinking that collector publications will also be auto-load balance across partitions by kafka?

…

On Wed, Dec 14, 2022 at 3:14 PM Yuri Shkuro ***@***.***> wrote: Do you have a place in mind in the documentation where this would be useful? — Reply to this email directly, view it on GitHub <#4110 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACQD3TALUXGPLCDQXUQKSTWNHP5JANCNFSM6AAAAAAS6QSKOA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- regards, Richard Warburton http://insightfullogic.com @RichardWarburto <http://twitter.com/richardwarburto>

[yurishkuro]

Each collector can write to any partition, the partition is selected based on trace-id, so that all spans for a traces come in the same partition.

[yurishkuro]

jaegertracing/documentation#597