Distributor

Distributor is the stage to route events, it supports routing events to different destinations through Jetstream message, Kafka message. Distributor receives Pulsar raw events and session mark events from its previous stage Sessionizer. Distributor configures an Esper Processor, users can configure EPL to selectively filter/mutate/route events to different destinations.

Rules are configured in EPL and can be changed dynamically with configure app. Sample EPL looks like below:

@OutputTo("OutboundMessageChannel")
@ClusterAffinityTag(colname="si")
@PublishOn(topics="Pulsar.MC/ssnzEvent")	
select * from PulsarEvent;
	
@OutputTo("OutboundMessageChannel")
@PublishOn(topics="Pulsar.RR1/Mobile")
select * from PulsarEvent(et='Mobile');

It means route all PulsarEvent to Jetstream topic "Pulsar.MC/ssnzEvent" with affinity key "si" and route Mobile Pulsar events to Jetstream topic "Pulsar.RR1/Mobile". The subscriber clusters(Jetstream clusters) listen to those topic and get the events. We support both consistent hash based schedule with affinity key and round-robin based scheduler. The subscriber clusters need to create netty context and InboundMessage channels to receive the events.

Support Kafka Message

@OutputTo("PulsarKafkaChannel")
@PublishOn(topics="Pulsar.mobile")
select * from PulsarEvent(et='Mobile');

It means route Mobile events to kafka topic "Pulsar.mobile".