QTopology is a distributed stream processing layer, written in node.js.
NPM package: https://www.npmjs.com/package/qtopology
Documentation https://qminer.github.io/qtopology
npm install qtopologyQTopology is a distributed stream processing layer, written in node.js.
It uses the following terminology, originating in Storm project:
- Topology - Organization of nodes into a graph that determines paths where messages must travel.
- Bolt - Node in topology that receives input data from other nodes and emits new data into the topology.
- Spout - Node in topology that reads data from external sources and emits the data into the topology.
- Stream - When data flows through the topology, it is optionaly tagged with stream ID. This can be used for routing.
When running in distributed mode, qtopology also uses the following:
- Coordination storage - must be resilient, receives worker registrations and sends them the initialization data. Also sends shutdown signals. Implementation is custom.
QTopologyprovidesREST-based service out-of-the-box, but the design is similar for other options likeMySQLstorage etc. - Worker - Runs on single server. Registers with coordination storage, receives initialization data and instantiates local topologies in separate subprocesses.
- Leader - one of the active workers is announced leader and it performs leadership tasks such as assigning of topologies to workers, detection of dead or inactive workers.
See documentation