Skip to content

Latest commit

 

History

History

distributed

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 

distributed patterns

pubsub

Use the native node module net to create a server that publishes events to subscribers.

server

var net = require('net')

if (require.main === module) {
    start()
}

function start () {
    var connections = []

    var server = net.createServer(function onConnection (conn) {
        var i = connections.length
        connections.push(conn)
        conn.on('close', () => connections.splice(i, 1))
    })
    server.listen(8000, () => console.log('listening on 8000'))

    setInterval(function () {
        connections.forEach(c => c.write(JSON.stringify({
            pid: process.pid,
            time: Date.now()
        })))
    }, 1000)
}

module.exports = start

client

var net = require('net')

if (require.main === module) {
    start()
}

function start () {
    var client = net.connect(8000)
    client.on('data', function (data) {
        console.log(JSON.parse(data))
    })
}

module.exports = start

reconnection

What happens if you start the client before the server? Or if the server goes offline?

Error: connect ECONNREFUSED 127.0.0.1:8000

To better decouple the client and server, you want the client to automatically try connecting until a server comes online. But that's too much work to do yourself -- look at these:

req rep

Request and reply semantics are like http requests. However, HTTP has more overhead compared to other socket protocols. Another protocol we can use is nanomsg. Nanomsg does some low level stuff for us. Our socket will connect and reconnect on their own, so we don't have to write any reconnection logic.

Server

var nano = require('nanomsg')
var rep = nano.socket('rep')

rep.bind('tcp://127.0.0.1:8000')
rep.on('data', function (data) {
    console.log('wooo', data.toString())
    rep.send(data.toString() + ' world')
})

client

var nano = require('nanomsg')
var req = nano.socket('req')

req.connect('tcp://127.0.0.1:8000')
req.send('hello')
req.on('data', function (msg) {
    console.log('received', msg.toString())
})

So this is weird. In node we use callback functions for response semantics, but this has a subscription API:

req.send('hello', function onResponse (err, data) {})

This means that we are not able to send multiple requests in parallel. This is part of the nanomsg protocol, which is based on zeroMq. The responder (server) is guaranteed to only get one message at a time, until it sends a response.

See also

dealer & router

In nanomsg/zeroMq context, parallel requests are done with a dealer/router structure. The server deals out requests round-robin to worker processes.

zmq-time-rep-cluster.js

multiplexing

Pipe several streams through a single socket. Check out this example

discovery

application sharding

Partitioning is a way to scale a stateful service, for example a websocket server. Instead of having stateless servers, we keep state and replicate it to accomodate failover.

dynamo db whitepaper

upring

This uses consistent hashing and the swim protocol to implement failover. Consistent hashses create a uniform distribution of connections amongst servers.

redis