proposal: control service synchronization

[tzaeschke]

It seems that path databases of control services (CS) are not synchronized when an AS is running multiple CS.

Impact

• Most clients are implemented such that they contact only one CS when asking for paths. If the CSes are not synchronized, the client will not see all path or even nor paths at all (if the CS never received any paths)
• If one CS acts as a fallback then won't work because it never got the paths from the other CS.
• In core ASes, the CSes may be queried by other CSes in other ASes. Many paths will not be made available if CSes do not contain all paths. Especially core ASes are likely to have multiple CS instances.

History

As far as I understand, CS synchronization was originally done with ZooKeeper, see #50. However, ZooKeeper was later removed, see #1025, and CS synchronization was dropped.

Sketch of solution

Any implementation should:

• synchronize paths fast, within seconds or minutes.
• If one CS crashes, it should be able to recover from missing updates or even be able to rebuild the whole path database
• Path that are received through CS synchronization within an AS probably do not need to be verified.

Proposal

One solution is for each CS to regularly poll other CS for updates. To avoid timing issues, we assume that all CS's clocks are reasonably in sync (We could also use a system with timestamps for each other CS, but that seems like overkill).
Every CS should locally keep track in the database of when it last synced with another CS. This timestamp can be used in queries to indicate for how long ago PSBs should be sent in a sync request. This also allows rebuilding path databases after a crash or prolonged downtime.

See also discussion here: https://scionproto.slack.com/archives/C8ADA9CEP/p1767972450040339

Alternatively, it should be possible to use a protobuf subscription/stream instead of polling. The stream would start with a request to send all PCBs after a given timestamp. After initial sync is achieved, the stream would remain open for the sender to immediately send new PCBs to all subscriber.

To be designed:

• Full topo model? Does every CS poll/subscribe to every other CS? That doesn't scale well but would be best for availability in case one CS disappears.
• Star topo model / chain topo model? Does every CS poll/subscribe to only one other SC? Which one? What happens if that one goes down?
• We also need to consider that for initial sync (after crash) one CS should receive ALL PCBs from its peer. However, after initial sync, peers should only send PCBs that it received from outside the AS (star topo, full topo) or from a parent (chain topo). This should avoid sending PCBs unnecessarily.**Initially: ** As a simple alternative it may be fine to forward all PCBs (after initial sync is finished) regardless of where they come from, as long as they are not already present in the local path DB; this would create some network overhead but considerably simplify the architecture.

I am happy to discuss this in detail later.

[oncilla]

Path that are received through CS synchronization within an AS probably do not need to be verified.

That assume communication is authenticated.

Alternatively, it should be possible to use a protobuf subscription/stream instead of polling. The stream would start with a request to send all PCBs after a given timestamp. After initial sync is achieved, the stream would remain open for the sender to immediately send new PCBs to all subscriber.

That sounds like a lot of complexity. What do you gain from this? The SCION control plane already only gives eventual consistency guarantees. Immediate updates seem unnecessary. Slightly related: twitchtv/twirp#70 (comment)

Full topo model? Does every CS poll/subscribe to every other CS? That doesn't scale well but would be best for availability in case one CS disappears.

What scale are we talking about here. What is your use case where you have more than a handful of CSes?

[tzaeschke]

@oncilla Thanks for the feedback.

That sounds like a lot of complexity. What do you gain from this?

It is a little bit faster, which may not matter much but I'd take it if it doesn't cost too much.
At first glance, I would have guessed that streaming is simpler than a polling, but I have never implemented a proto stream like this so I don't actually know (it looks like you looked at this more closely already). Anyway, this is listed under "To be designed" so it definitely needs more thought.

[oncilla]

It is a little bit faster, which may not matter much but I'd take it if it doesn't cost too much.

I think it costs a lot in terms of complexity. You need to worry about a lot more failure cases. And the amount of segments that are right now in the ecosystem is so small, I really would not go that route.

[JordiSubira]

Btw, this does not look like a bug to me. As you said in the history section, it is a feature that was deliberately dropped in the past. This does not mean it is not necessary to support certain use cases.

[tzaeschke]

Btw, this does not look like a bug to me. As you said in the history section, it is a feature that was deliberately dropped in the past. This does not mean it is not necessary to support certain use cases.

I am not sure how deliberate that was, there doesn't seem to be a discussion about what happens to CS synchronization.
I would also argue that the least that should be done is document that running multiple CS is problematic and not recommended?

[oncilla]

It was deliberately done to consolidate and simplify the deployments, which are still too complex.

Previously, deployment was incredibly complex because there was a beacon service, a path service, a certificate service, and synchronization via zookeeper. An ill-advised approach that drew the service boundaries at the wrong place.

We consolidated all of these services into a monolith, and further simplified deployment by choosing a single instance deployment because we wanted people to have as little friction as possible when playing with SCION.

FWIW, I think it is still by far to complex to run a SCION AS with the open source code. Introducing multiple instance deployments will only increase complexity for people testing out SCION.

[tzaeschke]

Apologies for my wording, I didn't mean to say the ZooKeeper was removed by indeliberately, I tried to refer to the missing discussion on CS synchronization, but it came out poorly. Again, apologies.
The problems and reason for removal are clearly documented in #1025 (which I linked above). What I couldn't find is a discussion on what happens when running multiple CS after ZK is removed.

As I understand it the discussion resulted in:

• We won't need multiple CS anytime soon.
• Starting multiple CS is technically possible but shouldn't be done because they won't synchronize.

I would argue that running a second CS as stand-by/backup is useful even without scalability concerns.

What I propose:

• Keep this proposal, but assign low priority. Reason: it is useful feature to have (eventually), even without scalability issues.
• Document somewhere that scionproto supports only a single CS.

Thoughts?

[JordiSubira]

Sure, we can keep the proposal, I was referring to the "bug" tag in this issue and change it to "proposal"