BP-41 Bookie Address changes tracking

[eolivelli]

• Allow the client to follow Bookie Address Changes
• If a Bookie changes network address the clients will be able to connect to the new addresess
• In order to leverage the most of this feature the Bookie must configure a fixed bookieId (see BP-41)
• add flag enableBookieAddressTracking to enable/disable this feature, as it needs a ZK watch for every active bookie in the cluster and if you have a static cluster there is no need to invalidate the local cache and issue more ZK requests
• handle ZK session expiration events, when the ZK client is not receiving watch notifications it could miss bookie address changes, so we have to eagerly flush the whole local cache of bookie addresses

This is part of BP-41
Master ticket #2396

[eolivelli]

also @nicoloboschi @Ghatage PTAL

[eolivelli]

gently nudging at @sijie @rdhabalia @jiazhai

[Ghatage]

Overall LGTM.
Can you please explain the upgrade scenario for this?
As I understand, after upgrading the bits we can still use the older bookie id but if we want, we can start the bookies with BP-41 enabled, and then we have to restart the clients with it enabled too? Is that right?

[Ghatage]

nit: We can use this to enable/disable, so not sure if naming variable as enabled is clear. It could just be value

[eolivelli]

changed

[Ghatage]

How frequently can a bookieID change? What factors drive its change?

[eolivelli]

I see these known cases:

• a bookie with a dynamic network address
• changing configuration (change DNS name....)
• you change configuration and add new endpoints (this will happen in the future, when for instance @sijie will contribute the pure TLS endpoint)

So basically I think they won't change very ofter

[eolivelli]

Can you please explain the upgrade scenario for this?
As I understand, after upgrading the bits we can still use the older bookie id but if we want, we can start the bookies with BP-41 enabled, and then we have to restart the clients with it enabled too? Is that right?

If you don't configure the bookieId the bookie will work as it worked in 4.11, so no problem for old clients.
If you configure bookieId in your bookie then only 4.12 clients will be able to use it.

If you want to leverage Bookie Address Change Tracking you have to:

• use bookidId
• use 4.12
• enable it in ClientConfiguration (it is enabled by default, in order to be friendly with bookies with dynamic addresses by default)

We cannot set a bookieId by default in order to be backward compatible

[Ghatage]

If you configure bookieId in your bookie then only 4.12 clients will be able to use it.

So can there be a case where we upgrade Bookies to 4.12 but disable this option and use the older hostname:port pairing
But then we won't be able to use 4.12 clients?
Sorry if I am misunderstanding this!

I don't see it as a problem, just wondering if we also need to call it out and update the documentation for this in the 4.12 release.

[eolivelli]

If you simply upgrade an existing cluster from 4.9/10/11 to 4.12 everything will work well.
You can set enableBookieAddressTracking=false safely as bookies are using hostname:port as bookieId and it is not expected that this address changes.

I am setting enableBookieAddressTracking=true by default because I think this is the future for BK and also in order to have all tests working with this feature.

The case of enableBookieAddressTracking=false is to save resources on ZK, by reducing the number of watcher.

If we were on ZK 3.6.x (client + server) we could leverage "Persistent Recursive Watches", that is a single watch for an entire tree of znodes, without the need to set it again after it triggers, and this will also mitigate the impact of these new additional watches

[eolivelli]

@jiazhai @rdhabalia Please help me merging this patch

[nicoloboschi]

LGTM