KAFKA-16024: SaslPlaintextConsumerTest#testCoordinatorFailover is flaky

[brandboat]

• broker.session.timeout.ms defaults to 9s. When a broker goes
  offline, the group coordinator may take up to this long to be
  re-elected.
• The commit callback retry timeout is currently 10 seconds, which
  leaves very little buffer. If the metadata hasn’t refreshed yet, the
  consumer may still send an OFFSET_COMMIT request to the offline
  coordinator, leading to transient failures.

This patch enable controlled.shutdown.enable to allow the broker to
notify the controller before shutting down. This speeds up the test by
triggering an immediate failover instead of waiting for the broker
session timeout (default: 9s) to expire.

Reviewers: TaiJuWu [email protected], PoAn Yang [email protected]

[TaiJuWu]

LGTM.
Run 100 times on local, no failure.

[FrankYang0529]

@brandboat In AbstractConsumerTest#brokerPropertyOverrides, it sets controlled.shutdown.enable=false. I think that is why we need to wait up to broker.session.timeout.ms for a new election. The initial intention was to speed up shutdown. If we modify the value to true for this case, can we avoid the flaky test?

[brandboat]

The initial intention was to speed up shutdown.

@FrankYang0529, good point! Interestingly, setting this ends up slowing down the test instead. I think it's ok to set controlled.shutdown.enable=true in this test case, of course. I have to run another 500 runs to make sure flaky won't happen again.

[brandboat]

I ran kafka.api.SaslSslConsumerTest.testCoordinatorFailover 500 times locally without any failures. ./gradlew core:test --tests kafka.api.SaslSslConsumerTest.testCoordinatorFailover runtime dropped from 31s → 19s on my local dev.

[FrankYang0529]

LGTM. Thanks for the patch.