BUG#37223451: Commit_order_queue::front() prone to race conditions

Background
----------

On an inbound replication channel, there is a Commit Order Manager,
which ensures that transactions are committed in the same order they
are received. This is achieved by making each transaction wait for
the preceding transaction (according to receive order) to begin
committing, before it commits.
When a replication worker finishes its commit, it finds out what is
the next worker waiting through the the method
`cs::apply::Commit_order_queue::front()` in order to wake it up.

Problem
-------

BUG#35206392 added the method
`cs::apply::Commit_order_queue::remove()` to
`cs::apply::Commit_order_queue` class.
Though it was missed that this method can race against
`cs::apply::Commit_order_queue::front()`, causing assert failures
and/or the incorrect replication worker to be awaken.

Solution
--------
`cs::apply::Commit_order_queue::front()` is extended to acquire
`m_push_pop_lock` thence avoiding race conditions.

Change-Id: I4ca20933779bcfe39e21f6d7812273ca4ac2b38e

Author: nacarvalho

sql/changestreams/apply/commit_order_queue.cc

@@ -183,6 +183,9 @@ void cs::apply::Commit_order_queue::push(value_type index) {
 
 cs::apply::Commit_order_queue::value_type
 cs::apply::Commit_order_queue::front() {
+  lock::Shared_spin_lock::Guard front_sentry{
+      this->m_push_pop_lock,
+      lock::Shared_spin_lock::enum_lock_acquisition::SL_SHARED};
   return this->m_commit_queue.front();
 }