RBMC: Check again for dead sibling service

During some bad path testing the sibling daemon on each BMC would make
it past the existing check done to make sure it was running and then
die. This would cause the wait for the sibling interface to be on D-Bus
to time out.  At that point each BMC became active since it thought the
sibling daemon was fine and just the sibling BMC had the problem.

Fix this by checking again if the sibling daemon is running when the
sibling interface still isn't on D-Bus after waiting for it.  If it
isn't, become passive.

Tested:

This is seen on each BMC:

```
Waiting for sibling interface and/or heartbeat: Present = False, Heartbeat = False
Done waiting for sibling. Interface present = False, heartbeat = False
Sibling service state is failed
Role = xyz.openbmc_project.State.BMC.Redundancy.Role.Passive due to: Sibling BMC service is not running
```

Signed-off-by: Matt Spinler <[email protected]>

Author: spinler

redundant-bmc/src/manager.cpp

@@ -49,7 +49,6 @@ Manager::Manager(sdbusplus::async::context& ctx,
     ctx.spawn(startup());
 }
 
-// clang-tidy currently mangles this into something unreadable
 // NOLINTNEXTLINE
 sdbusplus::async::task<> Manager::startup()
 {
@@ -72,13 +71,20 @@ sdbusplus::async::task<> Manager::startup()
         {
             co_await sibling->waitForSiblingUp(siblingTimeout);
 
-            if (previousRole == Role::Passive)
+            // Sibling service may have died.  Check again.
+            if (!sibling->getInterfacePresent())
+            {
+                passiveRoleInfo = co_await determinePassiveRoleIfRequired();
+            }
+
+            // If passive previously, let sibling go first.
+            if (!passiveRoleInfo && (previousRole == Role::Passive))
             {
                 co_await sibling->waitForSiblingRole();
             }
         }
 
-        updateRole(determineRole());
+        updateRole(passiveRoleInfo.value_or(determineRole()));
     }
 
     spawnRoleHandler();
@@ -118,7 +124,6 @@ void Manager::startHeartbeat()
     ctx.spawn(doHeartBeat());
 }
 
-// clang-tidy currently mangles this into something unreadable
 // NOLINTNEXTLINE
 sdbusplus::async::task<> Manager::doHeartBeat()
 {