clean shutdown of ganesha can lead to reclaim race windows

[jtlayton]

Suppose we have two parallel ganesha heads. One ganesha is then shut down, by initiating a shutdown via dbus or by sending it a SIGTERM. Eventually, ctdb notices this and migrates the IP addr and puts the other nodes into grace. This is racy.

In either case, ganesha will tear down all of the state held during shutdown. That includes any file locks held. By the time that we notice that a ganesha has gone down, any state held by that ganesha will have already been released. This opens a window where clients of other nodes in the cluster can race in and grab that state.

[jtlayton]

Note that ganesha now has a new prepare_shutdown export operation that we're using with FSAL_CEPH to avoid tearing down state on the MDS on a clean shutdown. You may want to add something similar to FSAL_GLUSTER.