Come up with an idea for using the storage hierarchy in DYAD

[ilumsden]

To support the Rabbits in DYAD, we need to decide how to handle the storage hierarchy.

Our initial plan is to use BFS on the Flux Resource Graph (RG) to identify whether a storage location is shared or not.

[ilumsden]

Our original idea will not work for the Rabbits.

Based on discussions with James Corbett, we have learned the following things about the Rabbits that will interfere with using the RG:

• Rabbits can be configured in 4 different modes
  • Lustre
  • XFS
  • GFS2
  • Raw
• Every one of these modes are local storage except Lustre, which is job-local shared storage
• GFS2 is unique because it (in theory) lets you run containerized software on the Rabbit nodes' AMD CPUs
• There is no guarantee that a Rabbit allocation (in any of the above modes) will be placed on the nearest Rabbit to the user's job
• El Capitan integration team is still undecided if Flux brokers will run on the Rabbit nodes
• Rabbit allocations can only be made from the system instance of Flux
  • In other words, child Flux instances cannot allocate Rabbit storage

Based on all of this, unless we have multiple jobs interacting with each other, Rabbit storage can be considered to be only local or shared for all nodes. There won't be a situation where we have "partially shared" resources.

[ilumsden]

Jae-Seung and I have discussed a new approach I have come up with. I'm going to take a bit more time to figure out how to present it before bringing in the rest of the group.