Add support for viewing data flow maps

[anjackson]

As part of the work to understand how digital preservation gets done in real institutions, I'm trying to understand what the overall flow of information is in these cases. i.e. tracing the path of the bitstreams from system to system at the overall organisational level.

I've come up with an approach based on space-time diagramming combined with Metro map styling. It takes a sequence of data event information and visualises it in quite a nice way. Some more features are needed still:

• More diagrams, including very simple OAIS patterns, UKWA etc. And a way to flick between institutions and workflows.
• When you click on one, update a panel that gives a place for lots of lovely gnarly detail and links. Or possibly copy these popovers
• Station names all have to be unique rather than just being labels. That should be changed.
• Multiple source-target pairs at the some timestamp would be useful sometimes.
• support a simple text markup (see below) and mapping it from YAML.
  • Nice error if a target of e.g. a delete does not exist.
  • Fix it so renders still work if places/data etc. are not declared first.
  • markerAt /fraction of line/ is quite useful, use @N@0.5

The 'simple test markup' looks like this:

# Data types and descriptions:
data sip "Submission Information Package" color="#ff0000"

# Locations where data can be stored:
location producer "Producer"
location ingest "Ingest Storage"
location archive "Archival Storage"
location access "Access Storage"

# Domains where locations are maintained:
domain dc "Designated Community"
domain ar "The Archive"
domain man "Management"

# Then the sequence of events in this dataflow...

# We start by transferring a package from an external party:
start sip@producer.dc
move sip@producer.dc sip@ingest.ar "Transfer to the archive"
space

# We then prepare the item for ingest to the archival storage storage system:
derive sip@ingest aip@ingest "Generate AIP from SIP"
copy sip@ingest aip@archive "Copy to archival storage" 
# And delete the temporary files:
delete sip@ingest, aip@ingest
space

# When access is requested, we generate an access copy:
copy aip@archive aip@access "Retrieve the AIP"
derive aip@access dip@access "Generate the DIP"
copy dip@access dip@consumer.dc "Send the DIP"
delete aip@access, dip@access

# And we're done:
end

[anjackson]

• Extend platforms for ```{dataflow}``` custom fenced code blocks???
  • Configuration | Observable Framework
    • markdown-it/markdown-it: Markdown parser, done right. 100% CommonMark support, extensions, syntax plugins & high speed i.e. by hacking the code block highlighting hook.
  • MystMD
    • Plugin mechanism https://mystmd.org/guide/javascript-plugins
  • DecapCMS https://decapcms.org/docs/custom-widgets/

[anjackson]

Okay, so the core idea is implemented and accessible at: https://www.digipres.org/workbench/dataflows/

Most of the TODO items above need to be split off into a 'future work' ticket. The goal right now is to get something workable out there and get some feedback before going much further.

[anjackson]

This basic diagram fails if the places are not defined:

start sip@producer
move sip@producer sip@store "Ingest SIP"
transform sip@store aip@store "SIP to AIP"
space
derive aip@store dip@store "Generate DIP"@N [0,1]
move dip@store dip@consumer "Access"
end

Error is: Uncaught (in promise) Error: Repeated coordinates 9,0

But adding

place store "Archival Storage" ar
place producer "Producer" dc
place consumer "Consumer" dc

in any order at all still works, so it seems like a bug.

[anjackson]

Okay, dropping scope and trying to tidy this up into a decently usable form. Decent error handling would be good.