Requirement: Allow addition of arbitrary headers by equipment manufacturers, operators, etc. without the need for authorization or approval by the FDSN

[krischer]

Allow addition of arbitrary headers by equipment manufacturers, operators, etc. without the need for authorization or approval by the FDSN.

[chad-earthscope]

There is no accommodation for arbitrary information in miniSEED 2.x except for blockette 2000, which is not appropriate for any simple entries such as a flag or a key-value pair.

The capability to include arbitrary information could be useful for specialized network operations (e.g. parameters determined at a station for low latency source estimates), laboratory sample measurements, unique installation environments (e.g. stations on glaciers/ice floes), and likely more. Arbitrary, custom headers also allows for the development and testing of new headers (without making invalid data) before approaching the FDSN with a proposal to include them in the standard.

A downside to allowing arbitrary headers is the potential for gathering information in archives that is not defined, rendering it useless for future interpretation. One way to mitigate this risk is to strongly encourage those that create their own headers to document their schema/definition using the same mechanism as the FDSN. Perhaps even providing some facility for hosting or linking to user-defined header schema documentation.

For reference, a lengthy earlier discussion on the justification and potential encoding of optional/extra headers in a record is here:
iris-edu/mseed3-evaluation#1

[crotwell]

I feel this is important for flexibility and growth of the NGF and would really like this to be a hierarchical key-value storage with a separation between FDSN standard keys and user-defined keys. While JSON may be too verbose for this use, something that allows the level of complexity is needed.

[krischer]

Summary

(Please let me know if I missed a point or misunderstood something)

Please vote on:

1. Do we want the ability to add arbitrary headers? (Yes/No)
2. Assuming yes to 1. This either requires vendor defined definitions or some kind of key-value data storage format (of course with additional definitions but at least its parse-able without more information). Which do we want? (More definitions/key-value data format)
3. Assuming we want a key-value data format. Which do we want? (Make a suggestion)

[crotwell]

1 yes
2 key-value
3 not sure, json5 would be easy to implement at a cost of extra bytes, messagepack or cbor might be more compact but more complicated to implement especially for data loggers. I lean towards json5, but think this would be best answered after some prototype implementations to better judge the tradeoffs.

[chad-earthscope]

1. Yes.
2. nestable, key-value. The FDSN should strongly encourage (data centers may require) that any non-FDSN-defined headers are documented using the same mechanism used by the FDSN.
3. not sure. CBOR is a good fit in some ways. If we stick to nested key-value pairs (nested Maps in the CBOR spec) and keys are always strings we can specify the schema using JSON Schema. We can go with other encodings, but this kind of flexibility is important.

[kaestli]

1. No! A standard with non-standard (maybe opaque) extensions influencing the interpretation of the standardized information is not a standard.
2. XML with schema reference (or anything els as self-defining as possible). Implicit reference to the timespan of a record must be avoided by any means (as then, re-alignment of records would change the information content in an uncontrollable way.) key-value does not allow for this (at what time did key x have value y?)
   3 --

[ozym]

1. Yes
2. Key-value
3. As per no. 3 in Requirement: The new format must be fully self-defining #5

[claudiodsf]

1. Do we want the ability to add arbitrary headers? (Yes/No)

Yes. This will allow a larger usage of the NGF by other communities. But we call for a standardization within the FDSN community.

2. Assuming yes to 1. This either requires vendor defined definitions or some kind of key-value data storage format (of course with additional definitions but at least its parse-able without more information). Which do we want? (More definitions/key-value data format)

Key-value

3. Assuming we want a key-value data format. Which do we want? (Make a suggestion)

See proposal of @crotwell in issue #5 and our response to question 3 in issue #5 (we prefer a binary format)

[ihenson-bsl]

1. Yes
2. Key-value
3. ?