Requirement: The new format must be fully self-defining

[krischer]

The NGF format should be self-defining in all key aspects

• miniSEED2 is not self-defining. For instance, the byte order of the FSDH is not identified and must be determined by an algorithmic check rather than being clearly specified in the format itself (this is because though SEED does define the byte order of the FSDH in blockette 50 fields 11 and 12, miniSEED2 does not have blockette 50).
• Eliminate the need to consult external references to understand how to make use of the content within the format. For example, the format should be parsable, even if the specific definition of the content is not known. The format should be self- defining, with perhaps the exception of encoding value lookups to enable efficient decoding of data that has been used without issue in miniSEED2 for decades.

[andres-h]

Being fully self-defining is obviously not possible unless huge amount of metadata is added to each record. No JSON!

[krischer]

I do agree that full self-definition without consulting the reference will be extremely difficult and does not mesh with the requirement of having a short header.

I assume the second bullet point should only be valid for the optional/additional data not part of the fixed header?

[andres-h]

I don't like the proposed fixed header at all. If I would vote, then #18/#20 would be NO from me.

[crotwell]

"Self-defining" is probably too strong a word here. Perhaps "self-contained" is better, in the sense that what is really desired is that a single NGF record, along with the specification and documents it references, contains sufficient information to parse the fixed headers and fully reconstruct the timeseries. Additional/optional data may be externally defined as long as it is parsable and "namespaced" in a way that allows systems that understand that particular additional content to take advantage of it.

We don't want to go down the rabbit hole of trying to define everything independent of outside resources, the format is for data exchange with seismologists, not with space aliens. So referring to outside definitions of URI, compression formats, byte order, time formats, etc should be acceptable within the specification and is unavoidable.

[krischer]

Summary

(Please let me know if I missed a point or misunderstood something)

This requirement is a bit undefined and not much discussion happened. Thus please vote on the following issues (2 & 3 largely mesh with #14):

1. Should the new data format be parse-able without resolving to heuristic checks? E.g. a mandatory byte order field. (Yes/No)
2. Should information not in the fixed header (assuming we have one) be in some form of standardized format (e.g. no binary blockette that is meaningless without the spec)? (Yes [some standardized format] / No [keep binary blockette layout or something similar])
3. Assuming 2. results in a "yes": What should this format be? Please propose one or more.

[crotwell]

1 Yes
2 Yes
3 In no particular order (and there may be others worth considering):
JSON although it is ascii and verbose
JSON5 ascii and verbose but less so than JSON
CBOR binary, but has some controversy
MessagePack binary, some controversy

The binary vs ascii difference is important, but it is critical is that it be a hierarchical key-value storage.

[ketchum-usgs]

1. Yes
2. No - No undefined add ons in the headers

[chad-earthscope]

1. Yes.
2. Yes, with FDSN defining "reserved" headers.
3. what @crotwell said.

[ozym]

1. Yes
2. Yes, assuming some form of namespacing.
3. As per @crotwell also, plus perhaps add BSON and UBJSON to the list (not an endorsement but for extra comparison)

[kaestli]

1. Yes (however byte order can be defined as part of the standard, not as flag in the header)
2. No - no undefined add ons in the headers.

[chad-earthscope]

@kaestli

2. No - no undefined add ons in the headers.

The description of this item does not include "undefined add ons". I think it is about how optional headers are structured.

[claudiodsf]

1. Should the new data format be parse-able without resolving to heuristic checks? E.g. a mandatory byte order field. (Yes/No)

Yes

2. Should information not in the fixed header (assuming we have one) be in some form of standardized format (e.g. no binary blockette that is meaningless without the spec)? (Yes [some standardized format] / No [keep binary blockette layout or something similar])

Yes. This will provide a flexible format, with still the possibility of having specific header add ons for a certain community (e.g., FDSN extensions)

3. Assuming 2. results in a "yes": What should this format be? Please propose one or more.

Yes, as per @crotwell. We prefer a binary format.

[ihenson-bsl]

1. Yes
2. Yes
3. Yes, as per @crotwell, prefer binary format.