Response Level and Information Equivalence with StationXML

[krischer]

Hi Chad and others,

I hope this is not too late. I like the proposal of having a JSON representation for station data however I would rather see this evolve into a JSON representation of the information contained in StationXML files and not only the text output from the FDSN web services. This means that the design goal of the StationJSON format should be that it can store the same information as StationXML files and one can convert from one to the other without loosing any information.

The JSON files at the network, station, and channel level would not change a lot I believe, but it would require additional definitions at the response level which should not be too much work.

I would be willing to look into actually doing that if there is interest in this. Potentially its even possible to auto-convert the XSD schema to a JSON schema.

Potential gains of a fully equivalent JSON representation of StationXML:

1. IMHO easier to edit then XML files.
2. Easier to parse and work with in almost all programming languages.
3. They could just be dumped into "JSON enabled" databases (PostgreSQL > 9.4, MongoDB, many NoSQL databases, ...). Some of these are pretty neat and can query on nested JSON files.
4. The "minimum" content characteristic desired for this format would be defined by the StationXML standard which I think is pretty sensible.
5. It could evolve together with the StationXML definition and thus does not require a full separate definition.

FDSN standards tend to stick around for a long time so I believe the effort of doing this are well worth it.

Cheers!

Lion

[crotwell]

I agree. If we are going to all the trouble of making a json version, it might as well be as capable of holding all the information that exists in the full xml version.

I am also willing to help with adding the responses if needed.

Philip

[CTrabant]

Mostly what you wrote are general advantages of JSON and I agree that some of them are compelling. What I'm left wondering is what problem would we be solving? Or is this more of a "build it and they will come" thing? Because we can doesn't mean we should.

I like the idea of evolving the XML and JSON schemas together but there is overhead to maintaining the schemas. Hmm. I'm not willing to commit yet.

Here's what I suggest: one or both of you try to mock up a complete StationJSON schema if you want to give it a try, the experience of mapping the schema will be instructive.

[crotwell]

In thinking about this, and given how JSON tends to be used, I wonder if maybe a slight alternative to putting responses in a monolithic StationJSON makes more sense. Instead of having the channel response as a subobject of the channel, have a schema that holds the response information as a top level object and retrieve it in a separate request. The object would still be closely aligned to StationXML's response. I just think that responses are large enough that they make sense as a separate query. Add to that the fact that they are often duplicated among similar channels, and so putting them inside the channel object is wasteful.

Another way of saying this would be that this JSON object could replace the old RESP file format as a way of storing/transporting responses independent of an actual channel.

I can take a stab at this if it seems plausible.

[crotwell]

Here is a first cut at response objects in json. Probably a little rough, but the big items are there.

#10

This is just the definition of the ResponseStage object and subobjects, have not yet linked it into channel or anything else. Maybe some discussion of how this might be used within a web service would be good before deciding how (or if) to proceed. I am leaning towards a linking model as opposed to an embedding model, but maybe their is room for both.

thanks
Philip

[crotwell]

Been thinking about this and I feel more strongly that a focus of being a JSON version of StationXML not the most useful. To take @krischer 's list, 1 and 2 are generic to the json vs xml debate. And while there is merit to the JSON is better than XML argument, it doesn't seem strong enough to me to justify parallel maintenance of a JSON and a XML that are structurally identical, just using a different language. There needs to be a use case besides just JSON is better.

Second, I feel like item 3 argues for NOT mimicking the stationXML structure directly. In particular, the containment philosophy of stationXML really only makes sense if you make one big query for the metadata you want and that is all you want. If you have interactivity with either an actual human or with client side business logic, you probably will do much more back and forth with the server, likely some variation on getting networks and/or stations, then picking a subset for which you return and get channels, and then only getting responses for a subset of channels. In this case, you end up with lots of repeated metadata, ie if you want to store stations, then the network metadata is repeated for each station. And if you need to store channels, then the station and network metadata is repeated for each channel. Both violate the don't store the same information in two (or more) places rule, as it very quickly becomes incompatible. And even if on the client side you strip it out for storage, you have the tension of if I get a channel, should I update the station and network with the "new" or is what I already have ok? Keeping all the pieces in sync becomes hard.

I feel the strongest use case for a StationJSON may very well be this more interactive back and forth style of interaction, which means that while StationJSON objects would mirror the attributes of stationXML objects, they would not follow the containment philosophy. Instead, if I want the response for a particular channel, what I get back is the bare minimum: the response object plus an identifier for the channel, but not the network, station and channel objects. Or maybe just one stage of the response as usually that is all I need. If I already have the channel, I am done and if I need it, I have enough of a references to can go back and ask for it. Side loading can also be allowed, but is a choice not a requirement.

Basically I feel that for many types of web service clients, many small transactions is vastly preferable to a few large ones. This is especially true in resource constrained scenarios, like web browser or mobile app, where parsing a large xml or json response is a real problem.

See issue #1 and JsonApi. I have used the JsonApi style interaction for some projects recently, and it just feels much, much cleaner than dealing with the current object within object within object style of StationXML. JsonApi feels nimble, while StationXML feels big and bloated.

Just my $0.02...

[krischer]

This would be pretty slick - I've never really thought about it this way. It would also enable some form of data-on-demand workflows, i.e. if one wants to rotate waveforms, the coordinates for the channels are downloaded, if one wants to remove the sensitivity, only that is downloaded, and so on. This could be extended by working with some kind of backend that transparently downloads whatever data people need on-demand so users would no longer have to explicitly download or request things.

So 👍 from my side!

[Jollyfant]

@krischer GraphQL does something like that. In practice, metadata comprises only a minor amount of data that we export compared to continuous waveform data. Shipping some redundant information is not a big deal usually, given that the format is designed to be inherently lightweight (i.e. not a gazillion <numeratorCoefficient> fields). Users probably are not going to care enough and hit the "download all" button anyway.

I agree with @crotwell that there needs to a better motivation than "just because". Transfer speed, parsing cost, and readability are definitely important. Another problem I have with StationXML is that the line between data and metadata right now is non-apparent. When we say "station metadata" we are really just talking about data (i.e. instrument specifics). Things like comments, data availability, and identifiers (doi) should in my opinion be excluded as they are actual metadata. The decision to nest data in the classical net->sta->cha->response structure can (and should) be influenced by the direction of recent mSEED3 developments. Ultimately, we are probably going to identify single streams instead of using the SNCL hierarchy anyway?