Discussion on methods

[ilpint]

I think that we should agree on some methods and I suggest only some points in no particular order that should be discussed, please add your points to the list

1. Decide in an early stage some definition to adopt to share a common vocabulary. For example should we declare what JSON definition we want to adopt? Either IETF rfc7159 or ECMA 404 or both?
2. It could be useful to try to adhere to some convention, already widely used for mapping XML and JSON, like JsonML or BadgerFish when choosing how to map elements and attributes in JSON. We could have the bonus of some already developed libraries, or just a documented convention to refer.
3. It could be useful to develop also an XSLT file for the transformation from StationXML, I use xslt transformations in my fdsnws/station service to create text and JSON output, so I could work on it when a first proposal of schema will be accepted. For a running example see the INGV service: MN net in JSON

[crotwell]

1. My understanding is that rfc7159 (superseded by rfc8259 just to say must use utf-8) and the emca404 are compatible and reference each other, so I don't think we need to worry about choosing. If we need to pick a spec to reference, rfc7493 (I-json) may be a better choice as it is the same JSON as 7159 but limits some things that are likely to break software anyway, like big numbers and duplicate keys.

2. I am reluctant to lean too heavily on autotranslate. If the purpose is to actually change and improve the structure and address known shortcomings, then it is hard to have a tool do it automatically. Maybe there are some ideas to pull from how they do conversions, but I would not want to be too rigid in following them. In general, my feeling is that a 1:1 translation from XML to JSON is a pointless exercise. The goal is not just to have JSON, but to change and improve the way we represent metadata. Yes, JSON is better then XML, but not that much better IMHO.

3. I agree a StationXML to JSON translator would be really useful and important, and probably is almost a requirement if this new format is going to gain usage. XSLT might be a way to do it, but a python script might be a bit easier to create and maintain.

[ilpint]

1. My understanding is that rfc7159 (superseded by rfc8259 just to say must use utf-8) and the emca404 are compatible and reference each other, so I don't think we need to worry about choosing. If we need to pick a spec to reference, rfc7493 (I-json) may be a better choice as it is the same JSON as 7159 but limits some things that are likely to break software anyway, like big numbers and duplicate keys.

Thanks for pointing out the correct rfc. What I was trying to say is that we should pick and declare a choice because there is a little of ambiguity in saying what JSON format is. Even a clear statement like we do not care of any differences is acceptable for me, if clearly expressed.

2. I am reluctant to lean too heavily on autotranslate. If the purpose is to actually change and improve the structure and address known shortcomings, then it is hard to have a tool do it automatically. Maybe there are some ideas to pull from how they do conversions, but I would not want to be too rigid in following them. In general, my feeling is that a 1:1 translation from XML to JSON is a pointless exercise. The goal is not just to have JSON, but to change and improve the way we represent metadata. Yes, JSON is better then XML, but not that much better IMHO.

Just to clarify, I would like to choose a convention for the parts of the XML where there is the possibility of 1:1 conversions, just to have a way to express an element or an attribute consistently. New objects that shall exists only in the JSON format can be expressed in every JSON permitted way. StationXML elements that need to be profoundly changed should be expressed in freestyle also. In my opinion choosing a good convention for the rest is a zero cost choice that can save work to us and to future implementer.

3. I agree a StationXML to JSON translator would be really useful and important, and probably is almost a requirement if this new format is going to gain usage. XSLT might be a way to do it, but a python script might be a bit easier to create and maintain.

I will try to contribute to both. XSLT translation could also help during transition of legacy fdsnws/station web services that produce StationXML facilitating early adoption in output produced by Data Centers just adding a final step after XML production.
It could be easily used in PHP code and maybe also used to translation directly made by web servers like nginx and apache.

I would add two more questions.

4. What about data center added namespaces in StationXML files?
5. Should we also consider suggesting an addition to the fdsnws/station service definition?

[crotwell]

Good points.

1. Agreed. Shall we go with rfc7493?

2. OK, maybe create a new issue and we can make a stab at a convention?

3. Yes. Only concern is that if there are 2 converters, we make very sure they produce the same output. Or at least declare one to be the official one and the other just an example. We do not want there to be confusion about the format due to inconsistencies.

4. Yes, that is something we will have to decide on. Maybe open a new issue just for that?

5. At a minimum, format=json would have to be added. But beyond that unsure how much ws should change. A flat JSON does open up possibilities for new queries. Open new issue for this as well?

Can you make separate issues for 2-5?

[ilpint]

1. RFC 7493 is still a Proposed Standard while RFC 8259 is an Internet Standard. Given the dates of the two and being of the same author, I think that RFC 7493 will never be an Internet Standard.
   2-5 Ok, I will open separate issues ASAP.

[ilpint]

6. Another proposal I'd like to present is the possible adoption of JSON-LD for our format. Using a linked data format could be useful in general when considering connections between station and other FDSN web services that could also be developed in JSON-LD. Furthermore, a "standard" linked format would be useful in general for different domains web services that reference seismic data or services.

[crotwell]

Seems well worth exploring. Are there any examples of scientific or other datasets using json-ld. A quick search mostly showed it being used inside web pages to give information to search engines.

[ilpint]

I'm fairly new to the concept of linked data, but I think since we're developing a new standard, we can try to imagine connections with other web services and data, not just within seismic science.

Exploring some resources, I discovered that the EU is significantly promoting Linked Open Data (LOD) using Semantic Web technologies such as Resource Description Framework (RDF) and SPARQL to improve interoperability, reusability, and cross-border data exchange. The European Data Portal and initiatives such as ISA2 are promoting their adoption.
RDF provides a standard method for structuring and linking data, while SPARQL enables queries, enabling the creation of a more connected and accessible European data landscape.

By definition JSON-LD is the lightweight data interchange format that enables the exchange of Linked Data, including Resource Description Framework (RDF) data, across the web. It extends JSON with a [context] mechanism to map JSON objects and properties to RDF concepts and Internationalized Resource Identifiers, allowing developers to easily convert JSON data to RDF and vice-versa.

Other than the European Data Portal, another example of use of RDF and JSON-LD is this Catalog of Open data of the Italian Ministry of Culture Catalogo dei dati aperti del Ministero della cultura. Some dataset are also provided in JSON-LD format Dataset dei luoghi culturali in Italia.

To learn more, I found it useful JSON-LD Best Practices best practice for developing API in JSON-LD.
Also useful: Data on the Web Best Practices.

Even if there was not so many examples of web services exposing data directly in JSON-LD there is a huge amount of scientific data referred by RDF resources for example this rdfportal.

Probably it is not necessary to develop a full RDF schema to have a JSON-LD compliant example, it could be whatever JSON with a "@context" added . But JSON-LD could be a future proof starting point, with a shared vocabulary already there, and more.

[crotwell]

To learn more, I found it useful JSON-LD Best Practices best practice for developing API in JSON-LD.

I find it a little concerning that this seems to be several years old, has obvious places where it is incomplete and yet the document hasn't changed in a long time. I am not opposed to borrowing ideas from JSON-LD, but it feels to me like it is a standard that never really took off, and so I am hesitant to base our spec on it without some evidence that it actually is more widely used.

On the other hand, it does provide a potential way to deal with elements/attributes from other namespaces.

Added a strawman to the wiki for JSON-LD.

[ilpint]

I added a mini project to convert stationXML files in JSON-LD using a very simple ontology for the seismic station metadata. It is a proof of concept.

1. Define ontology and context.
2. Use of an intermediate canonical data model to map ontology and StationXML
3. Build a tool to convert StationXML to JSON-LD
4. Build the environment to sustain the ontology use.