Sphinx Databus: Implement PDBx/MMCIF

[lizutah]

http://mmcif.wwpdb.org/pdbx-mmcif-home-page.htm

• download and test;
• python source file in sphinx that lets us use that scheme in that code;
• start creating plugins that use it;
• define a process for converting all places we pass data into subsets of that schema.

[keith923]

For what it's worth, the link in the issue description is dead.

My current understanding is that PDBx/mmCIF files are an alternative to PDB files. The format is documented as a "dictionary", and uses a "funky" format called STAR (which is the same format as the PDBx/mmCIF files). It is fully specified here. A brief introduction to the syntax is provided. There is also an interesting read at this site.

This STAR format is basically a set of key-value pairs with some added fluff for creating tables. It's very general, in a similar way as JSON or XML, but not as pretty as the former, and less verbose than the latter. It is this generallity that makes it possible to use the same format to represent both the molecular data, as well as the format for for the molecular data.

But wait, there's more! The specification for the molecular data, i.e., the PDBx/mmCIF files, is based on an even more abstract definition! I feel Excedrin headache number 42 coming on!

My thinking is that we'll need to support reading and writing the PDBx/mmCIF format, in addition to using the PDBx/mmCIF specification as the basis for our internal data structure. Along those lines, it seems reasonable to have code that reads the STAR formatted files into Python data structures that we can export as JSON. From there we'll be able to either generate a JSON schema for our data structure, or use the JSON as molecular data for the pipeline.

I'll add more specific information as I learn more.

[keith923]

This presentation is a good resource for what I was trying to describe above (wish I'd found it first). Slide 7 is particularly useful in showing how the format is defined.

[keith923]

I'm working on converting the XYZR file parser and geoflow plugin to use the JSON schema that I've generated from the PDBx/mmCIF schema.

[keith923]

I've got the XYZR and Geometric Flow plugins passing data using what I came up with using the PDBx/mmCIF schema. It's all currently in the databus branch.

In general, there is a new TypeManager class that plugins have access to in order to create value instances that are validated against the schema. There is also some rudimentary support for extending the schema. I think that it's a good start, and that it's something that we can build upon and refine as necessary.

As for the last "checkbox" at the top of this task... We are still using ad hoc data formats in the file reader/writer plugins and the stdout plugin. I think that I'll interpret that task to mean that we need to define something to hold lines of plain text in the schema, and then update the plugins.

As an aside, the entire pipeline source/sink type impedance matching is still defunct. It's not being used, and it's broken. I think that we need to specify keys in the schema that correspond to the data that they can source/sink, and then just check those when we create a pipeline. The question is whether or not we want to do that in this issue, or if there is another place that can be identified as more suitable. I'll check into that tomorrow, and consult @lizutah.

[lizutah]

create new issue from last check box and close.

[keith923]

As per my last comment, I've created a UDT for holding text, and the last checkbox is checked. Also, the source/sink validation/impedance matching is no longer defunct.