Decide on lifecycle of data

[mgeplf]

The quoted comments are from #10 (comment)

For a morphology, my impression was that changing the distribution was a new version whereas changing the metadata was not.

What is considered metadata? for a morphology, we have the attributes:

* name
* description
* mtype annotation
* license
* location
* species
* other annotations

Are they all metadata?

If someone refers to a morphology at time X when it has an mtype of SR_PC, but then someone changes the mtype, then the the same query now returns different results. If the user who made the query at time X stored the results, they will be surprised to find out that the results have changed, and we don't have a way to track this at the moment. When doing a single cell simulation, and the mtype has changed won't that be confusing? Even more so if the etype of a trace has changed (which is also metadata?)

Also, we don't have the concept of a version; so when you say distribution was a new version it's not clear to me what you mean. What is versioned, the entity?

With respect to provenance, we can store as a "usage" relationship the entity_id

Can you write up and with diagrams of what the lifecycle of these entities are? And formally define what has a version.
I think we all have different impressions of what means what, and it's important that we, and the users, are clear on what each of the actions means, and what sort of stability they will get from queries, when storing references to entities in configs/experiments, etc.

[jdcourcol]

annotations are not part of the entity and should not. An annotation points to an entity. So one can add annotations , update annotations as much as we want, this is not "changing" the entity.

[jdcourcol]

For a morphology, my impression was that changing the distribution was a new version whereas changing the metadata was not.
For a circuit, I would say that it is similar, considering the distribution is a folder, then you create a new version of the distribution. However, it is still the same circuit entity

With respect to provenance, we can store as a "usage" relationship the entity_id + the version of the distribution.
With respect to storing versions, we can ask the user at upload if he wants to keep the previous one. We can also identify is the previous version has been used or not to help. Same for cleaning.

[jdcourcol]

I will refine the above statement:

• anything that is a statement about the entity and does not about "using" the file/distribution, should point to the entity with just a regular foreign key (example: an m-type annotation, a experimentalist contribution..)
• if it relates to the content of the file, and changing the content of the file invalidates the "raison d'être" of the link, then we should have the foreign key + version stored on the pointing entity (example: a usage link , a validation based on the content of the file)

[jdcourcol]

one can have multiple file/path attached to the entity. In our current use case, I imagine that 1 is the primary distribution. The other are derived data and should be probably modelled as such (the distribution pointing to the entity not the other way around).

[mgeplf]

annotations are not part of the entity and should not. An annotation points to an entity. So one can add annotations , update annotations as much as we want, this is not "changing" the entity.

What if the annotation depends on the the metadata? For instance, there are annotations that apply to certain mtypes, if the mtype is mutable, that may invalidate the annotations that point to the entity.

For a morphology, my impression was that changing the distribution was a new version whereas changing the metadata was not.

This makes me think we aren't building a an entity service, but rather a versioned file store; but this goes back to what is considered metadata?

The other are derived data and should be probably modelled as such

Derived data is another requirement that needs to be defined and clarified, as it hasn't been discussed thus far.

[jdcourcol]

What if the annotation depends on the the metadata?
We considered that it is not a use case.

[mgeplf]

We considered that it is not a use case.

I think it would be worthwhile to write that sort of consideration and decision in documentation, so that is is widely known and understood.

Which part isn't a use case? Annotations being related to metadata or, that someone would change metadata that would invalidate annotations?

[mgeplf]

I wrote up what was discussed in the meeting we had:
#31

[dkeller9]

I think a new version or entity is needed every time the uploader wants to replace any essential fields of metadata (essential defined as being used in building, such as species or cell type) or data that is already in use by an experiment, unless they override this behavior. If the entity is not is use, then can just replace it without incident. If a new entity is made, perhaps a "deprecated" flag could be added to the old entity, as Gianluca suggested.

[mgeplf]

This ticket is too old, and the context has been forgotten.
I believe we are converging to a solution, perhaps in this #203
Close this in light of that