Add conditional restrictions to dictionary fields

[joneubank]

Implement conditional restrictions as described in: #199 Epic: Conditional Restrictions

This feature branch will collect the work for each part of the project as described in the linked Epic, once all parts are available it can be merged as a complete feature into the primary branch.

The outcome of this feature will be to have conditional restrictions available for fields in Lectern Dictionaries, and the script restrictions no longer available. This applies both to the dictionaries maintained by the lectern schema, and the validation implementation provided by the lectern client.

Breakdown

Task	Status	PR
Remove script restriction	In Progress

[edsu7]

Example uses greaterThanOrEqual.

But don't see this value in:
https://github.com/overture-stack/lectern/blob/da452d4dc016daa15738988d804547e47dfcc872/libraries/dictionary/README.md#comparedfieldsrule-data-structure

Also snakecase or camelcase?

[joneubank]

we want to consistently use camelCase for json properties. for constant values like "greaterThan" we could use upper snake case instead to indicate they are constant values: GREATER_THAN

I might've missed this property in the docs, thanks for pointing it out.

[edsu7]

This means that meta data cannot be nested or accept arrays.
Is this due to a technical limitation?

For PCGL currently this is the "workaround" ( a string mimicing an array of dictionary objects):
https://github.com/Pan-Canadian-Genome-Library/data-dictionary/blob/71fee8cc4cb2db40ec2e747f297a2ba9d2401e73/schemas/participant.json#L46C11-L46C148

[joneubank]

ok good feedback, string parsing is way worse... we were trying to keep this for simple usage and known property names... this shows you want to use it with any number of unknown properties. this can be changed I think.

[edsu7]

Would the following be an appropriate usage:

      {
        "name": "date_of_birth",
        "description": "Indicate participant's date of birth.",
        "valueType": "string",
        "meta": {
          "displayName": "Date Of Birth",
          "mappings": "All4One: Month,year of birth;mCODE STU3: Patient.birthDate;Phenopacket: Individual.date_of_birth;Beacon V2: Individual.date_of_birth",
          "source": "MOH"
        },
        "restrictions": {
          "if" : {
            "conditions": [
              {
                "fields": ["vital_status"],
                "match": {"value": "Deceased"}
              }
            ]
          },
          "then" : {
            "required": true,
            "regex": "#/regex/yyyy_mm"
          },
          "else" : {
            "empty" : true
          }
        }
      }

[joneubank]

Would the following be an appropriate usage:

@edsu7 looks semantically correct. The rules for this would be that when vital_status equals Deceased then this field (date_of_birth) must be provided (required) and must match the regex stored in the variable #/regex/yyyy_mm; otherwise (vital_status does not equal Deceased) this field must be empty.

That doesn't make sense for a field titled date_of_birth but assuming you meant a date_of_death then thats perfect.

[edsu7]

A couple of more examples - to mimic https://github.com/icgc-argo/argo-dictionary/blob/1a9783899c434c54b3585ac41a12f8099621c9eb/references/validationFunctions/chemotherapy/doseReduction.js
https://github.com/icgc-argo/argo-dictionary/blob/1a9783899c434c54b3585ac41a12f8099621c9eb/references/validationFunctions/chemotherapy/drugDose.js

{
  "name": "prescribed_cumulative_drug_dose",
  "description": "Indicate the total prescribed cumulative drug dose in the same units specified in chemotherapy_drug_dose_units.",
  "valueType": "number",
  "meta": {
    "displayName": "Prescribed Cumulative Drug Dose",
    "core": true,
  },
  "restrictions": {
    "if" : {
      "conditions": [
        {
          "fields": ["chemotherapy_drug_dose_units"],
          "match": {"exists":false]}
        },
        {
          "fields": ["actual_cumulative_drug_dose"],
          "match": {"exists":true]}
        }
      ],
      "case":"any"
    },
    "then" : {
      "empty": true
    },
    "else" : {
      "required": true
    }
  }
}

https://github.com/icgc-argo/argo-dictionary/blob/1a9783899c434c54b3585ac41a12f8099621c9eb/references/validationFunctions/biomarker/intervalOrId.js

{
  "name": "submitter_specimen_id",
  "description": "Unique identifier of the specimen, assigned by the data provider.",
  "valueType": "string",
  "meta": {
    "displayName": "Submitter Specimen Id",
    "core": true,
  },
  "restrictions": {
    "if" : {
      "conditions": [
        {
          "fields": ["submitter_primary_diagnosis_id","submitter_treatment_id","submitter_follow_up_id","test_interval"],
          "match": {"exists":true}
        }
      ],
      "case":"any"
    },
    "then" : {
      "empty": true
    },
    "else" : {
      "required": true,
      "regex": "^[A-Za-z0-9\-\._]{1,64}$"
    }
  }
}

[joneubank]

@edsu7 Replying to your examples. They seem to be mismatched with the links you gave so I'll interpret what you wrote and if that is what you expected then we're good!

1. If either of your conditions is found, then this field must be empty, otherwise it is required. So... if either chemotherapy_drug_dose_units has no value, OR actual_cumulative_drug_dose has a value, then prescribed_cumulative_drug_dose must not have a value. prescribed_cumulative_drug_dose is required whenever all those conditions are false (chemotherapy_drug_dose_units has a value and actual_cumulative_drug_dose has no value).

2. This restriction matches the second half of the script linked, the part that checks for the existence of an ID field. As you have written it, this condition is true when ALL the listed fields have a value. So this means if any of those are missing then this condition fails. This is because you have written "case":"any" in the if block, so it applies to the list of conditions, not the fields inside the one condition. It can be updated to match what you want by:

{
  "restrictions": {
    "if" : {
      "conditions": [
        {
          "fields": ["submitter_primary_diagnosis_id","submitter_treatment_id","submitter_follow_up_id","test_interval"],
          "match": {"exists":true},
          "case":"any"
        }
      ],
    },
    "then" : {
      "empty": true
    },
    "else" : {
      "required": true,
      "regex": "^[A-Za-z0-9\-\._]{1,64}$"
    }
  }
}

[joneubank]

Closing this PR as these changes were made before significant refactoring of the code base. The relevant changes will be replicated into a new feature branch based on the latest develop branch.

The proposed conditional restrictions spec has been committed to the main code base as part of the dictionary reference doc