Implemented and Tested Terminology Operations (#2719)

* Support for external terminology server * Fixed Issue with $export * Bug #1459: FIxed issue with $export error code when request type is Batch or Transaction. * Resources can now be validated against US Core * Added tests for $validate using external terminology service TODO: Need to make tests better. * Updated US-Core profiles in normative from STU3 -> R4. Wrote Tests for $Validate on a patient resource against us-core. * Improved added tests Added GenderIdentity USCORE-profile to normative Set "Definitions.zip" as "content" and "Copy Always" so that its path can be found Modified some pre-existing tests so that they account for profile validation. * Fixed appsettings * Removed unrelated bug-fix code for ExportNotSupportedInBundle * Validate tests pass on STU3 and R4, most pass on R5. Fixed issue with ZipSource which prevented defintion file summaries from being loaded. Removed last bit of code irrelevant work. * Added ValueSet-BirthSex to R4 and R5 Validate Tests passing for all versions * Cleaned up csproj for Stu3, R4, and R5 .web * Updated TestConfiguration.json to use new external terminology service endpoint * Changed endpoint back to hapi * Validate-Code operationHandler. OperationRequest, and OpeartionResponse. * ValueSet $Validate-Code is mostly working and testing is partially implemented using OntoServer as external TS * Added Testing for Get version of valueset $validate-code * Created Terminology Controller and profile validator now uses external terminology service by default * Implemented POST and GET $Validate-Code for both ValueSets and CodeSystems. Some E2E tests are implemented but more are to come. * Added more testing for $validate-code Implemented GET version of $lookup * $validate-code does not work for STU3 as the TS is on R4 Fixed error where some operations would return an error saying server could not read conformance statement. * Implemented $lookup and wrote E2E tests for said operation * Finished Implementing and testing $expand Cleaned up terminology related code. * Added logging when definitions folder is not found or external/fallback terminology service cannot be created Added ProfileValidationTerminologServer endpoint in the test configuration file * Fixed Test names in TerminologyOperationTests * Final code clean up before submitting PR Terminology related tests will be skipped if there is no external endpoint set. * Removed unused variable * Fixed bug where tests were failing because CreateAsync was using PUT request instead of POST * Added: Terminology Operation Filters for TerminologyControlle, Terminology Operations to Capability Statement Fixed: Batch tests failing because $lookup was assuming to not work, Tests failing due to InProcTestServer only available in local environment Improved code standard to fit with the rest of FHIR server codebase * Fixed Capability Statement * Added temporary operatioDefinition json files. * Added Operation definition for terminology operations. * Fixed problem with testConfiguration settings not being updated. Fixed small bug in TerminologyOperator * Removed code in CheckResults method from Terminology Operator that retried terminology operation if the error was related to conformance statement read issues. Fixed issue in bundle-batch that assumed $lookup was not implemented. * Added checks in terminology controller to make sure operation that is to be executed is enabled in appsettings. Added separate testconfiguration.json for each version of FHIR. Placed profiler resolver in a try catch if the profile validation terminology endpoint is empty * Removed old testconfiguration.json file that was causing tests to not be skipped when necessary. * Updated test configuration path in provision-deploy.yml Added terminology module logging messages to API.Resources.resx * Added break out of for each look once bool has been set to true in Lookup and Expand parameters filter. More code clean up * Fixed package.yml to account for separate, version specific, testconfiguration files. * Trying to fix build & deploy for test configuration * Corrected HAPI endpoint for STU3 in testconfiguration.json Corrected validate and terminology tests where they would break out of for each loop early. * Fixed tests that were not working correctly. * Fixed comment typo * Fixed Typos * Fixed test name typo. * $validate was already being added to the capability statement somewhere else within the FHIR service, so I removed it from where I was adding the terminology operations. Also fixed typo. * Added Terminology Operations Documentation. * Update TerminologyOperations.md * Update TerminologyOperations.md Co-authored-by: Jared Erwin <[email protected]>
microsoft · Sep 27, 2022 · f431f72 · f431f72
1 parent db9bdc6
commit f431f72
Show file tree

Hide file tree

Showing 83 changed files with 3,345 additions and 142 deletions.
diff --git a/build/jobs/package.yml b/build/jobs/package.yml
@@ -48,9 +48,23 @@ steps:
       artifactType: 'container'
 
   - task: PublishBuildArtifacts@1
-    displayName: 'publish test configuration jsons'
+    displayName: 'publish Stu3 test configuration jsons'
     inputs:
-      pathToPublish: './test/Configuration/'
+      pathToPublish: './test/Microsoft.Health.Fhir.Stu3.Tests.E2E/testconfiguration.json'
+      artifactName: 'deploy'
+      artifactType: 'container'
+
+  - task: PublishBuildArtifacts@1
+    displayName: 'publish R4 test configuration jsons'
+    inputs:
+      pathToPublish: './test/Microsoft.Health.Fhir.R4.Tests.E2E/testconfiguration.json'
+      artifactName: 'deploy'
+      artifactType: 'container'
+
+  - task: PublishBuildArtifacts@1
+    displayName: 'publish R5 test configuration jsons'
+    inputs:
+      pathToPublish: './test/Microsoft.Health.Fhir.R5.Tests.E2E/testconfiguration.json'
       artifactName: 'deploy'
       artifactType: 'container'
 

diff --git a/build/jobs/provision-deploy.yml b/build/jobs/provision-deploy.yml
@@ -46,8 +46,8 @@ jobs:
       ScriptType: inlineScript
       Inline: |
         Add-Type -AssemblyName System.Web
-     
-        $deployPath = "$(System.DefaultWorkingDirectory)/test/Configuration"
+
+        $deployPath = "$(System.DefaultWorkingDirectory)/test/Microsoft.Health.Fhir.${{parameters.version}}.Tests.E2E"
      
         $testConfig = (ConvertFrom-Json (Get-Content -Raw "$deployPath/testconfiguration.json"))
         $flattenedTestConfig = $(System.DefaultWorkingDirectory)/release/scripts/PowerShell/ConvertTo-FlattenedConfigurationHashtable.ps1 -InputObject $testConfig

diff --git a/docs/TerminologyOperations.md b/docs/TerminologyOperations.md
@@ -0,0 +1,146 @@
+# Connecting to External Terminology Service on Azure
+
+In order to run a terminology service, you must ensure that the correct version of the FHIR [definitions folder](http://hl7.org/fhir/R4/downloads.html "definitions folder") is loaded into your FHIR server. The Definitions should be loaded already with the Azure FHIR OSS repository, but may not be up to date and may have to be updated manually (Make sure definition version matches FHIR version). Data may have to be cleaned up as stated further down below.
+
+## Settings Up Configuration Settings
+
+[Here](https://confluence.hl7.org/display/FHIR/Public+Test+Servers) is a list of public test servers, some of which have terminology functionality.
+
+**Developer**
+
+As of date, there is no Microsoft Terminology Service; thus, we need to connect the FHIR server to an external terminology service. To do so, visit the [`appsettings.json`](https://github.com/microsoft/fhir-server/blob/feature/terminologyservice/src/Microsoft.Health.Fhir.Shared.Web/appsettings.json) file in the FHIR server repo and configure an external terminology service by adding an endpoint for ExternalTerminologyService and ProfileValidationTerminologyService.
+
+You may also need to enable each terminology operation in the `appsettings.json` file.
+
+**User**
+
+Open your FHIR Server resource through [Azure Portal](https://ms.portal.azure.com/#home) and under 'Settings' click on 'Configuration.'
+
+From there, find 'FhirServer__Operations__Terminology__ExternalTerminologyServer' and set to equal your terminology server endpoint.
+
+Do the same for 'FhirServer__Operations__Terminology__ProfileValidationTerminologyServer.'
+
+From there you can choose which terminology operations you want to enable through 'FhirServer__Operations__Terminology__ValidateCodeEnabled,' 'FhirServer__Operations__Terminology__LookupEnabled,' and 'FhirServer__Operations__Terminology__ExpandEnabled' by setting the value to true.
+
+
+## $validate FHIR resource
+[FHIR Documentation on validate](https://www.hl7.org/fhir/validation.html)
+
+[Microsoft Documentation on validate](https://docs.microsoft.com/en-us/azure/healthcare-apis/fhir/validation-against-profiles)
+
+
+To validate a FHIR resource, you must provide a profile that you can validate the resource against. 
+
+Specifically, if you are looking to validate against US Core Profiles, you should load those specific profiles to your FHIR server. You can find these profiles on the FHIR specification [here](http://hl7.org/fhir/us/core/history.html). To save storage, you can also just load the profiles that you are trying to validate against instead of loading all US Core profiles. Data may have to be cleaned and bundled up as stated further down below.
+
+Then, run a `POST` request to validate the FHIR resource with the following command against your FHIR server: `https://{LocalHost}/{ResourceType}/$validate?profile={USCoreProfile}` (Profile paramater is optional). The body of the request must contain the resource you are trying to validate.
+
+## $validate-code 
+[FHIR Documentation on validate-code for ValueSet](https://www.hl7.org/fhir/valueset-operation-validate-code.html)
+
+[FHIR Documentation on validate-code for CodeSystem](https://www.hl7.org/fhir/codesystem-operation-validate-code.html)
+
+There are currently two ways to validate a code that is in a ValueSet or CodeSystem.
+
+You can use a `GET` request to validate a code that is part of a ValueSet or CodeSystem that is already loaded into a FHIR server: `https://{LocalHost}/{ValueSet or CodeSystem}/{ResourceID}/$validate-code?system={system}&code={code}&display={optionalDisplay}`
+
+If the ValueSet or CodeSystem is not in your FHIR server, then you can use a `POST` request to validate the code: `https://{LocalHost}/{ValueSet or CodeSystem}/$validate-code` and the body of the request must contain a parameters resource with a code/coding along with a ValueSet or CodeSystem.
+
+## $expand
+[FHIR Documentation on expand](https://www.hl7.org/fhir/valueset-operation-expand.html)
+
+To get the expansion of a ValueSet, you must have a ValueSet in your FHIR server or you must provide the URL for the ValueSet to be accessed. 
+
+In the case that the ValueSet is already in the FHIR server you can use the following `GET` request:
+`https://{LocalHost}/ValueSet/{ResourceID}/$expand`
+
+If you wish to expand a ValueSet using a canonical URL, you must provide it in the URL parameter.
+`https://{LocalHost}/ValueSet/$expand?url={canonical ULR}`
+
+You can also use a `POST` request to expand a ValueSet where the body is a ValueSet resource:
+`https://{LocalHost}/ValueSet/$expand?`
+
+These requests can have "offset" and "count" as optional query parameters.
+
+## $lookup
+[FHIR Documentation on lookup](https://www.hl7.org/fhir/codesystem-operation-lookup.html)
+
+If looking up a code using a `GET` request, you must provide the system and code in the query parameters:
+`https://{LocalHost}/CodeSystem/$lookup?system={system}&code={code}`
+
+You can also use a `POST` request to look up a code by providing a parameters resource that contains a coding in the body of the request:
+`https://{LocalHost}/CodeSystem/$lookup`
+
+## Data Clean up
+
+Some of the FHIR resources downloaded from the FHIR specifications and US-Core Profile may require some clean up. It is suggested to clean the div element of the FHIR resources.
+
+Regex can be helpful here: 
+
+find     - "div": `<div (.*)>(.*)</div>`
+
+replace  - "div": `<div>PlaceHolder</div>`
+
+## Bundling US-Core Profiles
+'''
+
+        using Hl7.Fhir.Model;
+
+        using Hl7.Fhir.Serialization;
+
+        using System.Runtime.CompilerServices;
+
+
+        var parser = new FhirJsonParser();
+
+        var bundle = new Bundle();
+        bundle.Type = Bundle.BundleType.Batch;
+
+        foreach (var file in Directory.GetFiles(@"Location of US-Core Profiles", "*.json"))
+        {
+            var fileName = Path.GetFileName(file);
+            var resourceType = fileName.Split('-')[0];
+            Resource resource = null;
+            string url = null;
+            switch (resourceType)
+            {
+                case "SearchParameter":
+                    resource = parser.Parse<SearchParameter>(File.ReadAllText(file));
+                    url = ((SearchParameter)resource).Url;
+                    break;
+                case "StructureDefinition":
+                    resource = parser.Parse<StructureDefinition>(File.ReadAllText(file));
+                    url = ((StructureDefinition)resource).Url;
+                    break;
+                case "ValueSet":
+                    resource = parser.Parse<ValueSet>(File.ReadAllText(file));
+                    url = ((ValueSet)resource).Url;
+                    break;
+                default:
+                    break;
+            }
+
+            if (resource != null)
+            {
+                bundle.Entry.Add(Utils.CreateBundleEntry(resource, url));
+            }
+        }
+
+        var serializer = new FhirJsonSerializer();
+
+        File.WriteAllText(@"outputFile.json", serializer.SerializeToString(bundle));
+
+        return 0;
+
+        public class Utils
+        {
+            public static Bundle.EntryComponent CreateBundleEntry(Resource resource, string url)
+            {
+                Bundle.EntryComponent entry = new Bundle.EntryComponent();
+                entry.Request = new Bundle.RequestComponent() { Method = Bundle.HTTPVerb.POST, Url = resource.TypeName };
+                entry.FullUrl = url;
+                entry.Resource = resource;
+                return entry;
+            }
+        }
+'''
diff --git a/src/Microsoft.Health.Fhir.Api/Features/Filters/ExpandParametersFilter.cs b/src/Microsoft.Health.Fhir.Api/Features/Filters/ExpandParametersFilter.cs
@@ -0,0 +1,53 @@
+// -------------------------------------------------------------------------------------------------
+// Copyright (c) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License (MIT). See LICENSE in the repo root for license information.
+// -------------------------------------------------------------------------------------------------
+
+using System;
+using EnsureThat;
+using Hl7.Fhir.Model;
+using Microsoft.AspNetCore.Mvc.Filters;
+using Microsoft.Health.Fhir.Core.Exceptions;
+
+namespace Microsoft.Health.Fhir.Api.Features.Filters
+{
+    /// <summary>
+    /// Validate that the deserialized request body object is of type Hl7.Fhir.Model.Parameters.
+    /// </summary>
+    [AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
+    public sealed class ExpandParametersFilter : ActionFilterAttribute
+    {
+        public override void OnActionExecuting(ActionExecutingContext context)
+        {
+            EnsureArg.IsNotNull(context, nameof(context));
+
+            context.ActionArguments.TryGetValue("parameters", out var parameters);
+            context.ActionArguments.TryGetValue("idParameter", out var idParameter);
+            context.ActionArguments.TryGetValue("url", out var url);
+
+            bool hasValueSet = false;
+
+            if (parameters != null)
+            {
+                foreach (var paramComponent in ((Parameters)parameters).Parameter)
+                {
+                    if (string.Equals(paramComponent.Name, "valueSet", StringComparison.OrdinalIgnoreCase))
+                    {
+                        hasValueSet = true;
+                        break;
+                    }
+                }
+
+                if (!hasValueSet)
+                {
+                    throw new RequestNotValidException(Resources.ExpandMissingValueSetParameterComponent);
+                }
+            }
+
+            if (!((idParameter != null) ^ (url != null)) && parameters == null)
+            {
+                throw new RequestNotValidException(Resources.ExpandInvalidIdParamterXORUrl);
+            }
+        }
+    }
+}
diff --git a/src/Microsoft.Health.Fhir.Api/Features/Filters/LookupParametersFilter.cs b/src/Microsoft.Health.Fhir.Api/Features/Filters/LookupParametersFilter.cs
@@ -0,0 +1,53 @@
+// -------------------------------------------------------------------------------------------------
+// Copyright (c) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License (MIT). See LICENSE in the repo root for license information.
+// -------------------------------------------------------------------------------------------------
+
+using System;
+using EnsureThat;
+using Hl7.Fhir.Model;
+using Microsoft.AspNetCore.Mvc.Filters;
+using Microsoft.Health.Fhir.Core.Exceptions;
+
+namespace Microsoft.Health.Fhir.Api.Features.Filters
+{
+    /// <summary>
+    /// Validate that the deserialized request body object is of type Hl7.Fhir.Model.Parameters.
+    /// </summary>
+    [AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
+    public sealed class LookupParametersFilter : ActionFilterAttribute
+    {
+        public override void OnActionExecuting(ActionExecutingContext context)
+        {
+            EnsureArg.IsNotNull(context, nameof(context));
+
+            context.ActionArguments.TryGetValue("code", out var code);
+            context.ActionArguments.TryGetValue("system", out var system);
+            context.ActionArguments.TryGetValue("parameters", out var parameters);
+
+            if ((string.IsNullOrEmpty((string)code) || string.IsNullOrEmpty((string)system)) && parameters == null)
+            {
+                throw new RequestNotValidException(Resources.LookupInvalidMissingSystemOrCode);
+            }
+
+            bool hasCoding = false;
+
+            if (parameters != null)
+            {
+                foreach (var paramComponent in ((Parameters)parameters).Parameter)
+                {
+                    if (string.Equals(paramComponent.Name, "coding", StringComparison.OrdinalIgnoreCase))
+                    {
+                        hasCoding = true;
+                        break;
+                    }
+                }
+
+                if (!hasCoding)
+                {
+                    throw new RequestNotValidException(Resources.ParameterMissingCoding);
+                }
+            }
+        }
+    }
+}
diff --git a/src/Microsoft.Health.Fhir.Api/Features/Filters/ValidateCodeParametersFilter.cs b/src/Microsoft.Health.Fhir.Api/Features/Filters/ValidateCodeParametersFilter.cs
@@ -0,0 +1,80 @@
+// -------------------------------------------------------------------------------------------------
+// Copyright (c) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License (MIT). See LICENSE in the repo root for license information.
+// -------------------------------------------------------------------------------------------------
+
+using System;
+using EnsureThat;
+using Hl7.Fhir.Model;
+using Microsoft.AspNetCore.Mvc.Filters;
+using Microsoft.Health.Fhir.Core.Exceptions;
+
+namespace Microsoft.Health.Fhir.Api.Features.Filters
+{
+    /// <summary>
+    /// Validate that the deserialized request body object is of type Hl7.Fhir.Model.Parameters.
+    /// </summary>
+    [AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
+    public sealed class ValidateCodeParametersFilter : ActionFilterAttribute
+    {
+        public override void OnActionExecuting(ActionExecutingContext context)
+        {
+            EnsureArg.IsNotNull(context, nameof(context));
+
+            bool typeParameterExists = context.ActionArguments.TryGetValue("typeParameter", out var typeParameter);
+            context.ActionArguments.TryGetValue("code", out var code);
+            context.ActionArguments.TryGetValue("system", out var system);
+            context.ActionArguments.TryGetValue("parameters", out var parameters);
+
+            if ((typeParameterExists &&
+                !(string.Equals(typeParameter.ToString(), "CodeSystem", StringComparison.OrdinalIgnoreCase) || string.Equals(typeParameter.ToString(), "ValueSet", StringComparison.OrdinalIgnoreCase))) &&
+                parameters == null)
+            {
+                throw new RequestNotValidException(Resources.ValidateCodeInvalidResourceType);
+            }
+
+            if ((string.IsNullOrEmpty((string)code) || string.IsNullOrEmpty((string)system)) && parameters == null)
+            {
+                throw new RequestNotValidException(Resources.ValidateCodeMissingSystemOrCode);
+            }
+
+            bool hasCoding = false;
+            bool hasValueSetOrCodeSystem = false;
+
+            if (parameters != null)
+            {
+                if (((Parameters)parameters).Parameter.Count != 2)
+                {
+                    throw new RequestNotValidException(Resources.ValidateCodeInvalidParemeters);
+                }
+
+                foreach (var paramComponent in ((Parameters)parameters).Parameter)
+                {
+                    if (string.Equals(paramComponent.Name, "valueSet", StringComparison.OrdinalIgnoreCase))
+                    {
+                        hasValueSetOrCodeSystem = true;
+                    }
+                    else if (string.Equals(paramComponent.Name, "codeSystem", StringComparison.OrdinalIgnoreCase))
+                    {
+                        hasValueSetOrCodeSystem = true;
+                    }
+
+                    if (string.Equals(paramComponent.Name, "coding", StringComparison.OrdinalIgnoreCase))
+                    {
+                        hasCoding = true;
+                    }
+                }
+
+                if (!hasCoding)
+                {
+                    throw new RequestNotValidException(Resources.ParameterMissingCoding);
+                }
+
+                if (!hasValueSetOrCodeSystem)
+                {
+                    throw new RequestNotValidException(Resources.ValidateCodeInvalidResourceType);
+                }
+            }
+        }
+    }
+}
diff --git a/src/Microsoft.Health.Fhir.Api/Features/Routing/KnownRoutes.cs b/src/Microsoft.Health.Fhir.Api/Features/Routing/KnownRoutes.cs
@@ -43,6 +43,19 @@ internal class KnownRoutes
         public const string ValidateResourceType = ResourceType + "/" + Validate;
         public const string ValidateResourceTypeById = ResourceTypeById + "/" + Validate;
 
+        public const string ValidateCode = "$validate-code";
+        public const string ValidateCodeGET = ResourceType + "/" + IdRouteSegment + "/" + ValidateCode;
+        public const string ValidateCodePOST = ResourceType + "/" + ValidateCode;
+        public const string ValidateCodeDefinition = OperationDefinition + "/" + OperationsConstants.ValidateCode;
+
+        public const string LookUp = "CodeSystem" + "/" + "$lookup";
+        public const string LookUpDefinition = OperationDefinition + "/" + OperationsConstants.Lookup;
+
+        public const string Expand = "$expand";
+        public const string ExpandWithId = "ValueSet" + "/" + IdRouteSegment + "/" + Expand;
+        public const string ExpandWithoutId = "ValueSet" + "/" + Expand;
+        public const string ExpandDefinition = OperationDefinition + "/" + OperationsConstants.Expand;
+
         public const string Reindex = "$reindex";
         public const string ReindexSingleResource = ResourceTypeById + "/" + Reindex;
         public const string ReindexJobLocation = OperationsConstants.Operations + "/" + OperationsConstants.Reindex + "/" + IdRouteSegment;