Beta Release 2.0.0-beta.1 #225
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
points to lectern-js-client repo
* DictionaryMeta now recursive to accomodate nested meta data * Rename x->key in type definition * Typo correction in comment Co-authored-by: Anders Richardsson <[email protected]> * Fix Create Dictionary test fixtures to match new meta rules --------- Co-authored-by: Anders Richardsson <[email protected]>
* Create client package and copy files from js-lectern-client * Use rimraf in build scripts for cross platform compatibility * Target ESNext for internal library builds * Update copyright year of copied client files to 2024 * Remove -rf options from rimraf * Move lectern-client into @overture-stack org * Add mocha tests config file * Ensure we are using `===` not `==` * Remove `baseUrl` from tsconfig
* Collect client errors into common, order alphabetically * Create Singular<T> utility type to extract element types from array - helps simply type notation when processing DataRecords in client * Make a default RangeRestriction schema validator available * Update diff type FieldChange to potentially be undefined * Remove namespaces, error types moved to common * Refactor changeAnalysis with lectern shared types * Remove worker-threads code from client The client provides the logic for processing and validating data, any parallelization that is desired should be added to a server implementation and not in the reusable client. * Remove types that are duplicate of Lectern dictionary library * Move data processing functions out of schema-functions into their own directory * Reorganize validation and error message code * Move rest client code and update to Lectern shared types * Add Lectern dictionary library, remove worker-threads * Update client entry file for new code directories * Organize tests and fixtures with clean names Also updated messages in tests that have changed during code cleanup * updated lockfile for client package.json changes * Add standard copyright notice * Remove uneccessary comments left from refactor
* Create validation library * Move Data Record types to lectern-dictionary * Add regexp for boolean values for use in type validation * Move Validation functions into shared module * Convenience scripts for test/build for each sub-module * Adding references to validaiton library * Correct version regexp, add comments for string validations * Move validation pipelines alongside client processing code
- Version String regexp now correctly checks for . character instead of any character
There was no need to maintain two distinct paths for shared packages, they are now all collected as either packages or apps. All references to libraries have been removed in documentation and configs.
…215) * Move restrictions and data value types to dedicated files * Add test folders to TS path but exclude from build * Undefined checks where unchecked index access was done * Rename rangeToSymbol as rangeToText, change text priority to describe more restrictive conditions * Range checking utiltiy function with tests It should be mentioned here that the previously committed version of this had a mix up with the exclusive conditions, where the edge cases (value equal to the edge of the range) was incorrectly valdiated in the version imported from js-lectern-client. This is fixed now with tests to check these cases. * Generalize isArray type predicates, move isDefined to common * Add test folders to TS path but exclude from build * FieldRestriction type map and more granular codeList types * Move shared type definition to commonTypes * Use updated restriciton and name types in dictionary defs * Declare separate types for singular and array data record values * Code autoformatting * First pass types for reporting field resriction validation errors * Utility to convert a restriction test on a single value to apply to a whole array * Rewrite restriction validation functions to apply to single fields instead of data records * Simplify type selection * Fix edge case tests and work for isWithinRange * Documentation on process for refactored fieldValidation process * Fix generate script organization within monorepo * Refactor fieldValidation to handle each field individually Original implementation had record processing logic built into the restriction test logic. This refactor sepearates restriction tests to apply to a specific value only, based on the defined restriction rule. This will enable future development to resolve the restrictions on each field based on conditional restrictions and then test based on the resolved conditions. The next step is to add validation testing for full data records, full entity data sets, and then full dictionary data sets. These each loop over the previous element but also introduce new tests. For example, entity data sets enforce unique field and uniqueKey restrictions. Dictionary datasets enforce foreignKey restrictions. Unit tests have been added to fieldValidation function, and to the individual restriction tests. The restriction tests check the detailed set of values vs restriction rules, while the fieldValidation testing is checking that these restrictions are enforced based on schemaField definitions. Note that script validations are not being run in this code, and that the unique constraint cannot be applied to a field. There is additional changes required to move the unique constraint to a field level property (like isArray) so that it can't be affected by conditional restrictions. It will be validated in the Schema level validation (entity dataset). * WIP Documentation for Important Concepts * Move field validation code to `/validateField` * Add validateRecord functionality with basic documentation * Move validateField tests to directory matching source directory name * Tests for validateRecord * Editing TSDoc for validation type utils * Complete pnpm monorepo registration for scripts module * Infer predicates instead of declaring where possible * Create function `validateSchema` to validate a list of records with single schema Applies the `validateRecord` validation to every record, plus runs validations that require knowledge of the entire set of records for this schema: - unique - uniqueKey * Add copyright declaration to top of files * Clean regex test error message by removing the rule * Update test description text for rangeTest to match other tests * Test for validateSchema valid and unique restriction - fixes bug in testUniqueFieldRestriction that was using the wrong field name when generating the hash - adds DataSetHashMap type alias that was missing in previous commit - adds test utility to validate that test fixtures are valid vs the Lectern zod schemas * Add tests for uniqueKey restriction in validateSchema * Add tests for validateSchema uniqueKey with undefined values * Add tests for validateSchema to ensure it tests record and field restrictions * Update uniqueKey invalid test to check error detailed properties * Organize imports * Add `validateDictionary` function plus related tests and fixtures - applies foreign key restrictions - detects unrecognized schemas - collects foreign key errors with other record errors * Standardize test fixture validation message text * Remove range rule from test failure message * Add period to required test failure message * Move `resolveFieldRestrictions` function to its own file * Rename foreign key type and schema to `ForeignKeyRestriction` * Rename testField/restriction specs to match source file names * Remove empty spec files * Add copyright text to test fixtures * Result type can have Failure data, remove `Either` type * Add convertValue functionality - functions for converting values for field, record, schema, and dictionary - update to validation error type names to be shared with convert value types - includes test specs for each convert function exported * Fix simple typos * Update client processing functions to use new validation functions * Remove validation tests from client that are now done in validation package * Unique restriction rules enforced for array fields * Remove deprecated types and test implementations from legacy client * Remove old data conversion implementation from client processing * Remove old data valdiation pipeline from client processing * Remove tests around non-standard default value setting * Test fixtures for unique restrictions with arrays * Update property names in RecordError and TestResult types (#216) * Change `FieldDetails` property `value` to `fieldValue` * Change TestResult invalid property to be `details` * Rename convert functions as parse (#219)
* Combine common package with dictionary * Rename dictionary package to `@overture-stack/lectern-dictionary` - Updates all imports of the dictionary package - Update root README to reflect this change - Runs `pnpm update` to make use latest version of all dependencies - dependency udpate was required to get all packages onto the same version of zod and work with the zod-to-json-schema utility in the generate script - Relocate scripts to a single package in the `/script` dir. - Adds README to scripts package to describe available scripts - Designed to have a single scripts package with multiple utilities instead of separate package per script * Update dependency versions in validation package * Update lectern server package name in Dockerfile * Adds processSchema usage example to Client README * Updates Dictionary README to match packages format * Rename processing Success types to not say Error
… meta schemas and implement validation logic (#223) * Remove script restrictions; Move unique property out of restrictions These are breaking changes to the meta schema that need to be published as a major release. A document should be added to the repo describing these changes. * Field restrictions can be an array of restriction objects * Add tests for field restrictions of all forms * Test validation library uses all restrictions in array * Add document to detail major version changes * Adds conditional restrictions to dictionary restriction schema - performs code list and regex reference replacement recursively through conditional restrictions - performs reference replacement recursively through meta objects - applies conditional restriction checks when resolving restriction rules for each field - WIP: still requires many tests * Tests for references in meta and regex arrays * ConditionalRestrictionTest has optional case with proper default in validation * Single match object per condition, instead of array * Fix recursive conditional restriction parsing and add tests * Named container and volume in compose file * Fix import path * Empty field restriction validation tests * Remove development test file * Remove script restriction from test dictionary * Restriction schemas directly written for each field type - Generic conditional restriction function was removed because it could not be interpretted by the json schema generator - Although there is repeated code, directly writing the typed conditional restriction type schemas is easier to parse and hopefully maintain. They have their types enforced by a generic type even if the schema itself is not generated through a function. * Adds tests for conditional restriction match rules * Generated JSON Schema with Conditional Restrictions * Remove TODO statements that are not needed * Fix broken table in reference doc * Code cleanup by removing unused functions and comments
* Strictly define package exported functions, types, and files * Standardize package versions to `2.0.0-beta.1` * READMEs cleaned up for NPM publishing * Rename exported function variables to have same grammatic case * Remove redundancy in lectern-server description * README typos and clarifications * correct TypeScript casing typos * Remove unused imports, clean TSDoc comment blocks * Correcting typos and grammatical errors in documentation --------- Co-authored-by: Anders Richardsson <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Features
@overture-stackon npm