Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Beta Release 2.0.0-beta.1 #225

Merged
merged 28 commits into from
Aug 30, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
438a04b
update Jenkinsfile for cumulus migration (#196)
yalturmes Feb 8, 2024
0eaafe2
Update README.md (#197)
ciaranschutte Mar 1, 2024
cb0ed96
`.env` file location updates for monorepo
joneubank Mar 26, 2024
5c2fd7b
Meta properties can be nested (#201)
joneubank May 21, 2024
96fd13c
Add Lectern Client to monorepo (#208)
joneubank Jun 11, 2024
1ff4017
Refactor Client package to use shared Lectern types (#211)
joneubank Jun 14, 2024
c783f64
Move data validation functions into shared module (#212)
joneubank Jun 15, 2024
d3f55aa
Update generated dictionary JSON Schema
joneubank Jun 15, 2024
49930fd
Always build dictionary before generating json schema
joneubank Jun 15, 2024
bd4aac2
Auto-formatting correcting whitespace
joneubank Jun 15, 2024
8228b5f
Tests for version string (major.minor format)
joneubank Jun 15, 2024
f1753c2
Add missing copyright text
joneubank Jun 15, 2024
2a71623
Remove unused parameter
joneubank Jun 15, 2024
83815c0
Move all libraries into packages
joneubank Jun 18, 2024
6dc405b
Corrected a word: two categories
joneubank Jun 18, 2024
6784f6c
Add description and code use examples
joneubank Jun 18, 2024
b56d934
Use badges for published repository locations
joneubank Jun 18, 2024
d2f0a94
Remove unused deep-freeze dependency
joneubank Jun 19, 2024
3b7cb3c
TS minmum up to 5.5, upgrade deprecated rimraf
joneubank Jun 21, 2024
2c06589
Dependency version updates
joneubank Jun 22, 2024
1dee40a
Refactor Validation code in preparation for Conditional Restrictions …
joneubank Jul 27, 2024
9e74501
Add copyright header text
joneubank Jul 31, 2024
b484be4
TS comments for functions rangeToText and isValidValueType
joneubank Jul 31, 2024
29d8709
Consolidate Common and Dictionary packages (#220)
joneubank Aug 1, 2024
49c8f00
Conditional Restrictions - Add conditional restriction logic to field…
joneubank Aug 19, 2024
7169764
Update dictionary-reference to show unique as field property
joneubank Aug 22, 2024
97f1e9b
Cleanup for Beta-version Client Release (#224)
joneubank Aug 30, 2024
fc4fd77
Merge branch 'main' into develop
joneubank Aug 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@

# Allow select paths
!apps/*
!libraries/*
!packages/*
!package.json
!pnpm-lock.yaml
Expand Down
3 changes: 3 additions & 0 deletions Jenkinsfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
@Library(value='jenkins-pipeline-library@master', changelog=false) _
pipelineOVERTURELectern()

196 changes: 0 additions & 196 deletions Jenkinsfile.groovy

This file was deleted.

63 changes: 36 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Lectern - Data Dictionary Management and Validation

[<img hspace="5" src="https://img.shields.io/badge/chat--with--developers-slack-blue?style=for-the-badge">](http://slack.overture.bio)
[<img hspace="5" src="https://img.shields.io/badge/chat--with--developers-overture--slack-blue?style=for-the-badge">](http://slack.overture.bio)
[<img hspace="5" src="https://img.shields.io/badge/License-AGPL--3.0-blue?style=for-the-badge">](https://github.com/overture-stack/lectern/blob/develop/LICENSE)
[<img hspace="5" src="https://img.shields.io/badge/Code%20of%20Conduct-2.1-blue?style=for-the-badge">](CODE_OF_CONDUCT.md)
[<img hspace="5" src="https://img.shields.io/badge/Code%20of%20Conduct-blue?style=for-the-badge">](CODE_OF_CONDUCT.md)


Lectern is Overture's Data Dictionary Schema Manager, providing a system for defining Schemas that will validate the structured data collected by an application. The core of Lectern is a web-server application that handles storage and version management of data dictionaries. Lectern data dictionaries are collections of schemas that define the structure of tabular data files (like TSV). This application provides functionality to validate the structure of data dictionaries, maintain a list of dictionary versions, and to compute the difference between dictionary versions.
Expand All @@ -23,45 +23,58 @@ The repository is organized with the following directory structure:
.
├── apps/
│ └── server
├── libraries/
│ ├── common
│ └── dictionary
└── packages/
└── client
├── client
├── common
├── dictionary
└── validation
```

The modules in the monorepo are organized into three categories:
The modules in the monorepo are organized into two categories:

* __apps/__ - Standalone processes meant to be run. These are published to [ghcr.io](https://ghcr.io) as container images.
* __libraries/__ - Interal modules shared between other apps, libraries, and packages.
* __packages/__ - Packages published to [NPM](https://npmjs.com) meant to be imported into other TypeScript applications.
* __packages/__ - Reusable packages shared between applications and other packages. Packages are published to [NPM](https://npmjs.com).
* __scripts__ - Utility scripts for use within this repo.

| Component | Type | Package Name | Path | Published Location | Description |
| ----------------------------------- | ----------- | ------------------------------ | --------------------- | ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| [Lectern Server](apps/server/README.md) | Application | server | apps/server/ | [GHCR](https://github.com/overture-stack/lectern/pkgs/container/lectern) | Lectern Server web application. |
| [Lectern Client](packages/client/README.md) | Package | @overture-stack/lectern-client | packages/client | [NPM](https://www.npmjs.com/package/@overturebio-stack/lectern-client) | TypeScript Client to interact with Lectern Server and perform data validation. |
| [common](libraries/common/README.md) | Library | common | libraries/common/ | N/A | Non-specific but commonly reusable utilities. Includes shared Error classes. |
| [dictionary](libraries/dictionary/README.md) | Library | dictionary | libraries/dictionary/ | N/A | Dictionary meta-schema definition, includes TS types, and Zod schemas. This also exports all utilities for getting the diff of two dictionaries, and for validating data records with a Dictionary. |
| Component | Package Name | Path | Published Location | Description |
| --------------------------------------------------- | ---------------------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Lectern Server](apps/server/README.md) | @overture-stack/lectern-server | apps/server/ | [![Lectern GHCR Packages](https://img.shields.io/badge/GHCR-lectern-brightgreen?style=for-the-badge&logo=github)](https://github.com/overture-stack/lectern/pkgs/container/lectern) | Lectern Server web application. |
| [Lectern Client](packages/client/README.md) | @overture-stack/lectern-client | packages/client | [![Lectern Client NPM Package](https://img.shields.io/npm/v/@overture-stack/lectern-client?color=%23cb3837&style=for-the-badge&logo=npm)](https://www.npmjs.com/package/@overture-stack/lectern-client) | TypeScript Client to interact with Lectern Server and Lectern data dictionaries. This library provides a REST client to assist in fetching data from the Lectern server. It also exposes the functionality from the Lectern Validation library to use a Lectern data dictionary to validate data. |
| [Lectern Dictionary](packages/dictionary/README.md) | | @overture-stack/lectern-dictionary | [![Lectern Client NPM Package](https://img.shields.io/npm/v/@overture-stack/lectern-dictionary?color=%23cb3837&style=for-the-badge&logo=npm)](https://www.npmjs.com/package/@overture-stack/lectern-dictionary) | Dictionary meta-schema definition, includes TS types, and Zod schemas. This also exports all utilities for getting the diff of two dictionaries. |
| [Lectern Validation](packages/validation/README.md) | @overture-stack/lectern-validation | packages/validation/ | [![Lectern Validation NPM Package](https://img.shields.io/npm/v/@overture-stack/lectern-validation?color=%23cb3837&style=for-the-badge&logo=npm)](https://www.npmjs.com/package/@overture-stack/lectern-client) | Validate data using Lectern Dictionaries. |

## Developer Instructions

You can install all dependencies for the entire repo from the root (as defined the `pnpm-lock.yaml`) with the command:

`pnpm install`

Using `nx` will ensure all local dependencies are built, in the correct sequence, when building, running, or testing any of the applications and packages in the repo. To run a package.json script from any module - after installing dependencies - use a command of the form `pnpm nx <script> <package name>`. For example, to `build` the module `server` can be done with the command:
Using `nx` will ensure all local dependencies are built, in the correct sequence, when building, running, or testing any of the applications and packages in the repo. To run a package.json script from any module - after installing dependencies - use a command of the form `pnpm nx <script> <package name>`. For example, to `build` the server module `@overture-stack/lectern-server` can be done with the command:

`pnpm nx build server`
`pnpm nx build @overture-stack/lectern-server`

This will ensure that all dependencies of `server` are built in correct order before the `server` build is run.

To work with any module in this repository, follow the instructions in the README provide in the module directory.
Note that the full name of from the package must be used for this to work, so for the client the command would be:

`pnpm nx build @overture-stack/lectern-client`

For convenience, scripts have been added to the root level [`package.json`](./package.json) to run `build` and `test` scripts for every service using short names. These follow the pattern `pnpm <build|test>:<package short name/alias>`. For example, the same build command can be performed by:

`pnpm build:client`

To work with any module in this repository, follow the instructions in the README provide in that module's directory.

Get started by running the [Lecter Server application](apps/server/README.md).

### Common Commands

A few commonly reused scripts have been added to the root `package.json`. Run them from the root directory, or if you are in a sub directory then use `pnpm -w`.

For example, when your current working directory is not in the project root, you can still conveniently test every module in the monorepo with the command:

`pnpm -w test:all`

#### Build Everything

`pnpm build:all`
Expand All @@ -72,30 +85,26 @@ This will build all modules.

`pnpm test:all`

This will test everything, building all dependencies needed to fully test.
This will test all modules in the repo.

## Additonal Content
## Additional Content

In addition to the code for Lectern, this repository contains some useful reference material.

### Meta-Schema

Lectern provides a meta-schema definition that describes the structure of Lectern Dictionaries. The generated JSON Schema formatted copy of this schema can be found at [`./generated/DictionaryMetaSchema.json`](./generated/DictionaryMetaSchema.json).

This can be used as a programing language agnostic schema for external applications that want to validate, generate, or interact with Lectern Dictionaries.
This can be used as a programing language agnostic schema for external applications to validate Lectern Dictionaries.

> **Note:**
> [!NOTE]
>
> Don't manually update any files in the `./generated` path. This content is programatically generated from the source code.

### Sample Dictionaries



## Support & Contributions

We welcome community contributions! Please follow our [code of conduct](./code_of_conduct.md)

- Filing an [issue](https://github.com/overture-stack/ego/issues)
- Connect with us on [Slack](http://slack.overture.bio)
- Add or Upvote a [feature request](https://github.com/overture-stack/ego/issues?q=is%3Aopen+is%3Aissue+label%3Anew-feature)
- Add or Upvote a [feature request](https://github.com/overture-stack/ego/issues?q=is%3Aopen+is%3Aissue+label%3Anew-feature)
2 changes: 1 addition & 1 deletion apps/server/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ RUN chown -R ${APP_USER}:${APP_USER} $(npm config get prefix)
USER ${APP_USER}:${APP_USER}

RUN pnpm install --ignore-scripts
RUN pnpm nx build server
RUN pnpm nx build @overture-stack/lectern-server

EXPOSE 3000

Expand Down
Loading