🐛 Wrong metadata files not being caught

[lucasrodes]

Ike reported some metadata YAML files that shouldn't be valid, yet they went unnoticed:

• table object defined twice:

  etl/etl/steps/data/garden/war/2023-09-21/mie.meta.yml

  Line 75 in 787c746

  tables:

• display defined twice:

  etl/etl/steps/data/garden/tourism/2023-05-05/unwto.meta.yml

  Line 212 in 787c746

  display:

We should:

• Fix these
• Try to have our CI/CD detect these

[Marigold]

Both of these errors are correctly labeled with warnings by the YAML linter in VSCode (so the JSON Schema is working as intended).

The thing is, we don't strictly enforce YAML validity because many files still use deprecated fields like description and source, which would trigger a flood of errors. We have a few options:

1. Add description and source to the JSON Schema and perform strict validation

   • The downside is that these fields wouldn’t raise warnings in VSCode and would appear in autocomplete suggestions.

2. Migrate all description and source fields to the new metadata format

   • (How many datasets still need to be updated?)

Note: JSON Schema does have a "deprecated": true property, but it doesn’t seem to have any effect in VSCode.