Common serialization format for device configurations

[farshidtz]

A device service can have four types of configuration files:

• Server configurations (TOML/JSON?)
• Device(s) configurations (TOML/JSON)
• Profile(s) configurations (YAML/JSON)
• Provision Watcher (JSON)

(The list of supported serializations may not be complete.)

This is all very confusing and difficult to work with.

Can YAML be used as a common serialization for all config files?
If TOML is the preferred format (although nowadays not as common), why the device profile and provision watcher are different?

[iain-anderson]

Ideally these options would be simplified, however it is difficult to obtain agreement on what single format should be adopted. From our (commercial) perspective we'd like to see all of these files in JSON.

[lenny-goodell]

JSON is get for APIs, but not the best for files that are manually edited. This is why we have YAML and TOML for the static files. I could see standardizing all the static files to YAML or TOML, probably YAML since it is more widely used.

[iain-anderson]

Well, you choose your poison. JSON has quoting inconveniences; TOML's syntax for object arrays is a little clumsy; YAML invokes the holy war over indendation/spacing as syntax etc

[farshidtz]

For reference, adding a snippet of Device MQTT config in different serialization formats;

[Writable]
LogLevel = "INFO"
  # InsecureSecrets are required for when Redis is used for message bus
  [Writable.InsecureSecrets]
    [Writable.InsecureSecrets.DB]
    path = "redisdb"
      [Writable.InsecureSecrets.DB.Secrets]
      username = ""
      password = ""
    [Writable.InsecureSecrets.MQTT]
    path = "credentials"
      [Writable.InsecureSecrets.MQTT.Secrets]
      username = ""
      password = ""
  [Writable.Reading]
  ReadingUnits = true
  [Writable.Telemetry]
  Interval = "30s"
  PublishTopicPrefix  = "edgex/telemetry" # /<service-name>/<metric-name> will be added to this Publish Topic prefix
    [Writable.Telemetry.Metrics] # All service's metric names must be present in this list.
    # Device SDK provided Service Metrics
    EventsSent = false
    ReadingsSent = false
    # Common Security Service Metrics
    SecuritySecretsRequested = false
    SecuritySecretsStored = false
    SecurityConsulTokensRequested = false
    SecurityConsulTokenDuration = false
    [Writable.Telemetry.Tags] # Contains the service level tags to be attached to all the service's metrics
#    Gateway="my-iot-gateway" # Tag must be added here or via Consul Env Override can only chnage existing value, not added new ones.

Writable:
  LogLevel: INFO
  # InsecureSecrets are required for when Redis is used for message bus
  InsecureSecrets:
    DB:
      path: redisdb
      Secrets:
        username: ''
        password: ''
    MQTT:
      path: credentials
      Secrets:
        username: ''
        password: ''
  Reading:
    ReadingUnits: true
  Telemetry:
    Interval: 30s
    PublishTopicPrefix: edgex/telemetry # /<service-name>/<metric-name> will be added to this Publish Topic prefix
    Metrics: # All service's metric names must be present in this list.
      # Device SDK provided Service Metrics
      EventsSent: false
      ReadingsSent: false
      # Common Security Service Metrics
      SecuritySecretsRequested: false
      SecuritySecretsStored: false
      SecurityConsulTokensRequested: false
      SecurityConsulTokenDuration: false
    Tags: {} # Contains the service level tags to be attached to all the service's metrics
     # Gateway="my-iot-gateway" # Tag must be added here or via Consul Env Override can only chnage existing value, not added new ones.

{
  "Writable": {
    "LogLevel": "INFO",
    "InsecureSecrets": {
      "DB": {
        "path": "redisdb",
        "Secrets": {
          "username": "",
          "password": ""
        }
      },
      "MQTT": {
        "path": "credentials",
        "Secrets": {
          "username": "",
          "password": ""
        }
      }
    },
    "Reading": {
      "ReadingUnits": true
    },
    "Telemetry": {
      "Interval": "30s",
      "PublishTopicPrefix": "edgex/telemetry",
      "Metrics": {
        "EventsSent": false,
        "ReadingsSent": false,
        "SecuritySecretsRequested": false,
        "SecuritySecretsStored": false,
        "SecurityConsulTokensRequested": false,
        "SecurityConsulTokenDuration": false
      },
      "Tags": {}
    }
  }
}

[cloudxxx8]

In fact, all formats look good to me. However, we need to choose one for consistency.
Since we use JSON for API anyway, we can just use JSON for everything by default.
golang has build-in JSON parser, so we can reduce other parsers in the service if we want.
or, we can still support all the format in code, and provide one as the example by default.

[lenny-goodell]

I think we agreed on YAML and/or JSON.
YAML is easier to read and edit than JSON and is a common format used for configuration and compose files.

[farshidtz]

Here are the issues to track the progress:

• Change configuration file format to be YAML go-mod-bootstrap#424
• [GO] Switch TOML files to be YAML files device-sdk-go#1291
• Consume loading config from YAML and convert config TOML files to YAML edgex-go#4452