[REST] New REST APIs to parse/generate syntax for items and things

[lolodomo]

Related to #4509

Added REST APIs:

• GET /inbox/{thingUID}/syntax/generate to generate syntax for the thing associated to the discovery result
• GET /items/syntax/generate to generate syntax for all items in the registry
• POST /items/{itemname}/syntax/generate to generate syntax for an item in the registry or provided
• POST /items/{itemname}/syntax/parse to parse syntax for an item without updating the registry
• GET /things/syntax/generate to generate syntax for all things in the registry
• POST /things/{thingUID}/syntax/generate to generate syntax for a thing in the registry or provided
• POST /things/{thingUID}/syntax/parse to parse syntax for a thing without updating the registry

All these APIs have a parameter named "format" to request a particular output format. Of course, a syntax generator should be available for this format. Only "DSL" format is provided by this PR as this is currently our unique supported format for items and things in config files. So this parameter is set to "DSL" by default. In the future, new formats could be added and they will be automatically supported by these APIs.

All the APIs to generate syntax have a boolean parameter named "hideDefaultParameters" to hide or show the configuration parameters having the default value. They are hidden by default.

[lolodomo]

Here is an example of result for things:

Thing astro:moon:e2015c6f37 "Données Astro de Lune" @ "Paris" [geolocation="49.023461463214126,-1.1425781250000002"] {
	Channels:
		Type end : rise#end "Heure Fin" [offset=-1, forceEvent=true]
}
Bridge freeboxos:api:api "Freebox API" [appToken="xxxxx", httpsAvailable=true] {
	Thing active-player player-test "Player Revolution" [remoteCode="123456"]
	Thing player player4k "Player Mini 4k" [id=15]
	Thing revolution serveur-test "Serveur Revolution"
}
Bridge netatmo:account:compte "Netatmo" [clientId="yyyyy", clientSecret="zzzzz"] {
	Bridge weather-station indoor "Indoor" [id="11:22:33:44:55:66"] {
		Thing outdoor exterieur "Sonde extérieure" [id="22:33:44:55:66:77"]
		Thing rain pluviometre "Pluviomètre" [id="33:44:55:66:77:88"]
		Channels:
			Type co2-measurement : max-co2-hour "Max CO2 Last Day" [limit="MAX", period="1day"]
			Type co2-measurement : max-co2-week "Max CO2 Last Week" [limit="MAX"]
			Type co2-measurement : min-co2-hour "Min CO2 Last Day" [period="1day"]
			Type co2-measurement : min-co2-week "Min CO2 Last Week"
			Type noise-measurement : max-noise-hour "Max Noise Last Hour" [limit="MAX", period="1hour"]
			Type noise-measurement : min-noise-hour "Min Noise Last Hour" [period="1hour"]
			Type noise-timestamp : time-max-noise-week "Time Max Noise Last Week" [limit="DATE_MAX"]
			State Number : test-number [param1="Value 1", param2=true, param3=15, 20.5, 25.5, 30]
			Trigger String : test-string [titi=100, toto=true, false, true]
	}
}
Thing ntp:ntp:local "Local Time" [hostname="0.fr.pool.ntp.org", timeZone="Europe/Paris"] {
	Channels:
		Type string-channel : string "Date Heure" [DateTimeFormat="dd-MM-yyyy HH:mm:ss"]
}

and the same in flat mode:

Thing astro:moon:e2015c6f37 "Données Astro de Lune" @ "Paris" [geolocation="49.023461463214126,-1.1425781250000002"] {
	Channels:
		Type end : rise#end "Heure Fin" [offset=-1, forceEvent=true]
}
Thing freeboxos:api:api "Freebox API" [appToken="xxxxx", httpsAvailable=true]
Thing freeboxos:active-player:api:player-test "Player Revolution" (freeboxos:api:api) [remoteCode="123456"]
Thing freeboxos:player:api:player4k "Player Mini 4k" (freeboxos:api:api) [id=15]
Thing freeboxos:revolution:api:serveur-test "Serveur Revolution" (freeboxos:api:api)
Thing netatmo:account:compte "Netatmo" [clientId="yyyyy", clientSecret="zzzzz"]
Thing netatmo:weather-station:compte:indoor "Indoor" (netatmo:account:compte) [id="11:22:33:44:55:66"] {
	Channels:
		Type co2-measurement : max-co2-hour "Max CO2 Last Day" [limit="MAX", period="1day"]
		Type co2-measurement : max-co2-week "Max CO2 Last Week" [limit="MAX"]
		Type co2-measurement : min-co2-hour "Min CO2 Last Day" [period="1day"]
		Type co2-measurement : min-co2-week "Min CO2 Last Week"
		Type noise-measurement : max-noise-hour "Max Noise Last Hour" [limit="MAX", period="1hour"]
		Type noise-measurement : min-noise-hour "Min Noise Last Hour" [period="1hour"]
		Type noise-timestamp : time-max-noise-week "Time Max Noise Last Week" [limit="DATE_MAX"]
		State Number : test-number [param1="Value 1", param2=true, param3=15, 20.5, 25.5, 30]
		Trigger String : test-string [titi=100, toto=true, false, true]
}
Thing netatmo:outdoor:compte:indoor:exterieur "Sonde extérieure" (netatmo:weather-station:compte:indoor) [id="22:33:44:55:66:77"]
Thing netatmo:rain:compte:indoor:pluviometre "Pluviomètre" (netatmo:weather-station:compte:indoor) [id="33:44:55:66:77:88"]
Thing ntp:ntp:local "Local Time" [hostname="0.fr.pool.ntp.org", timeZone="Europe/Paris"] {
	Channels:
		Type string-channel : string "Date Heure" [DateTimeFormat="dd-MM-yyyy HH:mm:ss"]
}

[lolodomo]

And the result for items:

Group:Switch:OR (ON, OFF) DemoSwitchGroup [home-group]
Group X "Group X"
Group C "Group C" (X, U)
Group U "Group U" (X)
Color DemoColor (DemoSwitchGroup)
Contact DemoContact
DateTime DemoDateTime
Dimmer DemoDimmer (DemoSwitchGroup) { autoupdate="false" }
Number:Dimensionless DemoHumidity "Humidity [%.0f %%]" <humidity> (U, C) [Humidity, Measurement] { listWidget="" [action="", iconUseState=true], unit="%" }
Number:Dimensionless DemoHumidity2 "Humidity 2" <humidity> (U, C) [Humidity, Measurement] { listWidget="" [action="", iconUseState=true], unit="%" }
Image DemoImage
Location DemoLocation "Demo Location"
Number DemoNumber (U)
Number DemoNumber2 (U) { stateDescription="" [min=-10, max=20, step=0.5, pattern="%.1f"] }
Number DemoNumber3 "Demo Number 3 [%.1f]" (U) { stateDescription="" [min=-10, max=20, step=0.5, pattern="%.1f"] }
Number DemoNumber4 "Demo Number 4" (U) { stateDescription="" [min=-5, max=5, pattern="%.0f", readOnly=true] }
Number DemoNumber5 "Demo Number 5" (U)
Rollershutter DemoRollershutter
String DemoString (X)
String DemoString2 "My String 2 [- %s -]" (X)
String DemoString3 "My String 3 [= %s =]" (X)
String DemoString4 "My String 4 [+++ %s +++]" (X)
String DemoString5 "My String 5" (X)
String DemoString6 "My String 6 []" (X)
String DemoString7 "My String 7" (X)
Switch DemoSwitch "Demo Switch [%s]" <switch> (DemoSwitchGroup) [Control, Light] { matter="OnOffLight" [label="Lampe sur l\'étagère"] }
Number:Temperature DemoTemperature "Setpoint [%.1f °C]" <temperature> (C) [Setpoint, Temperature]
String test { commandDescription="" [options="0=value 0,1,2=Value 2"], stateDescription="" [options="0=Valeur 0,1=Valeur 1,2", pattern="%s"] }
Number test2 "Aujourd\'hui" <none> [Energy, Measurement]
Number:Pressure test_0_1 "Label Test" <if:noto-v1:cityscape-at-dusk> (DemoSwitchGroup, C) [Sensor] { commandDescription=" " [options="0=Valeur 0,1=Valeur 1"], ga="Lock" [inverted=true], unit="bar" }
Number:Pressure test_0_2 "Label , Test = 1" <if:noto-v1:cityscape-at-dusk> (DemoSwitchGroup, C) [Sensor] { commandDescription=" " [options="0=Valeur 0,1=Valeur 1"], ga="Lock" [inverted=true], unit="bar" }
String Donnees_Astro_de_Lune_Evenement_Periode "Événement Période" [Point] { channel="astro:moon:e2015c6f37:rise#event" [profile="system:trigger-event-string"] }
DateTime Donnees_Astro_de_Lune_Heure_Fin "Heure Fin [%1$td/%1$tm/%1$tY %1$tR]" [Point] { channel="astro:moon:e2015c6f37:rise#end" [profile="system:timestamp-offset", offset=30.0], channel="astro:moon:e2015c6f37:rise#start", autoupdate="false" }
DateTime Lune_Heure_Debut "Heure Début [%1$td/%1$tm/%1$tY %1$tR]" [Point] { channel="astro:moon:e2015c6f37:rise#start" }
Color MagicColor { channel="magic:color-light:magicColor:color" }
Dimmer MagicDimmer { channel="magic:dimmable-light:magicDimmer:brightness" }
Switch MagicOnOff { channel="magic:onoff-light:magicOnOff:switch" }

[lolodomo]

@kaikreuzer and @openhab/core-maintainers : for your information, the PR is now ready for review.

[lolodomo]

Go back to draft as I discovered a bug.

[lolodomo]

Problem with items sorting is solved. So ready again.

[spacemanspiff2007]

This is a really awesome feature and I'm really looking forward to it.

However the longer I think about the more I think it would be better if the conversion endpoint would not access existing things/items/etc. at all but these would have to be passed in.
It's easy to request and data and forward it to the corresponding endpoint with minimal overhead.
The benefit would be a single endpoint that does centralize all the conversion logic that is both easy to use, easy to understand and easy to maintain.
A query parameter/sub endpoint could select the output format and the body of the http endpoint could contain all the data.

e.g. a single

POST /fileFormat/create/dsl-items-file
POST /fileFormat/parse/dsl-items-file

POST /fileFormat/create/dsl-things-file
POST /fileFormat/parse/dsl-things-file

POST /fileFormat/create/future-file-format
POST /fileFormat/parse/future-file-format

The body would then be the normal DTOs like they are returned from the existing rest-api endpoints.
This would also be future proof and new file formats would not further pollute the existing items/things endpoints.
If one day we have a different file format that would allow e.g. defining things, items and transformations in one file the current approach would not work properly.
Also the data required to build a file format is properly documented as the endpoint input.

@lolodomo I'd love to hear your thoughts about this. What do you think about this approach? Do you - like me - think this would be an improvement?

[lolodomo]

I am OK with a new endpoint "/fileFormat". Tha's true that /items and /things endpoints are already largely cluttered.

The other advantage I see with a new endpoint is that the current solution has to deal with inconsistency between thing UID in the URI and the one in the provided thing data (body). So in this case, your solution avoid this problem and I like that.

You would like to remove the API that is for me the most important and the most magic, the one that will allow for a user just by calling one API without providing anything to create a file format for all his items or things currently existing in the registry. This is an easy way for the user to move to the current file format or a future file format. So I cannot agree with having no API based on the registries.

Now I agree with you that we need an API that takes a ThingDTO and produces the file file format and another that takes a file format and produces a ThingDTO. It is what I introduced yesterday. To be precise, the second is currently producing a EnrichedThingDTO and I have to clarify if we should use ThingDTO or EnrichedThingDTO in both APIs. Edit: thingsDTO is appropriate.
Currently the first API accepts a ThingDTO as input (body) but if none is provided, the thing is taken from the registry.
I agree with your idea to provide or produce a list of things, even if in practice Main UI will use these APIs with only one thing. This idea would also allow to reduce from 3 to 2 API because we could use the first API to produce file format for all things just by providing no thing data in body.

I prefer as it is now a query parameter to select the file format, rather than creating multiple sub-endpoints, one per format. It makes the API already OK for the future even if a new formats are introduced in the future.

So the compromise would be:

• POST /file-format/create/things taking a list of ThingDTO (body) and returning the file format, or considering all things in the registry if no data is provided in body
• POST /file-format/parse/things taking a file format (body) and returning a list of ThingDTO

Same principle for items.

Is it a reasonable compromise ?

[lolodomo]

Finally I think a separate API for things from registry remains better, keeping the POST APIs only for ThingDTO <-> syntax as you suggested.

GET /file-format/things-from-registry
POST /file-format/create/things
POST /file-format/parse/things

[spacemanspiff2007]

You would like to remove the API that is for me the most important and the most magic, the one that will allow for a user just by calling one API without providing anything to create a file format for all his items or things currently existing in the registry. This is an easy way for the user to move to the current file format or a future file format.

Fair point - things should be easy for the user.
I don't know you intend that things work because I'm not a java programmer and can't properly understand the code but wouldn't be generating stuff for the whole registry just be a matter of passing around responses?
e.g.

GET /items  >> POST /fileFormat/create/items-file
GET /things >> POST /fileFormat/create/things-file

What is your idea about resolving ambiguities?
E.g. if I have a TestItem in my installation and want to generate syntax for an item that is also named TestItem?

just by providing no thing data in body.

That would also work. If the body is empty ([]) the api could return the file format for the whole registry.

I'm still not a hundred percent sold on that because I think making requests and posting data between endpoints is cheap and it's something that's not done very often. The benefit is that there is a clear flow of data and an arbitrary input always produces the same output.

But if it's clear from the use of the endpoint that the item registry is getting accessed or that it's not getting accessed than I have no objections.

---

I prefer as it is now a query parameter to select the file format, rather than creating multiple sub-endpoints, one per format. It makes the API already OK for the future even if a new formats are introduced in the future.

This was my idea at first, too. However I then thought about how things will documented e.g. in api explorer and as far as I know there is currently no way to document the input/output data structure based on the query parameter.
E.g.
/file-format/create?format=thingsfile takes a different input than
/file-format/create?format=itemsfile
and I think it makes sense that the api documentation reflects that accordingly.

---

(You just wrote your response while I was typing the second paragraph out so I just left everything in)

Just a small suggestion: How about a sub-endpoint which returns the registry?
e.g.

GET  /file-format/existing/things
POST /file-format/create/things

[mherwege]

Would this allow us to get rid of the nearly parsers in the UI for items and things? These are hard to keep in sync with the xtext definitions in core.
And probably the hardest one is for sitemaps. I know this is not part of this PR yet, but I see potential applying the same principle to sitemaps. There is a lot of code doing parsing and generating DSL for sitemaps in the UI right now, which is a pain to maintain.

Combined with #4585, this would also open up an easier route to defining sitemaps in yaml, and switch between UI configuration, DSL and yaml in the UI.

One challenge I potentially see is showing errors in the UI. For sitemaps, as the parsing is done in the UI code, it will show (to a degree, and not always perfect) where the error in the sitemap DSL is. If the whole translation is done in core, the API response should be able to indicate where the error is as well, to be able visualize it in the UI. It shouldn't just be a log entry.

[lolodomo]

Fair point - things should be easy for the user. I don't know you intend that things work because I'm not a java programmer and can't properly understand the code but wouldn't be generating stuff for the whole registry just be a matter of passing around responses? e.g.

GET /items  >> POST /fileFormat/create/items-file
GET /things >> POST /fileFormat/create/things-file

Too much complicated to achieve by a lambda user.
For a developer, of course it is possible but it is two requests instead of one, a massive and useless additional network traffic generated between the client and the server.

What is your idea about resolving ambiguities?
E.g. if I have a TestItem in my installation and want to generate syntax for an item that is also named TestItem?

If the POST APIs are fully independent from our server registries, we have no conflicts to resolve. It is the role of the client to finally not request the creation of a new item/thing in the registry that already exists. I believe there is nothing new there.

Just a small suggestion: How about a sub-endpoint which returns the registry? e.g.

GET  /file-format/existing/things

That would be fine for me, Looks better than my proposal: GET /file-format/things-from-registry

[lolodomo]

Would this allow us to get rid of the nearly parsers in the UI for items and things? These are hard to keep in sync with the xtext definitions in core.

Yes, that is what we are discussing in a RFC I recently opened and that I finally started to embed in that PR.

And probably the hardest one is for sitemaps. I know this is not part of this PR yet, but I see potential applying the same principle to sitemaps. There is a lot of code doing parsing and generating DSL for sitemaps in the UI right now, which is a pain to maintain.

Yes, that was my intention to come to you in a second step to also consider sitemap. But I will do that in a next step and for that I would need first some explanations from you how it works today, what are the used APIs and how is triggered the code in UIComponentSitemapProvider that already generates DSL model. The syntax for sitemap is certainly more complex than the syntax for item or thing, but as we have the existing class UIComponentSitemapProvider with code I can simply reuse, it should be finally not too much complex, even maybe very easy ;)

Combined with #4585, this would also open up an easier route to defining sitemaps in yaml, and switch between UI configuration, DSL and yaml in the UI.

Yes, that is exactly my current intention started with items and things but that I can continue later with sitemaps.

One challenge I potentially see is showing errors in the UI. For sitemaps, as the parsing is done in the UI code, it will show (to a degree, and not always perfect) where the error in the sitemap DSL is. If the whole translation is done in core, the API response should be able to indicate where the error is as well, to be able visualize it in the UI. It shouldn't just be a log entry.

That is an interesting point, in my current (basic) implementation, the API that parses the DSL syntax will return a 400 status code and the errors are logged. One thing I could do easily is to provide what we currently log as output of the API in case of invalid syntax, so that it can be displayed in UI. But I will not be able to be more clever and more precise than what currently provides the core framework in logs when the DSL file is parsed.

[spacemanspiff2007]

Too much complicated to achieve by a lambda user.
For a developer, of course it is possible but it is two requests instead of one, a massive and useless additional network traffic generated between the client and the server.

I think the rest api users are only developers and even for large installations (>1k items) it's almost instant.
It's additional code that has to be created and maintained.
However if you want to provide the additional convenience that's fine with me. 🚀

It is the role of the client to finally not request the creation of a new item/thing in the registry that already exists. I believe there is nothing new there.

I fully agree 👍

One thing I could do easily is to provide what we currently log as output of the API in case of invalid syntax, so that it can be displayed in UI

That would be great. Would it then need to be logged at all?

[mherwege]

One thing I could do easily is to provide what we currently log as output of the API in case of invalid syntax, so that it can be displayed in UI. But I will not be able to be more clever and more precise than what currently provides the core framework in logs when the DSL file is parsed.

I think that’s a good start. We may then have to look at the errors generated by core at some point to refine the messages (e.g. the position of the error would always e useful if it can be found), but that’s a next step. Also the logs would benefit from improvements there.