From dda1d02e97eb647bcd5f02c07359f376a7231620 Mon Sep 17 00:00:00 2001 From: eric Date: Fri, 9 Oct 2026 01:51:35 -0500 Subject: [PATCH 01/11] [LORA] Fix buffer overflow when converting binary packets to hex TheengsUtils::_rawToHex() formats each byte with "%02X\r", so the last sprintf writes two bytes past the 2 * size hex digits (the '\r' and the terminating null). LORAtoX() sized its buffer 2 * packetSize + 1, one byte short, overflowing the stack for every binary LoRa packet. Co-Authored-By: Claude Opus 5.5 --- main/gatewayLORA.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/main/gatewayLORA.cpp b/main/gatewayLORA.cpp index d389fa01ad..7915aa4e83 100644 --- a/main/gatewayLORA.cpp +++ b/main/gatewayLORA.cpp @@ -459,7 +459,7 @@ void LORAtoX() { return; } // We have non-ascii data: create hex string of the data - char hex[packetSize * 2 + 1]; + char hex[packetSize * 2 + 2]; // _rawToHex writes "%02X\r" per byte: needs 2 * size + 2 TheengsUtils::_rawToHex(packet, hex, packetSize); // Terminate with a null character hex[packetSize * 2] = 0; From 8ae82001b9dcf7f893bf3bfe5c9edf231e7f3609 Mon Sep 17 00:00:00 2001 From: eric Date: Fri, 9 Oct 2026 01:51:44 -0500 Subject: [PATCH 02/11] [LORA] Add LoRaWAN ABP uplink decryption with optional JavaScript decoders Lets the LoRa gateway receive LoRaWAN devices using ABP on a single channel (e.g. Dragino sensors set to a fixed frequency and data rate), which previously could only be published as encrypted hex. - Build flag LORA_LORAWAN (ESP32): devices (DevAddr, NwkSKey, AppSKey, name, model) are declared at runtime through the "lorawan" key of the MQTTtoLORA/config command and stored in NVS with "save":true. - Uplinks from declared devices have their MIC verified, duplicate frame counters dropped and FRMPayload decrypted; they are published to LORAtoMQTT/ with fcnt, fport and the decrypted payload. Other packets keep the existing raw hex behaviour. - Build flag LORA_LORAWAN_JS adds Duktape to run TTN style decoders (Decoder(bytes, port) or decodeUplink(input)) uploaded as raw scripts to /LORAWANdecoder/, outside /commands/# because that subscription only accepts JSON up to JSON_MSG_BUFFER_MAX. NTP is started so decoders can timestamp with new Date(). - Optional per device "entities" declarations create Home Assistant discovery entities from the published fields. Without the flags the firmware is unchanged. Co-Authored-By: Claude Opus 5.5 --- docs/use/lora.md | 58 ++++ main/config_LORA.h | 27 ++ main/gatewayLORA.cpp | 39 ++- main/gatewayLORA_LoRaWAN.cpp | 556 +++++++++++++++++++++++++++++++++++ main/main.cpp | 6 + platformio.ini | 1 + 6 files changed, 678 insertions(+), 9 deletions(-) create mode 100644 main/gatewayLORA_LoRaWAN.cpp diff --git a/docs/use/lora.md b/docs/use/lora.md index 78694e2696..176f9dd848 100644 --- a/docs/use/lora.md +++ b/docs/use/lora.md @@ -81,6 +81,64 @@ will send binary 0x01C0FFEE * WiPhone message: `mosquitto_pub -t "home/OpenMQTTGateway/commands/MQTTtoLORA" -m '{"message":"test","type":"WiPhone","to":"123ABC","from":"FFFFFF"}'`\ will send "test" to a WiPhone with chip ID 123ABC +## Decoding LoRaWAN ABP devices + +The gateway can decrypt uplinks from LoRaWAN devices using ABP (activation by personalization), such as Dragino sensors switched to ABP mode. It is not a LoRaWAN network server, so this only suits a simple setup: +* The gateway listens on a single frequency, spreading factor and bandwidth. Set each device to send on that same channel (for example `AT+CHS=903900000` and a fixed data rate with ADR off on Dragino devices) and use sync word `0x34`. +* Only uplinks are handled. There is no join (OTAA), no downlink and no MAC commands, so turn off on the device any feature that waits for a downlink (confirmed uplinks, link checks, "rejoin when no downlink"). + +This feature is available on ESP32 only. It is not part of the default binaries: build with `'-DLORA_LORAWAN'`. To also decode the payload with a JavaScript decoder, add `'-DLORA_LORAWAN_JS'` and the Duktape library, for example in your `prod_env.ini`: +```ini +[env:ttgo-lora32-v21-lorawan] +extends = env:ttgo-lora32-v21 +lib_deps = + ${env:ttgo-lora32-v21.lib_deps} + ${libraries.duktape} +build_flags = + ${env:ttgo-lora32-v21.build_flags} + '-DLORA_LORAWAN' + '-DLORA_LORAWAN_JS' + '-DARDUINO_LOOP_STACK_SIZE=16384' +``` + +### Declaring a device +Add the device with its DevAddr and session keys through the LoRa configuration command. Use `"save":true` to keep it after a restart: + +`mosquitto_pub -t home/OpenMQTTGateway/commands/MQTTtoLORA/config -m '{"lorawan":{"devaddr":"0187F184","nwkskey":"<32 hex digits>","appskey":"<32 hex digits>","name":"Front door","model":"LDS02"},"save":true}'` + +You can declare up to 16 devices (`LORAWAN_MAX_DEVICES`), either one at a time or as an array. To change the name or model later, send only the `devaddr` and the fields to change. To delete a device, send `{"lorawan":{"devaddr":"0187F184","remove":true},"save":true}`. The LoRa state message lists the configured devices, but never their keys. + +For each uplink, the gateway checks the message integrity code (MIC) with the NwkSKey, drops repeated frames (same frame counter) and decrypts the payload with the AppSKey. It then publishes to `home/OpenMQTTGateway/LORAtoMQTT/`: +```json +{"id":"0187F184","name":"Front door","model":"LDS02","fcnt":67,"fport":10,"payload":"8C6C0100003B00000000","rssi":-34,"snr":10.5,"pferror":-2435,"packetSize":23} +``` +Packets from unknown devices, or that fail the MIC check, are published as raw `hex` like any other binary packet. + +### JavaScript payload decoders +With `LORA_LORAWAN_JS`, you can attach a decoder to each device. It uses the same formats as The Things Network: +* `function Decoder(bytes, port)` returns an object (TTN v2). +* `function decodeUplink(input)` receives `{bytes, fPort}` and returns `{data, warnings, errors}` (TTN v3). + +Publish the script as is, not wrapped in JSON, to `home/OpenMQTTGateway/LORAWANdecoder/`. The device must already be declared. The script is stored in flash and can be up to 3800 bytes (`LORAWAN_DECODER_MAX_SIZE`); an empty message removes it: + +`mosquitto_pub -t home/OpenMQTTGateway/LORAWANdecoder/0187F184 -f lds02_decoder.js` + +The fields of the decoded object are added to the published message: +```json +{"id":"0187F184","name":"Front door","model":"LDS02","fcnt":67,"fport":10,"payload":"8C6C0100003B00000000","BAT_V":3.18,"BAT_PCNT":"88.33","DOOR_OPEN_STATUS":1,"DOOR_STATE":"open","LAST_DOOR_OPEN_DURATION":0,"UTC_TIME":"2026-10-09T06:49:17.000Z","rssi":-34,"snr":10.5,"pferror":-2435,"packetSize":23} +``` +If the decoder fails, the message contains `decoder_error` instead. Decoders run on Duktape (ECMAScript 5.1 with some ES6 additions such as `const`), so arrow functions, `let` and template literals are not supported. When a decoder runs, the gateway sets its clock by NTP so that `new Date()` returns the current time. + +### Home Assistant entities +Add an `entities` list to the device declaration to create Home Assistant entities from the published fields: +```json +{"lorawan":{"devaddr":"0187F184","entities":[ + {"key":"DOOR_STATE","type":"binary_sensor","name":"Door","class":"door","on":"open","off":"close"}, + {"key":"BAT_V","name":"Battery voltage","class":"voltage","unit":"V"}, + {"key":"rssi","name":"RSSI","class":"signal_strength","unit":"dBm"}]},"save":true} +``` +`type` is `sensor` by default. `class` is the Home Assistant device class. Sensors use `state_class` `measurement` unless you set another one. Fields named like the existing LoRa sensors (`tempc`, `hum`, `moi`, `batt`, `count`) are discovered automatically. + More info on where the LoRa library is born [@sandeepmistry](https://github.com/sandeepmistry/arduino-LoRa/blob/master/API.md#radio-parameters) Tutorial on how to leverage LoRa for a mailbox sensor from [PricelessToolkit](https://www.youtube.com/channel/UCz75N6inuLHXnRC5tqagNLw): diff --git a/main/config_LORA.h b/main/config_LORA.h index 6e8ffac7ac..fac2584c1f 100644 --- a/main/config_LORA.h +++ b/main/config_LORA.h @@ -72,6 +72,33 @@ extern void XtoLORA(const char* topicOri, JsonObject& RFdata); #define repeatLORAwMQTT false // do we repeat a received signal by using MQTT with LORA gateway +/*-------------------LoRaWAN ABP decoding----------------------*/ +// Enable with '-DLORA_LORAWAN' (ESP32 only), add '-DLORA_LORAWAN_JS' and ${libraries.duktape} for JavaScript payload decoders +#ifdef LORA_LORAWAN +# ifndef LORAWAN_MAX_DEVICES +# define LORAWAN_MAX_DEVICES 16 +# endif +# define LORAWAN_NOT_HANDLED 0 // not a data uplink from a configured device (or MIC check failed) +# define LORAWAN_DECODED 1 +# define LORAWAN_DROP 2 // duplicate frame +extern void LORAWANsetup(); +extern int LORAWANtoJson(const uint8_t* packet, int packetSize, JsonObject& LORAdata); +extern void LORAWANConfig_fromJson(JsonObject& LORAdata); +extern void LORAWANState(JsonObject& LORAdata); +# ifdef ZmqttDiscovery +extern void launchLORAWANDiscovery(bool overrideDiscovery); +# endif +# ifdef LORA_LORAWAN_JS +// Decoder scripts are published raw (not JSON) to /LORAWANdecoder/. +// Deliberately outside /commands/#: that subscription only accepts JSON and is limited to JSON_MSG_BUFFER_MAX +# define subjectLORAWANdecoder "/LORAWANdecoder" +# ifndef LORAWAN_DECODER_MAX_SIZE +# define LORAWAN_DECODER_MAX_SIZE 3800 // bytes, NVS strings are limited to ~4000 bytes +# endif +extern void LORAWANdecoderFromMQTT(const char* topic, const char* payload); +# endif +#endif + /*-------------------PIN DEFINITIONS----------------------*/ //TTGO LORA BOARD ESP32 PIN DEFINITION diff --git a/main/gatewayLORA.cpp b/main/gatewayLORA.cpp index 7915aa4e83..4290e736a1 100644 --- a/main/gatewayLORA.cpp +++ b/main/gatewayLORA.cpp @@ -121,6 +121,9 @@ void createOrUpdateDeviceLORA(const char* id, const char* model, uint8_t flags) // This function always should be called from the main core as it generates direct mqtt messages // When overrideDiscovery=true, we publish discovery messages of known LORAdevices (even if no new) void launchLORADiscovery(bool overrideDiscovery) { +# ifdef LORA_LORAWAN + launchLORAWANDiscovery(overrideDiscovery); +# endif if (!overrideDiscovery && newLORADevices == 0) return; if (xSemaphoreTake(semaphorecreateOrUpdateDeviceLORA, pdMS_TO_TICKS(QueueSemaphoreTimeOutLoop)) == pdFALSE) { @@ -400,6 +403,9 @@ void LORAConfig_fromJson(JsonObject& LORAdata) { void setupLORA() { LORAConfig_init(); LORAConfig_load(); +# ifdef LORA_LORAWAN + LORAWANsetup(); +# endif # ifdef ZmqttDiscovery semaphorecreateOrUpdateDeviceLORA = xSemaphoreCreateBinary(); xSemaphoreGive(semaphorecreateOrUpdateDeviceLORA); @@ -454,17 +460,26 @@ void LORAtoX() { if (deviceId == WIPHONE) { _WiPhonetoX(packet, LORAdata); } else if (binary) { - if (LORAConfig.onlyKnown) { - THEENGS_LOG_TRACE(F("Ignoring non identifiable packet" CR)); +# ifdef LORA_LORAWAN + int lorawan = LORAWANtoJson(packet, packetSize, LORAdata); + if (lorawan == LORAWAN_DROP) return; + if (lorawan == LORAWAN_NOT_HANDLED) { +# endif + if (LORAConfig.onlyKnown) { + THEENGS_LOG_TRACE(F("Ignoring non identifiable packet" CR)); + return; + } + // We have non-ascii data: create hex string of the data + char hex[packetSize * 2 + 2]; // _rawToHex writes "%02X\r" per byte: needs 2 * size + 2 + TheengsUtils::_rawToHex(packet, hex, packetSize); + // Terminate with a null character + hex[packetSize * 2] = 0; + + LORAdata["hex"] = hex; +# ifdef LORA_LORAWAN } - // We have non-ascii data: create hex string of the data - char hex[packetSize * 2 + 2]; // _rawToHex writes "%02X\r" per byte: needs 2 * size + 2 - TheengsUtils::_rawToHex(packet, hex, packetSize); - // Terminate with a null character - hex[packetSize * 2] = 0; - - LORAdata["hex"] = hex; +# endif } else { // ascii payload std::string packetStrStd = (char*)packet; @@ -557,6 +572,9 @@ void XtoLORA(const char* topicOri, JsonObject& LORAdata) { // json object decodi // Load config from json if available LORAConfig_fromJson(LORAdata); +# ifdef LORA_LORAWAN + LORAWANConfig_fromJson(LORAdata); // LoRaWAN devices are stored separately, "save" applies to them too +# endif LORAConfig_apply(); stateLORAMeasures(); } @@ -594,6 +612,9 @@ String stateLORAMeasures() { LORAdata["enablecrc"] = LORAConfig.crc; LORAdata["invertiq"] = LORAConfig.invertIQ; LORAdata["onlyknown"] = LORAConfig.onlyKnown; +# ifdef LORA_LORAWAN + LORAWANState(LORAdata); +# endif LORAdata["origin"] = subjectGTWLORAtoMQTT; enqueueJsonObject(LORAdata); diff --git a/main/gatewayLORA_LoRaWAN.cpp b/main/gatewayLORA_LoRaWAN.cpp new file mode 100644 index 0000000000..ec1e2f1776 --- /dev/null +++ b/main/gatewayLORA_LoRaWAN.cpp @@ -0,0 +1,556 @@ +/* + Theengs OpenMQTTGateway - We Unite Sensors in One Open-Source Interface + + Act as a wifi or ethernet gateway between your 433mhz/infrared IR/BLE signal and a MQTT broker + Send and receiving command by MQTT + + LoRaWAN ABP uplink decoding for the LORA gateway: + - devices (DevAddr + session keys) are configured at runtime and stored in NVS + - the frame MIC is verified with the NwkSKey and the FRMPayload decrypted with the AppSKey + - optionally (LORA_LORAWAN_JS) the payload is decoded by a TTN style JavaScript decoder + (`function Decoder(bytes, port)` or `function decodeUplink(input)`) run with Duktape + - optionally Home Assistant discovery entities are declared per device + + This is meant for single channel reception of ABP devices configured on a fixed frequency and data rate, + it is not a LoRaWAN network server: no join, no downlink, no MAC command handling. + + This file is part of OpenMQTTGateway. + + OpenMQTTGateway is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + OpenMQTTGateway is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . +*/ +#include "User_config.h" + +#if defined(ZgatewayLORA) && defined(LORA_LORAWAN) +# ifndef ESP32 +# error "LORA_LORAWAN is only supported on ESP32" +# endif +# include + +# include + +# include "TheengsCommon.h" +# include "config_LORA.h" +# include "mbedtls/aes.h" +# ifdef ZmqttDiscovery +# include "config_mqttDiscovery.h" +# endif +# ifdef LORA_LORAWAN_JS +# include +# endif + +struct LoRaWANDevice { + uint32_t devAddr; + uint8_t nwkSKey[16]; + uint8_t appSKey[16]; + String name; + String model; + String entities; // serialized JSON array of Home Assistant entity declarations +# ifdef LORA_LORAWAN_JS + String decoder; +# endif + uint32_t lastFcnt; + bool seen; +}; + +static std::vector lorawanDevices; +static bool lorawanDiscoveryPending = false; + +// NVS keys are limited to 15 characters: "lw" + DevAddr for the device, "lj" + DevAddr for its decoder +# define LORAWAN_NVS_INDEX "lwdevs" + +static void lorawanNvsKey(char* key, const char* prefix, uint32_t devAddr) { + snprintf(key, 11, "%s%08X", prefix, devAddr); +} + +static bool lorawanParseKey(const char* hex, uint8_t* key) { + if (!hex || strlen(hex) != 32) return false; + for (int i = 0; i < 32; i++) { + if (!isxdigit(hex[i])) return false; + } + TheengsUtils::_hexToRaw(hex, key, 16); + return true; +} + +static bool lorawanParseDevAddr(const char* hex, uint32_t& devAddr) { + if (!hex || strlen(hex) != 8) return false; + for (int i = 0; i < 8; i++) { + if (!isxdigit(hex[i])) return false; + } + devAddr = strtoul(hex, NULL, 16); + return true; +} + +static LoRaWANDevice* lorawanFind(uint32_t devAddr) { + for (auto& dev : lorawanDevices) { + if (dev.devAddr == devAddr) return &dev; + } + return nullptr; +} + +/*-------------------- Crypto --------------------*/ + +static void lorawanAesBlock(const uint8_t* key, const uint8_t* in, uint8_t* out) { + mbedtls_aes_context ctx; + mbedtls_aes_init(&ctx); + mbedtls_aes_setkey_enc(&ctx, key, 128); + mbedtls_aes_crypt_ecb(&ctx, MBEDTLS_AES_ENCRYPT, in, out); + mbedtls_aes_free(&ctx); +} + +static void lorawanCmacSubkey(const uint8_t* in, uint8_t* out) { + uint8_t carry = 0; + for (int i = 15; i >= 0; i--) { + out[i] = (in[i] << 1) | carry; + carry = in[i] >> 7; + } + if (in[0] & 0x80) out[15] ^= 0x87; +} + +// AES-CMAC (RFC 4493) of B0 | msg, B0 being the LoRaWAN MIC block +static void lorawanMic(const uint8_t* key, const uint8_t* b0, const uint8_t* msg, size_t len, uint8_t* mac) { + uint8_t zero[16] = {0}, l[16], k1[16], k2[16], x[16], y[16]; + lorawanAesBlock(key, zero, l); + lorawanCmacSubkey(l, k1); + lorawanCmacSubkey(k1, k2); + lorawanAesBlock(key, b0, x); // B0 is always a complete, non final block + size_t blocks = (len + 15) / 16; + bool complete = blocks > 0 && (len % 16) == 0; + if (blocks == 0) blocks = 1; + for (size_t b = 0; b < blocks; b++) { + for (int i = 0; i < 16; i++) { + size_t idx = b * 16 + i; + uint8_t m = idx < len ? msg[idx] : (idx == len ? 0x80 : 0x00); + if (b == blocks - 1) m ^= complete ? k1[i] : k2[i]; + y[i] = x[i] ^ m; + } + lorawanAesBlock(key, y, x); + } + memcpy(mac, x, 16); +} + +// LoRaWAN 1.0.x FRMPayload encryption (AES-CTR like, uplink direction) +static void lorawanDecrypt(const uint8_t* key, uint32_t devAddr, uint32_t fcnt, const uint8_t* in, uint8_t* out, size_t len) { + uint8_t a[16] = {0x01, 0, 0, 0, 0, 0, + (uint8_t)devAddr, (uint8_t)(devAddr >> 8), (uint8_t)(devAddr >> 16), (uint8_t)(devAddr >> 24), + (uint8_t)fcnt, (uint8_t)(fcnt >> 8), (uint8_t)(fcnt >> 16), (uint8_t)(fcnt >> 24), 0, 0}; + uint8_t s[16]; + for (size_t i = 0; i < len; i += 16) { + a[15] = (uint8_t)(i / 16 + 1); + lorawanAesBlock(key, a, s); + for (size_t j = 0; j < 16 && i + j < len; j++) out[i + j] = in[i + j] ^ s[j]; + } +} + +/*-------------------- JavaScript decoder --------------------*/ + +# ifdef LORA_LORAWAN_JS +extern void ESPRestart(byte reason); + +// Decoders commonly timestamp their output (new Date()); the clock is otherwise only set for TLS connections. +// Must be called once the network is up. +static void lorawanStartNtp() { + static bool started = false; + if (started || time(nullptr) > 1000000000) return; + configTime(0, 0, NTP_SERVER); + started = true; +} + +static void lorawanDuktapeFatal(void* udata, const char* msg) { + THEENGS_LOG_ERROR(F("[LoRaWAN] JS fatal error: %s" CR), msg ? msg : ""); + ESPRestart(8); // Duktape fatal errors are mostly out of memory, the heap cannot be recovered +} + +// Runs the device decoder, merges the resulting object into LORAdata, returns false (and sets "decoder_error") on failure +static bool lorawanRunDecoder(const LoRaWANDevice& dev, uint8_t fport, const uint8_t* bytes, size_t len, JsonObject& LORAdata) { + lorawanStartNtp(); + duk_context* ctx = duk_create_heap(NULL, NULL, NULL, NULL, lorawanDuktapeFatal); + if (!ctx) { + LORAdata["decoder_error"] = "out of memory"; + return false; + } + bool ok = false; + if (duk_peval_lstring(ctx, dev.decoder.c_str(), dev.decoder.length()) != 0) { + LORAdata["decoder_error"] = duk_safe_to_string(ctx, -1); + } else { + duk_pop(ctx); + // TTN v3 `decodeUplink({bytes, fPort})` returning {data, errors}, else TTN v2 `Decoder(bytes, port)` + bool v3 = duk_get_global_string(ctx, "decodeUplink") && duk_is_function(ctx, -1); + if (!v3) { + duk_pop(ctx); + duk_get_global_string(ctx, "Decoder"); + } + if (!duk_is_function(ctx, -1)) { + LORAdata["decoder_error"] = "no Decoder(bytes, port) or decodeUplink(input) function"; + } else { + if (v3) duk_push_object(ctx); + duk_idx_t arr = duk_push_array(ctx); + for (size_t i = 0; i < len; i++) { + duk_push_uint(ctx, bytes[i]); + duk_put_prop_index(ctx, arr, i); + } + if (v3) { + duk_put_prop_string(ctx, -2, "bytes"); + duk_push_uint(ctx, fport); + duk_put_prop_string(ctx, -2, "fPort"); + } else { + duk_push_uint(ctx, fport); + } + if (duk_pcall(ctx, v3 ? 1 : 2) != DUK_EXEC_SUCCESS) { + LORAdata["decoder_error"] = duk_safe_to_string(ctx, -1); + } else { + if (v3) { + if (duk_get_prop_string(ctx, -1, "errors") && duk_is_array(ctx, -1) && duk_get_length(ctx, -1) > 0) { + duk_get_prop_index(ctx, -1, 0); + LORAdata["decoder_error"] = duk_safe_to_string(ctx, -1); + duk_pop(ctx); + } + duk_pop(ctx); + duk_get_prop_string(ctx, -1, "data"); + } + if (duk_is_object(ctx, -1) && !duk_is_array(ctx, -1)) { + const char* json = duk_json_encode(ctx, -1); + DynamicJsonDocument decoded(JSON_MSG_BUFFER); + DeserializationError error = deserializeJson(decoded, json); + if (error) { + LORAdata["decoder_error"] = error.c_str(); + } else { + for (JsonPair kv : decoded.as()) { + LORAdata[kv.key()] = kv.value(); + } + ok = !LORAdata.containsKey("decoder_error"); + } + } else if (!LORAdata.containsKey("decoder_error")) { + LORAdata["decoder_error"] = "decoder did not return an object"; + } + } + } + } + duk_destroy_heap(ctx); + if (!ok) THEENGS_LOG_WARNING(F("[LoRaWAN] decoder error for %X: %s" CR), dev.devAddr, LORAdata["decoder_error"].as()); + return ok; +} +# endif + +/*-------------------- Uplink decoding --------------------*/ + +int LORAWANtoJson(const uint8_t* p, int len, JsonObject& LORAdata) { + if (len < 12) return LORAWAN_NOT_HANDLED; + uint8_t mtype = p[0] >> 5; + if (mtype != 2 && mtype != 4) return LORAWAN_NOT_HANDLED; // (un)confirmed data up + uint32_t devAddr = p[1] | (p[2] << 8) | (p[3] << 16) | ((uint32_t)p[4] << 24); + LoRaWANDevice* dev = lorawanFind(devAddr); + if (!dev) return LORAWAN_NOT_HANDLED; + + int hdrLen = 8 + (p[5] & 0x0F); // MHDR + DevAddr + FCtrl + FCnt + FOpts + int msgLen = len - 4; // without MIC + if (msgLen < hdrLen) return LORAWAN_NOT_HANDLED; + // Rebuild the 32 bits frame counter from the 16 transmitted bits + uint32_t fcnt = (dev->lastFcnt & 0xFFFF0000) | (p[6] | (p[7] << 8)); + if (dev->seen && fcnt < dev->lastFcnt) fcnt += 0x10000; + + uint8_t b0[16] = {0x49, 0, 0, 0, 0, 0, p[1], p[2], p[3], p[4], + (uint8_t)fcnt, (uint8_t)(fcnt >> 8), (uint8_t)(fcnt >> 16), (uint8_t)(fcnt >> 24), 0, (uint8_t)msgLen}; + uint8_t mac[16]; + lorawanMic(dev->nwkSKey, b0, p, msgLen, mac); + if (memcmp(mac, p + msgLen, 4) != 0) { + THEENGS_LOG_WARNING(F("[LoRaWAN] MIC check failed for %X FCnt %u" CR), devAddr, fcnt); + return LORAWAN_NOT_HANDLED; + } + if (dev->seen && fcnt == dev->lastFcnt) { + THEENGS_LOG_NOTICE(F("[LoRaWAN] Duplicate FCnt %u from %X ignored" CR), fcnt, devAddr); + return LORAWAN_DROP; + } + dev->lastFcnt = fcnt; + dev->seen = true; + + char id[9]; + snprintf(id, sizeof(id), "%08X", devAddr); + LORAdata["id"] = id; + if (dev->name.length()) LORAdata["name"] = dev->name; + if (dev->model.length()) LORAdata["model"] = dev->model; + LORAdata["fcnt"] = fcnt; + if (mtype == 4) LORAdata["confirmed"] = true; + + if (msgLen > hdrLen) { + uint8_t fport = p[hdrLen]; + size_t n = msgLen - hdrLen - 1; + uint8_t plain[n > 0 ? n : 1]; + lorawanDecrypt(fport == 0 ? dev->nwkSKey : dev->appSKey, devAddr, fcnt, p + hdrLen + 1, plain, n); + char hex[n * 2 + 2]; // _rawToHex writes "%02X\r" per byte: needs 2 * size + 2 + TheengsUtils::_rawToHex(plain, hex, n); + hex[n * 2] = 0; + LORAdata["fport"] = fport; + LORAdata["payload"] = hex; +# ifdef LORA_LORAWAN_JS + if (fport != 0 && dev->decoder.length()) lorawanRunDecoder(*dev, fport, plain, n, LORAdata); +# endif + } + return LORAWAN_DECODED; +} + +/*-------------------- Configuration --------------------*/ + +static void lorawanToJson(const LoRaWANDevice& dev, JsonObject& jo, bool withKeys) { + char hex[34]; // _rawToHex writes "%02X\r" per byte: needs 2 * size + 2 + snprintf(hex, 9, "%08X", dev.devAddr); + jo["devaddr"] = hex; + if (withKeys) { + TheengsUtils::_rawToHex((byte*)dev.nwkSKey, hex, 16); + hex[32] = 0; + jo["nwkskey"] = hex; + TheengsUtils::_rawToHex((byte*)dev.appSKey, hex, 16); + hex[32] = 0; + jo["appskey"] = hex; + } + if (dev.name.length()) jo["name"] = dev.name; + if (dev.model.length()) jo["model"] = dev.model; + if (dev.entities.length()) { + if (withKeys) + jo["entities"] = serialized(dev.entities); + else + jo["entities"] = true; + } +# ifdef LORA_LORAWAN_JS + if (!withKeys) jo["decoder"] = dev.decoder.length() > 0; +# endif + if (!withKeys && dev.seen) jo["fcnt"] = dev.lastFcnt; +} + +static void lorawanSaveIndex() { + String index; + for (auto& dev : lorawanDevices) { + char hex[9]; + snprintf(hex, sizeof(hex), "%08X", dev.devAddr); + if (index.length()) index += ","; + index += hex; + } + preferences.begin(Gateway_Short_Name, false); + preferences.putString(LORAWAN_NVS_INDEX, index); + preferences.end(); +} + +static void lorawanSaveDevice(const LoRaWANDevice& dev) { + StaticJsonDocument doc; + JsonObject jo = doc.to(); + lorawanToJson(dev, jo, true); + String conf; + serializeJson(doc, conf); + char key[11]; + lorawanNvsKey(key, "lw", dev.devAddr); + preferences.begin(Gateway_Short_Name, false); + int result = preferences.putString(key, conf); + preferences.end(); + THEENGS_LOG_NOTICE(F("[LoRaWAN] device %X saved, result: %d" CR), dev.devAddr, result); +} + +static void lorawanEraseDevice(uint32_t devAddr) { + char key[11]; + preferences.begin(Gateway_Short_Name, false); + lorawanNvsKey(key, "lw", devAddr); + preferences.remove(key); + lorawanNvsKey(key, "lj", devAddr); + preferences.remove(key); + preferences.end(); +} + +// Applies one device declaration, returns false if invalid +static bool lorawanDeviceFromJson(JsonObject& jo, bool save) { + uint32_t devAddr; + if (!lorawanParseDevAddr(jo["devaddr"], devAddr)) { + THEENGS_LOG_ERROR(F("[LoRaWAN] invalid or missing devaddr (8 hex digits)" CR)); + return false; + } + LoRaWANDevice* dev = lorawanFind(devAddr); + if (jo["remove"] | false) { + if (dev) { + lorawanDevices.erase(lorawanDevices.begin() + (dev - lorawanDevices.data())); + THEENGS_LOG_NOTICE(F("[LoRaWAN] device %X removed" CR), devAddr); + } + if (save) { + lorawanEraseDevice(devAddr); + lorawanSaveIndex(); + } + return true; + } + LoRaWANDevice candidate = dev ? *dev : LoRaWANDevice{devAddr, {0}, {0}, "", "", "", +# ifdef LORA_LORAWAN_JS + "", +# endif + 0, + false}; + if (jo.containsKey("nwkskey") && !lorawanParseKey(jo["nwkskey"], candidate.nwkSKey)) { + THEENGS_LOG_ERROR(F("[LoRaWAN] invalid nwkskey (32 hex digits)" CR)); + return false; + } + if (jo.containsKey("appskey") && !lorawanParseKey(jo["appskey"], candidate.appSKey)) { + THEENGS_LOG_ERROR(F("[LoRaWAN] invalid appskey (32 hex digits)" CR)); + return false; + } + if (!dev && !(jo.containsKey("nwkskey") && jo.containsKey("appskey"))) { + THEENGS_LOG_ERROR(F("[LoRaWAN] new device %X requires nwkskey and appskey" CR), devAddr); + return false; + } + if (jo.containsKey("name")) candidate.name = jo["name"].as(); + if (jo.containsKey("model")) candidate.model = jo["model"].as(); + if (jo.containsKey("entities")) { + candidate.entities = ""; + if (jo["entities"].is()) serializeJson(jo["entities"], candidate.entities); + lorawanDiscoveryPending = true; + } + if (dev) { + *dev = candidate; + } else { + if (lorawanDevices.size() >= LORAWAN_MAX_DEVICES) { + THEENGS_LOG_ERROR(F("[LoRaWAN] maximum number of devices (%d) reached" CR), LORAWAN_MAX_DEVICES); + return false; + } + lorawanDevices.push_back(candidate); + } + THEENGS_LOG_NOTICE(F("[LoRaWAN] device %X configured" CR), devAddr); + if (save) { + lorawanSaveDevice(candidate); + lorawanSaveIndex(); + } + return true; +} + +// Handles the "lorawan" key of the LORA config command: an object or an array of device declarations +void LORAWANConfig_fromJson(JsonObject& LORAdata) { + if (!LORAdata.containsKey("lorawan")) return; + bool save = LORAdata["save"] | false; + if (LORAdata["lorawan"].is()) { + for (JsonObject jo : LORAdata["lorawan"].as()) lorawanDeviceFromJson(jo, save); + } else if (LORAdata["lorawan"].is()) { + JsonObject jo = LORAdata["lorawan"].as(); + lorawanDeviceFromJson(jo, save); + } +} + +// Adds the configured devices (without keys) to the LORA state message +void LORAWANState(JsonObject& LORAdata) { + JsonArray devices = LORAdata.createNestedArray("lorawan"); + for (auto& dev : lorawanDevices) { + JsonObject jo = devices.createNestedObject(); + lorawanToJson(dev, jo, false); + } +} + +void LORAWANsetup() { + lorawanDevices.clear(); + preferences.begin(Gateway_Short_Name, true); + String index = preferences.isKey(LORAWAN_NVS_INDEX) ? preferences.getString(LORAWAN_NVS_INDEX, "") : ""; + preferences.end(); + int start = 0; + while (start < (int)index.length()) { + int end = index.indexOf(',', start); + if (end < 0) end = index.length(); + String hex = index.substring(start, end); + start = end + 1; + uint32_t devAddr; + if (!lorawanParseDevAddr(hex.c_str(), devAddr)) continue; + char key[11]; + lorawanNvsKey(key, "lw", devAddr); + preferences.begin(Gateway_Short_Name, true); + String conf = preferences.isKey(key) ? preferences.getString(key, "") : ""; + preferences.end(); + StaticJsonDocument doc; + if (conf.length() == 0 || deserializeJson(doc, conf)) { + THEENGS_LOG_ERROR(F("[LoRaWAN] cannot load device %X" CR), devAddr); + continue; + } + JsonObject jo = doc.as(); + if (!lorawanDeviceFromJson(jo, false)) continue; +# ifdef LORA_LORAWAN_JS + lorawanNvsKey(key, "lj", devAddr); + preferences.begin(Gateway_Short_Name, true); + if (preferences.isKey(key)) lorawanFind(devAddr)->decoder = preferences.getString(key, ""); + preferences.end(); +# endif + } + THEENGS_LOG_NOTICE(F("[LoRaWAN] %d device(s) loaded" CR), lorawanDevices.size()); + lorawanDiscoveryPending = true; +} + +# ifdef LORA_LORAWAN_JS +// Decoder scripts are received raw (not JSON) on /LORAWANdecoder/; an empty payload removes the decoder +void LORAWANdecoderFromMQTT(const char* topic, const char* payload) { + const char* slash = strrchr(topic, '/'); + uint32_t devAddr; + if (!slash || !lorawanParseDevAddr(slash + 1, devAddr)) { + THEENGS_LOG_ERROR(F("[LoRaWAN] decoder topic must end with the 8 hex digits DevAddr: %s" CR), topic); + return; + } + LoRaWANDevice* dev = lorawanFind(devAddr); + if (!dev) { + THEENGS_LOG_ERROR(F("[LoRaWAN] decoder received for unknown device %X, configure it first" CR), devAddr); + return; + } + dev->decoder = payload; + char key[11]; + lorawanNvsKey(key, "lj", devAddr); + preferences.begin(Gateway_Short_Name, false); + if (dev->decoder.length()) { + size_t result = preferences.putString(key, dev->decoder); + THEENGS_LOG_NOTICE(F("[LoRaWAN] decoder for %X saved (%d bytes), result: %d" CR), devAddr, dev->decoder.length(), result); + } else { + preferences.remove(key); + THEENGS_LOG_NOTICE(F("[LoRaWAN] decoder for %X removed" CR), devAddr); + } + preferences.end(); +} +# endif + +/*-------------------- Home Assistant discovery --------------------*/ + +# ifdef ZmqttDiscovery +// Entities are declared per device, e.g. +// [{"key":"door","type":"binary_sensor","class":"door","on":"open","off":"closed"},{"key":"volt","class":"voltage","unit":"V"}] +// Must be called from the main loop as it publishes directly +void launchLORAWANDiscovery(bool overrideDiscovery) { +# ifdef LORA_LORAWAN_JS + if (overrideDiscovery) lorawanStartNtp(); // called on MQTT connection: the network is up +# endif + if (!overrideDiscovery && !lorawanDiscoveryPending) return; + lorawanDiscoveryPending = false; + for (auto& dev : lorawanDevices) { + if (!dev.entities.length()) continue; + StaticJsonDocument doc; + if (deserializeJson(doc, dev.entities)) { + THEENGS_LOG_ERROR(F("[LoRaWAN] invalid entities for %X" CR), dev.devAddr); + continue; + } + char id[9]; + snprintf(id, sizeof(id), "%08X", dev.devAddr); + String topic = String(subjectLORAtoMQTT) + "/" + id; + const char* deviceName = dev.name.length() ? dev.name.c_str() : id; + for (JsonObject e : doc.as()) { + const char* key = e["key"]; + if (!key) continue; + const char* type = e["type"] | "sensor"; + bool binary = strcmp(type, "binary_sensor") == 0; + String uniqueId = String(id) + "-" + key; + String valueTemplate = String("{{ value_json.") + key + " | is_defined }}"; + createDiscovery(type, + (char*)topic.c_str(), e["name"] | key, (char*)uniqueId.c_str(), + "", e["class"] | "", (char*)valueTemplate.c_str(), + e["on"] | (binary ? "true" : ""), e["off"] | (binary ? "false" : ""), e["unit"] | "", + 0, "", "", false, "", + deviceName, "", dev.model.c_str(), id, false, + binary ? stateClassNone : (e["state_class"] | stateClassMeasurement)); + } + } +} +# endif + +#endif diff --git a/main/main.cpp b/main/main.cpp index 43efb6b6fd..3855ec7eba 100644 --- a/main/main.cpp +++ b/main/main.cpp @@ -1154,6 +1154,12 @@ void setupMQTT() { mqtt->subscribe(String(mqtt_topic) + gateway_name + subjectMQTTtoX, receivingDATA, mqtt_max_payload_size); +# if defined(ZgatewayLORA) && defined(LORA_LORAWAN_JS) + // LoRaWAN payload decoder scripts are raw JavaScript, larger than the JSON commands + mqtt->subscribe( + String(mqtt_topic) + gateway_name + subjectLORAWANdecoder + "/+", [](char* topic, char* payload) { LORAWANdecoderFromMQTT(topic, payload); }, LORAWAN_DECODER_MAX_SIZE + 1); +# endif + # ifdef ZgatewayRF // subject on which other OMG will publish, this OMG will store these msg and by the way don't republish them if they have been already published mqtt->subscribe(subjectMultiGTWRF, receivingDATA, mqtt_max_payload_size); diff --git a/platformio.ini b/platformio.ini index ac6a5664df..972c04804a 100644 --- a/platformio.ini +++ b/platformio.ini @@ -125,6 +125,7 @@ newremoteswitch = https://github.com/1technophile/NewRemoteSwitch/archive/8eb980 ble = https://github.com/h2zero/NimBLE-Arduino/archive/2.5.0.zip irremoteesp = https://github.com/theengs/IRremoteESP8266/archive/arduinov3.zip lora = https://github.com/sandeepmistry/arduino-LoRa/archive/f4a1d27.zip +duktape = https://github.com/svaarala/duktape-releases/archive/refs/tags/v2.7.0.zip esppilight = https://github.com/theengs/ESPiLight/archive/b9dd7a1.zip rfWeatherStation = WeatherStationDataRx@0.3.1 rfm69 = https://github.com/LowPowerLab/RFM69/archive/2e915ea.zip From bceee91e8ff7267b4a9bd77056176679dd8684de Mon Sep 17 00:00:00 2001 From: eric Date: Fri, 9 Oct 2026 11:50:05 -0500 Subject: [PATCH 03/11] [LORA] WebUI page to manage LoRaWAN devices and decoders Adds "Configure LoRaWAN" (/lw) to the configuration menu when built with LORA_LORAWAN: - list of the declared devices with their decoder status and last FCnt - add/edit form for DevAddr, name, model, session keys, Home Assistant entities (JSON) and the JavaScript decoder; stored keys are never sent back to the browser (only a hint of their last bytes), an empty key field keeps the current key - "Test decoder" runs the decoder being edited on a payload, prefilled with the last decrypted payload of the device, without saving - delete button LORAWANconfigureDevice() now reports configuration errors to the caller, decoder storage is shared between MQTT and the WebUI (LORAWANsetDecoder), and device declarations are serialized with a larger buffer so longer entity lists are not truncated. Co-Authored-By: Claude Opus 5.5 --- main/config_LORA.h | 20 ++++ main/config_WebContent.h | 6 +- main/gatewayLORA_LoRaWAN.cpp | 106 ++++++++++------- main/webUI.cpp | 216 +++++++++++++++++++++++++++++++++++ 4 files changed, 303 insertions(+), 45 deletions(-) diff --git a/main/config_LORA.h b/main/config_LORA.h index fac2584c1f..3425a91222 100644 --- a/main/config_LORA.h +++ b/main/config_LORA.h @@ -81,6 +81,24 @@ extern void XtoLORA(const char* topicOri, JsonObject& RFdata); # define LORAWAN_NOT_HANDLED 0 // not a data uplink from a configured device (or MIC check failed) # define LORAWAN_DECODED 1 # define LORAWAN_DROP 2 // duplicate frame +# include +struct LoRaWANDevice { + uint32_t devAddr; + uint8_t nwkSKey[16]; + uint8_t appSKey[16]; + String name; + String model; + String entities; // serialized JSON array of Home Assistant entity declarations +# ifdef LORA_LORAWAN_JS + String decoder; +# endif + uint32_t lastFcnt; + bool seen; + String lastPayload; // last decrypted FRMPayload (hex), to test decoders from the WebUI + uint8_t lastPort; +}; +extern const std::vector& LORAWANdevices(); +extern bool LORAWANconfigureDevice(JsonObject& device, bool save, String* error); extern void LORAWANsetup(); extern int LORAWANtoJson(const uint8_t* packet, int packetSize, JsonObject& LORAdata); extern void LORAWANConfig_fromJson(JsonObject& LORAdata); @@ -96,6 +114,8 @@ extern void launchLORAWANDiscovery(bool overrideDiscovery); # define LORAWAN_DECODER_MAX_SIZE 3800 // bytes, NVS strings are limited to ~4000 bytes # endif extern void LORAWANdecoderFromMQTT(const char* topic, const char* payload); +extern bool LORAWANsetDecoder(uint32_t devAddr, const String& script); +extern bool LORAWANtestDecoder(const String& script, uint8_t fport, const uint8_t* bytes, size_t len, JsonObject& result); # endif #endif diff --git a/main/config_WebContent.h b/main/config_WebContent.h index 11d03697e2..ad9242c86f 100644 --- a/main/config_WebContent.h +++ b/main/config_WebContent.h @@ -67,7 +67,11 @@ #else # define configure_7 #endif -#define configure_8 +#if defined(ZgatewayLORA) && defined(LORA_LORAWAN) +# define configure_8 "

" +#else +# define configure_8 +#endif /*------------------- ----------------------*/ diff --git a/main/gatewayLORA_LoRaWAN.cpp b/main/gatewayLORA_LoRaWAN.cpp index ec1e2f1776..bad90a65e5 100644 --- a/main/gatewayLORA_LoRaWAN.cpp +++ b/main/gatewayLORA_LoRaWAN.cpp @@ -49,25 +49,16 @@ # include # endif -struct LoRaWANDevice { - uint32_t devAddr; - uint8_t nwkSKey[16]; - uint8_t appSKey[16]; - String name; - String model; - String entities; // serialized JSON array of Home Assistant entity declarations -# ifdef LORA_LORAWAN_JS - String decoder; -# endif - uint32_t lastFcnt; - bool seen; -}; - static std::vector lorawanDevices; + +const std::vector& LORAWANdevices() { + return lorawanDevices; +} static bool lorawanDiscoveryPending = false; // NVS keys are limited to 15 characters: "lw" + DevAddr for the device, "lj" + DevAddr for its decoder -# define LORAWAN_NVS_INDEX "lwdevs" +# define LORAWAN_NVS_INDEX "lwdevs" +# define LORAWAN_DEVICE_JSON_SIZE 3072 // device declaration with keys and entities, NVS strings are limited to ~4000 bytes static void lorawanNvsKey(char* key, const char* prefix, uint32_t devAddr) { snprintf(key, 11, "%s%08X", prefix, devAddr); @@ -172,7 +163,7 @@ static void lorawanDuktapeFatal(void* udata, const char* msg) { } // Runs the device decoder, merges the resulting object into LORAdata, returns false (and sets "decoder_error") on failure -static bool lorawanRunDecoder(const LoRaWANDevice& dev, uint8_t fport, const uint8_t* bytes, size_t len, JsonObject& LORAdata) { +static bool lorawanRunDecoder(const String& script, uint8_t fport, const uint8_t* bytes, size_t len, JsonObject& LORAdata) { lorawanStartNtp(); duk_context* ctx = duk_create_heap(NULL, NULL, NULL, NULL, lorawanDuktapeFatal); if (!ctx) { @@ -180,7 +171,7 @@ static bool lorawanRunDecoder(const LoRaWANDevice& dev, uint8_t fport, const uin return false; } bool ok = false; - if (duk_peval_lstring(ctx, dev.decoder.c_str(), dev.decoder.length()) != 0) { + if (duk_peval_lstring(ctx, script.c_str(), script.length()) != 0) { LORAdata["decoder_error"] = duk_safe_to_string(ctx, -1); } else { duk_pop(ctx); @@ -237,9 +228,13 @@ static bool lorawanRunDecoder(const LoRaWANDevice& dev, uint8_t fport, const uin } } duk_destroy_heap(ctx); - if (!ok) THEENGS_LOG_WARNING(F("[LoRaWAN] decoder error for %X: %s" CR), dev.devAddr, LORAdata["decoder_error"].as()); return ok; } + +// Runs a decoder script on the given bytes without any device (used to test decoders from the WebUI) +bool LORAWANtestDecoder(const String& script, uint8_t fport, const uint8_t* bytes, size_t len, JsonObject& result) { + return lorawanRunDecoder(script, fport, bytes, len, result); +} # endif /*-------------------- Uplink decoding --------------------*/ @@ -292,8 +287,11 @@ int LORAWANtoJson(const uint8_t* p, int len, JsonObject& LORAdata) { hex[n * 2] = 0; LORAdata["fport"] = fport; LORAdata["payload"] = hex; + dev->lastPayload = hex; + dev->lastPort = fport; # ifdef LORA_LORAWAN_JS - if (fport != 0 && dev->decoder.length()) lorawanRunDecoder(*dev, fport, plain, n, LORAdata); + if (fport != 0 && dev->decoder.length() && !lorawanRunDecoder(dev->decoder, fport, plain, n, LORAdata)) + THEENGS_LOG_WARNING(F("[LoRaWAN] decoder error for %X: %s" CR), devAddr, LORAdata["decoder_error"].as()); # endif } return LORAWAN_DECODED; @@ -341,7 +339,7 @@ static void lorawanSaveIndex() { } static void lorawanSaveDevice(const LoRaWANDevice& dev) { - StaticJsonDocument doc; + DynamicJsonDocument doc(LORAWAN_DEVICE_JSON_SIZE); JsonObject jo = doc.to(); lorawanToJson(dev, jo, true); String conf; @@ -365,10 +363,18 @@ static void lorawanEraseDevice(uint32_t devAddr) { } // Applies one device declaration, returns false if invalid -static bool lorawanDeviceFromJson(JsonObject& jo, bool save) { +// Logs a configuration error and reports it to the caller (WebUI) when requested +# define LORAWAN_CONFIG_ERROR(error, msg) \ + do { \ + THEENGS_LOG_ERROR(F("[LoRaWAN] %s" CR), msg); \ + if (error) *error = msg; \ + } while (0) + +// Applies one device declaration, returns false (and the reason in error, if given) if invalid +bool LORAWANconfigureDevice(JsonObject& jo, bool save, String* error) { uint32_t devAddr; if (!lorawanParseDevAddr(jo["devaddr"], devAddr)) { - THEENGS_LOG_ERROR(F("[LoRaWAN] invalid or missing devaddr (8 hex digits)" CR)); + LORAWAN_CONFIG_ERROR(error, "invalid or missing devaddr (8 hex digits)"); return false; } LoRaWANDevice* dev = lorawanFind(devAddr); @@ -388,31 +394,33 @@ static bool lorawanDeviceFromJson(JsonObject& jo, bool save) { "", # endif 0, - false}; + false, + "", + 0}; if (jo.containsKey("nwkskey") && !lorawanParseKey(jo["nwkskey"], candidate.nwkSKey)) { - THEENGS_LOG_ERROR(F("[LoRaWAN] invalid nwkskey (32 hex digits)" CR)); + LORAWAN_CONFIG_ERROR(error, "invalid nwkskey (32 hex digits)"); return false; } if (jo.containsKey("appskey") && !lorawanParseKey(jo["appskey"], candidate.appSKey)) { - THEENGS_LOG_ERROR(F("[LoRaWAN] invalid appskey (32 hex digits)" CR)); + LORAWAN_CONFIG_ERROR(error, "invalid appskey (32 hex digits)"); return false; } if (!dev && !(jo.containsKey("nwkskey") && jo.containsKey("appskey"))) { - THEENGS_LOG_ERROR(F("[LoRaWAN] new device %X requires nwkskey and appskey" CR), devAddr); + LORAWAN_CONFIG_ERROR(error, "a new device requires nwkskey and appskey"); return false; } if (jo.containsKey("name")) candidate.name = jo["name"].as(); if (jo.containsKey("model")) candidate.model = jo["model"].as(); if (jo.containsKey("entities")) { candidate.entities = ""; - if (jo["entities"].is()) serializeJson(jo["entities"], candidate.entities); + if (jo["entities"].is() && jo["entities"].size() > 0) serializeJson(jo["entities"], candidate.entities); lorawanDiscoveryPending = true; } if (dev) { *dev = candidate; } else { if (lorawanDevices.size() >= LORAWAN_MAX_DEVICES) { - THEENGS_LOG_ERROR(F("[LoRaWAN] maximum number of devices (%d) reached" CR), LORAWAN_MAX_DEVICES); + LORAWAN_CONFIG_ERROR(error, "maximum number of devices reached (LORAWAN_MAX_DEVICES)"); return false; } lorawanDevices.push_back(candidate); @@ -430,10 +438,10 @@ void LORAWANConfig_fromJson(JsonObject& LORAdata) { if (!LORAdata.containsKey("lorawan")) return; bool save = LORAdata["save"] | false; if (LORAdata["lorawan"].is()) { - for (JsonObject jo : LORAdata["lorawan"].as()) lorawanDeviceFromJson(jo, save); + for (JsonObject jo : LORAdata["lorawan"].as()) LORAWANconfigureDevice(jo, save, nullptr); } else if (LORAdata["lorawan"].is()) { JsonObject jo = LORAdata["lorawan"].as(); - lorawanDeviceFromJson(jo, save); + LORAWANconfigureDevice(jo, save, nullptr); } } @@ -464,13 +472,13 @@ void LORAWANsetup() { preferences.begin(Gateway_Short_Name, true); String conf = preferences.isKey(key) ? preferences.getString(key, "") : ""; preferences.end(); - StaticJsonDocument doc; + DynamicJsonDocument doc(LORAWAN_DEVICE_JSON_SIZE); if (conf.length() == 0 || deserializeJson(doc, conf)) { THEENGS_LOG_ERROR(F("[LoRaWAN] cannot load device %X" CR), devAddr); continue; } JsonObject jo = doc.as(); - if (!lorawanDeviceFromJson(jo, false)) continue; + if (!LORAWANconfigureDevice(jo, false, nullptr)) continue; # ifdef LORA_LORAWAN_JS lorawanNvsKey(key, "lj", devAddr); preferences.begin(Gateway_Short_Name, true); @@ -483,31 +491,41 @@ void LORAWANsetup() { } # ifdef LORA_LORAWAN_JS -// Decoder scripts are received raw (not JSON) on /LORAWANdecoder/; an empty payload removes the decoder -void LORAWANdecoderFromMQTT(const char* topic, const char* payload) { - const char* slash = strrchr(topic, '/'); - uint32_t devAddr; - if (!slash || !lorawanParseDevAddr(slash + 1, devAddr)) { - THEENGS_LOG_ERROR(F("[LoRaWAN] decoder topic must end with the 8 hex digits DevAddr: %s" CR), topic); - return; - } +// Sets (and stores) the decoder of a configured device, an empty script removes it +bool LORAWANsetDecoder(uint32_t devAddr, const String& script) { LoRaWANDevice* dev = lorawanFind(devAddr); if (!dev) { THEENGS_LOG_ERROR(F("[LoRaWAN] decoder received for unknown device %X, configure it first" CR), devAddr); - return; + return false; + } + if (script.length() > LORAWAN_DECODER_MAX_SIZE) { + THEENGS_LOG_ERROR(F("[LoRaWAN] decoder for %X exceeds %d bytes" CR), devAddr, LORAWAN_DECODER_MAX_SIZE); + return false; } - dev->decoder = payload; + dev->decoder = script; char key[11]; lorawanNvsKey(key, "lj", devAddr); preferences.begin(Gateway_Short_Name, false); if (dev->decoder.length()) { size_t result = preferences.putString(key, dev->decoder); THEENGS_LOG_NOTICE(F("[LoRaWAN] decoder for %X saved (%d bytes), result: %d" CR), devAddr, dev->decoder.length(), result); - } else { + } else if (preferences.isKey(key)) { preferences.remove(key); THEENGS_LOG_NOTICE(F("[LoRaWAN] decoder for %X removed" CR), devAddr); } preferences.end(); + return true; +} + +// Decoder scripts are received raw (not JSON) on /LORAWANdecoder/; an empty payload removes the decoder +void LORAWANdecoderFromMQTT(const char* topic, const char* payload) { + const char* slash = strrchr(topic, '/'); + uint32_t devAddr; + if (!slash || !lorawanParseDevAddr(slash + 1, devAddr)) { + THEENGS_LOG_ERROR(F("[LoRaWAN] decoder topic must end with the 8 hex digits DevAddr: %s" CR), topic); + return; + } + LORAWANsetDecoder(devAddr, payload); } # endif @@ -525,7 +543,7 @@ void launchLORAWANDiscovery(bool overrideDiscovery) { lorawanDiscoveryPending = false; for (auto& dev : lorawanDevices) { if (!dev.entities.length()) continue; - StaticJsonDocument doc; + DynamicJsonDocument doc(LORAWAN_DEVICE_JSON_SIZE); if (deserializeJson(doc, dev.entities)) { THEENGS_LOG_ERROR(F("[LoRaWAN] invalid entities for %X" CR), dev.devAddr); continue; diff --git a/main/webUI.cpp b/main/webUI.cpp index 134b09c01d..5c72b555de 100644 --- a/main/webUI.cpp +++ b/main/webUI.cpp @@ -1331,6 +1331,219 @@ void handleLA() { LORAConfig.onlyKnown ? "checked" : ""); sendFooterChunk(); } + +# ifdef LORA_LORAWAN +# include + +static String htmlEscape(const String& in) { + String out; + out.reserve(in.length() + 16); + for (size_t i = 0; i < in.length(); i++) { + char c = in[i]; + switch (c) { + case '&': + out += "&"; + break; + case '<': + out += "<"; + break; + case '>': + out += ">"; + break; + case '"': + out += """; + break; + case '\'': + out += "'"; + break; + default: + out += c; + } + } + return out; +} + +static const LoRaWANDevice* findLoRaWANDevice(const String& devAddrHex) { + uint32_t devAddr = strtoul(devAddrHex.c_str(), NULL, 16); + for (auto& dev : LORAWANdevices()) { + if (dev.devAddr == devAddr) return &dev; + } + return nullptr; +} + +static String keyHint(const uint8_t* key) { + char hint[64]; + snprintf(hint, sizeof(hint), "set, ends with ...%02X%02X - leave empty to keep", key[14], key[15]); + return hint; +} + +/** + * @brief /LW - Configure LoRaWAN devices: list (GET), edit form (GET ?d= or ?d=new), save/test/delete (POST) + * Form fields: da devaddr, nm name, md model, nk nwkskey, ak appskey, en entities (JSON array), dc decoder, + * tp test payload (hex), tf test port, nw 1 for a new device; buttons sv save, ts test decoder, rm delete + */ +void handleLW() { + WEBUI_TRACE_LOG(F("handleLW: uri: %s, args: %d, method: %d" CR), server.uri(), server.args(), server.method()); + WEBUI_SECURE + String message; + bool showForm = false; + bool isNew = server.arg("nw") == "1"; + String fDa = server.arg("da"), fNm = server.arg("nm"), fMd = server.arg("md"), fEn = server.arg("en"); + String fDc = server.arg("dc"), fTp = server.arg("tp"), fTf = server.hasArg("tf") ? server.arg("tf") : "1"; + String testOutput; + fDa.trim(); + fDa.toUpperCase(); + + if (server.hasArg("rm")) { + DynamicJsonDocument doc(128); + JsonObject jo = doc.to(); + jo["devaddr"] = fDa; + jo["remove"] = true; + String error; + message = LORAWANconfigureDevice(jo, true, &error) ? "Device " + fDa + " removed" : "Error: " + error; + } else if (server.hasArg("sv") || server.hasArg("ts")) { + showForm = true; + if (server.hasArg("ts")) { +# ifdef LORA_LORAWAN_JS + String hex = fTp; + hex.replace(" ", ""); + size_t n = hex.length() / 2; + uint8_t bytes[n > 0 ? n : 1]; + if (hex.length() % 2 || !TheengsUtils::_hexToRaw(hex.c_str(), bytes, n)) { + testOutput = "Test payload must be an even number of hex digits"; + } else { + DynamicJsonDocument result(JSON_MSG_BUFFER); + JsonObject ro = result.to(); + LORAWANtestDecoder(fDc, (uint8_t)fTf.toInt(), bytes, n, ro); + serializeJsonPretty(result, testOutput); + } +# endif + } else { + DynamicJsonDocument doc(4096); + JsonObject jo = doc.to(); + jo["devaddr"] = fDa; + jo["name"] = fNm; + jo["model"] = fMd; + String nk = server.arg("nk"), ak = server.arg("ak"); + nk.trim(); + ak.trim(); + if (nk.length()) jo["nwkskey"] = nk; + if (ak.length()) jo["appskey"] = ak; + String trimmedEntities = fEn; + trimmedEntities.trim(); + bool valid = true; + if (trimmedEntities.length()) { + DynamicJsonDocument entities(2048); + DeserializationError error = deserializeJson(entities, trimmedEntities); + if (error || !entities.is()) { + message = String("Error: entities must be a JSON array (") + (error ? error.c_str() : "not an array") + ")"; + valid = false; + } else { + jo["entities"] = entities.as(); + } + } else { + jo.createNestedArray("entities"); // clears the entities + } + String error; + if (valid && !LORAWANconfigureDevice(jo, true, &error)) { + message = "Error: " + error; + valid = false; + } +# ifdef LORA_LORAWAN_JS + if (valid) { + const LoRaWANDevice* dev = findLoRaWANDevice(fDa); + if (dev && dev->decoder != fDc && !LORAWANsetDecoder(dev->devAddr, fDc)) { + message = "Error: decoder not saved (maximum " + String(LORAWAN_DECODER_MAX_SIZE) + " bytes)"; + valid = false; + } + } +# endif + if (valid) { + message = "Device " + fDa + " saved"; + showForm = false; + } + } + } else if (server.hasArg("d")) { + showForm = true; + isNew = server.arg("d") == "new"; + if (!isNew) { + const LoRaWANDevice* dev = findLoRaWANDevice(server.arg("d")); + if (!dev) { + message = "Unknown device"; + showForm = false; + } else { + char hex[9]; + snprintf(hex, sizeof(hex), "%08X", dev->devAddr); + fDa = hex; + fNm = dev->name; + fMd = dev->model; + fEn = ""; + if (dev->entities.length()) { + DynamicJsonDocument entities(2048); + if (!deserializeJson(entities, dev->entities)) serializeJsonPretty(entities, fEn); + } +# ifdef LORA_LORAWAN_JS + fDc = dev->decoder; + fTp = dev->lastPayload; + if (dev->lastPayload.length()) fTf = String(dev->lastPort); +# endif + } + } + } + + char jsonChar[100]; + serializeJson(modules, jsonChar, measureJson(modules) + 1); + beginChunkedResponse(); + sendHeaderChunk((String(gateway_name) + " - Configure LoRaWAN").c_str()); + server.sendContent(script); + server.sendContent(style); + sendBodyChunk(body_header, jsonChar, gateway_name); + server.sendContent("
LoRaWAN devices"); + if (message.length()) server.sendContent("

" + htmlEscape(message) + "

"); + + if (!showForm) { + server.sendContent( + "" +# ifdef LORA_LORAWAN_JS + "" +# endif + ""); + for (auto& dev : LORAWANdevices()) { + char hex[9]; + snprintf(hex, sizeof(hex), "%08X", dev.devAddr); + String row = String(""; +# ifdef LORA_LORAWAN_JS + row += String(""; +# endif + row += String(""; + server.sendContent(row); + } + server.sendContent("
DevAddrNameModelDecoderFCnt
" + hex + "" + htmlEscape(dev.name) + "" + htmlEscape(dev.model) + "") + (dev.decoder.length() ? "yes" : "no") + "") + (dev.seen ? String(dev.lastFcnt) : String("-")) + "

"); + } else { + const LoRaWANDevice* dev = isNew ? nullptr : findLoRaWANDevice(fDa); + server.sendContent(String("
"); + server.sendContent("

DevAddr (8 hex digits)

"); + server.sendContent("

Name

"); + server.sendContent("

Model

"); + server.sendContent("

NwkSKey (32 hex digits)

"); + server.sendContent("

AppSKey (32 hex digits)

"); + server.sendContent("

Home Assistant entities (JSON array, optional)

"); +# ifdef LORA_LORAWAN_JS + server.sendContent("

Decoder (function Decoder(bytes, port) or function decodeUplink(input), max " + String(LORAWAN_DECODER_MAX_SIZE) + " bytes, empty for none)

"); + server.sendContent("

Test the decoder above (not saved): payload hex and port

"); + server.sendContent("

"); + if (testOutput.length()) server.sendContent("
" + htmlEscape(testOutput) + "
"); +# endif + server.sendContent("
"); + if (dev) { + server.sendContent("

"); + } + server.sendContent("

"); + } + server.sendContent(body_footer_config_menu); + sendFooterChunk(); +} +# endif # elif defined(ZgatewayRTL_433) || defined(ZgatewayPilight) || defined(ZgatewayRF) || defined(ZgatewayRF2) || defined(ZactuatorSomfy) # include @@ -1844,6 +2057,9 @@ void WebUISetup() { server.on("/wu", handleWU); // Configure WebUI # ifdef ZgatewayLORA server.on("/la", handleLA); // Configure LORA +# ifdef LORA_LORAWAN + server.on("/lw", handleLW); // Configure LoRaWAN devices +# endif # elif defined(ZgatewayRTL_433) || defined(ZgatewayPilight) || defined(ZgatewayRF) || defined(ZgatewayRF2) || defined(ZactuatorSomfy) server.on("/rf", handleRF); // Configure RF # endif From 73ed04470537709059ab2c4bbcd2598d2bce3465 Mon Sep 17 00:00:00 2001 From: eric Date: Fri, 9 Oct 2026 12:04:23 -0500 Subject: [PATCH 04/11] [LORA] Document the Configure LoRaWAN WebUI page Co-Authored-By: Claude Opus 5.5 --- docs/use/lora.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/use/lora.md b/docs/use/lora.md index 176f9dd848..9c64ec0000 100644 --- a/docs/use/lora.md +++ b/docs/use/lora.md @@ -102,7 +102,9 @@ build_flags = ``` ### Declaring a device -Add the device with its DevAddr and session keys through the LoRa configuration command. Use `"save":true` to keep it after a restart: +The easiest way is the WebUI: open **Configuration > Configure LoRaWAN**, then **Add device**. The page lists the declared devices. It lets you edit their name, model, session keys, Home Assistant entities and decoder, and test a decoder on a payload before saving. Stored keys are never shown, and an empty key field keeps the current key. + +You can also add the device with its DevAddr and session keys through the LoRa configuration command. Use `"save":true` to keep it after a restart: `mosquitto_pub -t home/OpenMQTTGateway/commands/MQTTtoLORA/config -m '{"lorawan":{"devaddr":"0187F184","nwkskey":"<32 hex digits>","appskey":"<32 hex digits>","name":"Front door","model":"LDS02"},"save":true}'` From b4c3d2a9d5657223730ca6eadc6a4caccddcd6ce Mon Sep 17 00:00:00 2001 From: eric Date: Fri, 9 Oct 2026 17:49:17 -0500 Subject: [PATCH 05/11] [LORA] WebUI Information page: one row per LoRaWAN device The Information page builds its table by replacing JSON punctuation and can't display nested values: the "lorawan" array of the LORA state was shown as stray rows ("lorawan | [devaddr | ...", "fcnt_down | 12}]"). Show each device as a single row ("LoRaWAN " with its fields joined) on that page; the MQTT state keeps the array. Co-Authored-By: Claude Opus 5.5 --- main/webUI.cpp | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/main/webUI.cpp b/main/webUI.cpp index 5c72b555de..3cd1457472 100644 --- a/main/webUI.cpp +++ b/main/webUI.cpp @@ -1781,6 +1781,32 @@ void handleTK() { * @brief /IN - Information Page * */ +# if defined(ZgatewayLORA) && defined(LORA_LORAWAN) +/** + * The Information page turns flat JSON into a table and can't display nested values: + * show each LoRaWAN device of the LORA state as one row instead of the nested "lorawan" array. + */ +static String lorawanInformation(const String& state) { + DynamicJsonDocument doc(JSON_MSG_BUFFER * 2); + if (deserializeJson(doc, state) || !doc["lorawan"].is()) return state; + for (JsonObject device : doc["lorawan"].as()) { + String value; + for (JsonPair kv : device) { + if (strcmp(kv.key().c_str(), "devaddr") == 0) continue; + if (value.length()) value += " / "; + value += kv.key().c_str(); + value += " "; + value += kv.value().as(); + } + doc[String("LoRaWAN ") + (device["devaddr"] | "")] = value; + } + doc.remove("lorawan"); + String output; + serializeJson(doc, output); + return output; +} +# endif + void handleIN() { WEBUI_TRACE_LOG(F("handleCN: uri: %s, args: %d, method: %d" CR), server.uri(), server.args(), server.method()); WEBUI_SECURE @@ -1810,7 +1836,11 @@ void handleIN() { # endif # if defined(ZgatewayLORA) informationDisplay += "1
LORA}2}1"; +# ifdef LORA_LORAWAN + informationDisplay += lorawanInformation(stateLORAMeasures()); +# else informationDisplay += stateLORAMeasures(); +# endif # endif # if defined(ZgatewayRF) informationDisplay += "1
RF}2}1"; From a83c3830b65788be27408a1b8cf6bb6128530e58 Mon Sep 17 00:00:00 2001 From: eric Date: Sat, 10 Oct 2026 10:46:57 -0500 Subject: [PATCH 06/11] [LORA] Use PRIX32 to print the 32-bit DevAddr Co-Authored-By: Claude Opus 5.5 --- main/gatewayLORA_LoRaWAN.cpp | 10 +++++----- main/webUI.cpp | 4 ++-- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/main/gatewayLORA_LoRaWAN.cpp b/main/gatewayLORA_LoRaWAN.cpp index bad90a65e5..98494790e6 100644 --- a/main/gatewayLORA_LoRaWAN.cpp +++ b/main/gatewayLORA_LoRaWAN.cpp @@ -61,7 +61,7 @@ static bool lorawanDiscoveryPending = false; # define LORAWAN_DEVICE_JSON_SIZE 3072 // device declaration with keys and entities, NVS strings are limited to ~4000 bytes static void lorawanNvsKey(char* key, const char* prefix, uint32_t devAddr) { - snprintf(key, 11, "%s%08X", prefix, devAddr); + snprintf(key, 11, "%s%08" PRIX32, prefix, devAddr); } static bool lorawanParseKey(const char* hex, uint8_t* key) { @@ -270,7 +270,7 @@ int LORAWANtoJson(const uint8_t* p, int len, JsonObject& LORAdata) { dev->seen = true; char id[9]; - snprintf(id, sizeof(id), "%08X", devAddr); + snprintf(id, sizeof(id), "%08" PRIX32, devAddr); LORAdata["id"] = id; if (dev->name.length()) LORAdata["name"] = dev->name; if (dev->model.length()) LORAdata["model"] = dev->model; @@ -301,7 +301,7 @@ int LORAWANtoJson(const uint8_t* p, int len, JsonObject& LORAdata) { static void lorawanToJson(const LoRaWANDevice& dev, JsonObject& jo, bool withKeys) { char hex[34]; // _rawToHex writes "%02X\r" per byte: needs 2 * size + 2 - snprintf(hex, 9, "%08X", dev.devAddr); + snprintf(hex, 9, "%08" PRIX32, dev.devAddr); jo["devaddr"] = hex; if (withKeys) { TheengsUtils::_rawToHex((byte*)dev.nwkSKey, hex, 16); @@ -329,7 +329,7 @@ static void lorawanSaveIndex() { String index; for (auto& dev : lorawanDevices) { char hex[9]; - snprintf(hex, sizeof(hex), "%08X", dev.devAddr); + snprintf(hex, sizeof(hex), "%08" PRIX32, dev.devAddr); if (index.length()) index += ","; index += hex; } @@ -549,7 +549,7 @@ void launchLORAWANDiscovery(bool overrideDiscovery) { continue; } char id[9]; - snprintf(id, sizeof(id), "%08X", dev.devAddr); + snprintf(id, sizeof(id), "%08" PRIX32, dev.devAddr); String topic = String(subjectLORAtoMQTT) + "/" + id; const char* deviceName = dev.name.length() ? dev.name.c_str() : id; for (JsonObject e : doc.as()) { diff --git a/main/webUI.cpp b/main/webUI.cpp index 3cd1457472..1aac8d25e8 100644 --- a/main/webUI.cpp +++ b/main/webUI.cpp @@ -1473,7 +1473,7 @@ void handleLW() { showForm = false; } else { char hex[9]; - snprintf(hex, sizeof(hex), "%08X", dev->devAddr); + snprintf(hex, sizeof(hex), "%08" PRIX32, dev->devAddr); fDa = hex; fNm = dev->name; fMd = dev->model; @@ -1510,7 +1510,7 @@ void handleLW() { "FCnt"); for (auto& dev : LORAWANdevices()) { char hex[9]; - snprintf(hex, sizeof(hex), "%08X", dev.devAddr); + snprintf(hex, sizeof(hex), "%08" PRIX32, dev.devAddr); String row = String("" + hex + "" + htmlEscape(dev.name) + "" + htmlEscape(dev.model) + ""; # ifdef LORA_LORAWAN_JS row += String("") + (dev.decoder.length() ? "yes" : "no") + ""; From 48f3b60b4de8a5bf33094d75c6cdef76c8348eda Mon Sep 17 00:00:00 2001 From: eric Date: Fri, 9 Oct 2026 13:43:17 -0500 Subject: [PATCH 07/11] [LORA] LoRaWAN downlinks: queue, schedules, triggers and confirmed uplink ACK - Per device downlink queue (MQTT "send" / WebUI), sent one per uplink in RX1 (RX2 if RX1 can no longer be reached), timed from a DIO0 RxDone interrupt; frames encrypted, MIC signed, FCntDown persisted in NVS. - Downlink rules per device: "schedules" (every N s, sent after the next uplink), "triggers" ("uplink" or decoded field values, with cooldown), receive window overrides (rx1_frequency/sf/bw, rx2_*, rx1_delay) and tx_power. - Region aware downlink power: 20 dBm on US915/AU915 (500 kHz downlinks) and EU868 RX2, gateway setting (max 14 dBm) on EU868 RX1. - Confirmed uplinks are acknowledged even without queued data. - WebUI: downlink rules, queue/clear, pending count and FCntDown. Status: frames sent in RX1 at 14 dBm were not applied by a Dragino LDS02 at -95..-117 dBm uplink RSSI; to be verified at short range with the device serial log. Co-Authored-By: Claude Opus 5.5 --- main/config_LORA.h | 29 +++- main/gatewayLORA_LoRaWAN.cpp | 316 +++++++++++++++++++++++++++++++++-- main/webUI.cpp | 53 +++++- 3 files changed, 375 insertions(+), 23 deletions(-) diff --git a/main/config_LORA.h b/main/config_LORA.h index 3425a91222..1dc042d00b 100644 --- a/main/config_LORA.h +++ b/main/config_LORA.h @@ -82,23 +82,40 @@ extern void XtoLORA(const char* topicOri, JsonObject& RFdata); # define LORAWAN_DECODED 1 # define LORAWAN_DROP 2 // duplicate frame # include +struct LoRaWANDownlink { + uint8_t fport; + String hex; // FRMPayload, not encrypted + String source; // "mqtt", "webui", "schedule", "trigger" +}; struct LoRaWANDevice { - uint32_t devAddr; - uint8_t nwkSKey[16]; - uint8_t appSKey[16]; + uint32_t devAddr = 0; + uint8_t nwkSKey[16] = {0}; + uint8_t appSKey[16] = {0}; String name; String model; String entities; // serialized JSON array of Home Assistant entity declarations + String downlinks; // serialized JSON object: receive windows overrides, "schedules" and "triggers" # ifdef LORA_LORAWAN_JS String decoder; # endif - uint32_t lastFcnt; - bool seen; + uint32_t lastFcnt = 0; + bool seen = false; String lastPayload; // last decrypted FRMPayload (hex), to test decoders from the WebUI - uint8_t lastPort; + uint8_t lastPort = 0; + uint32_t fcntDown = 0; // next downlink frame counter, persisted + std::vector queue; // sent one per uplink, in the receive windows following it + std::vector ruleLast; // millis() of the last schedule/trigger firing, by rule index }; extern const std::vector& LORAWANdevices(); extern bool LORAWANconfigureDevice(JsonObject& device, bool save, String* error); +extern bool LORAWANqueueDownlink(uint32_t devAddr, uint8_t fport, const String& hex, const char* source, String* error); +extern void LORAWANclearQueue(uint32_t devAddr); +# ifndef LORAWAN_MAX_QUEUE +# define LORAWAN_MAX_QUEUE 8 // pending downlinks per device +# endif +# ifndef LORAWAN_MAX_DOWNLINK_SIZE +# define LORAWAN_MAX_DOWNLINK_SIZE 51 // FRMPayload bytes, fits every data rate of the common regions +# endif extern void LORAWANsetup(); extern int LORAWANtoJson(const uint8_t* packet, int packetSize, JsonObject& LORAdata); extern void LORAWANConfig_fromJson(JsonObject& LORAdata); diff --git a/main/gatewayLORA_LoRaWAN.cpp b/main/gatewayLORA_LoRaWAN.cpp index 98494790e6..01d129bdd8 100644 --- a/main/gatewayLORA_LoRaWAN.cpp +++ b/main/gatewayLORA_LoRaWAN.cpp @@ -35,12 +35,14 @@ # ifndef ESP32 # error "LORA_LORAWAN is only supported on ESP32" # endif +# include # include # include # include "TheengsCommon.h" # include "config_LORA.h" +# include "esp_timer.h" # include "mbedtls/aes.h" # ifdef ZmqttDiscovery # include "config_mqttDiscovery.h" @@ -49,8 +51,17 @@ # include # endif +extern LORAConfig_s LORAConfig; +extern void LORAConfig_apply(); + static std::vector lorawanDevices; +// End of the last received packet (DIO0 = RxDone), the reference of the downlink receive windows +static volatile int64_t lorawanRxDoneUs = 0; +static void IRAM_ATTR lorawanRxDoneIsr() { + lorawanRxDoneUs = esp_timer_get_time(); +} + const std::vector& LORAWANdevices() { return lorawanDevices; } @@ -130,9 +141,9 @@ static void lorawanMic(const uint8_t* key, const uint8_t* b0, const uint8_t* msg memcpy(mac, x, 16); } -// LoRaWAN 1.0.x FRMPayload encryption (AES-CTR like, uplink direction) -static void lorawanDecrypt(const uint8_t* key, uint32_t devAddr, uint32_t fcnt, const uint8_t* in, uint8_t* out, size_t len) { - uint8_t a[16] = {0x01, 0, 0, 0, 0, 0, +// LoRaWAN 1.0.x FRMPayload encryption, symmetric (AES-CTR like), dir 0 = uplink, 1 = downlink +static void lorawanCrypt(const uint8_t* key, uint8_t dir, uint32_t devAddr, uint32_t fcnt, const uint8_t* in, uint8_t* out, size_t len) { + uint8_t a[16] = {0x01, 0, 0, 0, 0, dir, (uint8_t)devAddr, (uint8_t)(devAddr >> 8), (uint8_t)(devAddr >> 16), (uint8_t)(devAddr >> 24), (uint8_t)fcnt, (uint8_t)(fcnt >> 8), (uint8_t)(fcnt >> 16), (uint8_t)(fcnt >> 24), 0, 0}; uint8_t s[16]; @@ -237,6 +248,227 @@ bool LORAWANtestDecoder(const String& script, uint8_t fport, const uint8_t* byte } # endif +/*-------------------- Downlinks --------------------*/ + +static bool lorawanParseHex(const String& hex, uint8_t* out, size_t maxLen, size_t& len) { + if (hex.length() % 2 || hex.length() / 2 > maxLen) return false; + for (size_t i = 0; i < hex.length(); i++) { + if (!isxdigit(hex[i])) return false; + } + len = hex.length() / 2; + return len == 0 || TheengsUtils::_hexToRaw(hex.c_str(), out, len); +} + +// Queues a downlink, sent in the receive windows following the next uplink of the device +bool LORAWANqueueDownlink(uint32_t devAddr, uint8_t fport, const String& hexIn, const char* source, String* error) { + LoRaWANDevice* dev = lorawanFind(devAddr); + String hex = hexIn; + hex.replace(" ", ""); + hex.toUpperCase(); + uint8_t bytes[LORAWAN_MAX_DOWNLINK_SIZE]; + size_t len; + const char* problem = nullptr; + if (!dev) + problem = "unknown device"; + else if (fport < 1 || fport > 223) + problem = "fport must be 1 to 223"; + else if (!lorawanParseHex(hex, bytes, LORAWAN_MAX_DOWNLINK_SIZE, len)) + problem = "payload must be an even number of hex digits, up to LORAWAN_MAX_DOWNLINK_SIZE bytes"; + else if (dev->queue.size() >= LORAWAN_MAX_QUEUE) + problem = "downlink queue full"; + if (problem) { + THEENGS_LOG_ERROR(F("[LoRaWAN] downlink for %X not queued: %s" CR), devAddr, problem); + if (error) *error = problem; + return false; + } + dev->queue.push_back({fport, hex, source}); + THEENGS_LOG_NOTICE(F("[LoRaWAN] downlink queued for %X (%s): port %d %s, %d pending" CR), devAddr, source, fport, hex.c_str(), dev->queue.size()); + return true; +} + +void LORAWANclearQueue(uint32_t devAddr) { + LoRaWANDevice* dev = lorawanFind(devAddr); + if (dev) dev->queue.clear(); +} + +struct LoRaWANWindow { + long frequency; + int sf; + long bw; + int power; // dBm, PA_BOOST output: 2 to 20 +}; + +// Receive windows of the device: region defaults derived from the (single) uplink channel, overridable per device +static void lorawanWindows(JsonObject rules, LoRaWANWindow& rx1, LoRaWANWindow& rx2) { + long f = LORAConfig.frequency; + int sf = LORAConfig.spreadingFactor; + long bw = LORAConfig.signalBandwidth; + int power = LORAConfig.txPower; + if (f >= 902000000 && f <= 928000000 && bw == 125000) { + // US915 / AU915: RX1 on the 500 kHz channel (uplink channel modulo 8) with the same SF, RX2 923.3 MHz SF12 500 kHz; + // downlinks use 500 kHz (about 6 dB less sensitive than the 125 kHz uplinks) and up to 30 dBm are allowed: use the 20 dBm maximum + long base = f < 915200000 ? 902300000 : 915200000; + int channel = (f - base + 100000) / 200000; + rx1 = {923300000L + 600000L * (channel % 8), sf, 500000, 20}; + rx2 = {923300000L, 12, 500000, 20}; + } else if (f >= 863000000 && f <= 870000000) { + // EU868: RX1 on the uplink channel (14 dBm ERP limit), RX2 869.525 MHz SF12 125 kHz (869.4-869.65 MHz allows 500 mW) + rx1 = {f, sf, bw, power > 14 ? 14 : power}; + rx2 = {869525000L, 12, 125000, 20}; + } else { + rx1 = {f, sf, bw, power}; + rx2 = {f, sf, bw, power}; + } + rx1.frequency = rules["rx1_frequency"] | rx1.frequency; + rx1.sf = rules["rx1_sf"] | rx1.sf; + rx1.bw = rules["rx1_bw"] | rx1.bw; + rx2.frequency = rules["rx2_frequency"] | rx2.frequency; + rx2.sf = rules["rx2_sf"] | rx2.sf; + rx2.bw = rules["rx2_bw"] | rx2.bw; + rx1.power = rules["tx_power"] | rx1.power; + rx2.power = rules["tx_power"] | rx2.power; +} + +// Transmits the frame when esp_timer reaches atUs, then restores the uplink reception settings +static bool lorawanTransmitAt(int64_t atUs, const LoRaWANWindow& w, const uint8_t* frame, size_t len) { + LoRa.idle(); + LoRa.setFrequency(w.frequency); + LoRa.setSpreadingFactor(w.sf); + LoRa.setSignalBandwidth(w.bw); + LoRa.setCodingRate4(5); + LoRa.setPreambleLength(8); + LoRa.setSyncWord(LORAConfig.syncWord); + LoRa.disableCrc(); // LoRaWAN downlinks have no payload CRC + LoRa.enableInvertIQ(); // and use inverted IQ + LoRa.setTxPower(w.power); + LoRa.beginPacket(); + LoRa.write(frame, len); + bool sent = false; + int64_t wait = atUs - esp_timer_get_time(); + if (wait > 500) { + if (wait > 3000) delay((wait - 2000) / 1000); + while (esp_timer_get_time() < atUs) { + } + LoRa.endPacket(); // starts the transmission and waits for TxDone + sent = true; + } + LORAConfig_apply(); + LoRa.receive(); + return sent; +} + +static void lorawanSaveFcntDown(const LoRaWANDevice& dev) { + char key[11]; + lorawanNvsKey(key, "lc", dev.devAddr); + preferences.begin(Gateway_Short_Name, false); + preferences.putUInt(key, dev.fcntDown); + preferences.end(); +} + +// Does the decoded uplink match a trigger condition: "uplink" or an object of field values that must all be equal +static bool lorawanTriggerMatches(JsonVariant when, JsonObject& LORAdata) { + if (when.is()) return strcmp(when.as(), "uplink") == 0; + if (!when.is()) return false; + for (JsonPair kv : when.as()) { + if (!LORAdata.containsKey(kv.key())) return false; + String expected, actual; + serializeJson(kv.value(), expected); + serializeJson(LORAdata[kv.key()], actual); + if (expected != actual) return false; + } + return true; +} + +// Applies the schedules and triggers of the device, then sends the first queued downlink (and/or the ACK of a +// confirmed uplink) in RX1, or RX2 if RX1 cannot be reached anymore +static void lorawanProcessDownlink(LoRaWANDevice& dev, bool confirmedUplink, int64_t rxDoneUs, JsonObject& LORAdata) { + DynamicJsonDocument rulesDoc(LORAWAN_DEVICE_JSON_SIZE); + if (dev.downlinks.length()) deserializeJson(rulesDoc, dev.downlinks); + JsonObject rules = rulesDoc.as(); + uint32_t now = millis(); + size_t ruleCount = rules["schedules"].size() + rules["triggers"].size(); + if (dev.ruleLast.size() != ruleCount) dev.ruleLast.assign(ruleCount, 0); + size_t rule = 0; + for (JsonObject schedule : rules["schedules"].as()) { + uint32_t every = schedule["every"] | 0; + // schedules fire at the first uplink, then at the first uplink after each interval + if (every && (dev.ruleLast[rule] == 0 || now - dev.ruleLast[rule] >= every * 1000UL)) { + if (LORAWANqueueDownlink(dev.devAddr, schedule["fport"] | 0, schedule["hex"] | "", "schedule", nullptr)) dev.ruleLast[rule] = now ? now : 1; + } + rule++; + } + for (JsonObject trigger : rules["triggers"].as()) { + uint32_t cooldown = trigger["cooldown"] | 0; + if (lorawanTriggerMatches(trigger["when"], LORAdata) && (dev.ruleLast[rule] == 0 || now - dev.ruleLast[rule] >= cooldown * 1000UL)) { + if (LORAWANqueueDownlink(dev.devAddr, trigger["fport"] | 0, trigger["hex"] | "", "trigger", nullptr)) { + dev.queue.insert(dev.queue.begin(), dev.queue.back()); // triggers answer this uplink first + dev.queue.pop_back(); + dev.ruleLast[rule] = now ? now : 1; + } + } + rule++; + } + if (dev.queue.empty() && !confirmedUplink) return; + if (rxDoneUs == 0) { + LORAdata["downlink_error"] = "no RxDone timestamp"; + return; + } + + bool hasData = !dev.queue.empty(); + uint8_t frame[13 + LORAWAN_MAX_DOWNLINK_SIZE]; + size_t n = 0; + frame[n++] = 0x60; // unconfirmed data down + for (int i = 0; i < 4; i++) frame[n++] = (uint8_t)(dev.devAddr >> (8 * i)); + frame[n++] = (confirmedUplink ? 0x20 : 0x00) | (dev.queue.size() > 1 ? 0x10 : 0x00); // FCtrl: ACK, FPending + frame[n++] = (uint8_t)dev.fcntDown; + frame[n++] = (uint8_t)(dev.fcntDown >> 8); + if (hasData) { + const LoRaWANDownlink& dl = dev.queue.front(); + uint8_t payload[LORAWAN_MAX_DOWNLINK_SIZE]; + size_t len = 0; + lorawanParseHex(dl.hex, payload, LORAWAN_MAX_DOWNLINK_SIZE, len); + frame[n++] = dl.fport; + lorawanCrypt(dev.appSKey, 1, dev.devAddr, dev.fcntDown, payload, frame + n, len); + n += len; + } + uint8_t b0[16] = {0x49, 0, 0, 0, 0, 1, frame[1], frame[2], frame[3], frame[4], + (uint8_t)dev.fcntDown, (uint8_t)(dev.fcntDown >> 8), (uint8_t)(dev.fcntDown >> 16), (uint8_t)(dev.fcntDown >> 24), 0, (uint8_t)n}; + uint8_t mac[16]; + lorawanMic(dev.nwkSKey, b0, frame, n, mac); + memcpy(frame + n, mac, 4); + n += 4; + + LoRaWANWindow rx1, rx2; + lorawanWindows(rules, rx1, rx2); + int64_t rx1At = rxDoneUs + (int64_t)(rules["rx1_delay"] | 1) * 1000000LL; + const char* window = "rx1"; + bool sent = lorawanTransmitAt(rx1At, rx1, frame, n); + if (!sent) { + window = "rx2"; + sent = lorawanTransmitAt(rx1At + 1000000LL, rx2, frame, n); + } + if (!sent) { + LORAdata["downlink_error"] = "receive windows missed"; + THEENGS_LOG_WARNING(F("[LoRaWAN] downlink for %X: receive windows missed" CR), dev.devAddr); + } else { + JsonObject dl = LORAdata.createNestedObject("downlink"); + dl["fcnt"] = dev.fcntDown; + dl["window"] = window; + dl["power"] = strcmp(window, "rx1") == 0 ? rx1.power : rx2.power; + if (confirmedUplink) dl["ack"] = true; + if (hasData) { + dl["fport"] = dev.queue.front().fport; + dl["payload"] = dev.queue.front().hex; + dl["source"] = dev.queue.front().source; + dev.queue.erase(dev.queue.begin()); + } + THEENGS_LOG_NOTICE(F("[LoRaWAN] downlink sent to %X in %s, FCnt %u" CR), dev.devAddr, window, dev.fcntDown); + dev.fcntDown++; + lorawanSaveFcntDown(dev); + } + if (dev.queue.size()) LORAdata["downlink_pending"] = dev.queue.size(); +} + /*-------------------- Uplink decoding --------------------*/ int LORAWANtoJson(const uint8_t* p, int len, JsonObject& LORAdata) { @@ -247,6 +479,7 @@ int LORAWANtoJson(const uint8_t* p, int len, JsonObject& LORAdata) { LoRaWANDevice* dev = lorawanFind(devAddr); if (!dev) return LORAWAN_NOT_HANDLED; + int64_t rxDoneUs = lorawanRxDoneUs; int hdrLen = 8 + (p[5] & 0x0F); // MHDR + DevAddr + FCtrl + FCnt + FOpts int msgLen = len - 4; // without MIC if (msgLen < hdrLen) return LORAWAN_NOT_HANDLED; @@ -281,7 +514,7 @@ int LORAWANtoJson(const uint8_t* p, int len, JsonObject& LORAdata) { uint8_t fport = p[hdrLen]; size_t n = msgLen - hdrLen - 1; uint8_t plain[n > 0 ? n : 1]; - lorawanDecrypt(fport == 0 ? dev->nwkSKey : dev->appSKey, devAddr, fcnt, p + hdrLen + 1, plain, n); + lorawanCrypt(fport == 0 ? dev->nwkSKey : dev->appSKey, 0, devAddr, fcnt, p + hdrLen + 1, plain, n); char hex[n * 2 + 2]; // _rawToHex writes "%02X\r" per byte: needs 2 * size + 2 TheengsUtils::_rawToHex(plain, hex, n); hex[n * 2] = 0; @@ -294,6 +527,7 @@ int LORAWANtoJson(const uint8_t* p, int len, JsonObject& LORAdata) { THEENGS_LOG_WARNING(F("[LoRaWAN] decoder error for %X: %s" CR), devAddr, LORAdata["decoder_error"].as()); # endif } + lorawanProcessDownlink(*dev, mtype == 4, rxDoneUs, LORAdata); return LORAWAN_DECODED; } @@ -319,10 +553,20 @@ static void lorawanToJson(const LoRaWANDevice& dev, JsonObject& jo, bool withKey else jo["entities"] = true; } + if (dev.downlinks.length()) { + if (withKeys) + jo["downlinks"] = serialized(dev.downlinks); + else + jo["downlinks"] = true; + } # ifdef LORA_LORAWAN_JS if (!withKeys) jo["decoder"] = dev.decoder.length() > 0; # endif if (!withKeys && dev.seen) jo["fcnt"] = dev.lastFcnt; + if (!withKeys) { + jo["fcnt_down"] = dev.fcntDown; + if (dev.queue.size()) jo["downlink_pending"] = dev.queue.size(); + } } static void lorawanSaveIndex() { @@ -358,7 +602,9 @@ static void lorawanEraseDevice(uint32_t devAddr) { lorawanNvsKey(key, "lw", devAddr); preferences.remove(key); lorawanNvsKey(key, "lj", devAddr); - preferences.remove(key); + if (preferences.isKey(key)) preferences.remove(key); + lorawanNvsKey(key, "lc", devAddr); + if (preferences.isKey(key)) preferences.remove(key); preferences.end(); } @@ -370,6 +616,23 @@ static void lorawanEraseDevice(uint32_t devAddr) { if (error) *error = msg; \ } while (0) +// Checks the downlink rules: {"rx1_frequency":..,"schedules":[{"every":s,"fport":p,"hex":".."}],"triggers":[{"when":"uplink"|{..},"fport":p,"hex":"..","cooldown":s}]} +static const char* lorawanCheckRules(JsonObject rules) { + uint8_t bytes[LORAWAN_MAX_DOWNLINK_SIZE]; + size_t len; + for (const char* list : {"schedules", "triggers"}) { + if (rules.containsKey(list) && !rules[list].is()) return "schedules and triggers must be arrays"; + for (JsonObject rule : rules[list].as()) { + int fport = rule["fport"] | 0; + if (fport < 1 || fport > 223) return "each schedule/trigger needs an fport from 1 to 223"; + if (!lorawanParseHex(String(rule["hex"] | ""), bytes, LORAWAN_MAX_DOWNLINK_SIZE, len)) return "each schedule/trigger needs a hex payload"; + if (strcmp(list, "schedules") == 0 && (rule["every"] | 0) <= 0) return "each schedule needs an interval in seconds (every)"; + if (strcmp(list, "triggers") == 0 && !rule.containsKey("when")) return "each trigger needs a condition (when)"; + } + } + return nullptr; +} + // Applies one device declaration, returns false (and the reason in error, if given) if invalid bool LORAWANconfigureDevice(JsonObject& jo, bool save, String* error) { uint32_t devAddr; @@ -389,14 +652,11 @@ bool LORAWANconfigureDevice(JsonObject& jo, bool save, String* error) { } return true; } - LoRaWANDevice candidate = dev ? *dev : LoRaWANDevice{devAddr, {0}, {0}, "", "", "", -# ifdef LORA_LORAWAN_JS - "", -# endif - 0, - false, - "", - 0}; + LoRaWANDevice candidate; + if (dev) + candidate = *dev; + else + candidate.devAddr = devAddr; if (jo.containsKey("nwkskey") && !lorawanParseKey(jo["nwkskey"], candidate.nwkSKey)) { LORAWAN_CONFIG_ERROR(error, "invalid nwkskey (32 hex digits)"); return false; @@ -416,6 +676,18 @@ bool LORAWANconfigureDevice(JsonObject& jo, bool save, String* error) { if (jo["entities"].is() && jo["entities"].size() > 0) serializeJson(jo["entities"], candidate.entities); lorawanDiscoveryPending = true; } + if (jo.containsKey("downlinks")) { + candidate.downlinks = ""; + if (jo["downlinks"].is() && jo["downlinks"].size() > 0) { + const char* problem = lorawanCheckRules(jo["downlinks"].as()); + if (problem) { + LORAWAN_CONFIG_ERROR(error, problem); + return false; + } + serializeJson(jo["downlinks"], candidate.downlinks); + } + candidate.ruleLast.clear(); + } if (dev) { *dev = candidate; } else { @@ -430,6 +702,18 @@ bool LORAWANconfigureDevice(JsonObject& jo, bool save, String* error) { lorawanSaveDevice(candidate); lorawanSaveIndex(); } + // Commands: {"clear_queue":true}, {"send":{"fport":1,"hex":"0100003C"}} (or an array) + if (jo["clear_queue"] | false) LORAWANclearQueue(devAddr); + if (jo.containsKey("send")) { + JsonArray sends = jo["send"].is() ? jo["send"].as() : JsonArray(); + bool ok = true; + if (sends.isNull()) { + ok = LORAWANqueueDownlink(devAddr, jo["send"]["fport"] | 0, String(jo["send"]["hex"] | ""), "mqtt", error); + } else { + for (JsonObject send : sends) ok = LORAWANqueueDownlink(devAddr, send["fport"] | 0, String(send["hex"] | ""), "mqtt", error) && ok; + } + if (!ok) return false; + } return true; } @@ -479,6 +763,10 @@ void LORAWANsetup() { } JsonObject jo = doc.as(); if (!LORAWANconfigureDevice(jo, false, nullptr)) continue; + lorawanNvsKey(key, "lc", devAddr); + preferences.begin(Gateway_Short_Name, true); + if (preferences.isKey(key)) lorawanFind(devAddr)->fcntDown = preferences.getUInt(key, 0); + preferences.end(); # ifdef LORA_LORAWAN_JS lorawanNvsKey(key, "lj", devAddr); preferences.begin(Gateway_Short_Name, true); @@ -487,6 +775,8 @@ void LORAWANsetup() { # endif } THEENGS_LOG_NOTICE(F("[LoRaWAN] %d device(s) loaded" CR), lorawanDevices.size()); + pinMode(LORA_DI0, INPUT); + attachInterrupt(digitalPinToInterrupt(LORA_DI0), lorawanRxDoneIsr, RISING); // DIO0 is RxDone in receive mode lorawanDiscoveryPending = true; } diff --git a/main/webUI.cpp b/main/webUI.cpp index 1aac8d25e8..446e03dd8b 100644 --- a/main/webUI.cpp +++ b/main/webUI.cpp @@ -1390,11 +1390,27 @@ void handleLW() { bool isNew = server.arg("nw") == "1"; String fDa = server.arg("da"), fNm = server.arg("nm"), fMd = server.arg("md"), fEn = server.arg("en"); String fDc = server.arg("dc"), fTp = server.arg("tp"), fTf = server.hasArg("tf") ? server.arg("tf") : "1"; + String fDl = server.arg("dl"), fQp = server.hasArg("qp") ? server.arg("qp") : "1", fQh = server.arg("qh"); String testOutput; fDa.trim(); fDa.toUpperCase(); - if (server.hasArg("rm")) { + if (server.hasArg("qs") || server.hasArg("qc")) { + showForm = true; + const LoRaWANDevice* dev = findLoRaWANDevice(fDa); + String error; + if (!dev) { + message = "Error: save the device before queuing downlinks"; + } else if (server.hasArg("qc")) { + LORAWANclearQueue(dev->devAddr); + message = "Downlink queue cleared"; + } else if (LORAWANqueueDownlink(dev->devAddr, (uint8_t)fQp.toInt(), fQh, "webui", &error)) { + message = "Downlink queued, it will be sent after the next uplink of the device"; + fQh = ""; + } else { + message = "Error: " + error; + } + } else if (server.hasArg("rm")) { DynamicJsonDocument doc(128); JsonObject jo = doc.to(); jo["devaddr"] = fDa; @@ -1444,6 +1460,20 @@ void handleLW() { } else { jo.createNestedArray("entities"); // clears the entities } + String trimmedDownlinks = fDl; + trimmedDownlinks.trim(); + if (valid && trimmedDownlinks.length()) { + DynamicJsonDocument downlinks(2048); + DeserializationError error = deserializeJson(downlinks, trimmedDownlinks); + if (error || !downlinks.is()) { + message = String("Error: downlink rules must be a JSON object (") + (error ? error.c_str() : "not an object") + ")"; + valid = false; + } else { + jo["downlinks"] = downlinks.as(); + } + } else if (valid) { + jo.createNestedObject("downlinks"); // clears the rules + } String error; if (valid && !LORAWANconfigureDevice(jo, true, &error)) { message = "Error: " + error; @@ -1477,6 +1507,11 @@ void handleLW() { fDa = hex; fNm = dev->name; fMd = dev->model; + fDl = ""; + if (dev->downlinks.length()) { + DynamicJsonDocument downlinks(2048); + if (!deserializeJson(downlinks, dev->downlinks)) serializeJsonPretty(downlinks, fDl); + } fEn = ""; if (dev->entities.length()) { DynamicJsonDocument entities(2048); @@ -1507,7 +1542,7 @@ void handleLW() { # ifdef LORA_LORAWAN_JS "Decoder" # endif - "FCnt"); + "FCntPending"); for (auto& dev : LORAWANdevices()) { char hex[9]; snprintf(hex, sizeof(hex), "%08" PRIX32, dev.devAddr); @@ -1515,7 +1550,7 @@ void handleLW() { # ifdef LORA_LORAWAN_JS row += String("") + (dev.decoder.length() ? "yes" : "no") + ""; # endif - row += String("") + (dev.seen ? String(dev.lastFcnt) : String("-")) + ""; + row += String("") + (dev.seen ? String(dev.lastFcnt) : String("-")) + "" + String(dev.queue.size()) + ""; server.sendContent(row); } server.sendContent("

"); @@ -1534,7 +1569,17 @@ void handleLW() { server.sendContent("

"); if (testOutput.length()) server.sendContent("
" + htmlEscape(testOutput) + "
"); # endif - server.sendContent("
"); + server.sendContent("

Downlink rules (JSON object, optional): schedules queue a command every N seconds (sent after the next uplink), triggers answer an uplink matching when (\"uplink\" or field values); receive windows can be overridden (rx1_frequency, rx1_sf, rx1_bw, rx2_..., rx1_delay) and the downlink power (tx_power, dBm)

"); + server.sendContent("
"); + if (dev) { + String queue = "

Send a downlink (queued, sent in the receive windows after the next uplink; downlink FCnt " + String(dev->fcntDown) + ")
"; + for (auto& dl : dev->queue) queue += "pending: port " + String(dl.fport) + " " + htmlEscape(dl.hex) + " (" + htmlEscape(dl.source) + ")
"; + queue += "

"; + queue += "

"; + if (dev->queue.size()) queue += "

"; + server.sendContent(queue); + } + server.sendContent(""); if (dev) { server.sendContent("

"); } From f6e949e19dfdb4b691fc2352883d830bfa9c4f7d Mon Sep 17 00:00:00 2001 From: eric Date: Fri, 9 Oct 2026 16:18:00 -0500 Subject: [PATCH 08/11] [LORA] LoRaWAN: accept ABP frame counter restarts, selectable receive window - ABP devices often restart their frame counter when they reboot; a frame whose MIC only verifies with the raw 16 bits counter is accepted as a counter restart (published with "fcnt_reset": true) instead of being dropped as unknown. - Downlink rule "window": "auto" (RX1, else RX2), "rx1" or "rx2". A Dragino LDS02 in single channel mode (AT+CHS) computes its RX1 channel from a hopping channel index, so RX1 is unpredictable; RX2 (923.3 MHz SF12 500 kHz) is fixed. Verified: downlink received in RX2 (device RSSI -64 dBm) and applied. Co-Authored-By: Claude Opus 5.5 --- main/gatewayLORA_LoRaWAN.cpp | 26 ++++++++++++++++++++++---- main/webUI.cpp | 2 +- 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/main/gatewayLORA_LoRaWAN.cpp b/main/gatewayLORA_LoRaWAN.cpp index 01d129bdd8..10a7cc05c1 100644 --- a/main/gatewayLORA_LoRaWAN.cpp +++ b/main/gatewayLORA_LoRaWAN.cpp @@ -441,9 +441,12 @@ static void lorawanProcessDownlink(LoRaWANDevice& dev, bool confirmedUplink, int LoRaWANWindow rx1, rx2; lorawanWindows(rules, rx1, rx2); int64_t rx1At = rxDoneUs + (int64_t)(rules["rx1_delay"] | 1) * 1000000LL; + // "window": "auto" (RX1, else RX2), "rx1" or "rx2" (e.g. for devices with an unpredictable RX1 channel) + const char* windowRule = rules["window"] | "auto"; const char* window = "rx1"; - bool sent = lorawanTransmitAt(rx1At, rx1, frame, n); - if (!sent) { + bool sent = false; + if (strcmp(windowRule, "rx2") != 0) sent = lorawanTransmitAt(rx1At, rx1, frame, n); + if (!sent && strcmp(windowRule, "rx1") != 0) { window = "rx2"; sent = lorawanTransmitAt(rx1At + 1000000LL, rx2, frame, n); } @@ -492,8 +495,23 @@ int LORAWANtoJson(const uint8_t* p, int len, JsonObject& LORAdata) { uint8_t mac[16]; lorawanMic(dev->nwkSKey, b0, p, msgLen, mac); if (memcmp(mac, p + msgLen, 4) != 0) { - THEENGS_LOG_WARNING(F("[LoRaWAN] MIC check failed for %X FCnt %u" CR), devAddr, fcnt); - return LORAWAN_NOT_HANDLED; + // ABP devices usually restart their frame counter when they reboot: accept a valid frame with the 16 bits counter + uint32_t fcnt16 = p[6] | (p[7] << 8); + bool reset = false; + if (fcnt != fcnt16) { + b0[12] = 0; + b0[13] = 0; + lorawanMic(dev->nwkSKey, b0, p, msgLen, mac); + reset = memcmp(mac, p + msgLen, 4) == 0; + } + if (!reset) { + THEENGS_LOG_WARNING(F("[LoRaWAN] MIC check failed for %X FCnt %u" CR), devAddr, fcnt); + return LORAWAN_NOT_HANDLED; + } + THEENGS_LOG_NOTICE(F("[LoRaWAN] %X frame counter restarted (%u after %u)" CR), devAddr, fcnt16, dev->lastFcnt); + LORAdata["fcnt_reset"] = true; + fcnt = fcnt16; + dev->seen = false; // not a duplicate of the previous counter value } if (dev->seen && fcnt == dev->lastFcnt) { THEENGS_LOG_NOTICE(F("[LoRaWAN] Duplicate FCnt %u from %X ignored" CR), fcnt, devAddr); diff --git a/main/webUI.cpp b/main/webUI.cpp index 446e03dd8b..2480e20e5b 100644 --- a/main/webUI.cpp +++ b/main/webUI.cpp @@ -1569,7 +1569,7 @@ void handleLW() { server.sendContent("

"); if (testOutput.length()) server.sendContent("
" + htmlEscape(testOutput) + "
"); # endif - server.sendContent("

Downlink rules (JSON object, optional): schedules queue a command every N seconds (sent after the next uplink), triggers answer an uplink matching when (\"uplink\" or field values); receive windows can be overridden (rx1_frequency, rx1_sf, rx1_bw, rx2_..., rx1_delay) and the downlink power (tx_power, dBm)

"); + server.sendContent("

Downlink rules (JSON object, optional): schedules queue a command every N seconds (sent after the next uplink), triggers answer an uplink matching when (\"uplink\" or field values); receive windows can be overridden (rx1_frequency, rx1_sf, rx1_bw, rx2_..., rx1_delay, window: auto/rx1/rx2) and the downlink power (tx_power, dBm)

"); server.sendContent("
"); if (dev) { String queue = "

Send a downlink (queued, sent in the receive windows after the next uplink; downlink FCnt " + String(dev->fcntDown) + ")
"; From 43f7419961a95be9601c6d69bc05a42c66146994 Mon Sep 17 00:00:00 2001 From: eric Date: Fri, 9 Oct 2026 16:26:46 -0500 Subject: [PATCH 09/11] [LORA] Document LoRaWAN downlinks Co-Authored-By: Claude Opus 5.5 --- docs/use/lora.md | 38 +++++++++++++++++++++++++++++++++++++- 1 file changed, 37 insertions(+), 1 deletion(-) diff --git a/docs/use/lora.md b/docs/use/lora.md index 9c64ec0000..f4c903a4d4 100644 --- a/docs/use/lora.md +++ b/docs/use/lora.md @@ -85,7 +85,7 @@ will send "test" to a WiPhone with chip ID 123ABC The gateway can decrypt uplinks from LoRaWAN devices using ABP (activation by personalization), such as Dragino sensors switched to ABP mode. It is not a LoRaWAN network server, so this only suits a simple setup: * The gateway listens on a single frequency, spreading factor and bandwidth. Set each device to send on that same channel (for example `AT+CHS=903900000` and a fixed data rate with ADR off on Dragino devices) and use sync word `0x34`. -* Only uplinks are handled. There is no join (OTAA), no downlink and no MAC commands, so turn off on the device any feature that waits for a downlink (confirmed uplinks, link checks, "rejoin when no downlink"). +* There is no join (OTAA) and no MAC command handling. Downlinks are sent only when you queue them (see below), and confirmed uplinks are acknowledged. Turn off on the device any feature that waits for MAC answers, such as link checks. This feature is available on ESP32 only. It is not part of the default binaries: build with `'-DLORA_LORAWAN'`. To also decode the payload with a JavaScript decoder, add `'-DLORA_LORAWAN_JS'` and the Duktape library, for example in your `prod_env.ini`: ```ini @@ -141,6 +141,42 @@ Add an `entities` list to the device declaration to create Home Assistant entiti ``` `type` is `sensor` by default. `class` is the Home Assistant device class. Sensors use `state_class` `measurement` unless you set another one. Fields named like the existing LoRa sensors (`tempc`, `hum`, `moi`, `batt`, `count`) are discovered automatically. +### Downlinks +LoRaWAN devices are usually class A: they only listen for a short time 1 s (RX1) and 2 s (RX2) after each of their uplinks. You can't send a command whenever you want. You queue it, and the gateway sends it in the receive window that follows the next uplink of the device. One queued downlink is sent per uplink, and up to 8 can be queued per device (`LORAWAN_MAX_QUEUE`). Downlinks are encrypted and signed with the device keys. The downlink frame counter is kept in flash. + +Queue a command with the `send` key, as an object or an array. The payload is the hex FRMPayload, up to 51 bytes: + +`mosquitto_pub -t home/OpenMQTTGateway/commands/MQTTtoLORA/config -m '{"lorawan":{"devaddr":"0187F184","send":{"fport":1,"hex":"01000E10"}}}'` + +Use `{"lorawan":{"devaddr":"0187F184","clear_queue":true}}` to drop the pending commands. You can also queue and clear commands from the device page of **Configure LoRaWAN**, which shows the pending commands and the downlink frame counter. The uplink message that triggered a downlink reports it: +```json +{"id":"0187F184","fcnt":5,"fport":10,"payload":"8D500100007300000300","downlink":{"fcnt":0,"window":"rx2","power":20,"fport":1,"payload":"01000E10","source":"mqtt"},"downlink_pending":1} +``` + +Uplinks with a confirmed data type are acknowledged in the same way, even if no command is queued. + +#### Downlink rules +The `downlinks` object of a device declaration holds commands sent automatically, and receive window settings: +```json +{"lorawan":{"devaddr":"0187F184","downlinks":{ + "window":"rx2", + "schedules":[{"every":86400,"fport":1,"hex":"01015180"}], + "triggers":[{"when":{"DOOR_STATE":"open"},"fport":1,"hex":"A601","cooldown":3600}]}},"save":true} +``` +* `schedules`: queues the command every `every` seconds. It is sent after the first uplink, then after the first uplink following each interval. +* `triggers`: answers an uplink that matches `when`. Use `"uplink"` to match any uplink, or an object of decoded field values that must all be equal. The command is sent in the receive window of that uplink, before the queued ones. `cooldown` (seconds) limits how often it fires. +* `window`: `auto` (default) uses RX1, or RX2 if RX1 can no longer be reached. `rx1` and `rx2` force one window. +* `rx1_frequency`, `rx1_sf`, `rx1_bw`, `rx2_frequency`, `rx2_sf`, `rx2_bw` and `rx1_delay` (seconds, default 1) override the receive windows. +* `tx_power` (dBm) overrides the downlink power. + +By default the receive windows follow the region of the configured frequency. For US915 and AU915, RX1 uses the 500 kHz channel matching the uplink channel with the same spreading factor, and RX2 uses 923.3 MHz, SF12, 500 kHz. Both are sent at 20 dBm, because 500 kHz downlinks are about 6 dB less sensitive than 125 kHz uplinks. For EU868, RX1 uses the uplink channel at no more than 14 dBm, and RX2 uses 869.525 MHz, SF12, 125 kHz at 20 dBm. Elsewhere, both windows use the uplink settings and the configured power. + +::: tip Dragino devices in single channel mode +With `AT+CHS`, Dragino devices (e.g. LDS02) transmit on the fixed channel, but compute their RX1 channel from their internal channel hopping, so RX1 changes from one uplink to the next. RX2 is fixed: use `"window":"rx2"`. Their downlink commands (e.g. `01000E10` for a 1 hour uplink interval) were accepted on port 1 in testing. +::: + +ABP devices usually restart their frame counter when they reboot. The gateway accepts such a restart (the message contains `"fcnt_reset":true`) instead of rejecting the frames. + More info on where the LoRa library is born [@sandeepmistry](https://github.com/sandeepmistry/arduino-LoRa/blob/master/API.md#radio-parameters) Tutorial on how to leverage LoRa for a mailbox sensor from [PricelessToolkit](https://www.youtube.com/channel/UCz75N6inuLHXnRC5tqagNLw): From cabb446681b93511ee29bbb14cee44507eb15163 Mon Sep 17 00:00:00 2001 From: eric Date: Fri, 9 Oct 2026 17:30:53 -0500 Subject: [PATCH 10/11] [LORA] LoRaWAN: acknowledge repeated confirmed uplinks When a device misses the ACK of a confirmed uplink it retransmits the same frame (same FCnt). The gateway dropped it as a duplicate before the downlink step, so the retries were never acknowledged and the device kept retrying until NbTrans was exhausted. Duplicates of confirmed uplinks are now acknowledged again, without republishing the uplink nor evaluating schedules and triggers. Co-Authored-By: Claude Opus 5.5 --- main/gatewayLORA_LoRaWAN.cpp | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/main/gatewayLORA_LoRaWAN.cpp b/main/gatewayLORA_LoRaWAN.cpp index 10a7cc05c1..945afd7599 100644 --- a/main/gatewayLORA_LoRaWAN.cpp +++ b/main/gatewayLORA_LoRaWAN.cpp @@ -381,7 +381,7 @@ static bool lorawanTriggerMatches(JsonVariant when, JsonObject& LORAdata) { // Applies the schedules and triggers of the device, then sends the first queued downlink (and/or the ACK of a // confirmed uplink) in RX1, or RX2 if RX1 cannot be reached anymore -static void lorawanProcessDownlink(LoRaWANDevice& dev, bool confirmedUplink, int64_t rxDoneUs, JsonObject& LORAdata) { +static void lorawanProcessDownlink(LoRaWANDevice& dev, bool confirmedUplink, int64_t rxDoneUs, JsonObject& LORAdata, bool applyRules) { DynamicJsonDocument rulesDoc(LORAWAN_DEVICE_JSON_SIZE); if (dev.downlinks.length()) deserializeJson(rulesDoc, dev.downlinks); JsonObject rules = rulesDoc.as(); @@ -389,7 +389,7 @@ static void lorawanProcessDownlink(LoRaWANDevice& dev, bool confirmedUplink, int size_t ruleCount = rules["schedules"].size() + rules["triggers"].size(); if (dev.ruleLast.size() != ruleCount) dev.ruleLast.assign(ruleCount, 0); size_t rule = 0; - for (JsonObject schedule : rules["schedules"].as()) { + for (JsonObject schedule : (applyRules ? rules["schedules"].as() : JsonArray())) { uint32_t every = schedule["every"] | 0; // schedules fire at the first uplink, then at the first uplink after each interval if (every && (dev.ruleLast[rule] == 0 || now - dev.ruleLast[rule] >= every * 1000UL)) { @@ -397,7 +397,7 @@ static void lorawanProcessDownlink(LoRaWANDevice& dev, bool confirmedUplink, int } rule++; } - for (JsonObject trigger : rules["triggers"].as()) { + for (JsonObject trigger : (applyRules ? rules["triggers"].as() : JsonArray())) { uint32_t cooldown = trigger["cooldown"] | 0; if (lorawanTriggerMatches(trigger["when"], LORAdata) && (dev.ruleLast[rule] == 0 || now - dev.ruleLast[rule] >= cooldown * 1000UL)) { if (LORAWANqueueDownlink(dev.devAddr, trigger["fport"] | 0, trigger["hex"] | "", "trigger", nullptr)) { @@ -515,6 +515,8 @@ int LORAWANtoJson(const uint8_t* p, int len, JsonObject& LORAdata) { } if (dev->seen && fcnt == dev->lastFcnt) { THEENGS_LOG_NOTICE(F("[LoRaWAN] Duplicate FCnt %u from %X ignored" CR), fcnt, devAddr); + // A repeated confirmed uplink means our ACK was missed: acknowledge it again (without republishing it) + if (mtype == 4) lorawanProcessDownlink(*dev, true, rxDoneUs, LORAdata, false); return LORAWAN_DROP; } dev->lastFcnt = fcnt; @@ -545,7 +547,7 @@ int LORAWANtoJson(const uint8_t* p, int len, JsonObject& LORAdata) { THEENGS_LOG_WARNING(F("[LoRaWAN] decoder error for %X: %s" CR), devAddr, LORAdata["decoder_error"].as()); # endif } - lorawanProcessDownlink(*dev, mtype == 4, rxDoneUs, LORAdata); + lorawanProcessDownlink(*dev, mtype == 4, rxDoneUs, LORAdata, true); return LORAWAN_DECODED; } From 9ca00323b9961514097d3e56867a3126d52b0e3f Mon Sep 17 00:00:00 2001 From: eric Date: Fri, 9 Oct 2026 17:41:49 -0500 Subject: [PATCH 11/11] [LORA] Document confirmed uplink retransmissions Co-Authored-By: Claude Opus 5.5 --- docs/use/lora.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/use/lora.md b/docs/use/lora.md index f4c903a4d4..558ff0f5bb 100644 --- a/docs/use/lora.md +++ b/docs/use/lora.md @@ -153,7 +153,7 @@ Use `{"lorawan":{"devaddr":"0187F184","clear_queue":true}}` to drop the pending {"id":"0187F184","fcnt":5,"fport":10,"payload":"8D500100007300000300","downlink":{"fcnt":0,"window":"rx2","power":20,"fport":1,"payload":"01000E10","source":"mqtt"},"downlink_pending":1} ``` -Uplinks with a confirmed data type are acknowledged in the same way, even if no command is queued. +Uplinks with a confirmed data type are acknowledged in the same way, even if no command is queued. When a device misses an ACK, it retransmits the same frame. The gateway acknowledges the retransmission again without publishing it twice. Confirmed uplinks (e.g. `AT+CCONFIRM=1` and `AT+CNBTRIALS=1,3` on Dragino devices) help a distant device make sure its events arrive. Keep in mind that the ACK is a downlink, so at the edge of the range the device may miss it and retransmit, which costs battery. #### Downlink rules The `downlinks` object of a device declaration holds commands sent automatically, and receive window settings: