Skip to content

[SYS] Re-enable serial logs on esp32c3_lolin_mini - #2429

Open
MatthieuGrr wants to merge 1 commit into
1technophile:developmentfrom
MatthieuGrr:feat/c3-lolin-mini-serial-logs
Open

MatthieuGrr wants to merge 1 commit into
1technophile:developmentfrom
MatthieuGrr:feat/c3-lolin-mini-serial-logs

Conversation

@MatthieuGrr

Copy link
Copy Markdown
Contributor

Description:

Follow-up to #2428, as announced there: re-enables serial logging on esp32c3_lolin_mini.

The env was built with LOG_LEVEL_SILENT and WM_DEBUG=0 because serial output delayed every message when no listener was connected (native USB CDC). #2428 makes HWCDC writes non-blocking, so this workaround is no longer needed and the default log level applies again. esp32c3_lolin_mini_with_serial is left untouched, but it is now nearly identical to esp32c3_lolin_mini: it only re-sets the default LOG_LEVEL_NOTICE and adds two CONFIG_ESP_CONSOLE_* defines. Happy to remove it (from environments.ini and environments.json) in this PR or in a follow-up, as you prefer.

Testing

Build: esp32c3_lolin_mini on this commit, no warnings, flash 96.5 % (1,896,311 / 1,966,080 bytes), RAM 17.5 %.

ESP32-C3 board (not a Lolin, but same chip and native USB-Serial/JTAG port) flashed with this env, plugged into a Linux host, BLE scan interval set to 100 to load the loop. Same env (this PR's esp32c3_lolin_mini + MQTT presets) built twice: on development @ 9b09fcd (with #2428) and on 5e7d3cb (without it). Three phases each: port read continuously, port opened and never read, port closed.

With #2428 Without #2428
Port read no drop, ~260 BTtoMQTT msg/min no drop, ~43 msg/min
Port open, never read (1 h) no drop, ~260 msg/min, uptime steady loop stalled ~4 min after start: BLE traffic 9 → 0 msg/min, LWT offline 5 times over 17 min, SYStoMQTT queued then flushed at once; recovered without reboot, then ~43 msg/min
Port closed (20 min) no drop, ~265 msg/min no drop, ~44 msg/min
Free heap (min) 912 B under full load 20 KB (6× less traffic)

With #2428 the gateway ran 1 h 45 through all three phases without a single LWT offline (and was still online after more than 2 h). Opening the port reboots the board once in both builds (DTR/RTS untouched), unrelated to this change.

Checklist:

  • The pull request is done against the latest development branch
  • Only one feature/fix was added per PR and the code change compiles without warnings
  • I accept the DCO.

The silent log level was a workaround for the native USB CDC stall fixed
in 1technophile#2428 (non-blocking HWCDC writes). With that fix, logging no longer
delays message processing when no serial listener is connected.

Signed-off-by: Matthieu <93468141+MatthieuGrr@users.noreply.github.com>

@1technophile 1technophile left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for following up on #2428. Re-enabling logs on this env makes sense now that the USB-CDC writes are non-blocking, but a few things are worth checking first (details inline):

  • With logs compiled in, the WebUI console capture (addLog()) becomes reachable on this env, and it has an unlocked shared buffer that can overflow by 1–2 bytes under task preemption.
  • The soak in the description shows a 912 B minimum free heap. It would help to see the same soak on the current LOG_LEVEL_SILENT build for comparison.

Outside the diff:

  • esp32c3_lolin_mini_with_serial is now effectively identical to the base env. Its extra CONFIG_ESP_CONSOLE_* -D flags don't affect the precompiled Arduino libs, and -DCONFIG_ESP_CONSOLE_UART=ESP_CONSOLE_USB_SERIAL_JTAG redefines a boolean sdkconfig macro with an undefined token (redefinition warning; any #if CONFIG_ESP_CONSOLE_UART then evaluates to 0). I'd suggest deleting that env here, from both environments.ini and .github/workflows/environments.json, so CI and the web installer don't ship two indistinguishable C3 images.

Comment thread environments.ini
; Serial output currently causes a delay on each message, if no serial listener is connected.
; As a default we disable serial output. Use the _with_serial environment if you
; need serial output for debugging purposes.
'-DLOG_LEVEL=LOG_LEVEL_SILENT' ; disable serial logging.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With LOG_LEVEL_SILENT gone, every THEENGS_LOG_* call also goes through SerialWeb::write() → addLog() (main/webUI.cpp) once someone has opened the WebUI console and log_buffer is allocated. addLog() uses the global line (malloc'd ROW_LENGTH = 512) and lineIndex with no lock. If the loop task passes the lineIndex >= ROW_LENGTH - 2 check at index 509 and is then preempted by procBLETask (priority 2, same core on the single-core C3), which also logs, the loop task resumes and writes line[511] plus the terminator at line[512]. That is a heap overflow, which shows up later as Bad head or a panic.

The bug isn't C3-specific, but until now this env never ran that path because all logging compiled out. A mutex around addLog() (or a per-call local buffer) would fix it, either here or in a separate PR before this lands.

Comment thread environments.ini
; Serial output currently causes a delay on each message, if no serial listener is connected.
; As a default we disable serial output. Use the _with_serial environment if you
; need serial output for debugging purposes.
'-DLOG_LEVEL=LOG_LEVEL_SILENT' ; disable serial logging.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory margin: the soak in the description reaches a 912 B minimum free heap with logging on. Both columns of the table come from the log-enabled env, so nothing shows how much of that comes from re-enabling logs versus what the current silent build reaches under the same ~260 msg/min load. Could you rerun the soak on the current env for comparison? With under 1 KB headroom, a transient WiFiClient, ArduinoJson or mbedTLS allocation during a BLE burst can fail and drop publishes or reboot the gateway.

Related: procBLETask is created with a fixed 8000 B stack (main/gatewayBT.cpp) and coreTask with 5120 B. Those margins were sized with logging compiled out. It would be worth logging uxTaskGetStackHighWaterMark() for both during the soak to confirm the added log formatting frames still fit.

Comment thread environments.ini
; Serial output currently causes a delay on each message, if no serial listener is connected.
; As a default we disable serial output. Use the _with_serial environment if you
; need serial output for debugging purposes.
'-DLOG_LEVEL=LOG_LEVEL_SILENT' ; disable serial logging.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor, but relevant now that logs are visible: with SERIAL_TX_TIMEOUT_MS=0 from #2428, HWCDC::flush() doesn't wait for the TX ring to drain. In main.cpp, THEENGS_LOG_WARNING("Rebooting for reason code %d") is followed immediately by ESP.restart(), so that line (and any OTA or erase messages still in the ring) usually never reaches the host, even though it's the line that explains the reset. A short bounded drain before restart (e.g. wait until Serial.availableForWrite() recovers, capped at ~100 ms) would keep it.

Comment thread environments.ini
; Serial output currently causes a delay on each message, if no serial listener is connected.
; As a default we disable serial output. Use the _with_serial environment if you
; need serial output for debugging purposes.
'-DLOG_LEVEL=LOG_LEVEL_SILENT' ; disable serial logging.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flash headroom: with the log strings compiled back in, the image is 1,896,311 / 1,966,080 B (96.5%) of the min_spiffs app slot, leaving about 69 KB. That's fine today, but the next Theengs Decoder or NimBLE bump could push this env over (section .flash.text will not fit). It's worth knowing about, and maybe noting in the env comment.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants