clang+LTO makes USB Serial unacceptably slow

[Ivan-Voronyuk]

TL;DR

Enabling -flto (or CMAKE_INTERPROCEDURAL_OPTIMIZATION ON) causes USB Serial to output only one "block" per second in Release mode. The issue disappears in Debug builds.

---

Detailed Description

Minimal Repro Code

#include <pico/stdio.h>
int main() {
  stdio_init_all();
  uint32_t cnt = 0;
  while (1) stdio_printf("!!! %d\n", cnt++);
}

• Behavior with LTO:

  • Outputs exactly one line per second (vs. ~1.8k lines/s without LTO).
  • Replacing stdio_printf with printf doesn't make a difference
  • std::cout << "!!! " << cnt++ << std::endl; emits exactly one symbol per second.

  Issue only occurs with CMAKE_BUILD_TYPE=Release. Debug builds work normally.

---

How to Reproduce

1. Code: Use the main.cpp above.
2. CMakeLists.txt:

cmake_minimum_required(VERSION 3.13...3.27)
set(CMAKE_INTERPROCEDURAL_OPTIMIZATION TRUE)  # <-- Triggers the bug
set(PICO_SDK_PATH "<your_sdk_path>")
set(PICO_TOOLCHAIN_PATH "<llvm_toolchain_path>")  # Tested with clang-18, LLVM-ET-Arm-18.1.3-Linux-x86_64 toolchain
set(PICO_COMPILER pico_arm_cortex_m0plus_clang)
include(.../pico_sdk_import.cmake)

project(SLOW)
pico_sdk_init()

# Configure stdio
set(PICO_STDIO_UART OFF)
set(PICO_STDIO_USB ON)

add_executable(main main.cpp)
target_link_libraries(main PRIVATE pico_stdlib)
pico_enable_stdio_usb(main 1)
pico_enable_stdio_uart(main 0)
pico_set_float_implementation(main pico)
pico_add_extra_outputs(main)

---

This issue on tinyusb repo: hathach/tinyusb#3120

[geurtv]

Don't know if this should be considered a clang(*) or a pico-sdk bug: with LTO enabled the linker discards all the auto_init_mutex mutexes; section ".mutex_array" is empty. In your case it's print_mutex in stdio.c that causes problems. I don't know what memory print_mutex now occupies but mutex_try_enter_block_until() in function stdout_serialize_begin() now times-out every time (1000 ms timeout). Removing static from macro auto_init_mutex or adding __attribute__((used)) to it fixes the issue ... but I wouldn't be surprised if there will be other (hidden) issues ...

Note that LTO with gcc also fails. -flto + -Wl,--wrap doesn't work due to a one or two decade old linker bug.

Maybe pico-sdk can detect whether LTO is enabled and then abort with an error instead.

(*) I used clang 20.1.0-ATfl

[Ivan-Voronyuk]

Declaring mutex_t as volatile in mutex.h also fixes the issue.
It seems to me that mutexes should be volatile as they exist in order to sync code across different cores so it might be a good solution.

PS: I use clang-18.1.3 and LLVM-ET-Arm-18.1.3 toolchain.

[lurch]

See also https://github.com/raspberrypi/pico-sdk/issues?q=is%3Aissue%20lto

[kilograham]

Does

#define auto_init_mutex(name) static _attribute_((section(".mutex_array"))) __used mutex_t name

help?