run_command "crashing" my VPS OCI Linux ARM

[dhaern]

This is the 2nd time this happen but when the model use run_command with a relative moderate size command my vps "crash" and stop respond (maybe with a high RAM usage), I'm running Ubuntu 24.04 OCI ARM64 with 24gb RAM and I had enough free RAM using OWUI. Never happened with my custom Jupyter Multikernel tool for OWUI, only with open terminal using run_command both times.

Here the crash from my browser outside my vps, this stopped to respond and NoMachine crashed.

[dhaern]

I think there may be a serious performance problem in the run_command / /execute path.

I’m running Open Terminal 0.11.15 on Ubuntu 24.04 ARM64, and I’ve now had the whole VPS effectively hang twice when a model used run_command with a moderately large command. Not huge output, not anything obviously extreme, but enough that the machine became barely responsive for a few minutes. In both cases it eventually recovered on its own.

What makes this suspicious is that it does not look like a simple OOM case. I still had free RAM available, but during the incident the VPS became sluggish enough that even SSH / remote desktop responsiveness was affected.

From what I can see, the Open Terminal service itself does not appear to be ballooning in resident memory, so I’m not sure this is a classic memory leak. It feels more like something in the /execute implementation may be causing very heavy I/O or inefficient subprocess output handling under certain commands.

I can see successful requests like POST /execute?wait=120&tail=250, so this is not an obvious crash with a traceback. But the correlation with run_command is now strong enough that I wanted to report it in case this path has known performance issues.

After reading the implementation, one thing that stood out is that on Linux the /execute path appears to use a PTY by default, write command output into JSONL logs in small chunks, and flush to disk on every chunk before reading the log back to build the response. That may be fine for light interactive use, but it looks like it could become very expensive for non-interactive commands that produce bursts of output, even if the total output is not especially large.

[frenzybiscuit]

What are you trying to do?

Can you provide an example command on how to test this out?

[dhaern]

What are you trying to do?

Can you provide an example command on how to test this out?

This is a very random "crash" and hard to reproduce, anyways with the new:

OPEN_TERMINAL_LOG_FLUSH_INTERVAL=1
OPEN_TERMINAL_LOG_FLUSH_BUFFER=65536

I can lower the 4KB PTY and making this configurable helps to performance but I think the real root problem here was OWUI huge historial + context compression tool making it even bigger and jumping the OWUI RAM consumption from 4-5gb to 17gb and "crashing".

[tjbck]

The root cause is per-chunk fsync() on every 4KB PTY read (~250 fsyncs/sec on high-output commands).

Flushing is now configurable. For your ARM VPS, set:

OPEN_TERMINAL_LOG_FLUSH_INTERVAL=1
OPEN_TERMINAL_LOG_FLUSH_BUFFER=65536

Default behaviour is unchanged. Let me know if this resolves it.