Problem
coli chat currently prints a completion footer with an approximate token count and elapsed time, for example:
└─ ~370 tok · 313s
For local inference, especially with large disk-streamed models, the actual generation throughput is an important runtime metric. Seeing tokens/second after each response would make it much easier to evaluate tuning changes, different execution paths, storage configurations, warm/cold behavior, and general inference performance.
The current token count is estimated from the generated text. Simply hard-coding another calculation into the existing footer would work, but would make the implementation less flexible and would still couple measurement and presentation.
It would be preferable to have a reusable generation-statistics mechanism that can use exact token counts when available and fall back to the current estimate otherwise.
Proposed solution
Add configurable generation statistics to coli chat, while separating statistics collection/calculation from terminal rendering.
A possible user-facing interface could be:
coli chat --stats full
coli chat --stats compact
coli chat --stats off
Optionally, an environment variable such as COLI_CHAT_STATS could define the default.
Example output:
Full:
└─ ~370 tok · 313s · ~1.18 tok/s
Compact:
└─ ~370 tok · ~1.18 tok/s
Off:
No statistics footer.
The statistics implementation should preferably support two sources for token counts:
- Exact completion-token counts supplied by the engine/server when available.
- The existing character-based estimate as a fallback.
The output could distinguish between exact and estimated measurements.
Estimated:
└─ ~370 tok · 313s · ~1.18 tok/s
Exact:
└─ 372 tok · 313s · 1.19 tok/s
This would allow the implementation to move to exact token accounting later without changing the chat UI or its public interface.
The existing interrupted state should remain compatible, for example:
└─ ~30 tok · 23s · ~1.30 tok/s · ⏹ interrupted
The exact internal implementation is intentionally left open. The main request is to avoid coupling the throughput calculation directly to a hard-coded footer and instead provide a reusable and configurable statistics path.
Alternatives considered
The simplest alternative would be to extend the existing footer directly and calculate:
estimated_tokens / elapsed_time
This would produce the desired tokens/second value with negligible performance overhead.
However, this would keep token estimation, throughput calculation, and terminal formatting coupled together. It would also require another change later if exact completion-token counts become available from the engine or OpenAI-compatible server.
Another alternative would be a fully user-defined format string. This would provide maximum flexibility, but seems unnecessarily complex for this use case and would introduce validation and compatibility concerns.
A small set of predefined modes such as off, compact, and full seems sufficient while keeping the implementation maintainable.
Scope and compatibility
This should be a CLI/UI-level feature and should not require changes to model formats or inference behavior.
The feature should remain compatible with:
- CPU inference
- Metal inference
- CUDA inference
- local/private chat engines
- attached
coli serve sessions
- different model architectures
- existing zero-dependency operation
The statistics calculation itself should have negligible performance impact because it occurs after or alongside generation and only requires simple token accounting and elapsed-time calculation.
If exact completion-token counts are unavailable on a particular execution path, the existing approximate token-count method can remain the fallback.
Existing behavior can be preserved by choosing an appropriate default mode, so scripts and current workflows should not require modification.
Problem
coli chatcurrently prints a completion footer with an approximate token count and elapsed time, for example:└─ ~370 tok · 313sFor local inference, especially with large disk-streamed models, the actual generation throughput is an important runtime metric. Seeing tokens/second after each response would make it much easier to evaluate tuning changes, different execution paths, storage configurations, warm/cold behavior, and general inference performance.
The current token count is estimated from the generated text. Simply hard-coding another calculation into the existing footer would work, but would make the implementation less flexible and would still couple measurement and presentation.
It would be preferable to have a reusable generation-statistics mechanism that can use exact token counts when available and fall back to the current estimate otherwise.
Proposed solution
Add configurable generation statistics to
coli chat, while separating statistics collection/calculation from terminal rendering.A possible user-facing interface could be:
coli chat --stats fullcoli chat --stats compactcoli chat --stats offOptionally, an environment variable such as
COLI_CHAT_STATScould define the default.Example output:
Full:
└─ ~370 tok · 313s · ~1.18 tok/sCompact:
└─ ~370 tok · ~1.18 tok/sOff:
No statistics footer.
The statistics implementation should preferably support two sources for token counts:
The output could distinguish between exact and estimated measurements.
Estimated:
└─ ~370 tok · 313s · ~1.18 tok/sExact:
└─ 372 tok · 313s · 1.19 tok/sThis would allow the implementation to move to exact token accounting later without changing the chat UI or its public interface.
The existing interrupted state should remain compatible, for example:
└─ ~30 tok · 23s · ~1.30 tok/s · ⏹ interruptedThe exact internal implementation is intentionally left open. The main request is to avoid coupling the throughput calculation directly to a hard-coded footer and instead provide a reusable and configurable statistics path.
Alternatives considered
The simplest alternative would be to extend the existing footer directly and calculate:
estimated_tokens / elapsed_timeThis would produce the desired tokens/second value with negligible performance overhead.
However, this would keep token estimation, throughput calculation, and terminal formatting coupled together. It would also require another change later if exact completion-token counts become available from the engine or OpenAI-compatible server.
Another alternative would be a fully user-defined format string. This would provide maximum flexibility, but seems unnecessarily complex for this use case and would introduce validation and compatibility concerns.
A small set of predefined modes such as
off,compact, andfullseems sufficient while keeping the implementation maintainable.Scope and compatibility
This should be a CLI/UI-level feature and should not require changes to model formats or inference behavior.
The feature should remain compatible with:
coli servesessionsThe statistics calculation itself should have negligible performance impact because it occurs after or alongside generation and only requires simple token accounting and elapsed-time calculation.
If exact completion-token counts are unavailable on a particular execution path, the existing approximate token-count method can remain the fallback.
Existing behavior can be preserved by choosing an appropriate default mode, so scripts and current workflows should not require modification.