sm-logtool

sm-logtool is a terminal-first log explorer for SmarterMail logs. It ships with:

• A Textual wizard UI (browse) for interactive searching.
• A console search command (search) for quick scripted checks.
• Log staging that copies or unzips source logs before analysis.
• Conversation/entry grouping for supported SmarterMail log kinds.
• Syntax-highlighted results in both TUI and CLI output.
• Live progress, execution mode, and cancel support for long TUI searches.
• Parallel multi-target search with safe serial fallback when needed.

Requirements

• Python 3.10+
• Linux (project classifiers currently target POSIX/Linux)

Deployment Model

sm-logtool does not require installation on the same host as SmarterMail, but it is designed for that workflow. In practice, you typically SSH to the mail server and run searches there.

The tool stages logs into a separate working directory so the original SmarterMail logs remain untouched during analysis and sub-searches.

Install

Install from PyPI (recommended):

pipx install sm-logtool

Alternative with pip:

python -m pip install sm-logtool

This installs the sm-logtool command.

Update

Update an existing install from PyPI:

pipx upgrade sm-logtool
# or
python -m pip install --upgrade sm-logtool

If you use the fuzzy-search speedup extra, update with extras:

pipx install --force "sm-logtool[speedups]"
# or
python -m pip install --upgrade "sm-logtool[speedups]"

Recommended Speedups (Strongly Recommended)

For significantly better fuzzy-search performance, install with the optional speedups extra:

pipx install "sm-logtool[speedups]"
# or
python -m pip install "sm-logtool[speedups]"

sm-logtool automatically uses the accelerator when available and automatically falls back to the built-in matcher when it is not installed. Skipping this extra can materially reduce fuzzy-search responsiveness and overall usability on large logs.

Configuration

Configuration is YAML with these keys:

• logs_dir: source SmarterMail logs directory.
• staging_dir: working directory used for copied/unzipped logs.
• default_kind: default log kind (for example smtp).
• theme: Textual UI theme name (for example Cyberdark, Cybernotdark, or textual-dark). Results syntax highlighting follows the selected UI theme palette and theme name.

Example:

logs_dir: /var/lib/smartermail/Logs
staging_dir: /var/tmp/sm-logtool/logs
default_kind: smtp
theme: Cyberdark

Repository template:

• config.example.yaml is a sample config for local bootstrap.
• The repository does not track a live runtime config.yaml.

Bootstrap a per-user config from the template:

mkdir -p ~/.config/sm-logtool
cp config.example.yaml ~/.config/sm-logtool/config.yaml

If staging_dir does not exist yet, the app creates it automatically. On startup and quit, staged files older than 14 days are pruned automatically.

Default config location is per-user:

• ~/.config/sm-logtool/config.yaml

Config resolution order:

1. --config /path/to/config.yaml
2. SM_LOGTOOL_CONFIG
3. ~/.config/sm-logtool/config.yaml

When the default path is used and the file does not exist, sm-logtool creates it automatically with SmarterMail-oriented defaults.

Usage

Top-level help:

sm-logtool --help
sm-logtool --version

Launch the TUI

sm-logtool
# or
sm-logtool browse --logs-dir /var/lib/smartermail/Logs

Browse Mode Workflow

Wizard flow:

1. Choose log kind.
2. Select one or more log dates. Today is selected by default; deselect it if unwanted.
3. Enter search term and choose search mode (Literal/Wildcard/Regex/Fuzzy) plus result mode (Show all related traffic/Only matching rows).
4. Review results, copy selection/all, and optionally run sub-search. Right-click on results to open copy actions.

Core actions are always visible in the top action strip:

• Ctrl+Q quit
• Ctrl+R reset search state
• Ctrl+U open command palette/menu

Search-step footer shortcuts:

• Ctrl+F focus search input
• Ctrl+Left previous search mode
• Ctrl+Right next search mode
• Ctrl+Up increase fuzzy threshold (fuzzy mode only)
• Ctrl+Down decrease fuzzy threshold (fuzzy mode only)

Clipboard note:

• Results copy actions use terminal clipboard protocol (OSC 52). Clipboard support depends on your terminal/multiplexer chain configuration.

Date selection shortcuts:

• Arrow keys to move
• Click to toggle a date
• Space to toggle a date
• Enter to select the highlighted date and continue
• Today is selected by default when the date step opens

Run console search

sm-logtool search --kind smtp --date 2024.01.01 "example.com"

Minimum examples:

# Search newest log for default_kind from config.yaml (default: smtp)
sm-logtool search "somebody@example.net"

# Search newest delivery log
sm-logtool search --kind delivery "somebody@example.net"

# Wildcard mode: '*' any chars, '?' single char
sm-logtool search --mode wildcard "Login failed: User * not found"

# Regex mode: Python regular expression
sm-logtool search --mode regex "Login failed: User \\[(sales|billing)\\]"

# Fuzzy mode: approximate matching with configurable threshold
sm-logtool search --mode fuzzy --fuzzy-threshold 0.72 \
  "Authentcation faild for user [sales]"

# Result mode: only show direct matching rows
sm-logtool search --result-mode matching-only "blocked"

Target resolution:

1. If --log-file is provided (repeatable), those files are searched.
2. Else if --date is provided (repeatable), those dates are searched.
3. Else the newest available log for --kind is searched.

Search options:

• --logs-dir: source logs directory. Optional when logs_dir is set in the active config file.
• --staging-dir: staging directory. Optional when staging_dir is set in the active config file.
• --kind: log kind. Optional when default_kind is set in the active config file.
• --date: YYYY.MM.DD date to search. Repeat to search multiple dates.
• --log-file: explicit file to search. Repeat to search multiple files.
• --list: list available logs for the selected kind and exit.
• --list-kinds: list supported kinds and exit.
• --mode: search mode (literal, wildcard, regex, or fuzzy).
• --fuzzy-threshold: similarity threshold for --mode fuzzy from 0.00 to 1.00 (default 0.75).
• --result-mode: output mode (related or matching-only). related (default) shows full grouped traffic for matched identifiers.
• --case-sensitive: disable default case-insensitive matching.

Search mode behavior:

• literal: exact substring matching (default).
• wildcard: * matches any sequence and ? matches one character.
• regex: Python re syntax (PCRE-like, but not full PCRE).
• fuzzy: approximate line matching using a similarity threshold. Installing sm-logtool[speedups] is strongly recommended for this mode.

Result mode behavior:

• related: show full grouped conversations for matched identifiers (default).
• matching-only: show only rows that directly match the search term.

Regex checker note:

• If an online regex builder does not offer Python mode, use PCRE/PCRE2 and stick to common features; some PCRE-only constructs may not work.

Convert Terminal Themes (Visual Utility)

Use the built-in visual converter:

sm-logtool themes --source ~/.config/sm-logtool/theme-sources

Theme file locations (per-user):

• Source theme files to import: ~/.config/sm-logtool/theme-sources
• Converted themes saved by Theme Studio: ~/.config/sm-logtool/themes
• Both directories are created automatically on first run of sm-logtool browse or sm-logtool themes.

These locations are user-home paths, so imported/converted themes are local user settings, not repository files.

Theme Studio workflow:

• Supported source files: .itermcolors, .colors, .colortheme.
• Toggle mapping profiles (balanced / vivid / soft) in the UI and preview both chrome and syntax colors live before saving.
• Toggle ANSI-256 quantization in the UI for non-truecolor terminals.
• Click preview elements to select a mapping target, then:
  • [ / ] cycle mapping source (auto, semantic colors, ansi0..ansi15)
  • - / = cycle mapping target
  • c clear current override
• Selection-row states are auto-corrected before save so Selected, Active, and Selected+Active remain distinct.
• sm-logtool browse auto-loads saved converted themes from that directory.
• Safety: when using --config or SM_LOGTOOL_CONFIG, in-app theme switching does not auto-write the config file.

Testing with a temporary config (recommended for development):

sm-logtool --config /tmp/sm-logtool-test.yaml themes
sm-logtool --config /tmp/sm-logtool-test.yaml browse

Supported Log Kinds

Search handlers currently exist for:

• smtp, imap, pop
• delivery
• administrative
• imapretrieval
• activation, autocleanfolders, calendars, contentfilter, event, generalerrors, indexing, ldap, maintenance, profiler, spamchecks, webdav

Log discovery expects SmarterMail-style names such as: YYYY.MM.DD-kind.log or YYYY.MM.DD-kind.log.zip.

Development

Install with test, lint, and type-check tooling:

python -m pip install -e ".[test,lint,typecheck]"

Run standards checks quickly:

python -m pytest -q test/test_line_length_policy.py \
  test/test_public_docstrings.py

Run lint checks before running tests:

python -m ruff check .

Run static type checks:

python -m mypy sm_logtool

Run the full test suite with both frameworks used in this repository:

pytest -q
python -m unittest discover test

Additional Docs

• Contributing
• Code of Conduct
• Search Design Notes
• Syntax Highlighting Notes
• Release Notes Template

License

This project is licensed under AGPL-3.0. See LICENSE.