DataSynth

DataSynth generates realistic synthetic email conversations and supporting documents for eDiscovery testing, ML datasets, and email-processing validation. It provides a Rust terminal interface backed by a packaged Python generation engine.

Generated messages are MIME .eml files with Message-ID, In-Reply-To, References, and DataSynth thread metadata. PDF and DOCX attachments are embedded in their messages and retained as numbered sidecars for combined-PDF workflows.

Install a release

The Linux release archive contains everything needed to run DataSynth. Python, pip, Rust, and Cargo are not required.

Verify the download, then extract and run it:

sha256sum --check datasynth-v0.1.0-linux-x86_64.tar.gz.sha256
tar -xzf datasynth-v0.1.0-linux-x86_64.tar.gz
cd datasynth-v0.1.0-linux-x86_64

export GEMINI_API_KEY="..."
# or: export OPENROUTER_API_KEY="..."

./datasynth

DataSynth stores settings and output outside the installation directory:

Data	Linux default
Settings	~/.config/datasynth/settings.json
Generated output	~/.local/share/datasynth/output/
Generated roster	~/.local/share/datasynth/roster.json

XDG_CONFIG_HOME and XDG_DATA_HOME are respected. For isolated runs or testing, set DATASYNTH_CONFIG_DIR and DATASYNTH_DATA_DIR.

Terminal workflow

1. Choose Gemini or OpenRouter and select a model.
2. Load and select a topic.
3. Generate two fictional companies and their employee rosters.
4. Configure the target count, attachment rate, and thread action weights.
5. Inspect the prompt preview.
6. Generate .eml files and attachments.
7. Convert a run to PDF and optionally combine it.
8. Apply Bates numbering to a combined PDF.

Engine CLI

Developers can use the worker directly:

uv run datasynth-worker generate \
  --files 25 \
  --attachments 30 \
  --topic "Quarterly Review" \
  --roster path/to/roster.json \
  --provider gemini \
  --model gemini-2.5-flash

uv run datasynth-worker convert \
  --folder path/to/generated/run \
  --combine

uv run datasynth-worker bates \
  --file path/to/generated/run/run_combined.pdf \
  --prefix CASE

Omitting --provider runs the deterministic local/faker path and does not require an API key. This is useful for smoke testing.

Run datasynth-worker <command> --help for all options. Existing output directories are preserved by default; generate --overwrite is required to replace their numbered files.

Development

Prerequisites are Rust 1.80 or newer, Python 3.11 or newer, and uv or a standard Python virtual environment.

./start.sh

The launcher installs the locked Python environment and starts the release-mode TUI. It does not require an API key merely to open the application.

Run the checks independently with:

cargo fmt -- --check
cargo clippy --locked --all-targets -- -D warnings
cargo test --locked

uv run ruff check datasynth tests
uv run ruff format --check datasynth tests
uv run pytest

Build a Linux release

./scripts/build-release.sh

The script builds:

• datasynth, the Rust terminal frontend;
• datasynth-worker, a standalone Python worker with its runtime and font;
• a versioned .tar.gz archive and SHA-256 checksum under dist/.

The frontend locates a sibling datasynth-worker automatically. Developers can override it with DATASYNTH_WORKER=/path/to/datasynth-worker.

Repository layout

assets/data/        Embedded topic, company, and person lists
datasynth/          Python generation and document-processing package
fonts/              Bundled Unicode font
src/                Rust terminal frontend
tests/              Python unit and integration tests
scripts/            Release build tooling

License

DataSynth is proprietary software. See LICENSE.