Repository navigation
Conversation
Session::grouped(label, f) runs f as one undo step on the active document. On success the group leaves exactly one undo entry holding the workbook from before the group; on error it restores the workbook, selection, undo and redo stacks as they were and returns the error. group_begin, group_end and group_rollback do the same across separate calls, for an action that runs one step per call. While a group is open on a document, undoable commands on it add no undo steps of their own, so a long group neither grows the history nor pushes older entries past the undo limit, and Undo and Redo are disabled there. Groups nest: the outermost one makes the step, and a failing inner group rolls back only its own changes. Closing a document drops its groups, and a command that panics drops the groups it opened. history.list reports whether a group is open. Insert Copied Cells folds its nested steps into one undo step by counting revisions; inside a group the nested steps push no entries, so it now leaves the history alone there instead of removing entries from before the group.
A macro recorder needs more than a journal index: the journal is trimmed past 10,000 entries, it also holds the commands that other commands run (so playback would run them twice), and it keeps no selection, which relative references need. Session now tracks how deeply execute calls are nested. While recording, a buffer (record.rs) keeps each successful, journaled top-level call with the document uid, the active sheet (index and name) and the selection it ran in, then its id and parameters without secrets, as the journal stores them. Queries, failed commands and the command that starts recording are not kept. The buffer holds up to 100,000 steps and counts any it drops. Session::start_recording, stop_recording, is_recording and depth are the interface. Automate › Record Actions keeps recording through the journal as before.
Active content such as macros must not run only because a workbook was opened. DocState gets a trust state: Local for workbooks made in the session (new workbooks and samples) and FromFile for anything opened with file.open. Active content in a FromFile workbook asks before it runs. A second window on a workbook shares its trust. The clipboard carries the trust of its source document. Pasting formulas, formats (conditional formats hold formulas) or validation from a FromFile workbook into another workbook makes that workbook FromFile too; pasting values, column widths or notes carries nothing active and changes nothing. Sheet moves and copies stay inside one workbook, so its trust is unchanged. document.inspect reports the trust.
GridCraft keeps workbook data in hidden defined names so it travels inside the xlsx file. The existing items write one name holding one string literal, which past Excel's 255-character literal and 8,192-character formula limits gives a file Excel does not accept. cmd/stash.rs stores an item `key` as hidden workbook-scope names _gc_<key>_0, _gc_<key>_1, ..., each a formula of string literals of at most 255 characters (UTF-16 units, as Excel counts them) joined with &, at most 8,192 characters per name. Reading concatenates the chunks in order and stops at a gap; it also accepts the older single _sc_<key> name. Names are matched case-insensitively and only at workbook scope. Keys are limited to letters, digits, _ and ., items to 4,194,304 characters, and junk around or inside the literals, duplicate or out-of-order chunks, non-JSON and over-deep JSON all read safely. keys(prefix) lists the stored items that start with a prefix. Custom views, scripts, the watch list and scenarios keep their current storage; nothing reads or writes the new names yet. docs/architecture.md describes the engine building blocks added in this series: undo groups, the recording buffer, document trust and this storage.
…its JSON-lines protocol
Python code for cells runs in Monty 1.1.0 (MIT), a Python interpreter
written in Rust for sandboxed code, inside a separate worker process:
the app binary started again with --python-worker. Code such as
'x' * 10**12 makes an interpreter allocate until the process aborts, and
10**10**8 never checks the interpreter's time limit; catch_unwind can
stop neither, so only a separate process that the app can kill is a
reliable harness. RustPython was considered first, but every recent
release depends on the LGPL-3.0 malachite crates, which would be linked
into the app binary.
The new crate gridcraft-python (layer 1, on core) holds:
- protocol: one JSON object per line. The parent sends Init {limits,
init_code}, Run {id, code}, XlValue {id, reply} and Reset; the worker
answers Ready {init_error}, Xl {id, reference, headers} and Done {id,
output, stdout, stderr, error {line, message}, elapsed_ms}. Worker
frames start with a marker, so other text that reaches its standard
output is never taken for a frame. Values travel as
gridcraft_core::Value.
- worker (feature `python`, not on wasm32): one Monty REPL whose
namespace is shared by every cell, with the initialization code run at
start and on Reset. print output is collected per run (1 MB, then cut
off). xl() asks the parent and waits for the reply; file, environment
and network access are refused with PermissionError. Limits: the time
limit, recursion depth 200, 10,000 xl() calls per run and 1,000,000
output cells. A closed input pipe ends the worker even while code
runs.
- conversions: None to empty, bool, int and float to numbers (NaN,
infinities and integers too large for a double give #NUM!), str to
text, a list to a column, a list of lists to a table padded with #N/A,
a dict to a key/value table, and any other object to its repr. xl()
gives a scalar for one cell, a list of rows for a range, and a list of
dicts keyed by the first row with headers=True.
- client: PyWorker starts the worker and reads its frames on a thread,
so poll() never blocks longer than asked. A run past its time limit
plus one second is killed (#TIMEOUT!), a watchdog reads the worker's
resident set size every 50 ms and kills it above 512 MB, and a worker
that exits fails its runs; the next run starts a fresh worker. An
in-process transport runs the worker on a thread for tests.
apps/gridcraft and apps/gridcraft-cli check for --python-worker first
thing in main, before the logger, panic hook or window.
Tests: protocol round trips, conversions in both directions, error
lines, refused host access and the output cap; in-process runs with a
shared namespace, xl(), initialization code, reset and initialization
errors; and, on a real process, `while True: pass` and 10**10**9 time
out, 'x' * 10**12 is stopped by the memory watchdog, a killed worker is
replaced, and the host keeps running throughout.
…! errors
Cell gains `py: Option<Arc<PyCode>>`, read with a serde default and not
written for ordinary cells, so older native files load and ordinary
cells serialize as before. PyCode holds the code, the output mode
(Python object or cell values), the last result (value, type name,
preview, whether it converted to cells) and a stale flag for manual
calculation. The cell's `value` stays the cached result, so formulas,
display and saving read a Python cell like any other. is_blank,
CellStore::has and input_text (which returns the code) take the code
into account; Cell::has_content and clear_content cover value, formula
and code together. The model also reads and writes the file form,
PY("chunk"&"chunk",0|1) with string literals of at most 255 characters,
and finds the xl("...") literals in code, skipping comments and other
strings. Code is limited to 7,000 characters so that the formula stays
under the 8,192-character formula limit.
Edits: cell.set over a Python cell, Clear Contents, Clear All and the
cells a merge empties remove the code; Clear Formats keeps it. Copy and
paste (All, Formulas), Fill, sort, cut-move, sheet copy and undo carry
it unchanged; Paste Values and Paste Link do not, and paste operations
do not combine numbers with a Python cell. Fill copies Python cells
instead of extending them as a series. cell.set with =PY(...) is
refused with a message that points to python.set. The sheet-protection
check compares the code, so a locked Python cell cannot be changed.
cell.get reports {python: code, output, typeName, preview, stale}.
The xl() literals follow reference-moving edits: rewrite_all_formulas
(row and column insert and delete, cell insert and delete, cut-move),
sheet rename and sheet delete adjust each literal like a formula
reference, and replace it only when it changed.
Errors: CellError gains Python, Busy, Timeout and Blocked, and ALL has
14 entries, so the lexer reads them as literals. ERROR.TYPE gives 11 for
#BLOCKED!, the number Excel returns. Microsoft documents no ERROR.TYPE
number for #PYTHON!, #BUSY! or #TIMEOUT!; they take 19, 20 and 21, after
the numbers Excel is known to use. The xlsx writer caches the four as
#VALUE!, as it already does #SPILL! and #CALC!, because they are not
error codes of the file format.
Tests cover the =PY refusal, removal by entry and clearing, undo, copy,
paste values, fill and sort, xl() rewrites through row insert, sheet
rename, cut-move and column delete, native JSON round trips, and the
new errors.
…rker per workbook After every top-level command that changed a workbook, Session::execute hands the workbook before and after to the Python runtime (engine/src/python.rs), which finds the cells to run: Python cells whose code changed, whose xl() literals read cells that changed (cell store and spill differences, sheet by sheet; references resolve like a formula's, so names, tables and Sheet! references work), or whose literal names something only known at run time; every Python cell after the first of those, in row-major order sheet by sheet, because they share one namespace; all of them when a call is xl(<non-literal>) or a sheet with Python cells disappeared; and the cells after a removed Python cell. Python cell positions are cached per sheet version and patched from the edit's band differences, so a workbook without Python pays only for the bands an edit touched. Nothing runs when a workbook is opened. Each cell runs once per refresh: run, write the value (an array spills through calc::lay_python_spill, or shows #SPILL! when blocked), let calc recalculate its formula dependents, then the next cell. Results go straight into the workbook and are not undoable, as for data tables: one undo takes back the edit and the results it caused. A Python cell fed, through formulas or directly, by a later Python cell in the same refresh gets #CALC!. xl() is answered from the current workbook and refused for non-references, missing sheets and names, deleted (#REF!) cells and ranges over the output cap. Calc keeps a Python cell's spill: cells_changed no longer drops it as it does a replaced formula's, recalc_all lays Python spills out again from the kept results, and after each edit the runtime lays out again the spills of Python cells showing #SPILL! or whose area the edit touched, so clearing a blocker spills again without running the code. Runs happen inside the command for headless callers (CLI, MCP, tests). With set_python_background(true) a host steps them with step_python(); waiting cells show #BUSY!, and an edit, undo or reset during a refresh restarts it from the earlier of its next cell and the new first cell, discarding the result in flight. Undo, redo and the panic restore mark the namespace dirty: the next refresh resets it and runs every cell. A failure that took the worker down (time limit, memory, crash) starts a fresh worker, which runs the earlier cells again once before the queue continues. In manual and automatic-except-tables calculation, affected cells are only marked stale until Calculate Now or Calculate Sheet runs them. Workbooks opened from a file (Trust::FromFile) run nothing, and new or edited code there shows #BLOCKED!. Without a worker launcher (the web build, or a host that set none) cells show #BLOCKED!. Limits: Prefs.python_timeout_seconds (5 s) per cell, 512 MB per worker, recursion depth 200, 1 MB of output kept, 1,000,000 output cells. Every run is recorded in DocState.py_log, a ring buffer of 500 entries with the cell, status, duration, output and error line. gridcraft-python gains run.rs (the launcher, run outcomes, and the default launcher that apps set at start-up) and a web stand-in for the client with the same API, so the engine needs no cfg of its own. Both apps set the default launcher to their own binary. Tests: order and the shared namespace, xl() dependencies with formula dependents, the Python to formula to Python chain, the cycle error, spills blocked, unblocked and kept through a structural edit, errors with their line and output, time limits, ranges, names and other sheets, the trust gate, manual calculation, undo resetting the namespace, background runs with #BUSY! and restart, and a real worker process killed mid-refresh with the earlier cell run again.
…om files
cmd/python.rs registers the Python commands, all reachable from the
control channel, the CLI and MCP:
- python.insert {cell?} (Formulas > Python, Ctrl+Alt+Shift+P) makes the
cell a Python cell with no code yet;
- python.set {cell?, code, output?} sets the code (at most 7,000
characters) and runs it, answering with the cell's value; the same
code again keeps its result without running;
- python.clear {cell?} (Convert to Normal Cell) turns Python off,
clearing the code and value, undoably; the cells after it run again;
- python.output {cell?, mode: object|value} shows the result as a
Python object (its type name, with no spill) or converts it to values;
- python.reset {clearDiagnostics?} starts Python afresh and runs every
Python cell again;
- python.diagnostics {sheet?, errorsOnly?} returns the last 500 runs;
- python.initialization {code?} returns, or sets and then runs again
with, the workbook's initialization code, kept in the hidden workbook
data item py_init;
- python.options {timeoutSeconds?} reads or sets the per-cell time
limit (0.1 to 3600 s, default 5);
- python.enable lets Python run in a workbook opened from a file
(FromFile becomes Trusted) and runs its Python cells.
None of them opens a dialog. With empty parameters they act on the
active cell or report state, or return an error.
Trust gate: a workbook opened from a file runs no Python; code set
there shows #BLOCKED!, and python.reset does not change that;
python.enable does, by making the workbook Trusted, a new Trust state
for a file whose active content the person allowed to run.
document.inspect reports it as `trusted`. A locked cell on a protected sheet refuses
python.set through the existing protection check, which compares
Python code.
python.set, insert, output and clear refresh inside the command so they
can report the result, and tell Session::execute that the refresh is
done (only as the top-level command; inside another command, that
command's refresh covers them).
Tests: set, rerun avoidance, clear and undo, insert, output modes, the
code limit, parameter errors, protection, reset and diagnostics
filters, initialization code (first, on change, its errors), the trust
gate and enabling, a missing worker, the time-limit setting, and a CLI
test that drives python.set through gridcraft-cli with Python in a real
worker process: `while True: pass` gives #TIMEOUT!, 'x' * 10**12 gives
#PYTHON!, and the next cell still runs.
…s for Python content
A Python cell is written to xlsx as an ordinary formula,
PY("chunk"&"chunk",0|1), with string literals of at most 255
characters, 1 for a Python object result and 0 for values, and no
_xlfn. prefix, together with its cached value. A spilling result is
written like a dynamic array (cm="1", t="array" ref=...) with the
spilled values cached, so other spreadsheet applications show the
values (and #NAME? if they recalculate). The reader turns a formula
that is PY( followed by string literals, and optionally ,0 or ,1, back
into a Python cell, reading the formula's own text so that line breaks
in the code survive, and keeps the cached spilled values as the cell's
result, so a reopened workbook shows its arrays without running
anything. Python errors are cached as #VALUE!, as #SPILL! and #CALC!
already are. The initialization code travels in the hidden workbook
data names. Native JSON keeps Cell.py whole: code, output mode, the
last result and the exact error.
Opening any file adopts PY(...) formulas from every format
(io::adopt_python) and then applies limits to Python content
(io::enforce_python_limits): a cell with more than 7,000 characters of
code keeps only its cached value, with a warning, and a kept result
that is not a well-formed array of at most 1,000,000 cells is dropped.
calc::lay_python_spill refuses such an array too (#SPILL!), so a
crafted file cannot make it loop over a claimed size. Opening a file
still runs nothing: the workbook is FromFile, shows its cached values,
and runs Python only after python.enable.
python.set and python.insert refuse a table's header row, which holds
the column's name, as Excel keeps formulas out of headers.
Catalog: Formulas > Python gains Insert Python, Reset, Diagnostics and
Initialization; `cargo xtask parity` gives 278 / 294 (94.6%).
Tests: xlsx round trip of values, a formula over a Python cell, a
spilling table, an object-mode cell, 7,000 characters of code with
quotes and line breaks, an error and initialization code, checking the
written PY() formulas; the native JSON round trip; hostile content (code
over the limit in JSON and xlsx, a result claiming far more cells than
it holds); and a malformed result that never spills.
When the person has Python 3.11 or newer, Python cells run on it, so
pandas and numpy work; otherwise they run on the built-in Monty worker.
GridCraft never installs an interpreter or packages.
Finding the interpreter (gridcraft-python `interp`): the search order
is the path from python.options (a bare name is looked up on PATH),
VIRTUAL_ENV, CONDA_PREFIX, python3 then python on PATH, and on Windows
the `py -3` launcher and the usual install folders, newest first. The
Microsoft Store shortcuts under %LOCALAPPDATA%\Microsoft\WindowsApps
are skipped with a reason, since they open the Store instead of running
Python. Each candidate is probed under a 5 s deadline: first with -I
for its version (anything older than 3.11 is rejected with the version
it reported), then with -P -X utf8 for sys.executable, the pandas and
numpy versions (from package metadata, without importing pandas) and
whether it is externally managed (PEP 668). Cells run with
sys.executable, so a virtual environment and the program behind
`py -3` are used directly. The search takes a Host trait, so tests run
it against fake machines, including a fake Windows one. It runs on its
own thread and never on the web build. The engine caches the result per
session and searches again when the interpreter option changes, on
python.reset and on python.options {detect: true}. Apps opt in with
interp::enable_search(); tests and library users keep the built-in
Python or the launcher they set. Prefs.python_interpreter keeps the
path.
Running on it: the parent's process handling is unchanged (the deadline
kill, the memory watchdog, the restart after a crash and the JSON-lines
protocol, including the xl() round trip). Only the child differs:
`<python> -P -X utf8 -u -c <shim>` instead of the app binary with
--python-worker. -P keeps the current folder off sys.path; -I and -s
are not used because they hide user site-packages, where
`pip install --user` puts pandas.
- crates/python/src/shim.py, embedded with include_str!, is the worker
for CPython. It is data that the user's interpreter runs, not app
code. It keeps frames on a private copy of standard output and points
file descriptor 1 at standard error, so output from C code or child
processes cannot break a frame; reads input on a thread and exits
when GridCraft goes away; collects print output per run up to the
limit; stops a run at the time limit with a timer and interrupt_main
(the parent's kill covers C calls that cannot be interrupted), and
installs the SIGINT handler itself, since a parent started with SIGINT
ignored passes that on; and returns the last expression's value.
SystemExit is a cell error, not an exit.
- Conversions: xl() gives a pandas DataFrame when pandas is installed
(imported on first use; the first row is the header with
headers=True) and lists otherwise, as the built-in Python does.
Results: DataFrame (with the index only when it is not the default
RangeIndex), Series, one- and two-dimensional ndarrays, lists and
dicts spill; numpy scalars become numbers or booleans; datetime, date
and Timestamp become date serials; NaN and infinities become #NUM!;
None, NaT and NA become empty; anything else is an object with its
repr. The limits on output cells and xl() calls apply.
- The recursion limit on CPython is at least 1000, since pandas calls
deeper than the built-in Python's 200 allows.
- Workers start in a process group of their own (a job object on
Windows) through process-wrap (MIT OR Apache-2.0), so a kill also
stops processes the user's code started; CREATE_NO_WINDOW keeps a
console window from appearing on Windows. The worker's standard error
is drained on a thread, keeping its last 8 KB.
- A worker that exits or times out before its Ready message is started
once more and its waiting runs are sent again.
- The engine picks the launcher per run: one set for the session, else
the detected CPython, else the built-in worker. In the background,
cells wait as #BUSY! while the search finishes; headless callers wait
for it. A new search stops the workers so that they start again on
what it finds, from a fresh namespace, and an options change runs the
cells again.
python.options takes `interpreter` and `detect` and reports the
backend; python.diagnostics reports it too: kind, summary line, path,
version, pandas, numpy, source and every rejected candidate with its
reason.
The user's CPython has the access of the person's own account: files,
network and processes. The trust gate is what keeps a workbook opened
from a file from running code on it before python.enable.
Tests: the search against fake machines and this machine; shim protocol
tests that run when this machine has Python 3.11 or newer and are
skipped with a message otherwise (xl() round trip, conversions, print
capture, errors with lines, the time limit, the shim's own and the
parent's kill, a loop stopped with SIGINT ignored, reset, and a child
process killed with the worker's group); pandas tests that run only
when pandas imports; the retry before Ready; and an engine run of cells
on the detected CPython.
…against crash loops The built-in Python (Monty) is the fallback when no CPython is found. It now refuses code it cannot run before any of it runs, instead of running part of it, and no failure can hang the application or make the worker restart in a loop. - Imports: a small tokenizer (crates/python/src/imports.rs) reads the import statements of a cell, with strings and comments blanked, bracketed and backslash-continued lines joined, and statements split on `;` and `:`. A module Monty does not have (pandas, numpy, scipy, statistics and others) refuses the cell with #PYTHON! and "This cell needs CPython: `<module>` isn't available in the built-in Python. Install Python 3.11 or newer and set it in Python Options.", at the import's line. The scan is bounded and never fails. - Constructs: the cell is first compiled on its own; a NotImplementedError from Monty's compiler (match, del, yield, type aliases, wildcard imports and others) refuses it with "This cell needs CPython: the built-in Python doesn't support <what>." Other compile errors still come from the run, as before. - Messages, on both backends: a time limit gives #TIMEOUT! "Stopped after N s (an endless loop?)", whether the interpreter stopped itself or the parent killed it; RecursionError gives #PYTHON! "Too much recursion", as does a worker that died of a stack overflow (read from the end of its standard error); the memory watchdog gives "Used more than N MB". - Crash-loop guard: when the worker dies while running a cell (a kill at the deadline, the memory limit, a crash), the cell is remembered with a fingerprint of its code and of what its xl() calls read. It does not run again (#PYTHON!, "Not run: this cell stopped Python the last time it ran", status `refused` in the diagnostics) until one of those changes or python.reset runs everything again. A call whose reference is only known at run time counts any edit as a change. - On CPython, ModuleNotFoundError names the command for that interpreter, `"<sys.executable>" -m pip install <package>` (with the pip names of sklearn, cv2, PIL, yaml and a few others). When the interpreter is externally managed (PEP 668), it says to make a virtual environment instead and choose its python in python.options. GridCraft never runs pip. Tests: Monty refusals (pandas, statistics, from numpy, match, wildcard imports; nothing of a refused cell prints), the scanner on strings, comments, multi-line statements and hostile input, the exact time limit, recursion and memory messages, a worker dying of a stack overflow, the guard (refused after a kill, run again when its input changes, on reset and on a code change), and the pip hint on this machine's CPython.
…admap) - architecture.md: the python crate in the layer table, and a Python in cells section: worker processes, the protocol, the built-in and CPython backends, the engine runtime, the trust gate (with the Trusted state), files and the web build. - file-format-parity.md: Python cells in xlsx, other imports and native JSON, the initialization code and the limits on open. - function-parity.md: Python in cells, and #PYTHON!, #BUSY!, #TIMEOUT! and #BLOCKED! with their ERROR.TYPE numbers (19, 20 and 21 are GridCraft's own; Microsoft documents none for them). - cli.md: running Python cells headless, which Python runs them, and the trust gate for files opened with --in. - ROADMAP.md, target-app-parity.md and gaps.md: progress log, catalog 278 / 294, and the UI that is still to come. - AGENTS.md and CLAUDE.md: python in the layering list, and the CPython shim as the one exception to the Rust-only rule.
…rkbook
python.stop kills the active workbook's worker and drops its queued run.
Every Python cell that had not finished shows #PYTHON! ("Stopped") and a
"stopped" row in Diagnostics; nothing runs again until the next edit or
python.reset, which start from a fresh namespace. Cells that finished
keep their values. With nothing running it returns {stopped: 0}.
The package installs that follow use the same command to cancel an
install.
…person approves
GridCraft keeps one virtual environment of its own for Python cells,
made from the CPython it found (it never downloads an interpreter), in
the application's data folder: ~/Library/Application Support/GridCraft/
python-env on macOS, $XDG_DATA_HOME/gridcraft/python-env (by default
~/.local/share) on Linux and BSD, and %LOCALAPPDATA%\GridCraft\python-env
on Windows. GRIDCRAFT_PYTHON_ENV or python.options {environment} move it;
tests always do. The search for Python tries it after VIRTUAL_ENV and
CONDA_PREFIX and before PATH, and the probe reports whether a candidate
is a venv or a conda environment and which distributions it has.
- python.install {packages, approve, declare?} installs nothing without
approve: true; otherwise it fails with EngineError::Approve, whose
message names the packages and where they would go. It refuses while
Python is turned off for a workbook opened from a file. Requirements
are checked strictly: a PEP 508 name, plain extras and a version
specifier only (no URLs, paths, options, markers or other
characters), at most 32; import names such as sklearn and PIL map to
their pip names.
- Packages go into the person's own venv or conda environment when that
runs cells, otherwise into GridCraft's environment, made first with
uv venv --seed or python -m venv; never into a system or externally
managed Python.
- Installs run uv pip install when uv is on PATH, else python -m pip
install, always with --only-binary, from an argument list, on a
thread with a 10-minute deadline; python.stop cancels them. Failures
are explained (no network, no wheel for this platform or Python, pip
or venv missing). When an install succeeds, the search runs again and
every workbook's Python cells run again.
- python.installStatus reports the running or last install, with a
one-line summary and the versions installed; every install is a row
in python.diagnostics (kept whatever its sheet filter, and kept when
old runs are trimmed) and a record in the application log.
- python.packages declares the packages a workbook needs, kept in
hidden names so they travel in xlsx and native files, and checked and
capped when read; python.diagnostics reports each as found or missing.
- python.environment reports GridCraft's environment and its packages,
creates it, and deletes it (only a folder holding a pyvenv.cfg, and
only with approve: true).
- The CPython shim no longer appends a pip command to a
ModuleNotFoundError; the run records the missing module's pip name
(missing), for python.install.
- EngineError::Approve is added for commands that need the person's
approval. The macro engine pull request adds the same variant.
- CLI tests point GRIDCRAFT_PYTHON_ENV at a folder that does not exist,
so they never use the real environment. A test that installs from the
package index is ignored by default (it needs the network).
…warm-up after installs
A package's first import after an install can take longer than a cell's
5 s limit (a cold import of scikit-learn took about 22 s on macOS). The
cell then showed #TIMEOUT!, and the crash-loop guard refused to run it
again, so a working install looked like a failed one.
- On CPython, time spent importing modules not loaded yet does not
count against the time limit, up to python.options {importSeconds}
(1 to 600, 60 by default, kept in Prefs). The shim wraps
builtins.__import__ (the outermost import on the main thread) and
replaces its timer with a 50 ms watchdog that checks the run's time
less its import time. While an import runs it sends
FromWorker::Extend {id, ms, module}, and the parent moves its own
deadline by that much, never past the allowance; the parent's kill
stays the backstop. The built-in Python never sends it.
- A worker loads numpy and pandas, when installed, before it reports
ready, under an allowance of its own, and its start-up does not count
as the first cell's time.
- A time limit that interrupts an import replaces the worker, since a
package may be half loaded; the message names the module ("Stopped
after 61 s while importing sklearn (the first import after an install
can be slow)"), and RunOutcome carries it.
- A cell the time limit stopped mid-import, or on its first run on a
new worker, runs once more on a fresh worker after the cells before
it, shown as a "retried" row in Diagnostics; stopped again, it is
refused as before.
- Every install ends with a warm-up that imports what it installed once
under a 120 s deadline (python.installStatus {stage} says so while it
runs); a warm-up failure is a warning on an install that worked. uv
installs pass --compile-bytecode.
This was referenced Oct 12, 2026
Contributor
Author
|
Closed: the application-managed environment and package installs are not part of the restructured design described on #469. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #469 (Python in cells: engine), which depends on #462. Until #469 is merged, this pull request also shows its commits; the three commits listed under Changes are the ones to review.
Summary
This pull request lets GridCraft install Python packages for Python cells, and keeps a slow first import of a newly installed package from looking like a failure. It is engine-side only: everything is reachable through commands, and therefore through the CLI, MCP and the control channel. There is no user interface; the approval dialog, the install progress and the related Python Options belong to a later pull request.
python.installinstalls packages only when the call carriesapprove: true. Without it, the command fails with the question it would ask.New commands:
python.stop,python.install,python.installStatus,python.packagesandpython.environment.python.optionsgainsenvironmentandimportSeconds, andpython.diagnosticsgainspackagesand install rows. None of them opens a dialog.Design
Where the environment lives
~/Library/Application Support/GridCraft/python-env$XDG_DATA_HOME/gridcraft/python-env(by default~/.local/share/gridcraft/python-env)%LOCALAPPDATA%\GridCraft\python-envpython.options {environment}or theGRIDCRAFT_PYTHON_ENVenvironment variable moves it. Tests always move it into a temporary folder, so they never touch the real one. The CLI tests point it at a folder that does not exist. The search for Python (interp.rs) now tries this environment afterVIRTUAL_ENVandCONDA_PREFIXand beforePATH. The probe also reports whether a candidate is a venv or a conda environment, and which distributions it has.python.environmentreports the environment and its packages.{action: "create"}makes it, and{action: "delete", approve: true}removes it. Only a folder that holds apyvenv.cfgis ever deleted.Installers: uv, pip and venv
When
uvis onPATH, the environment is made withuv venv --seedand packages are installed withuv pip install --python <env> --only-binary :all: --compile-bytecode.--seedputs pip in as well, so the environment keeps working without uv.When
uvis absent, GridCraft uses the interpreter's own tools:python -m venvandpython -m pip install --only-binary=:all: --require-virtualenv. These are slower, and they need thevenvmodule and pip; some Linux distributions package these separately (for examplepython3-venv). When one of them is missing, the error says so and suggests installing it or uv. A conda environment is always installed into with its own pip.Only ready-made packages (wheels) are installed, so no package is built from source on the user's machine. Every program is run from an argument list, never through a shell. An install runs on a background thread with a 10-minute deadline, and
python.stopcancels it. Its output is kept, at most 64 KB of it. A failure is explained from that output: no network, no wheel for this platform or Python version, or pip or venv missing.The approval gate
python.install {packages, approve, declare?}withoutapprove: truereturnsEngineError::Approveand starts nothing. The message names the packages, given by their pip names, and the environment they would go into. It also states that installing downloads code from the internet and runs it on this computer.python.environment {action: "delete"}needsapprove: truein the same way.Trust::FromFile) cannot install anything untilpython.enablehas been run for it.-ror--index-url, environment markers and other characters are refused, and at most 32 requirements are accepted. Import names that differ from their pip names (sklearn,PIL,cv2,yamland others) are mapped to the pip name.python.packages. They are kept in hidden names, so they travel in xlsx and native files. They are checked and capped again when a file is read, and declaring them never installs anything.EngineError::Approveis new in this branch. The macro engine pull request (#465) adds the same variant with the same message format. Whichever pull request is merged second will have a small conflict inEngineError, which is resolved by keeping one copy.When an install succeeds, the search for Python runs again, workers restart on the new environment, and every workbook's Python cells run again.
python.installStatusreports the running or the last install, with a one-line summary such as "Installed scikit-learn 1.9.1" and the versions installed. Every install is also a row inpython.diagnosticsand a record in the application log. Install rows are kept whatever the sheet filter says, and they are kept when old runs are trimmed.Time limits and imports
The shim wraps
builtins.__import__, and only the outermost import on the main thread is timed. Itsthreading.Timeris replaced by a 50 ms watchdog that compares the run's time, less its import time, with the cell's limit. The import time that is not counted is capped bypython.options {importSeconds}, which accepts 1 to 600 and defaults to 60.While an import runs, the shim sends
FromWorker::Extend {id, ms, module}heartbeats. The parent moves its own deadline by that amount, never past the allowance. The parent's kill stays the backstop for code that never returns to Python. The built-in Python (Monty) never sends these heartbeats.retriedrow. If the cell is stopped again, the crash-loop guard refuses it as before.python.installStatus {stage}reports this step while it runs. If the warm-up fails, the install still counts as successful and carries a warning.Web build
Installs are compiled out of wasm32:
env::AVAILABLEis false, and the commands return an error saying Python is not available in the web version.Changes
#PYTHON!("Stopped").crates/python/src/env.rs(new) covers requirement checks, pip names, the environment folder, install steps, the install runner and failure explanations.interp.rsadds the environment to the search and the venv, conda and distribution probe.crates/engine/src/pyenv.rs(new) contains the five commands,EngineError::Approve,Prefs::python_environment, the install rows inPyRunLog, andmissingon runs that failed to import a module.ModuleNotFoundError. The engine records the missing module's pip name instead.GRIDCRAFT_PYTHON_ENV.docs/architecture.md,docs/cli.mdandROADMAP.md.FromWorker::Extend,Limits::import_ms, andPyError::{restart, importing}.Prefs::python_import_seconds, the retry, and the warm-up step.docs/architecture.md,docs/cli.md,docs/gaps.md§22 andROADMAP.md.Not included, and left for later pull requests:
Known issue: the first import of scikit-learn
This issue is mitigated, not fixed. On macOS, in a new environment, the first import of scikit-learn after installing it took about 22 s. Later imports took under 1 s. The time appears to go into loading scipy's and numpy's native libraries for the first time, at roughly 0.5 to 1 s each. This looks like the operating system checking each new binary, but the cause has not been confirmed. Compiling bytecode made no measurable difference.
Before this change, that first import exceeded the 5 s cell limit, and the crash-loop guard then refused the cell. With the import allowance, the warm-up after the install and the single retry, a first
import sklearnin a cell succeeded in testing. The behaviour on other platforms and with other large packages has not been measured.docs/gaps.md§22 records the issue.Dependencies and licences
This pull request adds no Rust dependencies;
Cargo.tomlandCargo.lockare unchanged. At run time, GridCraft uses these programs when they are present on the user's machine. None of them is bundled or downloaded by GridCraft.venvandensurepipPackages that the user approves are installed from the package index that their installer is configured to use, under each package's own licence.
Testing
cargo xtask ci: formatting and clippy (-D warnings) pass for each commit. The test step fails only on the three tests that already fail on main:io::tests::csv_save_keeps_the_separator_it_was_opened_with;tests::saving_an_opened_semicolon_csv_keeps_semicolons.These are fixed by CSV/text export: General numbers without exponents (integers below 1e16 exact);
&switches to scientific past 21 characters #449 and fix(i18n): correct scattered wrong French translations and add 2 missing keys #454.cargo test --workspace --no-fail-fast: every other test passes.every_command_survives_empty_paramscovers the new commands.cargo xtask assets,layers,wasm,parityandkeytipspass.New engine tests (
tests/python_install.rs) cover:python.stopcancelling an install;New tests for the time limits:
importSecondsis validated and saved.Tests that need CPython 3.11 or newer skip without it.
GRIDCRAFT_TEST_PYTHONcan point them at a specific interpreter.One test installs a real package from the package index into a temporary environment and imports it in a cell. It needs the network, so it is
#[ignore]by default and CI stays offline. Run it withcargo test -p gridcraft-engine -- --ignored a_package_installs_for_real. It passed locally on macOS with uv.Headless check: with
GRIDCRAFT_PYTHON_ENVset to a temporary folder,gridcraft-cli runwas used to:python.install {packages: ["six"]}without approval fail with the approval question;six;Generative AI use: The code and descriptions in this pull request were written with the assistance of Claude Opus 5.5 (Claude Code). I directed the work, chose its scope, and reviewed and tested the changes.