Skip to content

sphinxdocs: implement a persistent worker mode #2878

@rickeylev

Description

@rickeylev

Right now, sphinxdocs runs as a fresh process every time. For iterative development,
this adds unnecessary overhead in the setup and execution of sphinx. It also creates
barriers to make use of Sphinx's incremental rebuilding features.

By implementing the sphinx action as a persistent worker, it should be possible to reduce the startup overhead and better detect what has changed.

I see two basic implementation options:

ideal: The worker is the sphinx process. Build requests then re-use the existing sphinx process. This eliminates any startup overhead. This requires

  1. Implementing a wrapper that uses sphinx as a library. Essentially running a while True: wait_for_request(); build() loop.
  2. That Sphinx be OK with a single process handling multiple builds. i.e. that the global process state (caches etc) is OK to reuse between builds.

backup: The worker calls sphinx-build as a subprocess. The persistent worker API tells which files changed, so the worker can massage state so that sphinx's incremental (timestamp based) building works better.

The env-related event api might also be relevent to this: https://www.sphinx-doc.org/en/master/extdev/event_callbacks.html

Notably, events to intercept how source files are read. e.g. intercept how they're read and keep an in-memory cache based on hash.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions