Support for running nbqa on quatro format (.qmd) markdown files

[hguturu]

Closes #880

Two main changes:

• Added check for .qmd when using Jupytext
• Added conversion of leading quartro cell options #| that get changed to # | by black/ruff back to #|
  (_restore_quarto_cell_options) - in theory this can be dropped if black and ruff natively support this format
  • black is evaluating how to handle this situation - Exceptions to comment spacing psf/black#3668
  • ruff also has the issue - Add support for Quarto notebooks  astral-sh/ruff#6140
  • No longer needed since Quarto authors confirm it can handle both #| and # | for python due to formatters - Quarto Cell Options Specification quarto-dev/quarto-cli#12364 (comment)
• Added test_jupytext.test_qmd and supporting resource tests/data/notebook_for_testing.qmd to test new functionality
• Updated README.md with .qmd support feature
• Updated requirements-dev.txt and setup.cfg with quarto-cli dependency

20250324:

• Refactored usage of _restore_quarto_cell_options so other formats don't break
  • Removed logic all together since both #| and # | are supported
• Discovered new cell option behavior and checking quarto team if valid behavior or bug in quarto - Quarto Cell Options Specification quarto-dev/quarto-cli#12363
  • Test case added for new multi-line cell options

PR simplified to just checking .qmd extension since special handling logic for quarto #| are no longer needed. Authors also confirm {python} is required for runable code blocks (the same ones we format) quarto-dev/quarto-cli#12364 (reply in thread). All other cases of references to python are test with syntax highlighting.

Code tested locally with tox -e py and all tests passing, so should be ready for another round of github actions.

[hguturu]

@MarcoGorelli PR should be ready for another round of workflow runs by the testsuite. Logic has also been simplified.

[malcolmbarrett]

I've been following this, but I noticed while playing with the current implementation that it adds a bunch of metadata to the file's yaml, e.g., in the case of the example in the PR

jupyter:
  jupytext:
    text_representation:
      extension: .qmd
      format_name: quarto
      format_version: '1.0'
      jupytext_version: 1.16.7
  kernelspec:
    display_name: Python 3
    language: python
    name: python3

Would that be possible to avoid? I don't know how useful it is for the underlying tools, but it certainly takes up a lot of space in an area that often contains important information

[hguturu]

That is actually not being done by nbqa, but rather jupytext and quarto convert.

nbqa converts qmd to json via jupytext at https://github.com/nbQA-dev/nbQA/pull/881/files#diff-d72c6dc865400b582f957459d2f13e4030da8a9f8afdec522def74313d29a25fR109.

And then converts back to qmd via jupytext at https://github.com/nbQA-dev/nbQA/pull/881/files#diff-30fd9daaa53315a90c29babeec62d45b8705ebe8f57133a0034a01db15a7860aR229.

The jupytext conversion back to qmd - https://github.com/mwouts/jupytext/blob/main/src/jupytext/jupytext.py#L238.

The jupytext metadata is added by https://github.com/mwouts/jupytext/blob/main/src/jupytext/jupytext.py#L185 and the kernelspec metadata is added by quarto convert.

[malcolmbarrett]

Yes, it seems unavoidable without modifying the upstream processes. Thanks for the info!

[hguturu]

@MarcoGorelli PR should be ready for another round of workflow runs by the testsuite. Logic has also been simplified.

Thanks for running the suite. It looks my test case was not robust to changing jupytext version. I have now fixed that.

[MarcoGorelli]

Sure thanks - if you want to make a separate pr which just fixes a typo or something then after that ci will run without needing approval

[hguturu]

Sure thanks - if you want to make a separate pr which just fixes a typo or something then after that ci will run without needing approval

Fortunately, I think we are all passing now? Ready to merge?

The current failure with docs seems to be related to doc-links which is a false positive? Both of the "broken" links seem to be working?

[MarcoGorelli]

thanks @hguturu !

i don't use nbqa anymore (haven't in years), so I'd just like to please ask:

• @malcolmbarrett could you take a look at the changes please, and approve if they look good to you?
• @hguturu if some issue is reported, will you be available to fix it?

if so, happy to ship it and make a new release

[malcolmbarrett]

The code lgtm.

As for maintenance of the feature, I do think the best outcome is native support for .qmd files in ruff, not a workaround like this or others I've taken. I don't know if the ruff maintainers will do that or accept a contribution along those lines, but I don't think any approach I've taken has an acceptable impact on the document itself. My question about the metadata earlier is an example. I think a Python formatter/linter should just touch Python code. It seems inevitable here that that will not be the case, in part because of the use of quarto convert, which I doubt will ever be able to make that guarantee even if other upstream issues were addressed.

That's my 2 cents as to whether it's worth maintaining. I have a limited use case (applying ruff to Quarto files), so maybe other users will find this useful for applying other tools to Quarto files.

edit: I am still interested in using it in the short term while I wait out a hopefully native implementation in ruff, but I also have another temporary workaround (convert to Jupyter, apply ruff, convert back to Quarto) that works fine but with the same limitations

[hguturu]

Yes, I will be available to fix issues if the are reported.

I agree with Malcolm on having the other tools support qmd files natively would be ideal, but this will be a good workaround until that happens. I figure ruff will eventually add it since they have this issue open (and the issues for broader jupytext and markdown code block support). Additionally, it is nice for other tools which may never add support.

I also believe the concerns about metadata changes already effect other markdown formats supported? Since myst would also go through jupytext. Also since quarto recognizes jupytext as a valid way to run quarto notebooks (https://quarto.org/docs/tools/jupyter-lab.html#jupytext), I figure they will work together to ensure the compatibility doesn't break.