Doom's performance optimizations break `org-drill` (in general: certain usages of `org-map-entries`)

[RBckmnn]

I confirm that...

• I have searched the issue tracker, documentation, FAQ, Discourse, and Google, in case this issue has already been reported/resolved.
• I have read "How to Debug Issues", and will use it to provide as much information about this issue as possible.
• The issue can be reproduced on the latest available commit of Doom.
• The issue can be reproduced on a stable release of Emacs, such as 27, 28, or 29. (Unstable versions end in .50, .60, or .9x)

Expected behavior

I ran into this issue while using the org-drill package (but it seems to be a general problem). The package allows for spaced repetition practice of flash cards. A flash card corresponds to a heading in an org file. During a drill session a part of the content of such a heading is shown to the user and the user then has to indicate how well he remembered the part not displayed.

Current behavior

I have noticed that org-drill behaved weirdly for quite some time now. The most common problems are that in a practice session org-drill sometimes shows headings in a collapsed manner (the contents of the heading should be expanded) or it shows parts of the org file that do not correspond to a heading at all. Please see below for my attempts to resolve the issue.

Steps to reproduce

I have produced a minimal working example that exposes the underlying issue. To reproduce:

1. Download the added file, rename it to file1.org
2. Define the following function:

(defun foo ()
  (interactive)
  (org-map-entries
   (lambda ()
     (sit-for 0.5)
     (org-id-get-create))
   nil '("./file1.org")))

3. Open the file downloaded in 1. Make sure that both headlines in the file are collapsed and point is positioned at the beginning of the file. Then invoke the foo function.

The function should add an id property to every heading in the file but it does so only for the second heading in the file.
The root cause of this seems to be the +org--restart-mode-h function which runs because org-map-entries calls org-agenda-prepare-buffers in its implementations which kicks off Doom's performance optimizations for org. In fact, turning +org--restart-mode-h into a no-op seems to solve all the issues with org-drill.
file1.txt

System Information

https://pastebin.com/VavSwE7T

[hlissner]

Does this fix the issue for you?

;;; add to $DOOMDIR/config.el
(defadvice! fixed-+org--optimize-backgrounded-agenda-buffers-a (fn file)
  :override #'+org--optimize-backgrounded-agenda-buffers-a
  (if-let (buf (org-find-base-buffer-visiting file))
      buf
    (funcall fn file)))

I realize the optimization isn't really needed for org buffers that are already open. If this advice works, It'll inform a proper solution for the module.

EDIT: Fixed snippet.

[RBckmnn]

Hello Henrik,

thanks for you speedy reply. The advice you proposed caused an error. I assume you meant

;; add to $DOOMDIR/config.el
(defadvice! fixed-+org--optimize-backgrounded-agenda-buffers-a (fn1 fn2 file)
  :around #'+org--optimize-backgrounded-agenda-buffers-a
  (if-let (buf (org-find-base-buffer-visiting file))
      buf
    (funcall fn1 fn2 file)))

It seems that this fixes my minimal example. However, org-drill still seems to be broken, probably because it supports drilling from any number of org files (which are not necessarily opened yet).

(see the definition of org-drill-scope here and its usage in org-drill-map-entries here )

[hlissner]

This actually doesn't seem related to +org--optimize-backgrounded-agenda-buffers-a, but more on where the cursor is in already-open agenda buffers. I.e. If the cursor is on the second heading, org-map-entries only operates on that heading (and any after it), but if the cursor is at the beginning of the buffer (or the buffer hasn't been visited yet), it operates on all headings correctly. This occurs whether or not the +org--optimize-backgrounded-agenda-buffers-a advice exists.

I suspect saveplace (or maybe a bug in org-map-entries), but I'm looking more into it.