Notebook cell execution

[davidbrochart]

Problem

Today at the Jupyter Server meeting we talked about notebook restoration, where RTC could be a solution (see jupyter-server/jupyter_server#900). In that solution, the frontend just displays live changes to the notebook shared model. But currently, a notebook cell execution is either None or an integer greater than 0, so there is no way to show that a cell is executing.

Proposed Solution

We could encode the "cell executing" state in the execution count, for instance as "*". I think it could even be part of nbformat, because it brings some information about the state of the notebook at the time it was saved.

[fcollonval]

Taking the following scenario with the above proposal:

• User A open a notebook
• User A runs all cells among which one takes a long time
• While the execution is processing the long running cell, user A saves the document
• User A needs to shutdown the application (included server and kernel) for some external reasons before the long running cell finishes.
• Two hours later user A reopens the notebook
• The long running cell display a * that does not mean I'm running

So for me a state information should not be part of the document.

There are a couple of other scenarii I can think of that will make the * storage confusing; e.g. sharing a notebook file with a cell in that state with coworkers, open a read-only notebook with such state,... .

[echarles]

So for me a state information should not be part of the document.

After reading twice @fcollonval scenario, I tend to agree that the state information should not be part of the document.

This drives I think to a more general discussion about "what is a document?, what is a notebook?, what is a kernel?, what is a server? what is a state?" in RTC land.

[davidbrochart]

• The long running cell display a * that does not mean I'm running

No, it means I was not finished when the notebook was saved, and it's aligned with the cell outputs. For instance, imagine a cell that prints intermediary results and a final result. The saved notebook won't show the final result, which is reflected in the execution state as not finished.

So for me a state information should not be part of the document.

It is already, implicitly through the outputs. IMO the execution state just makes it more explicit.

[echarles]

No, it means I was not finished when the notebook was saved,

This is a developer perspective. How can user eyes go from * to not finished when the notebook was saved?

It is already, implicitly through the outputs. IMO the execution state just makes it more explicit.

You are taking for granted that the current implementation (the outputs are part of the CRDT) is the right approach, which I am not convinced.

[davidbrochart]

How can user eyes go from * to not finished when the notebook was saved?

Let me reformulate to just not finished executing, which is the current situation.

You are taking for granted that the current implementation (the outputs are part of the CRDT) is the right approach, which I am not convinced.

No, I am just taking for granted that outputs are part of the notebook format, CRDT or not.

[fcollonval]

Let me reformulate to just not finished executing, which is the current situation.

No the current situation is more specific than that; we displayed a * are:

• cell is currently executed in the kernel
• cell is queued for execution

So there are always an assumption it is a transient information linked to some live interaction with the kernel.

---

You are raising a good point about the outputs that could be reflecting only a partial execution. So I lean towards proposing adding an information in the document about a cell being partly executed that should be unrelated to the * used in the UI. That could be stored in the execution_count or else where.

[davidbrochart]

• cell is queued for execution

Although execution being queued is really kernel specific. For instance, akernel has the ability to run cells concurrently, so they might never be queued. If cells are queued, it's only because the kernel is executing blocking code.
So for me the * really means not finished executing, but we don't know if the cell started executing. Let's say it is scheduled for execution. And I think this * is the exact information that we want to store in the document.

[echarles]

No, I am just taking for granted that outputs are part of the notebook format, CRDT or not.

I stil think that nbformat and the CRDT shared models are different things. Similar but different, so governed by different rules and schemas.

... If cells are queued, it's only because the kernel is executing blocking code. So for me the * really means not finished executing, but we don't know if the cell started executing. Let's say it is scheduled for execution. And I think this * is the exact information that we want to store in the document.

You clearly define two different states, but foresee only one indicator for the user. This could be better, like eg..

• . = queued for execution
• * = being executed.

However, the static representation of the notebook in a ipynb file with nbformat should not be discussed at the same level as the runtime status of that same notebook. Hence my point above that the rules and schemas should be different.

[echarles]

I would even be tempted to say that the output of a cell not being fully executed should not be part of that saved version (the ipynb file) of the notebook, as showing incomplete and potentially giving wrong conclusions to the reader. I don't expect everyone to agree with that, but I expect this to be a discussion point.

[davidbrochart]

You clearly define two different states, but foresee only one indicator for the user.

My point is that there are not two different states, only one which is scheduled for execution. The * visual indicator corresponds to this state.

[davidbrochart]

I opened #197 which adds an execution_state field to a cell, that encodes the execution state as an "idle" or "busy" string.