Support callback dependencies

[mdekstrand]

Supporting 'callback' dependencies and outputs (for lack of a better term) would enable a number of interesting possibilities for using DVC to control processes that store data outside of DVC's file access, without requiring an explosion of remote access possibilities.

This would generalize what is possible with HTTP outputs and dependencies.

An example of what this could look like in a DVC file:

deps:
- cmd: python step-status.py --my-step
  md5: <checksum>

Instead of consulting a file, as with path:, DVC would run the specified command (which should be relatively quick), and compute the MD5 hash of its output. That command could do whatever it needs to in order to get the data status.

My specific use case is using DVC to control a large data import process that processes raw data files, loads them into PostgreSQL, and performs some additional computations involving intermediate files. I would implement a script that extracts data status from the PostgreSQL database so that DVC can check whether a step is up-to-date with the actual data currently in the database. I could implement this with HTTP dependencies and something like PostgREST, but that would introduce additional infrastructure requirements for people using the code (namely, a running PostgREST server).

[mdekstrand]

This is probably one particular instantiation of #1577.

[efiop]

Hi @mdekstrand ! We currently have a so called "callback stages" that are ran every time. With those, you could do something like

dvc run -o marker python step-status.py --my-step
dvc run -d marker -d ... -o ... mycommand

this way, dvc would always run marker.dvc and if it changes the marker file, it will trigger reproductions down the pipeline. Would something like this work for you?

[mdekstrand]

Almost, but not quite. The problem is forcing ordering between generating the marker file, and whatever step came before to populate the database with the content that step-status checks. As soon as I make the callback stage depend on another, to force ordering, it ceases to be a callback stage. For example:

dvc run -o step1-out.txt -d ... python step1.py
dvc run -o step1-check.txt -d step1-out.txt python step-status.py --my-step
dvc run -d step1-check.txt -d step1

Without this, running a repro might run the step-status command before the data it requires is ready.

[efiop]

@mdekstrand Thank you for the explanation! Indeed, our current callback stage is not very useful in that scenario. And how about an explicit flag smth like always_reproduce: true, that would make arbitrary stage always run on repro. Would that be suitable in your scenario? I'm asking about that because we actually have plans to introduce it #1407 instead of the current fragile no-deps assumption.

As to your idea with

deps:
- cmd: python step-status.py --my-step
  md5: <checksum>

it seems like we don't really need the md5 of the output, we could simply use the return code of that command as an indicator. E.g. <0 - error, 0 - didn't change, >0 - changed.

[mdekstrand]

The always_reproduce: true solution would make the desired functionality possible, I believe, but the dependency graph would be cluttered with status steps.

I don't think I like the return code option, though, because then the status script must know how to compare current state against previous state. If DVC checksums the scripts output, then all the script needs to be able to do is emit a stable summary or description of current state, and DVC's existing logic can take care of determining if that represents a change or not.

[efiop]

@mdekstrand great point! In that case, what if we make the command return the checksum itself through stdout instead of us computing md5 of its output? That has the potential of being used not only for dependencies but also for outputs, as a cheap alternative-checksum plugin. There are a lot of things to consider with it though.

[shcheklein]

Just a temporary workaround that comes to my mind. To make an intermediate stage effectively a "callback" stage, we can make it depend (along with other things, like DB upload pipeline) on an artificial callback stage that for example just dumps the current timestamp.

We can even reuse this dummy callback stage everywhere to make any number of stages always reproducible.

I love the idea to have cmd dependencies. It's simple to implement and solves a very good use case in a neat way.

@dmpetrov @Suor @pared @MrOutis your thoughts?

[pared]

Seems like good feature request. I don't see too much problems with implementation at first sight. Probably some graceful exception handling will be required (when status check will be performed on non existing data). Also, I think that some kind of communiciation with user might be a good idea. Like Your executable dependency returned : X, do you want to proceed? In order to avoid situation where check returns some error and we assume it was desired output.

[Suor]

I am in favor of cmd dependencies, looks generic and could be used as simple as

cmd: psql -U user -c 'select id from some_table order by id desc limit 1'

and many other alike.

We need to come up with command line interface though. How should this look in dvc run?

[Suor]

How about dvc run --dep-cmd="psql -U ..." ...?