Detached processes in deno

[manikawnth]

Currently node has the capability to run a child process in detached mode (setsid()), which is lacking in deno (?)

It seems the Tokio Command doesn't expose any specific functionality for this.
Is it possible for Deno to provide this functionality?

Also, this specific default (kill if handle is dropped from the scope) in rust std and tokio is overridden in Deno (cli/ops/process.rs). Is that related to detached mode?

fn op_run(...) {
     ....
    // We want to kill child when it's closed
    c.kill_on_drop(true);
}

Is there a better place to ask such questions than cluttering the issues section?

[ry]

Correct, Deno does not have this functionality yet. I would be happy to add it.

Let's figure out what the JS API should look like. We try to not randomly invent new APIs, but instead look at how Node, Go, Rust, Python have exposed it and see if we can use the most common nomenclatures. Would you mind researching this? This would help move the feature forward.

Is there a better place to ask such questions than cluttering the issues section?

This is the right place for a feature request.

[manikawnth]

Firstly Deno has done the right thing by not making detached as default, 'coz that's the default nature of both win API and posix API

C has the cleanest way of not having anything in language and having OS specific APIs

Node has flat argument structure. while detached and shell are applicable to both OSs, uid and gid of unix are generally exposed which throws a ENOTSUP (not supported) error on non-supported platforms like windows

go has syscall.StartProcess() which has separate SysProcAttr structs for each OS. It took the correct route of freezing syscall package and delegating it to sys/ specific package. But it has a real low-level interface for spawning, like doing Syscall(SYS_FORK..) then do Setsid() and then Exec(...) which is right way. Windows has a fairly easy parameterized CreateProcess(...)

Python's subprocess is intended to replace old os.spawn() and they've similar function signature subprocess.run(...) as Deno. The Popen(...) constructor has start_new_session which is similar to setsid, but it's applicable only for Linux. Windows users have to create a separate STARTUPINFO object for all process creation parameters

Rust took the c route of not having anything directly interfaced but left to users via OS specific extension traits (impl CommandExt for Command)
All langugages have moved/moving towards either OS specific interfaces or configurations,

Largely I can think of two options:

1. Like Node, keep a flat option structure and throw "Not supported" error as the developer should really know what he's doing. Or
2. Pass the OS specific config similar to Python, e.g. like a Typescript either/or type and all the OS specific variables to be interfaced under Deno namespace. This way we can do clean check across using Deno.build.os. Pseudo-syntax:
   type ProcessOpts = Deno.WinOpts.process | Deno.UnixOpts.process

[buckle2000]

How useful is this feature? You can always spawn a separate process by running the script again with different arguments. It would be hard to design security around this.

[manikawnth]

Detachment is just one aspect of it. The sub-process creation also has setting uid, gids to the process, which is needed, if we're looking deno as potential replacement for bash and python.

And moreover underlying rust is handling it cleanly, so it's a matter of exposing it with a neat interface.

[jcc10]

I would like to comment that on linux children can outlive the parent by default.

Here is the test I have been using:
https://gist.github.com/jcc10/3543f9bba8275738cac7cbd417010884

[guest271314]

How useful is this feature?

With a Bash Native Messaging host I can do something like

local_server() {
  if pgrep -f './deno run --allow-net --allow-read --allow-run --unstable deno_server.js' > /dev/null; then
    pkill -f './deno run --allow-net --allow-read --allow-run --unstable deno_server.js' & send_message '"Local server off."' 
  else
    ./deno run --allow-net --allow-read --allow-run --unstable deno_server.js & send_message '"Local server on."'
  fi
}
local_server

in the browser so the Native Messaging host does not have to remain open we use runtime.sendNativeMessage which exits when the message is returned to the client instead of runtime.connectNative for a long-lived connnection

 chrome.runtime.sendNativeMessage('native_messaging_espeakng'
  , {}, (nativeMessage) => console.log({nativeMessage}))

When using a Deno Native Messaging host the Deno server exits when the parent Native Messaging host exists.

  let process = Deno.run({
        cmd: [`pgrep`, `-f`, `deno_server.js`],
        stdout: 'piped',
        stderr: 'piped',
      });

  let rawOutput = await process.output();
  process.close();
  if (rawOutput.length) {     
      process = Deno.run({
        cmd: [`pkill`, `-f`, `./deno_server.js`],
        stdout: 'piped',
        stderr: 'piped',
      });
      
      await process.status();
      process.close();
      message = '"Local server off."';
      
  } else {
      process = Deno.spawn({
        cmd: ['./deno_server.js'],
        stdout: 'piped',
        stderr: 'piped',
      });

      await Promise.race([null, process.status()]);
      message = '"Local server on."';     
  }

I was not expecting that behaviour. Scouring the Manual I was expecting to at least locate a way to keep the child process active if/when the parent process exits. Then I located several issues illuminating the fact this is not currently possible in Deno. Providing users with the option to keep child processes open sounds reasonable to me.

[sigmaSd]

@guest271314 if you use the new Command api, I think you can use ref() to keep the child alive
const p = (new Deno.Command()).spawn(); p.ref()

[guest271314]

Is that documented anywhere? Which method do the commands get passed to. Command() or spawn()?

[crowlKats]

https://deno.land/[email protected]?unstable=&s=Deno.Command

[guest271314]

I'll try that out. Thanks.

[guest271314]

@sigmaSd @crowlKats

This is not working as expected

    process = new Deno.Command('./deno_server.js', {
      stdout: "piped"
    });
    const child = process.spawn();
    child.ref();

the process still exits.

[bartlomieju]

What's the content of deno_server.js?

[guest271314]

What's the content of deno_server.js?

#!/usr/bin/env -S ./deno run --allow-net --allow-read --allow-run --unstable --v8-flags="--expose-gc,--jitless"
// https://deno.land/[email protected]/runtime/http_server_apis_low_level
// Start listening on port 8443 of localhost.
const server = Deno.listenTls({
  port: 8443,
  certFile: 'certificate.pem',
  keyFile: 'certificate.key',
  alpnProtocols: ['h2', 'http/1.1'],
});
// console.log(`HTTP webserver running.  Access it at:  https://localhost:8443/`);
// Connections to the server will be yielded up as an async iterable.
for await (const conn of server) {
  // In order to not be blocking, we need to handle each connection individually
  // without awaiting the function
  serveHttp(conn);
}

async function serveHttp(conn) {
  // This "upgrades" a network connection into an HTTP connection.
  const httpConn = Deno.serveHttp(conn);
  // Each request sent over the HTTP connection will be yielded as an async
  // iterator from the HTTP connection.
  for await (const requestEvent of httpConn) {
    // The native HTTP server uses the web standard `Request` and `Response`
    // objects.
    let body = null;
    if (requestEvent.request.method === 'POST') {
      let json = await requestEvent.request.json();
      const process = Deno.run({
        cmd: json,
        stdout: 'piped',
        stderr: 'piped',
      });
      body = await process.output();
    }
    // The requestEvent's `.respondWith()` method is how we send the response
    // back to the client.
    requestEvent.respondWith(
      new Response(body, {
        headers: {
          'Content-Type': 'application/octet-stream',
          'Cross-Origin-Opener-Policy': 'unsafe-none',
          'Cross-Origin-Embedder-Policy': 'unsafe-none',
          'Access-Control-Allow-Origin':
            'chrome-extension://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
          'Access-Control-Allow-Private-Network': 'true',
          'Access-Control-Allow-Headers':
            'Access-Control-Request-Private-Network',
          'Access-Control-Allow-Methods': 'OPTIONS,POST,GET,HEAD',
        },
      })
    );
  }
}

Note, that is called from a Native Messaging host

#!/usr/bin/env -S ./deno run --allow-run --unstable --v8-flags="--expose-gc,--jitless"
async function getMessage() {
  const header = new Uint32Array(1);
  await Deno.stdin.read(header);
  const output = new Uint8Array(header[0]);
  await Deno.stdin.read(output);
  return output;
}

async function sendMessage(_) {
  let message = '';
  let process = Deno.run({
    cmd: ['pgrep', '-f', './deno_server.js'],
    stdout: 'piped',
    stderr: 'piped',
  });
  let rawOutput = await process.output();
  process.close();
  if (rawOutput.length) {
    process = Deno.run({
      cmd: ['pkill', '-f', './deno_server.js'],
      stdout: 'null',
      stderr: 'null',
    });
    process.close();
    message = '"Local server off."';
  } else {
    process = Deno.run({
      cmd: ['./deno_server.js'],
      stdout: 'null',
      stderr: 'null',
    });
/*
    process = new Deno.Command('./deno_server.js', {
      stdout:'null'
    });
    const child = process.spawn();
    child.ref();
    await process.output();
*/
    message = '"Local server on."';
  }
  message = new TextEncoder().encode(message);
  const header = Uint32Array.from(
    {
      length: 4,
    },
    (_, index) => (message.length >> (index * 8)) & 0xff
  );
  const output = new Uint8Array(header.length + message.length);
  output.set(header, 0);
  output.set(message, 4);
  await Deno.stdout.write(output.buffer);
}

async function main() {
  while (true) {
    const message = await getMessage();
    await sendMessage(message);
    gc();
  }
}

try {
  main();
} catch (e) {
  Deno.exit();
}

Sending the message I need to use runtime.connectNative for the server to not exit when the Native Messaging host exits after sending the single message to client (the browser)

// Deno exits child process when parent process exits
// https://github.com/denoland/deno/issues/5501
// Use runtime.connectNative()
const handleMessage = async (nativeMessage) => {
  port.onMessage.removeListener(handleMessage);
  port.onDisconnect.addListener((e) => {
    console.log(e);
    if (chrome.runtime.lastError) {
      console.log(chrome.runtime.lastError);
    }
  });
  parent.postMessage(nativeMessage, name);
  // Wait 100ms for server process to start
  await new Promise((resolve) => setTimeout(resolve, 100));
  const controller = new AbortController();
  const { signal } = controller;
  parent.postMessage('Ready.', name);
  onmessage = async (e) => {
    if (e.data instanceof Array) {
      try {
        const { body } = await fetch('https://localhost:8443', {
          method: 'post',
          cache: 'no-store',
          credentials: 'omit',
          body: JSON.stringify(e.data),
          signal,
        });
        parent.postMessage(body, name, [body]);
      } catch (err) {
        parent.postMessage(err, name);
      }
    } else {
      if (e.data === 'Done writing input stream.') {
        port.onMessage.addListener((nativeMessage) => {
          parent.postMessage(nativeMessage, name);
          port.disconnect();
        });
        port.postMessage(null);
      }
      if (e.data === 'Abort.') {
        port.disconnect();
        controller.abort();
        close();
      }
    }
  };
};
const port = chrome.runtime.connectNative('native_messaging_espeakng');
port.onMessage.addListener(handleMessage);
port.postMessage(null);

Ideally instead of using a "long-lived" connection to the host from the client I just send a single message (https://developer.chrome.com/docs/extensions/reference/runtime/#method-sendNativeMessage) from the client to the host to turn the server on and off, as I do using Bash https://github.com/guest271314/native-messaging-espeak-ng/tree/master

onload = async () => {
  chrome.runtime.sendNativeMessage(
    'native_messaging_espeakng',
    {},
    async (nativeMessage) => {
      parent.postMessage(nativeMessage, name);
      await new Promise((resolve) => setTimeout(resolve, 100));
      const controller = new AbortController();
      const { signal } = controller;
      parent.postMessage('Ready.', name);
      onmessage = async (e) => {
        if (e.data instanceof ReadableStream) {
          try {
            const { value: file, done } = await e.data.getReader().read();
            const fd = new FormData();
            const stdin = await file.text();
            fd.append(file.name, stdin);
            const { body } = await fetch('http://localhost:8000', {
              method: 'post',
              cache: 'no-store',
              credentials: 'omit',
              body: fd,
              signal,
            });
            parent.postMessage(body, name, [body]);
          } catch (err) {
            parent.postMessage(err, name);
          }
        } else {
          if (e.data === 'Done writing input stream.') {
            chrome.runtime.sendNativeMessage(
              'native_messaging_espeakng',
              {},
              (nativeMessage) => {
                parent.postMessage(nativeMessage, name);
              }
            );
          }
          if (e.data === 'Abort.') {
            controller.abort();
          }
        }
      };
    }
  );
};

#!/bin/bash
# https://stackoverflow.com/a/24777120
send_message() {
  message="$1"
  # Calculate the byte size of the string.
  # NOTE: This assumes that byte length is identical to the string length!
  # Do not use multibyte (unicode) characters, escape them instead, e.g.
  # message='"Some unicode character:\u1234"'
  messagelen=${#message}
  # Convert to an integer in native byte order.
  # If you see an error message in Chrome's stdout with
  # "Native Messaging host tried sending a message that is ... bytes long.",
  # then just swap the order, i.e. messagelen1 <-> messagelen4 and
  # messagelen2 <-> messagelen3
  messagelen1=$(( ($messagelen      ) & 0xFF ))               
  messagelen2=$(( ($messagelen >>  8) & 0xFF ))               
  messagelen3=$(( ($messagelen >> 16) & 0xFF ))               
  messagelen4=$(( ($messagelen >> 24) & 0xFF ))               
  # Print the message byte length followed by the actual message.
  printf "$(printf '\\x%x\\x%x\\x%x\\x%x' \
        $messagelen1 $messagelpen2 $messagelen3 $messagelen4)%s" "$message"
}
local_server() {
  if pgrep -f 'php -S localhost:8000' > /dev/null; then
    pkill -f 'php -S localhost:8000' & send_message '"Local server off."' 
  else
    php -S localhost:8000 & send_message '"Local server on."'
  fi
}
local_server

[bartlomieju]

@guest271314 could you boil it down to something easier to reproduce?

[guest271314]

No, not really. You cannot really reproduce a Native Messaging connection using substitute means.

Once you create your first Native Messaging host and extension you'll get the hang of it. If you can build Deno you can install a Native Messaging host on Chromium or Chrome. Firefox is a little different. Here are the instructions for each branch. Compare the result for yourself. https://github.com/guest271314/native-messaging-espeak-ng/tree/master. I think you might even be able to substitute the Deno server for php -S localhost if you don't have PHP installed. https://github.com/guest271314/native-messaging-espeak-ng/tree/deno-server.

If you have any questions don't hesitate to ask.

[guest271314]

@bartlomieju

I don't even think you have to install or build espeak-ng to reproduce what happens in this case. Just use chrome.runtime.sendNativeMessage in nativeTransferableStreams.js from this https://github.com/guest271314/native-messaging-espeak-ng/blob/fed906b878ad69af337f1a3bccc10bee477f25e5/nativeTransferableStream.js to start the local Deno server - not the runtime.connectNative() that is currently in the deno-server branch. You will need to either generate your own certificate or just use Deno.listen(), I don;t think it matters as to the issue I described. Then run the current nativeTransferableStreams.js in the deno-server branch to compare. You can also compare to the Bash and PHP version and see the PHP server called rom Bash does not exit immediately when the single message from Native Messaging client is sent and the Native Messaging host exits.

FWIW This is a generic Deno Native Messaging host that just echos input beginning with an array of 200000 https://github.com/guest271314/native-messaging-deno.

[jokeyrhyme]

Try to reproduce this with a child process that just has a setInterval(...) in it or something simple and long-lived, we just need to see whether we can get a child process to out-live its parent process

We don't need to see communications between the processes to demonstrate whether .ref() works or not

[guest271314]

.ref() does not work.

This

process = new Deno.Command('./deno_server.js', {
  stdout:'null'
});
const child = process.spawn();
child.ref();
await process.output();

does not keep the process created by Deno.run() running when the calling script exists - both scripts exit.

I always fetch the latest deno executable

$ ./deno --version

deno 1.29.1 (release, x86_64-unknown-linux-gnu)
v8 10.9.194.5
typescript 4.9.4

[guest271314]

Two videos demonstrating what happens.

Both version I use the same .ref() code. The parent process is a deno process and the child process is a deno process.

"deno_subprocess_exits.webm" shows what happens when we open a process from a parent then the parent exists immediately afterwards.

deno_subprocess_exits.webm

"deno_subprocess_keep_parent_process_open.webm" shows what happens when we keep the parent process open until the child process, "deno_server.js", serves the response then the client closes the parent process.

deno_subprocess_keep_parent_process_open.webm

[bartlomieju]

.ref() does not work.

This

process = new Deno.Command('./deno_server.js', {
  stdout:'null'
});
const child = process.spawn();
child.ref();
await process.output();

does not keep the process created by Deno.run() running when the calling script exists - both scripts exit.

I always fetch the latest deno executable

$ ./deno --version

deno 1.29.1 (release, x86_64-unknown-linux-gnu)
v8 10.9.194.5
typescript 4.9.4

That is expected behavior - ref() is only meant to keep event loop alive if the subprocess is still running. You are looking for detached subprocesses, which are currently not implemented and hence why this issue is open.

[guest271314]

You are looking for detached subprocesses, which are currently not implemented and hence why this issue is open.

I had no clue about ref() until I found this issue.

I could could verify deno executable does not support detached subprocesses in my comparison to the Bash script which is effective the same as the Deno script algorithmically.

Thank you for just saying Deno doesn't support that yet.

[gaoxiaoliangz]

Any updates on this? It seems that using spawn from node:child_process with deatched set to true is not working either.

[jtoppine]

Not sure if it helps anyone, but curiously:

• if you create a subprocess without awaiting it, unref it, and let the main process naturally come to end (or press Ctrl+C), the subprocess will be killed along with the main process.

• if you create a subprocess without awaiting it, unref it, and then end the main process with Deno.exit(0), the subprocess stays alive after the main process

Adding explicit Deno.exit() actually worked as a nice workaround for my problem, where I did want to just start a child process on the background and leave it there, while the main program is short lived. Feels like a weird hack, though. (is it actually a bug in Deno.exit(), one would think a script should behave the same when programmatically exited vs naturally come to an end?)

I feel like this behaviour is a bit confusing, and it would do good to have all this child process lifetime stuff documented in the Deno.Command docs.

Being able to explicitly set child processes as detached or... not detached, would be a nice feature.

[mehlian]

Not sure if it helps anyone, but curiously:

* if you create a subprocess without awaiting it, unref it, and let the main process naturally come to end (or press Ctrl+C), the subprocess will be killed along with the main process.

* if you create a subprocess without awaiting it, unref it, **and then end the main process with Deno.exit(0)**, the subprocess stays alive after the main process

Adding explicit Deno.exit() actually worked as a nice workaround for my problem, where I did want to just start a child process on the background and leave it there, while the main program is short lived. Feels like a weird hack, though. (is it actually a bug in Deno.exit(), one would think a script should behave the same when programmatically exited vs naturally come to an end?)

I feel like this behaviour is a bit confusing, and it would do good to have all this child process lifetime stuff documented in the Deno.Command docs.

Being able to explicitly set child processes as detached or... not detached, would be a nice feature.

Minimum repro code for this:

function run_background_process(execPath: string, options: Deno.CommandOptions)
{
    options.stderr = "null";
    options.stdin = "null";
    options.stdout = "null";

    let proc = new Deno.Command(execPath, options);
    let child = proc.spawn();
    child.output();
    child.unref();

    Deno.exit(0);
}

However, this is a bug/hack as it does not allow multiple detached processes to run.
For example, this:

run_background_process(`explorer`, {});
run_background_process(`explorer`, {});

will start only one "File Explorer" process on Windows.

But if we remove Deno.exit(0); from run_background_process function and use it like this:

run_background_process(`explorer`, {});
run_background_process(`explorer`, {});
Deno.exit(0);

then we will fire multiple detached processes.

[Yohe-Am]

Another important usecase the lack of detached (setsid) blocks is sub-process tree cleanup through posix process groups. Invaluable when cleaning up after tests. Can't wait for 2.0