Nice way to handle env variable?

[Prizrako]

Some methods to set environment varialbes for single command, instead of this:

os.Setenv("XB_AUTHOR_DATE", dateStr)
os.Setenv("XB_COMMITTER_DATE", dateStr)

Maybe in old shell way:

XB_AUTHOR_DATE="2021-04-05" XB_COMMITTER_DATE="2021-04-05" xb_push xxxx

[No-one-important]

would
script.Env("variable", "value").Exec("command")
be a suitable solution

[bitfield]

I'm not seeing where this adds much value over, for example:

script.Exec("sh -c 'export MSG=hello; echo $MSG'").Stdout()

[No-one-important]

on windows that syntax is not possible

[bitfield]

That's a fair point, but what about os.Setenv, for example?

[No-one-important]

It works but I think it would cleaner to have it inline and builtin to the module.

[bitfield]

Useful abstractions, as John Ousterhout says, should be deep: they should conceal lots of functionality behind a simple API. Is there any way we could do more than simply call os.Setenv? Some example programs that need to set their environment would be useful. What about reading an envfile, for example?

[josegonzalez]

I think it might be nice to not use os.Setenv() and instead only set them for a Pipeline - maybe something like a wrapper around script.Exec("sh -c 'export KEY=value; true'"), but for potentially multiple key/value pairs? In this way, we scope environment variables to just the pipeline rather than the entire golang process that is invoking/utilizing the pipeline.

Note that the wrapper would probably need to properly escape the environment variables, so it wouldn't be as simple as a fmt.Sprintf() call. Maybe thats enough deepness for it to be implemented?

[bitfield]

That sounds good, @josegonzalez—can you write a short piece of example code to show what you mean?

[josegonzalez]

Maybe something like the following:

environ := map[string]string{
  "HASURA_GRAPHQL_JWT_SECRET": "{\"type\": \"HS256\", \"key\": \"256BITKEYHERE$\"}"
}
contents, err := script.Env(env).Exec("env")

[bitfield]

Okay, but this looks pretty similar to:

os.Setenv("HASURA_GRAPHQL_JWT_SECRET", `{"type": "HS256", "key": "256BITKEYHERE$"}`)
contents, err := script.Exec("env")

Can you describe what problem you think an Env method on the pipe would be solving here?

[No-one-important]

os.Setenv would be global to the entire script this would allow it be local to the pipe

[bitfield]

Okay, but why is that important? Can you think of a situation where you'd want to set some environment variable for a command run by a pipe, but not for the program as a whole?

[No-one-important]

I'm not sure of the usecases but it just allows the same flexibility that the bash syntax allows but for all OSs

[bitfield]

Well, if there are no use cases for something, that's just another way of saying we don't need it, isn't it? There are lots of feature ideas that aren't bad in themselves—they're great features! It's just that we don't need them, so we don't add them.

[bitfield]

Closing pending real use cases.

[lambrospetrou]

@bitfield I was actually looking at this today for a use-case I have... As part of the Skybear.NET platform, I am running a few subprocesses per request/job. I want each of those command Execs to have their own environment, since I pass account specific things and I don't want to put them on the global process level environment.

Having a way to pass environment variables to modify the cmd.Env of the command created byNewPipe().XYZ().Exec("my command") would be awesome.

Right now, I am probably going to revert to using the os package since I cannot do it with script, which is sad since I love the UX of this package.

I could do the method above to put everything into the command itself with exports, but that's going to be cumbersome and error-prone to maintain, especially for "unsetting variables".

[bitfield]

@lambrospetrou that's great! Could you comment with a sketch of the kind of program you'd like to write, showing a convenient way to set the required environment variables? I don't yet have a good idea of the right API for this.

[lambrospetrou]

Without spending ages thinking about it, my first intuition would be something like below.

Examples

I am using the Go definition of an environment as []string but your version can adapt accordingly if you don't like that (e.g. using a map).

Entirely new environment

Similar to the WithStdout/WithStderr helpers.

customEnv := []string{"PATH=/something", "VAR=something-else"}
script.NewPipe().WithEnv(customEnv).Exec("my long command")

Append to the default environment

By default, right now the command will inherit the parent process environment. So the WithExtraEnv helper will explicitly do that, by taking the os.Environ() and calling append(os.Environ(), extraEnv...) to add the extra provided environment variables.

extraEnv := []string{"VAR2=something2"}
script.NewPipe().WithExtraEnv(extraEnv).Exec("my long command")

Unset environment variables

This is still tricky, but it's hard to automagically do by nature. With the above two options though someone can easily either provide the complete environment they want, hence they can just generate the environment they need (even taking os.Environ() and manipulating it), or overwrite existing variables by passing empty values since the last value wins (see example https://go.dev/play/p/inKF-5GswlK ).

References

• https://pkg.go.dev/os#Environ
• https://pkg.go.dev/os/[email protected]#Command

[bitfield]

Rather than have two methods, I'd prefer to have one that behaves the same way as setting cmd.Env: its argument completely replaces the current environment.

That way, if you want to control the whole environment:

script.NewPipe().
  WithEnvironment([]string{"FOO=bar"}).
  Exec("myprog")

That solves the problem of how to unset variables. On the other hand, to add extra variables to the current environment, you can use append as in the stdlib example.

script.NewPipe().
  WithEnvironment(append(os.Environ(), "FOO=bar")).
  Exec("myprog")

Would this API work for your use case?

[lambrospetrou]

Yeap! This can work.

[mahadzaryab1]

@bitfield are you looking for someone to work on this? i've never contributed to an open source project and this issue seems like a fun one to work on so I'd love to contribute here if possible.

[bitfield]

@mahadzaryab1 sure, all contributions are very welcome!

[josegonzalez]

Ah I have a use-case, sorry for not posting.

Basically I have a web process that can - in a single api call - spawn a bunch of processes serially with different environment variables. I'd like to not leak the current process state/have to track which variables to unset and reset at a later date.

[mahadzaryab1]

thanks @bitfield! i'll start working on this right away

[mahadzaryab1]

hi @bitfield! i've got a PR open at #208 and would appreciate any feedback you have