Write standard logs to stdout and JSON logs to a file

[apgrucza]

I have a job that runs AWS Nuke weekly. It outputs logs in the standard format, which is good for reviewing the results because it is human-readable. But sometimes I want to do some analysis on the results which is much easier when the logs are in JSON format.

I'd like to keep the stdout logs as-is but be able to also specify a file to write logs to (either in JSON format, or in a configurable format).

[ekristen]

@apgrucza interesting idea, I'm not sure the current logging system can handle that, I do have some tentative plans to overhaul it but not anytime soon.

How are you thinking this would work? We'd need a way to state output to stdout and output to a log file or is stdout always a thing and you can just specify an additional output and format?

[mbarneyjr]

We have a similar process that runs AWS Nuke regularly, and would like to perform queries against our logs to see what actually happened in a given run, which is easy with JSON-structured logging. Expanding on the suggestion, I think supporting either standard/JSON logs to both stdout/a file would be best. Perhaps the config could support a list of log destinations, which could be stdout, a file, or something else, with a configuration property for log format

Also, we run this process in a central account that assumes a role into other accounts. The ability to add fields to the logs that get emitted would help us add things like account, or region to the each log line would allow us to filter/query/aggregate our logs across any dimension we need

[apgrucza]

If using logrus, the readme says you can use hooks to log to multiple places simultaneously.

I'm going to assume the only destinations we need to support are stdout and file. If someone wants logs written to stderr then they can just redirect stdout to stderr. If someone doesn't want stdout logs at all then they can redirect them to /dev/null.

I'm not sure if we would want to allow different log options (log-level, log-caller, log-disable-color, log-full-timestamp) per destination, or if it's even possible with log hooks. If not, then it would be enough to just add these new CLI options:

• --log-file-path
• --log-file-format (should default to json)

But if we want to allow different log options per destination then I think it would get too messy configuring this via CLI options. Using the config file would be better. But I also would like to retain the ability to set the log level via the AWS_NUKE_LOG_LEVEL environment variable, so I would not want the logging environment variables (and the corresponding --log CLI options) to be ignored either. In this case, the existing CLI options and environment variables should continue to work for stdout, and file logging is configured in the config file:

log_file:
  path: aws-nuke-log.json
  format: json
  level: trace
  caller: true
  full-timestamp: true

If you forsee the need to support log destinations other than these, then the above config could be changed to a YAML array instead.

[ekristen]

Thanks for the feedback. I will try and take a look at this when I can.