add Elastic Stack and Elastic Package version to test results report

[teresaromero]

Resolves #991

Elastic stack version and elastic-package running version are now logged into the running test output when elastic-stack test is run.

[teresaromero]

@efd6 should this information be across all the tests? (static, asset, pipeline, etc...)

Let me know if the elastic-package version format is ok or you have any other thought

[efd6]

AFIUI the version is a constant over the table, correct? If that's the case, the additional verbosity is not a win.

In the output of benchstat for example, the constant data is provide as a preamble to the table.

$ benchstat new.bench old.bench 
goos: darwin
goarch: amd64
pkg: github.com/elastic/go-libaudit/v2/aucoalesce
cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
                    │  new.bench  │             old.bench              │
                    │   sec/op    │   sec/op     vs base               │
CoalesceMessages-16   4.191µ ± 3%   4.594µ ± 4%  +9.62% (p=0.000 n=10)

                    │  new.bench   │              old.bench              │
                    │     B/op     │     B/op      vs base               │
CoalesceMessages-16   2.753Ki ± 0%   2.786Ki ± 0%  +1.21% (p=0.000 n=10)

                    │ new.bench  │             old.bench             │
                    │ allocs/op  │ allocs/op   vs base               │
CoalesceMessages-16   13.00 ± 8%   14.00 ± 0%  +7.69% (p=0.003 n=10)

In terms of other thought, I'd love it if the tool emitted JSON instead of a table (or at least optionally instead of), since then any format can by chosen by the user after processing the machine-readable format.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: c0d5ac6

History

• 💚 Build #6572 succeeded c580a91

[teresaromero]

AFIUI the version is a constant over the table, correct? If that's the case, the additional verbosity is not a win.

As you said, its not a result perse, but a constant defining the test. I've added as a command log when running each test. To be part of the table/xml i guess it should be part of the result and redundant to each test case.

In terms of other thought, I'd love it if the tool emitted JSON instead of a table (or at least optionally instead of), since then any format can by chosen by the user after processing the machine-readable format.

i've added the option to make a json file/output with the test results

[jsoriano]

As this is printed by each test action, this is printed multiple times, once for each type of test, when running elastic-package test without specifying a subcommand. Is this intended?

[teresaromero]

it is intentional, as the version from the stack is taken from the clients / profile.

would it make sense to create a esClient right at the begining of the test cmd to get the version?

[jsoriano]

would it make sense to create a esClient right at the begining of the test cmd to get the version?

I don't think so, as not all tests need it. Let's leave it here by now.

[jsoriano]

I think it would be better to move this to a separate PR. Or was this report intended to include the version too? 🤔

[teresaromero]

Initially i thought this data should appear at the report (table, xml) but by the feedback i got i understood it needed to be printed as a log. The propsal for json y guess is independent as it still reflects the results of the test and not the "environment" (versions) that have been used for them.

eitherway, i am extracting the commit to a new branch and reverting from this PR

@efd6 could you describe with more detail what is the request for this changes? should the version still be part of the results or just be logged into the test output? should the new json option reflect the versions too? thanks

[teresaromero]

opening #3101 to track json format changes