Artillery on a Github Runner without NPM repository

[eliezercazares]

Hello,

I am working for a company, that, in order to not reveal compromising info, I can say simply that in general I have this scenario:

Artillery runs on Github Actions via a Workflow using a Runner with limited internet that our company has in order to reach specific applications behind VPNs.
So this runner basically has access mostly to internal apps, and some internet, but i.e. I can't do "npm install" on those runners.

I advised our devops team that we should be able and they might look into a solution. They offered to come up with Artifactory later... but meanwhile... I wonder:

• How to install artillery offline... I read about npm pack, but the node_modules for Artillery have local paths of my local computer in them... if I pack them, they will fail because of the path won't they? Also, I am not sure if global packages can be installed like this.

I think Artillery needs to be installed globally which is also giving a headache to my small resources head. 😅

Version info: 2.0.0-27

Running this command:

npm install 

I expected to have all the dependencies there

but our Runner executing Artillery has no access to Node Repositories. .-.

While devops provides a solution like an artifactory,

I still would like to execute the tests to show my colleagues about Artillery.

files and stuff involved in this issue:
github workflow yaml
github runners
firewall blocking node repository (npm install rendered useless)
private ecosystem because we test internal apps
npm install won't work
copying node_modules won't work (Seems artillery has to be installed globally)

Thanks for your help!

Hopefully our Devops team will be able to put Artillery in an Artifactory if possible, and meanwhile I'll find a solution.

Thanks again!

[hassy]

Hi @eliezercazares 👋

Is using a custom Docker image in your GitHub runner an option? We publish official Docker images for each release of Artillery that contains all of the dependencies: https://hub.docker.com/r/artilleryio/artillery/tags

[eliezercazares]

Hi @eliezercazares 👋

Is using a custom Docker image in your GitHub runner an option? We publish official Docker images for each release of Artillery that contains all of the dependencies: https://hub.docker.com/r/artilleryio/artillery/tags

Hi Hassy! Super happy to read from you, I think it is because I've seen other Workflows using Docker Containers, I will share with my Devops team to try and get this on our Workflows. :D

Thanks a lot! I will keep posted on this topic, even though is not an issue, for future reference for others.

[eliezercazares]

Well, I am having some issues...

This is the workflow file:

name: Run Functional Tests

on:
    push:
        branches: [ main ]
    pull_request:
        branches: [ main ]

jobs:
    run-artillery-in-docker:
        runs-on: AnInternalRunner
        container:
            image: artilleryio/artillery:latest
            volumes:
                 - ${{ github.workspace }}:/usr/artillery/
        steps:
            - name: Checkout the repo
                uses: actions/checkout@v2
            - name: Run sample test in docker
                run: run /usr/artillery/scenarios/fls/happy_path.yml

This returns in Github Workflows:

Warning: /usr/artillery/scenarios/fls/happy_path.yml is not a artillery command.

I was not able to see my files, but dumb me didn't add the checkout action, I have updated this comment to reflect this.

the files are there inside docker container now, but I am still facing the same issue:

The command "run /usr/artillery/scenarios/fls/happy_path.yml" does not work with the mentioned error: "Warning: /usr/artillery/scenarios/fls/happy_path.yml is not a artillery command.".

I went through the documentation which has a single command :

docker run --rm -it -v ${PWD}:/scripts \
  artilleryio/artillery:latest \
  run /scripts/test_script.yaml

However I believe the workflow file translated it properly?

I tried the entire day and I was able to learn a bit and progress a bit, but now this.

I will keep researching on my side. It's probably a Layer 8 error.

[eliezercazares]

UPDATE:

This issue I was facing is a duplicate of:
#1394

Resolved in the same way. :D
Basically, explicitly refer the artillery bin, I guess using artillery so recursively messed up the command in the docker image, or I'm some dumb which is more likely.

[eliezercazares]

By the way @hassy now that I got it running, it seems there is an issue with the plugin json-include for artillery running in docker.
Module could not be found (MODULE_NOT_FOUND) warn for plugin json-include

I am almost certain this one it is a bug, because everything else is running now, I even tried a docker contained "npm install -g " for artillery json include.

[eliezercazares]

@hassy hi, I hope you are doing very very well! one quick question about "expect plugin"..

I know we have assertion for "equals", but do we have more, like "less than", "greater than", I ask because in my project I need to count the amount of JSON elements removed from a request in the response. Sometimes should be equal, somestimes less, sometimes more.

I am using the processor file to create boolean variables to compare in the YAML using "expect plugin" equals... but it would be so nice to make it more legible, readable for everybody.

[eliezercazares]

I am now aware of the class Expectations.js in Expect-Plugin, hopefully I can contribute with something to the framework.

In my environment, we mostly work with json body in the requests, and we count products in request versus products our API can provide of that list.

so being able to compare bigger than, smaller than, equal, not equal, it's important.

the only problem I see right now is that the method I can base my contribution on takes an array.

I am thinking of perhaps verifying the array has only two values, and to indicate the comparison.

or perhaps a string representation that later I can split by the comparison operator and perform it... I will see...

[eliezercazares]

@hassy
I wrote this little function that should be able to verify if all elements in an array are different.

I am looking at how to enable it for usage, I so far, only have the method, which I would gladly hand over for addition. :D

I've put the following method in my global installation path of artillery, under home/node_modules/artillery/node_modules/artillery-plugin-expect/lib/expectations.js and it works :D
Basically it's inside the expect plugin in the node_modules of artillery global.

I would be sooo happy to contribute to Artillery. @hassy

function expectNotEqual(expectation, body, req, res, userContext) {
    debug('check not equal');
    debug('expectation:', expectation);
    debug('body:', typeof body);

    let result = {
        ok: false,
        expected: 'all values must be different',
        type: 'notEqual'
    };

    const values = _.map(expectation.notEqual, (str) => {
        return String(template(String(str), userContext));
    });

    const unique = _.uniq(values);
    result.ok = values.length === unique.length;
    result.got = `${ values.join(', ' )}`;

    return result;
}

Oh and of course it has to be added in module.exports in expectations.js, but what do I tell ya! you're the author :D

[eliezercazares]

In the end I was able to resolve all the issues, and I would like to share some more Assertions I have created.

• Not equal
• GreaterThan
• EqualOrGreaterThan
• LesserThan
• EqualOrLesserThan

Which can use variables from i.e.

post:
    url: ""
    capture: 
        - json: "$.products.length"
          as: productsInResponse
    expect:
        - greaterThan:
            - "{{ productsInRequest }}"
            - "{{ productsInResponse }}"
        - notEqual:
            - "{{ productsInRequest }}"
            - "{{ productsInResponse }}"
            - "{{ productsRemoved }}"

If in Greater Than, Lesser Than comparisons more than 2 values are provided, the assertion fails telling you can only compare two values at once.

I would love to send this for review! @hassy

[hassy]

@eliezercazares sorry for the delay! Those assertions sound very useful but I wonder if we can come up with a more generic ways of doing comparisons 🤔

[eliezercazares]

@eliezercazares sorry for the delay! Those assertions sound very useful but I wonder if we can come up with a more generic ways of doing comparisons 🤔

If I can help with that let me know! I think it would pair perfectly with a new ability, to capture from req.json just like we can capture within the yaml variables from the response.

Because in the end, afterResponse i.e. we will have enough variables to compare req vs response.

[bernardobridge]

I wonder if an interface like this:

expect:
   - isTrue: "{{productsInRequest}} == {{productsInResponse}}"

or

expect:
   - isTrue:
       expression: "=="
       value: "{{productsInRequest}}"
       test: "{{productsInResponse}}"

might be a good idea. It's generic enough to encompass all the cases above. I believe we already do something similar in ensure with the conditions there (using filtrex), so it would be consistent with that too, and that could be used for idea on how to implement.