Using Consul backend yields invalid memory address or nil pointer dereference

[harmw]

Observing the following:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x402171]

goroutine 1 [running]:
panic(0xab3060, 0xc8200120b0)
    /usr/local/opt/go/libexec/src/runtime/panic.go:481 +0x3e6
main.readFromConsul(0xc8200e5220, 0x18, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/dragoslav/workspace/go/src/github.com/magneticio/vamp-workflow-agent/kv.go:72 +0x4a1
main.readFromKeyValueStore(0xc8200e5220, 0x18, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/dragoslav/workspace/go/src/github.com/magneticio/vamp-workflow-agent/kv.go:17 +0x182
main.main()
    /Users/dragoslav/workspace/go/src/github.com/magneticio/vamp-workflow-agent/main.go:67 +0x712

The Marathon JSON:

{
    "id": "vamp/vamp-workflow-agent",
    "instances": 1,
    "cpus": 0.1,
    "mem": 300,
    "labels": {"team":"purple"},
    "args": ["-storeType=consul", "-rootPath=/vamp/workflows", "-storeConnection=consul.blaze:8500"],
    "container": {
        "type": "DOCKER",
        "docker": {
            "portMappings": [{
                "hostPort": 0,
                "containerPort": 5000,
                "protocol": "tcp"
            }],
            "image": "magneticio/vamp-workflow-agent:0.9.0",
            "network": "BRIDGE",
            "forcePullImage": true,
            "parameters": [{
                "key": "dns",
                "value": "172.17.42.1"
            },{
                "key": "log-driver",
                "value": "fluentd"
            },{
                "key": "log-opt",
                "value": "tag=docker.vamp-workflow-agent"
            }]
        }
    },
    "env": {
        "SERVICE_5000_TAGS": "infrastructure,team-purple"
    }
}

I haven't taken a look at the Go code, so just sharing the observation for now 😃

[harmw]

Tried to run the agent manually using the following:

docker run -it -e VERSION=v6.2.2 -e VAMP_WORKFLOW_VERSION=0.9.0 -e VAMP_KEY_VALUE_STORE_ROOT_PATH=/vamp/workflows/metrics -e VAMP_KEY_VALUE_STORE_TYPE=consul -e VAMP_KEY_VALUE_ROOT_PATH=/vamp/workflows -e VAMP_KEY_VALUE_STORE_CONNECTION=consul.blaze:8500 -e VAMP_URL=http://vamp.blaze --rm --entrypoint /bin/sh  magneticio/vamp-workflow-agent:0.9.0

Breaks:

/ # /usr/local/vamp/vamp-workflow-agent
12:21:00.235 main NOTI ==>
██╗   ██╗ █████╗ ███╗   ███╗██████╗
██║   ██║██╔══██╗████╗ ████║██╔══██╗
██║   ██║███████║██╔████╔██║██████╔╝
╚██╗ ██╔╝██╔══██║██║╚██╔╝██║██╔═══╝
 ╚████╔╝ ██║  ██║██║ ╚═╝ ██║██║
  ╚═══╝  ╚═╝  ╚═╝╚═╝     ╚═╝╚═╝
                       workflow agent
                       version 0.9.0
                       by magnetic.io

12:21:00.235 main NOTI ==> Starting Vamp Workflow Agent
12:21:00.235 main INFO ==> Key-value store type          : consul
12:21:00.235 main INFO ==> Key-value store connection    : consul.blaze:8500
12:21:00.235 main INFO ==> Key-value store root key path : /vamp/workflows/metrics
12:21:00.235 main INFO ==> Workflow file path            : /usr/local/vamp
12:21:00.235 main INFO ==> Reading workflow from         : /vamp/workflows/metrics/workflow
12:21:00.235 main INFO ==> Initializing Consul connection: consul.blaze:8500
12:21:00.235 main INFO ==> Consul getting '/vamp/workflows/metrics/workflow' key value.
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x402171]

goroutine 1 [running]:
panic(0xab3060, 0xc82000a0a0)
    /usr/local/opt/go/libexec/src/runtime/panic.go:481 +0x3e6
main.readFromConsul(0xc82012a600, 0x20, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/dragoslav/workspace/go/src/github.com/magneticio/vamp-workflow-agent/kv.go:72 +0x4a1
main.readFromKeyValueStore(0xc82012a600, 0x20, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/dragoslav/workspace/go/src/github.com/magneticio/vamp-workflow-agent/kv.go:17 +0x182
main.main()
    /Users/dragoslav/workspace/go/src/github.com/magneticio/vamp-workflow-agent/main.go:67 +0x712

[harmw]

The auto-deployed workflow agents do work however...

[iamtew]

This looks familiar to magneticio/vamp#806.

Until Vamp has fully started all the necessary data in Consul won't be populated, and this will start failing. I guess the auto-deployed agents do work because they wait for Vamp to fully start and populate its settings in Consul, but in cases when we try to start the agents before it's been fully initialized we run in to this issue.

[harmw]

current scenario is deploying 0.9.2, this succeeds and it will deploy workflows. Both workflows fail however, with the above error. They will keep redeploying (powered through Marathon) throughout the day. Just deployed 0.9.2 with katana workflows, that launches just fine (though hits yet another error, though not the above).