Running `fjord deploy` exits with promisify-child-process error

[iulspop]

Hello,

I'm trying to give Fjord a go and encoutered an error that's hard to debug from a user point of view.

I installed fjord_cli globally using npm. I ran fjord setup in an empty folder and it added some settings JSON file succesfully. I then ran fjord deploy and I got this error message:

✖ Failed to deploy fjord-shared-fjord_app resources. Exiting.
grep process exited with code 1
Error: Error: Process exited with code 1
    at ChildProcess.done (/home/joy/.nvm/versions/node/v16.10.0/lib/node_modules/fjord_cli/node_modules/promisify-child-process/index.cjs:70:19)
    at ChildProcess.emit (node:events:402:35)
    at maybeClose (node:internal/child_process:1064:16)
    at Socket.<anonymous> (node:internal/child_process:450:11)
    at Socket.emit (node:events:390:28)
    at Pipe.<anonymous> (node:net:672:12) {
  code: 1,
  signal: null,
  stdout: [Getter],
  stderr: [Getter]
}

It looks like the uncaught error occurs in the promisify-child-process package. In the source, in the deploy.js file under src/commands, the spawn function from promisify-child-process is used to run an cdk deploy command for each part of Fjord. One of those is causing this error.

Also, nothing is written to the cdk.out folder. From a quick look at the documentation for promisify-child-process, it's because "spawn and fork don't capture stdout and stderr unless you pass an encoding or maxBuffer option" and no such options are passed to the spawn function calls.

I replaced the empty options object passed to the spawn function calls with {encoding: 'utf8'} as recommended in the docs. This already gave a much clearer error message (though the logs aren't written to the cdk.out folder, not sure if it's meant for that):

✖ Failed to deploy fjord-shared-fjord_app resources. Exiting.
grep process exited with code 1
Error: Error: Process exited with code 1
    at ChildProcess.done (/home/joy/.nvm/versions/node/v16.10.0/lib/node_modules/fjord_cli/node_modules/promisify-child-process/index.cjs:70:19)
    at ChildProcess.emit (node:events:402:35)
    at maybeClose (node:internal/child_process:1064:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5) {
  code: 1,
  signal: null,
  stdout: '\n' +
    'Error: Fjord settings -- consumerGroups -- consumer group #1 is missing a valid KAFKA_TOPICS param (must be a string of space-separated topics)\n' +
    '    at /home/joy/.nvm/versions/node/v16.10.0/lib/node_modules/fjord_cli/lib/settings.js:36:72\n' +
    '    at Array.forEach (<anonymous>)\n' +
    '    at getSettings (/home/joy/.nvm/versions/node/v16.10.0/lib/node_modules/fjord_cli/lib/settings.js:35:22)\n' +
    '    at Object.<anonymous> (/home/joy/.nvm/versions/node/v16.10.0/lib/node_modules/fjord_cli/lib/FjordApp.js:8:18)\n' +
    '    at Module._compile (node:internal/modules/cjs/loader:1101:14)\n' +
    '    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)\n' +
    '    at Module.load (node:internal/modules/cjs/loader:981:32)\n' +
    '    at Function.Module._load (node:internal/modules/cjs/loader:822:12)\n' +
    '    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:79:12)\n' +
    '    at node:internal/main/run_main_module:17:47\n' +
    '\n',
  stderr: '/home/joy/.nvm/versions/node/v16.10.0/lib/node_modules/fjord_cli/lib/settings.js:88\n' +
    "    throw new Error('Incorrect Fjord settings parameters. Check logs above.')\n" +
    '    ^\n' +
    '\n' +
    'Error: Incorrect Fjord settings parameters. Check logs above.\n' +
    '    at getSettings (/home/joy/.nvm/versions/node/v16.10.0/lib/node_modules/fjord_cli/lib/settings.js:88:11)\n' +
    '    at Object.<anonymous> (/home/joy/.nvm/versions/node/v16.10.0/lib/node_modules/fjord_cli/lib/FjordApp.js:8:18)\n' +
    '    at Module._compile (node:internal/modules/cjs/loader:1101:14)\n' +
    '    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)\n' +
    '    at Module.load (node:internal/modules/cjs/loader:981:32)\n' +
    '    at Function.Module._load (node:internal/modules/cjs/loader:822:12)\n' +
    '    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:79:12)\n' +
    '    at node:internal/main/run_main_module:17:47\n' +
    'Subprocess exited with error 1\n'
}

So, it looks like the FjordSettings.json isn't configured correctly. I must set the KAFKA_TOPICS. The documentation for the Fjord CLI says: "update the FjordSettings.json file with server and consumer groups information", but without further clarification. Can you add a section to clarify how to configure the FjordSettings.json file correctly? Thank you.

Note:
I installed and configured the AWS CLI correctly. I was able to spin up an EC2 instance from the CLI, so my credentials are configured correctly. I installed aws-cdk via npm install -g and I'm able to run the cdk synth command from the command line, so I figure it's installed correctly.

[dejwakh]

Thanks for reaching out and for the detailed issue, Iuliu!

The FjordSettings.json file should look something like this:

{
  "name": "Fjord App",
  "server": {
    "NAME": "Fjord-Server",
    "JWT_KEY": "",
    "API_TOPICS": "stocks weather",
    "SEC_PER_PULSE": "30"
  },
  "consumerGroups": [
    {
      "NAME": "MyGroup",
      "KAFKA_TOPICS": "stocks weather",
      "API_TOPICS": "stocks weather",
      "FROM_BEGINNINGS": "false false",
      "BROKERS": "123.456.789.111:9093",
      "SECURITY": "none",
      "MEMBERS_COUNT": "1",
      "CONCURRENT_PARTITIONS": "1",
      "STARTING_DELAY_SEC": "0"
    }
  ]
}

Note that for all three parameters (KAFKA_TOPICS, API_TOPICS, FROM_BEGINNINGS), the value is a string that can contain any number of space-separated items, but they should be the same number of items. It can be 1, 5, or 2 (like the example above).

The consumerGroups is an array that can also contain any number of consumer group objects. In the above example, there's only one consumerGroup.

Do you have a Kafka broker setup too? You should be able to deploy Fjord to AWS without a Kafka broker set up, but of course it wouldn't do anything without the connection to Kafka.

Try that and let me know if that worked for you or not.