CLI reference

• Destination
• Streamsets
  • Create streamsets
  • Edit streamsets
  • List all streamsets
  • Delete streamsets
  • Balance pipelines between streamsets
• Source
  • Create source
  • Edit source
  • List all sources
  • Delete source
• Pipeline
  • Create pipeline
  • Edit pipeline
  • List all pipelines
  • Delete pipeline
  • Pipeline info
  • Pipeline logs
  • Reset pipeline offset
  • Start pipeline
  • Stop pipeline
  • Force stop pipeline
  • Destination logs
  • Update pipeline
  • Force delete pipeline
• Backup
• Restore
  • Restore Database
  • Restore StreamSets

Destination

1. Data destination config.

Anodot API token - You can copy it from Settings > API tokens > Data Collection in your Anodot account. Also here you can configure a proxy to connect to Anodot. For directory pipelines, you also need to use an API key Instructions

agent destination

Example:

> agent destination
Use proxy for connecting to Anodot? [y/N]: y
Proxy uri: http://squid:3181
Proxy username []:
Proxy password []:
Destination url [https://app.anodot.com]:
Anodot api data collection token: tokenhere
Anodot access key: apikey
Destination configured

Options:

• -t, --token TEXT
• --proxy / --no-proxy
• --proxy-host TEXT
• --proxy-user TEXT
• --proxy-password TEXT

StreamSets

StreamSets is an engine that runs pipelines. You can install multiple StreamSets instances on different hosts and they will be used by the agent to run pipelines

1. Add streamsets

agent streamsets add [OPTIONS]

Options:

• --url TEXT
• --username TEXT
• --password TEXT
• --agent-ext-url TEXT

2. Edit streamsets

agent streamsets edit [OPTIONS] URL

Arguments: URL - streamsets url, required argument

3. List all StreamSets

agent streamsets list

4. Delete streamsets

agent streamsets delete URL

Arguments: URL - streamsets url, required argument

5. Balance pipelines between StreamSets

Splits existing pipelines evenly between all StreamSets so every StreamSets instance runs the same number of pipelines

agent streamsets balance

---

Source

Data sources management

1. Create source

agent source create [OPTIONS]

Options:

• -a, --advanced
• -f, --file FILENAME - path to config file

2. Edit source

agent source edit [OPTIONS] NAME

Options:

• -a, --advanced
• -f, --file FILENAME - path to config file

Arguments: NAME - source name, required argument

3. List all sources

agent source list

4. Delete source

agent source delete NAME

Arguments: NAME - source name, required argument

---

Pipeline

Pipelines management

1. Create pipeline

agent pipeline create [OPTIONS]

Options:

• -a, --advanced
• -f, --file FILENAME - path to config file

2. Edit pipeline

agent pipeline edit [OPTIONS] PIPELINE_ID

Options:

• -a, --advanced
• -f, --file FILENAME - path to config file

Arguments: PIPELINE_ID - pipeline id, required argument

3. List all pipelines

agent pipeline list

4. Delete pipeline

agent pipeline delete PIPELINE_ID

Arguments: PIPELINE_ID - pipeline id, required argument

5. Pipeline info

Show pipeline status, errors if any, statistics about the number of records sent

agent pipeline info [OPTIONS] PIPELINE_ID

Options: -l, --lines INTEGER

Arguments: PIPELINE_ID - pipeline id, required argument

6. Pipeline logs

agent pipeline logs [OPTIONS] PIPELINE_ID

Options:

• -l, --lines INTEGER
• -s, --severity [INFO|ERROR]

Arguments: PIPELINE_ID - pipeline id, required argument

7. Reset pipeline offset

agent pipeline reset PIPELINE_ID

Arguments: PIPELINE_ID - pipeline id, required argument

8. Start pipeline

agent pipeline start PIPELINE_ID

Arguments: PIPELINE_ID - pipeline id, required argument

9. Stop pipeline

agent pipeline stop PIPELINE_ID

Arguments: PIPELINE_ID - pipeline id, required argument

10. Force stop pipeline

A pipeline can be force stopped only if it is in the STOPPING state

agent pipeline force-stop PIPELINE_ID

Arguments: PIPELINE_ID - pipeline id, required argument

11. Destination logs

Enable destination response logs for a pipeline (for debugging purposes only)

agent pipeline destination-logs [OPTIONS] PIPELINE_ID

Options:

• --enable / --disable

12. Update pipeline

Recreates a pipeline with existing configuration

agent pipeline update PIPELINE_ID

Arguments: PIPELINE_ID - pipeline id, required argument

13. Force delete pipeline

Delete everything about pipeline if safe delete failed

agent pipeline force-delete PIPELINE_ID

Arguments: PIPELINE_ID - pipeline id, required argument

Backup

Backing up Agent's database into a single dump file

agent backup

Restore

1. Restore database

Restores Agent's Database from the dump file. Just

agent restore database FILE_NAME

Arguments: FILE_NAME - absolute path to the database dump file

2. Restore StreamSets

Recreates or updates existing pipelines, their offsets and statuses using database info. If StreamSets instance related to the pipeline is not available, the pipeline will not be restored

agent restore streamsets

Example Backup and Restore