0.5.0

Changes: 0.4.1...0.5.0
Milestone: https://github.com/ohsu-comp-bio/funnel/milestone/3?closed=1

• Breaking changes
  • Configuration
    • The configuration file structure has been refactored to simplify, remove large duplicated blocks and deep nesting. Most structures live at the root level now.
    • Added some basic config validation to catch misspelled or unknown fields, a common source of issues.
• Failure tolerance
  • Added retries with exponential backoff and jitter to database and RPC clients.
• Databases
  • Added Google Cloud Datastore database backend.
• Events
  • Added “task created” event type.
  • Added full support for writing events to Kafka.
  • Added storage of system log events.
• Web dashboard
  • Display per-task system logs.
  • Tweaked display of large text fields such as stdout/err and input content.
• Storage
  • Ensure all file handles are closed consistently.
  • Added retries to Swift storage.
  • Produce warning on empty directory download.
  • Better defaults for chunk size in Swift backend.
  • Added generic S3 (i.e. not Amazon S3) storage backend/client, based on the Minio client library.
  • Also added the ability to configure/enable multiple, separate S3 backends simultaneously.
  • Added HTTP storage backend, which currently supports read-only operations (write/put is not supported).
• Removed the autoscaler code. This code was getting old and outdated, and nobody seemed to be using it. A fresh version will be rewritten in the future.
• Lots of other bugfixes.