Excessive throttling when using some AWS services

[atanunq]

First of all, thank you for the very useful plugin!

Problem

Using in_cloudwatch_logs, we are ingesting logs from a few different AWS services, most notably Lambda and AppSync:

# AppSync config
...
log_group_name            /aws/appsync/apis/
use_log_group_name_prefix true

# Lambda config
...
log_group_name            /aws/lambda/
use_log_group_name_prefix true
use_todays_log_stream     true # Get only log streams prefixed with today's YYYY/MM/DD

However, both of these services create a log stream for each request, which results in having a lot streams with few logs lines in each (even with very short retention configured). With the current configuration, get_log_events API calls are made for each stream, even though no new log lines are expected for older streams. After a while, the get_log_events quota is reached and throttling comes into play.

To mitigate that, I've been thinking about a few different approaches:

• Option to enable deletion of the log stream after reading it. Maybe through matching a regex from the last line to know when we're done. E.g until ^REPORT RequestId: .*$ is matched, in the case of Lambda logs and ^.* End Request$ for AppSync.
• Option to enable a rolling start date. E.g get only log streams that are up to 30 minutes old and check only them for new logs.

I am happy to work towards a solution, if any of that makes sense