Conversation
The analytics summary endpoint could start returning no data if the 2 day cache expires and new data hasn't been generated by then. While this points to some underlying issue, we'd prefer to keep returning the last cached value (even if it's stale), rather than having this error. As part of this, also make the timeouts configurable so we can try to better tune this as needed.
- Fetch the pieces of data out of the analytics JSONB and store them as normal columns for faster, more direct access. - Eliminate custom `array_accum` function usage for better performance of aggregating. - Include organization name in cache key so the cache items are more obvious. - Include other items in the cache key that may affect the cache results so that when these settings are tweaked, the cache is regenerated. - Fix newer pg_dump versions including `\restrict` and `\unrestrict` in dump which we don't want for our own schema SQL file.
The unnesting of the user ids in the changes from bb756db was causing other stuff to be summed up incorrectly. This revamps things again, but matching the summing original logic (with test coverage of how multiple user IDs are handled now), but still performs better.
AWS `search.max_buckets` is limited to 10,000. So use a composite aggregation query so that these can be paginated to fetch all of the unique user ids.
This verifies that not only is the latest data present, but that all of the months/days are also populated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Various fixes for the summary analytics endpoint to prevent returning incomplete data or returning no data at all when things become uncached. The complete data is now cached indefinitely until new, complete data can replace the cache. Various improvements also made to the performance of gathering the aggregated data out of postgresql.