Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance issues when processed_messages gets "large" #137

Open
bendavies opened this issue Feb 19, 2025 · 0 comments
Open

Performance issues when processed_messages gets "large" #137

bendavies opened this issue Feb 19, 2025 · 0 comments

Comments

@bendavies
Copy link

bendavies commented Feb 19, 2025

Hi,

Loading the dashboard is taking 20-30 seconds for us when processed_messages becomes a few million.
We currently have 6,552,864 rows in that table, so not really that big.

I know this table can be purged, but 6 million rows doesn't seem like enough to warrant that.

query 1 that takes up to 20 seconds:

SELECT DISTINCT
	id_15,
	MIN(sclr_16) AS dctrn_minrownum
FROM
	(
		SELECT
			p0_.run_id AS run_id_0,
			p0_.attempt AS attempt_1,
			p0_.message_type AS message_type_2,
			p0_.description AS description_3,
			p0_.dispatched_at AS dispatched_at_4,
			p0_.received_at AS received_at_5,
			p0_.finished_at AS finished_at_6,
			p0_.wait_time AS wait_time_7,
			p0_.handle_time AS handle_time_8,
			p0_.memory_usage AS memory_usage_9,
			p0_.transport AS transport_10,
			p0_.tags AS tags_11,
			p0_.failure_type AS failure_type_12,
			p0_.failure_message AS failure_message_13,
			p0_.results AS results_14,
			p0_.id AS id_15,
			ROW_NUMBER() OVER (
				ORDER BY
					p0_.finished_at DESC
			) AS sclr_16
		FROM
			processed_messages p0_
	) dctrn_result
GROUP BY
	id_15
ORDER BY
	dctrn_minrownum ASC
LIMIT
	15;

Second query which takes 10 seconds to load the statistics page:

SELECT
	count(DISTINCT p0_.id) AS sclr_0
FROM
	processed_messages p0_
GROUP BY
	p0_.message_type;

I've not looked into improving these yet. just raising for visibility.

Thanks,
Ben

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant