Skip to content

Expose percent-of-total metrics for report rows in the Reporting API - #24983

Merged
caddoo merged 15 commits into
6.x-devfrom
dev-14950
Aug 11, 2026
Merged

Expose percent-of-total metrics for report rows in the Reporting API#24983
caddoo merged 15 commits into
6.x-devfrom
dev-14950

Conversation

@caddoo

@caddoo caddoo commented Aug 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description:

Fixes DEV-14950 (#10739)

Report tables show each row's value as a % of the report total in the hover tooltip, but the Reporting API never exposed those percentages. This adds them as columns on every report row, eg nb_visits_percent_of_total next to nb_visits.

How it works:

  • A new PercentOfReportTotal processed metric is registered in the DataTablePostProcessor for every metric the report processes totals for (Metrics::getMetricIdsToProcessReportTotal() plus Report::getMetricNamesToProcessReportTotals()), using the totals the ReportTotalsCalculator already computes. Rates and averages don't get one, and neither do reports without a dimension (VisitsSummary.get etc) where a % of total is meaningless.
  • Values are quotients formatted like every other percent metric, so format_metrics=0 gives the raw value and the default gives eg 12.3%. Subtable and flattened rows are relative to the first level report total, same as the UI tooltips.
  • On by default per the decision on the ticket. percent_of_total=0 turns it off, and totals=0 also disables it since the totals are the source.
  • The percentages match what the UI already shows since both use the same totalsUnformatted metadata, and they flow into all export formats and ImageGraph (pass columns=nb_visits_percent_of_total) for free.
  • Referrers loads subtables through inner API requests, those now pass percent_of_total=0 so the outer request stays in control of the output.
  • Non-additive metrics (unique visitors, users) don't get a percent column. Their report total is a plain sum of non-summable values, so the percentage could exceed 100% and would differ depending on which archived record serves the report (the flat vs hierarchical Actions record, caught by MixedArchivingAggregationTest).

Note for CSV/TSV consumers: the new columns change the header and column count. Anyone parsing by position can pass percent_of_total=0 to get the previous output. This should be called out in the release notes and developer changelog.

Backport: none, this is an additive feature so Matomo 6 only, 5.x takes bugfix/security only.

All system test expected files are updated (synced from CI artifacts, then every diff was verified to be a pure addition of percent columns).

Review

Cross-plugin impact (scanned local premium + public plugin clones)

  • No plugin uses percent_of_total as a param or column name, no conflicts.
  • SearchEngineKeywordsPerformance embeds subtables via inner Request::processRequest calls like Referrers did, so it needs the same percent_of_total => 0 on its inner requests (follow up PR in that repo).
  • Premium plugins overriding getMetricNamesToProcessReportTotals (FormAnalytics, MCCA, PaidAdvertisingPerformance, SEKP) get percent columns for their custom metrics automatically, which is the intended behaviour.
  • Roughly 37 plugin repos have system test expected files that will need regenerating once this merges, the largest being GoogleAnalyticsImporter, CustomReports, CrashAnalytics, RollUpReporting and Funnels.

Submodule companion PRs (draft, merge after this PR)

The gitlinks in this PR point at those branches so CI can fetch them.

Checklist

  • [✔] I have understood, reviewed, and tested all AI outputs before use
  • [✔] All AI instructions respect security, IP, and privacy rules

caddoo added 4 commits August 3, 2026 16:27
@caddoo
Expose percent-of-total metrics for report rows in the Reporting API
a07d361 
Adds a {metric}_percent_of_total processed metric to every report row,
calculated from the report totals (ReportTotalsCalculator) for the
metrics reports process totals for. Values follow format_metrics like
other percent metrics and can be disabled with percent_of_total=0
(or totals=0).

The percentages match the ratio tooltips the HTML table visualization
already shows, and flow into all export formats and ImageGraph.
@caddoo
Add developer changelog entry for percent-of-total metrics
4729173
@caddoo
Exclude percent-of-total columns from scheduled email reports
39d15ec 
Scheduled reports render every processed report column, the new
percent-of-total columns would add up to six columns to the emailed
HTML/PDF/CSV tables.
@caddoo
Update system test expected files for percent-of-total metrics
9bae96c 
Synced from CI run 30784690250, with these exceptions reviewed and
handled locally:
- Reverted unrelated sync noise (Live idVisit flake in TimezonesTest,
  trailing-newline-only changes in BotTracking and glossary files).
- PercentOfReportTotal now also resolves totals keyed by metric id,
  which happens when queued filters are disabled for a request. This
  makes comparison rows carry their own series' percentages (computed
  against that series' report total) instead of missing them, so the
  DataComparison expected files were regenerated locally.
- Renderers now translate percent-of-total column names based on the
  base metric translation (translateColumnNames=1 CSV exports).
@caddoo caddoo added this to the 6.0.0 milestone Aug 3, 2026
@caddoo
caddoo marked this pull request as draft August 3, 2026 05:50
caddoo added 3 commits August 3, 2026 20:45
@caddoo
Sync remaining expected files and export expectations for percent-of-…
5bd40dc 
…total

- ReportExporting UI text expectations regenerated with the percent
  columns the exports now contain.
- ActionsDataTable_flattened screenshot: two rows with fully tied
  metrics swap position because array_multisort falls back to
  comparing row objects, which now include the percent columns.
- Remaining system test expected files from CI run 30787649801
  (Goals item reports and ranking-query tests gained percents through
  the metric-id keyed totals support).
@caddoo
Skip percent-of-total metrics for pivoted tables, fix BC test comparison
9967a00 
Pivoted tables have one column per pivot dimension value, computing
percent-of-total metrics there crashed PivotByDimension's column
normalization (array_combine count mismatch) and would not be
meaningful anyway.

BackwardsCompatibility1XTest compares against OneVisitorTwoVisits
expected files, the 1.x fixture cannot produce the same percentages
for metrics its archives do not contain, so the percent variants of
already-removed fields are removed as well.
@caddoo
Sync ArchiveCronTest expected file for percent-of-total metrics
a8c206f
This was referenced Aug 4, 2026
Merged
Merged
Merged
Merged
caddoo added 3 commits August 4, 2026 16:04
@caddoo
Exclude non-additive metrics from percent-of-total, converge expected…
f5d7825 
… files

Unique visitor and user totals are plain sums of non-summable values, so
their percent-of-total was not a meaningful denominator and its presence
differed between the flat and the hierarchical Actions record a report is
served from (caught by MixedArchivingAggregationTest).

Also converges the ArchiveInvalidation and BlobReportLimiting expected
files, whose multi-dataset test methods abort on the first failing
dataset and therefore only surfaced one new file per CI round, and
advances the submodule gitlinks to branches containing their updated
expected files.
@caddoo
Merge remote-tracking branch 'origin/6.x-dev' into dev-14950
a5cc5fb 
# Conflicts:
#	plugins/MarketingCampaignsReporting
@caddoo
Advance MarketingCampaignsReporting for non-min_php expected files
893b583
@caddoo
caddoo marked this pull request as ready for review August 4, 2026 05:47
@chippison
chippison requested a review from a team August 5, 2026 20:52
chippison
chippison reviewed Aug 6, 2026
View reviewed changes
Comment thread plugins/CoreHome/Columns/Metrics/PercentOfReportTotal.php
Comment thread core/API/DataTablePostProcessor.php Outdated
Comment thread plugins/CoreHome/Columns/Metrics/PercentOfReportTotal.php
Comment thread plugins/API/ProcessedReport.php
caddoo added 2 commits August 6, 2026 15:17
@caddoo
Address review findings for percent-of-total metrics
22cc54d 
- PercentOfReportTotal now refreshes its total from the table's totalsUnformatted
  metadata in beforeCompute(), so DataRounding's percent recompute divides the
  rounded row values by the rounded totals instead of the stale unrounded total
  (a rounded row of 10 over an unrounded total of 7 gave 142.9%).
- DataRounding also processes table level totals when rows of a rounding enabled
  site are part of a table even though no requested site has rounding enabled
  (MultiSites tables requested with another ambient idSite), keeping the output
  independent of the site the request was made for.
- Truncation summary rows get their percent-of-total columns recomputed from the
  summed metric values when the percent columns were computed before the generic
  filters ran (eg filter_sort_column=nb_visits_percent_of_total), instead of
  keeping the sum of the rounded per row quotients.
- percent_of_total is read with Request::getBoolParameter, so 'false' disables
  the metrics like '0' does.
- API.getProcessedReport describes the percent columns in metadata.metricTypes
  and metadata.metricsDocumentation.
- New coverage: DataRoundingCoverageTest checks the raw quotient and formatted
  percentage against the rounded values, PercentOfTotalQueryParamTest covers the
  boolean param value and the truncation summary row. Regenerated the affected
  expected files.
@caddoo
Sync getProcessedReport expected files for percent-of-total metadata
3bf2204 
Regenerated from CI artifacts (run 31068050774): every diff adds the percent
column entries to metadata.metricTypes and metadata.metricsDocumentation.
Advances the CustomVariables gitlink for its expected file.
@caddoo
caddoo requested a review from chippison August 6, 2026 04:58
chippison
chippison reviewed Aug 6, 2026
View reviewed changes
Comment thread plugins/API/ProcessedReport.php Outdated
caddoo added 3 commits August 7, 2026 09:47
@caddoo
Keep metricsDocumentation hidden for hideMetricsDoc=1
f98488f 
getReportMetadata() removes the metricsDocumentation block when hideMetricsDoc
is set; adding the percent-of-total documentation unconditionally recreated the
key with only the percent entries in it. Only add the documentation when the
block is still present, like removeEmptyColumns() already does.
@caddoo
Retrigger CI
2212ed3
@caddoo
Retrigger CI after Actions outage
767c231
@caddoo
caddoo requested a review from chippison August 7, 2026 01:48
chippison
chippison approved these changes Aug 7, 2026
View reviewed changes

@chippison chippison left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick fixes.

Looks good now

@caddoo
caddoo merged commit b565835 into 6.x-dev Aug 11, 2026
329 of 333 checks passed
@caddoo
caddoo deleted the dev-14950 branch August 11, 2026 03:32
sgiehl pushed a commit to matomo-org/plugin-Provider that referenced this pull request Aug 11, 2026
@caddoo @sgiehl
Add percent-of-total metric columns to system test expected files
9ed994c 
Matomo core adds always-on {metric}_percent_of_total columns to report
rows in API responses (matomo-org/matomo#24983, Matomo 6). These expected
files include the new columns and only pass against a Matomo 6 core that
contains that change.
sgiehl pushed a commit to matomo-org/plugin-AnonymousPiwikUsageMeasurement that referenced this pull request Aug 11, 2026
@caddoo @sgiehl
Add percent-of-total metric columns to system test expected files
84d9d27 
Matomo core adds always-on {metric}_percent_of_total columns to report
rows in API responses (matomo-org/matomo#24983, Matomo 6). These expected
files include the new columns and only pass against a Matomo 6 core that
contains that change.
sgiehl pushed a commit to matomo-org/plugin-CustomVariables that referenced this pull request Aug 11, 2026
@caddoo @sgiehl
Add percent-of-total metric columns to system test expected files
2a9df44 
Matomo core adds always-on {metric}_percent_of_total columns to report
rows in API responses (matomo-org/matomo#24983, Matomo 6). These expected
files include the new columns and only pass against a Matomo 6 core that
contains that change.
sgiehl pushed a commit to matomo-org/plugin-Provider that referenced this pull request Aug 11, 2026
@caddoo
Add percent-of-total metric columns to system test expected files (#79)
53cc054 
Matomo core adds always-on {metric}_percent_of_total columns to report
rows in API responses (matomo-org/matomo#24983, Matomo 6). These expected
files include the new columns and only pass against a Matomo 6 core that
contains that change.
sgiehl pushed a commit to matomo-org/plugin-AnonymousPiwikUsageMeasurement that referenced this pull request Aug 11, 2026
@caddoo
Add percent-of-total metric columns to system test expected files (#115)
08d6509 
Matomo core adds always-on {metric}_percent_of_total columns to report
rows in API responses (matomo-org/matomo#24983, Matomo 6). These expected
files include the new columns and only pass against a Matomo 6 core that
contains that change.
sgiehl pushed a commit to matomo-org/plugin-CustomVariables that referenced this pull request Aug 11, 2026
@caddoo
Add percent-of-total metric columns to system test expected files (#139)
6ae495b 
* Add percent-of-total metric columns to system test expected files

Matomo core adds always-on {metric}_percent_of_total columns to report
rows in API responses (matomo-org/matomo#24983, Matomo 6). These expected
files include the new columns and only pass against a Matomo 6 core that
contains that change.

* Describe percent-of-total metrics in processed report metadata expected file
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chippison chippison chippison approved these changes

Assignees

No one assigned

Labels

None yet

Milestone

6.0.0

Development

Successfully merging this pull request may close these issues.

2 participants

@caddoo @chippison