Tech Report: Technologies - major.minor versions granularity

[max-ostapenko]

Related to HTTPArchive/httparchive.org#984

As the aggregation changes we have new schemas, and new tables for tech report.

I placed them in reports dataset:

• tech_crux (successor of core_web_vitals.technologies)
• tech_report_adoption
• tech_report_categories
• tech_report_core_web_vitals
• tech_report_lighthouse
• tech_report_page_weight
• tech_report_technologies
• tech_report_versions

Notes:

• removed a few columns from tech_crux (as compared to core_web_vitals.technologies):

  • category
  • origins_with_good_cwv_2023 and origins_with_good_cwv_2024 - deduplicated in origins_with_good_cwv

• removed empty similar_technologies column from technologies

• all the metrics have 'ALL' version that aggregates at technology level and expected to match the current values:
  

• corresponding to the current approach tech_report_versions has full adoption data from crawl.pages and tech_report_adoption has the smaller absolute values because of the JOIN with CrUX.
  

• example of the technology versions:

SELECT
  version,
  origins
FROM `reports.tech_report_versions`
WHERE technology = 'WordPress' AND
  client = 'mobile'
ORDER BY
  origins DESC

[max-ostapenko]

@tunetheweb @rviscomi FYI
removed a few columns from tech_crux (as compared to core_web_vitals.technologies which is to be deprecated):

• category
• origins_with_good_cwv_2023 and origins_with_good_cwv_2024 - deduplicated in origins_with_good_cwv

[max-ostapenko]

Some test cases for the version pattern:

SELECT
  version,
  REGEXP_EXTRACT(version, r'\d+(?:\.\d+)?') AS major_minor
FROM UNNEST(['1.2.3', '01976.2.83', '0003.3.4', '0.0.1', '1.2', 'version 5.1.2', '8']) AS version

version	major_minor
1.2.3	1.2
01976.2.83	01976.2
0003.3.4	0003.3
0.0.1	0.0
1.2	1.2
version 5.1.2	5.1
8	8

[max-ostapenko]

@rviscomi @tunetheweb After expanding the pattern to major + minor it's now obvious how messy the data is.
I've tried with a stricter (semver approach) ^((?:0|[1-9]\d*)\.(?:0|[1-9]\d*))\.(?:0|[1-9]\d*) and more relaxed \d+(?:\.\d+)? patterns, but neither one is great:

Examples of the technologies that are omitted in stricter version:

I was thinking to do something like IF(stricter IS NOT NULL, stricter, relaxed).
WDYT? Any suggestions?

[rviscomi]

I think relaxed is fine - we're just echoing what the site declared its version to be. When we aggregate technologies by version, the most popular ones will bubble up to the top anyway.

[max-ostapenko]

Then, if no more questions, it's ready to be merged.

[max-ostapenko]

Or considering we don't want to look into a long tail, maybe we limit to top 50 versions per technology?