Repository navigation
Make project identification the core functionality of this repository - #5
Merged
Merged
Conversation
devin-ai-integration
Bot
force-pushed
the
main
branch
2 times, most recently
from
November 18, 2026 20:08
c5ef7d3 to
dfcb8fe
Compare
…orpus identify(url; commit) returns an Identification(project, version, provenance) and only returns versions that look like plausible version numbers for the project. Hand-verified corrections live in overrides.toml. test/corpus.toml holds 3,031 known-good mappings harvested from GeneralMetadata.jl (scripts/build_corpus.jl) and gates the weekly data refresh. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…d-separator tags, refresh gate, per-commit known failures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… push Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…utrank Repology's URL data Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ose a PR when a refresh regresses it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… version 1.5.6.pl02 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…data had 1.5.6) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… downstream subpackages Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…exported URL to another project through a PR Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ck between projects sharing a repository (SDL2/SDL3) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… qdrant and cudss corpus entries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…, letter-prefixed and _rc versions, and curated repositories/tag schemes for projects Repology's data dropped Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…SPIRV-LLVM-Translator and Win32-OpenSSH Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…e corpus passed and no exported URL changed project, otherwise propose a PR Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration
Bot
force-pushed
the
main
branch
from
November 18, 2026 20:23
dfcb8fe to
fede20b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
identify()with provenance, project-awarevercmpvia libversion_jll, a 3,031-entry regression corpus that checks project and version, and a scheduled Repology import that fits a standard GitHub runner and is gated on that corpus.Why
SecurityAdvisories depends on mapping JLL sources to upstream projects and versions, and that mapping has been the fragile step. Expat 2.8.3 was recorded as
8_3because a stale Repology pin missed it and the generic tag parser fell back to a guess. Nothing marked results as guesses, nothing caught regressions when the data changed, and the data only refreshed when someone ran the workflow by hand.This makes Repology.jl the single place that answers "what upstream project and version is this source?". Every answer now says how it was obtained.
What changed
identifywith provenance (src/identify.jl)identify(url; commit=nothing, use_overrides=true)returnsIdentification(project, version, provenance).identify_upstream(url, hash)is still there as a wrapper, so GeneralMetadata works without changes.Resolvers run in a fixed order, and
provenancerecords which one answered:overridesourcesinadditional_info.toml(skipped whenuse_overrides=false)url-patternurl_patternsinadditional_info.tomlrepology-urltag-schemegit-taggit ls-remoterepositoryheuristicVersion validation: a version is returned only if it is plausible for that project. Malformed values like
8_3and placeholders likegitcome back asnothing.Tag parsing: the parser handles mixed separators (
R_2_8_3,REL_16_9,V_9_2_P1,OCCT V7_9_2,RELEASE.2025-…). It no longer mistakes all-digit date tags for commit hashes and no longer cuts versions short after a numeric prefix.Curation in one file (
additional_info.toml)url_patterns,repositories,tag_patterns,cpes,version_scheme/version_flagsandsources.url_patternsandrepositoriesnow take precedence over Repology's own data. That's needed where Repology assigns an upstream tarball or repository to a downstream subpackage, e.g. FFmpeg 7.1.1 →ffmpeg-static-libs-for-sunshine, rsync →rsync3, tiff →libtiff-5.tag_patternsdecide which one a tag belongs to. For example,libsdl-org/SDL.gitgivessdl2forrelease-2.*andsdl3forrelease-3.*.vercmp(src/vercmp.jl)vercmp(a, b; flags)calls libversion'sversion_compare4directly throughlibversion_jll.vercmp(project, a, b)uses that project's version scheme:version_schemeislibversionorsemver, e.g. ImageMagick's7.1.2-1is a patch release after7.1.2.version_flagspasses libversion flags, e.g.P_IS_PATCHfor OpenSSH's9.2p1.X-N,XpN, a letter suffix) that the scheme doesn't settle,vercmp(project, a, b)returnsnothing. A form counts as settled if the project is curated or has already shipped a release in that form.Regression corpus (
test/corpus.toml,test/known_failures.toml)The corpus has 3,031 source → upstream mappings from GeneralMetadata (built by
scripts/build_corpus.jl): 1,806 archives and 1,225 git+commit sources.manualentries were hand-corrected in GeneralMetadata.jll-versionentries are ones where the upstream version equalled the JLL version.Where Repology was right and GeneralMetadata wrong, the corpus was corrected. Examples: xorriso
1.5.6.pl02(not1.5.6), cudss0.7.0.20,deno(notdeno-bin),qdrant(notrust:qdrant).test/known_failures.tomllists the 6 current misses as@test_broken:Regenerate it with
scripts/known_failures.jl. Git entries are only checked whenREPOLOGY_TEST_NETWORKis set.Results on the same data, without overrides:
These numbers were measured before the later curation and the data refresh, which brought known misses down to 6.
Import fits a standard runner (
scripts/update_repology_info.jl)Scheduled, corpus-gated refresh (
.github/workflows/repology.yml)REPOLOGY_ALLOW_FIXED=1, then regeneratesknown_failures.toml.automatic/repology-data-refresh-<branch>, and a PR is opened or updated. It shows theknown_failures.tomldiff and a table of project changes. The reviewer then fixesadditional_info.toml, correctstest/corpus.toml, or accepts the change.1.5.6forxorriso-1.5.6.pl02.tar.gz) and opened [automatic] Repology data refresh with corpus regressions JuliaComputing/Repology.jl#3. That PR fixed the corpus entry and three downstream-subpackage mappings, and it is merged here, along with the project checks from Check the project of every corpus entry; review refreshes that move exported URLs to another project JuliaComputing/Repology.jl#4.Data
data/comes from the 2026-11 dump, generated by the workflow.Settings needed on this repo
Not in this PR
identifyand storingprovenance, and SecurityAdvisories will useprovenanceandvercmp(project, …). Both are separate changes.