Support on-demand content in repair_metadata #848

jobselko · 2025-05-11T20:30:50Z

This PR extends the repair_metadata endpoint for repositories to also support on-demand content if the associated remote supports the PyPI json version API.

For on-demand content, we expect that:

PythonPackageContent always has correct name and version, and one ContentArtifact
RemoteArtifact always has correct sha256

Closes #849

pulp_python/app/tasks/repair.py

closes pulp#849

gerrod3

Good work on the logic, the grouping of the packages by name-version and then further grouping by shared url of the RAs looks right (it is definitely a bit complex!)

gerrod3 · 2025-06-02T05:07:03Z

pulp_python/app/tasks/repair.py

+                    grouped_by_url[ra.remote.url].append((package, ra))
+
+            # Prioritize the URL that can serve the most packages
+            for url, pkg_ra in sorted(


pkg_ra is a list of (package, ra) tuples, so I think this variable should be named to reflect it.

Would pkg_ra_pairs be better?

gerrod3 · 2025-06-02T05:07:51Z

pulp_python/app/utils.py

@@ -189,6 +191,19 @@ def artifact_to_python_content_data(filename, artifact, domain=None):
    return data


+def fetch_json_release_metadata(name: str, version: str, remote: Remote) -> dict:


You didn't change this to use the remote's downloader.

gerrod3 · 2025-06-02T05:11:59Z

pulp_python/app/tasks/repair.py

+    # Keep track of on-demand packages that need to be repaired
+    # A package is removed from this variable once it is successfully repaired
+    pkgs_not_repaired = set(on_demand_content.values_list("pk", flat=True))


Why not make this an empty set and then add the packages that we failed to repair. Should be less expensive.

gerrod3 · 2025-06-02T05:20:50Z

pulp_python/app/tasks/repair.py

+                    continue
+
+                for package, ra in pkg_ra:
+                    if package.pk not in pkgs_not_repaired:


Let's use group_list (probably should change it to a set) to track if we repaired the packaged yet. And then remove packages from it when we processed it.

gerrod3 · 2025-06-02T05:24:09Z

pulp_python/app/tasks/repair.py

+                        )
+                        batch = []
+                        set_of_update_fields.clear()
+                    pkgs_not_repaired.remove(package.pk)


Suggested change

pkgs_not_repaired.remove(package.pk)

group_set.remove(package.pk)

pkgs_not_repaired.update(group_set)

After a group is finished processing, any left over packages we didn't remove can then be added to pkgs_not_repaired

gerrod3 · 2025-06-02T05:26:45Z

pulp_python/app/tasks/repair.py

+                # All packages have the same URL, so pick a random remote
+                remote = pkg_ra[0][1].remote
+                try:
+                    json_data = fetch_json_release_metadata(name, version, remote)


Suggested change

# All packages have the same URL, so pick a random remote

remote = pkg_ra[0][1].remote

try:

json_data = fetch_json_release_metadata(name, version, remote)

if not group_set:

break # No packages left to repair, move onto the next group

# All packages have the same URL, so pick a random remote

remote = pkg_ra[0][1].remote

try:

json_data = fetch_json_release_metadata(name, version, remote)

By switching to tracking the repaired packages by group you can break out of the RAs loop early if there is no remaining packages to process.

gerrod3 · 2025-06-02T05:35:18Z

pulp_python/app/tasks/repair.py

+                    changed = False
+                    for field, value in new_data.items():
+                        if getattr(package, field) != value:
+                            setattr(package, field, value)
+                            set_of_update_fields.add(field)
+                            changed = True
+                    if changed:
+                        batch.append(package)
+                    if len(batch) == 1000:
+                        total_repaired += len(batch)
+                        PythonPackageContent.objects.bulk_update(
+                            batch, set_of_update_fields
+                        )
+                        batch = []
+                        set_of_update_fields.clear()


Since this is now repeated twice we can probably move it into an inline function.

jobselko self-assigned this May 11, 2025

github-actions bot added no-changelog no-issue labels May 11, 2025

jobselko force-pushed the repair_on_demand_content branch 4 times, most recently from fc1102d to 99180e4 Compare May 16, 2025 18:54

jobselko force-pushed the repair_on_demand_content branch 8 times, most recently from 9e27aaf to 04c2e4f Compare May 22, 2025 11:33

github-actions bot removed no-issue no-changelog labels May 22, 2025

jobselko force-pushed the repair_on_demand_content branch 2 times, most recently from 9875d50 to 925e392 Compare May 22, 2025 11:38

jobselko changed the title ~~DRAFT: Add on_demand content to repair_metadata~~ Support on-demand content in repair_metadata May 22, 2025

jobselko marked this pull request as ready for review May 22, 2025 11:59

jobselko requested a review from gerrod3 May 22, 2025 12:25

jobselko force-pushed the repair_on_demand_content branch 2 times, most recently from 2b14e83 to 79d7acf Compare May 22, 2025 12:51

github-actions bot added the no-issue label May 22, 2025

jobselko force-pushed the repair_on_demand_content branch from 79d7acf to b2b74fa Compare May 22, 2025 12:53

github-actions bot removed the no-issue label May 22, 2025

jobselko force-pushed the repair_on_demand_content branch 2 times, most recently from 42e9834 to 0b91e31 Compare May 22, 2025 13:42

jobselko commented May 22, 2025

View reviewed changes

pulp_python/app/tasks/repair.py Outdated Show resolved Hide resolved

jobselko marked this pull request as draft May 22, 2025 18:03

jobselko force-pushed the repair_on_demand_content branch from 0b91e31 to 7c3cb5c Compare May 25, 2025 20:40

Support on-demand content in repair_metadata

95f6e5f

closes pulp#849

jobselko force-pushed the repair_on_demand_content branch from 7c3cb5c to 95f6e5f Compare May 25, 2025 21:44

github-actions bot added the no-issue label May 25, 2025

jobselko marked this pull request as ready for review May 26, 2025 11:43

gerrod3 requested changes Jun 2, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support on-demand content in repair_metadata #848

Support on-demand content in repair_metadata #848

Uh oh!

jobselko commented May 11, 2025 •

edited

Loading

Uh oh!

Uh oh!

gerrod3 left a comment

Uh oh!

gerrod3 Jun 2, 2025

Uh oh!

jobselko Jun 2, 2025

Uh oh!

gerrod3 Jun 2, 2025

Uh oh!

gerrod3 Jun 2, 2025

Uh oh!

gerrod3 Jun 2, 2025

Uh oh!

gerrod3 Jun 2, 2025

Uh oh!

gerrod3 Jun 2, 2025

Uh oh!

gerrod3 Jun 2, 2025

Uh oh!

Uh oh!

		@@ -189,6 +191,19 @@ def artifact_to_python_content_data(filename, artifact, domain=None):
		return data


		def fetch_json_release_metadata(name: str, version: str, remote: Remote) -> dict:

	pkgs_not_repaired.remove(package.pk)
	group_set.remove(package.pk)
	pkgs_not_repaired.update(group_set)

Support on-demand content in repair_metadata #848

Are you sure you want to change the base?

Support on-demand content in repair_metadata #848

Uh oh!

Conversation

jobselko commented May 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

gerrod3 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jobselko commented May 11, 2025 •

edited

Loading