Tbain/253 add tags count

[tbain]

Description

This implements openedx/modular-learning#253 , the task to add tag usage counts to the tags table under the taxonomies table. The frontend piece is where the results of this aggregation work is displayed is part of a separate pr to openedx/frontend-app-authoring. This change adds a subquery annotation onto the django query for retrieving tags. The original implementation of the counts for tags only counted raw usage of each tag. This feature/PR aggregatea sum of any tag and child tag usage with sibling de-duplication for the same usage (e.g. when two sibling nodes are used against the same course, module, etc. we still only need to count that as '1' for any parent/grandparent nodes) as specified in the AC for the issue above, so it was replaced with this more complicated bit of logic that sums across tag usage based on various courses, sections, modules, and libraries that might use a tag.

• update:
  The count logic is done in-memory, since we saw noticeable performance issues with trying to stay in the QuerySet/Django paradigm for calculating the counts. This makes the code a little less straightforward, since we break it out into a somewhat odd in-memory python application of the logic, but it still works as intended and resolves as many performance pain points as possible while still adhering to the counting requirements that end up necessitating such code.

• update:
  The count logic was moved out to the API level so that all the query data could complete and the logic to add the counts would not affect the ability to process the data as a QuerySet until the very end. This necessitated removing all the usage_count unit tests out of the _models and _api levels to the _view levels, so re-implemented that logic as appropriate (with a lot of AI help to speed it up)

AI Usage Disclosure: Claude was used via intelliJ IDE integration was used through the authoring process to work through complicated logic, craft the foundation of the unit tests, and also simplify it/make it more pythonic/alleviate performance concerns.

Supporting information

Github issue with AC: openedx/modular-learning#253

Testing instructions

Refer to the AC in the Github Issue. Steps to verify this is implemented and working via UX (Note, depends on the frontend part of this ticket):

1. Navigate to the "Studio home" page
2. Navigate into an existing Course (or create a course and navigate into it)
3. In the "Course Outline" page, add tag(s) from an existing taxonomy to the course, module, or section. Ensure at least one of the tags you add is a sub-tag of a root tag.
4. Navigate back to the "Studio home" page
5. Click the "Taxonomies" tab to navigate to the Taxonomies page
6. Navigate into the Taxonomy that corresponds to the tag you added in step 3
7. Observe that, if a tag is used, there is now an additional column on the table named "Usage Count" that is populated with bubbles that display the count of tags usages, if applicable
8. Ensure that the tag you added in Step 3 properly associates the incremented count from its usage, and ensure that the usage count properly aggregates up the lineage based on the sub tag you selected in step 3

Other information

Include anything else that will help reviewers and consumers understand the change.

• Does this change depend on other changes elsewhere?
  • this ticket is backwards compatible with the current implementation in frontend-app-authoring, since by default the frontend does not request the counts.
• Any special concerns or limitations? For example: deprecations, migrations, security, or accessibility.
  • none at this time

[openedx-webhooks]

Thanks for the pull request, @tbain!

This repository is currently maintained by @axim-engineering.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

---

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[jesperhodge]

There seem to be changes missing. For example, src/taxonomy/data/api.ts.
Could you

• review this PR and make sure that all necessary changes are in this branch? Compare to the open Unicon PR.
• review discussions in the Unicon PR and either resolve them or copy them here to be addressed here.
• fix any pipeline errors
  ?

[mgwozdz-unicon]

Since we're no longer using recursive SQL for this, is it possible to update the PR description for accuracy?

[tbain]

There seem to be changes missing. For example, src/taxonomy/data/api.ts. Could you

* review this PR and make sure that all necessary changes are in this branch? Compare to the open Unicon PR.

* review discussions in the Unicon PR and either resolve them or copy them here to be addressed here.

* fix any pipeline errors
  ?

• src/taxonomy/data/api.ts, as an example, was a file in the front-end changes. I compared everything with the Backend changes/openedx-core and this is the correct set of files
• All comments/issues to address from the aforementioned PR have been addressed with this one, so this PR is up to date
• Working on that - I had missed a test suite that was affected by the changes so address that, still working on a strange quality issue where it's complaining about the time the unit test suite takes

[Copilot]

Pull request overview

Adds rolled-up, de-duplicated tag usage counts (including ancestor rollups) to the tag listing query so the Taxonomies UI can display accurate “Usage Count” values per tag.

Changes:

• Replaced the prior per-tag direct usage counting subquery with a dynamic, depth-aware subquery that rolls counts up to ancestors with per-object de-duplication.
• Updated existing API/model tests to reflect rolled-up counts and added a broader set of usage-count test cases.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 9 comments.

File	Description
src/openedx_tagging/models/base.py	Centralizes and updates include_counts behavior by annotating tag querysets with rolled-up, de-duplicated usage_count via a subquery.
tests/openedx_tagging/test_models.py	Updates expected usage counts and adds multiple new test scenarios validating ancestor rollup and sibling de-duplication.
tests/openedx_tagging/test_api.py	Updates autocomplete/search test expectations to reflect rolled-up usage counts returned by the API when include_counts=True.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[bradenmacdonald]

Feel free to ping me for review here once the AC are clarified and the comments from Copilot etc are addressed.

[tbain]

Feel free to ping me for review here once the AC are clarified and the comments from Copilot etc are addressed.

@bradenmacdonald I think this is ready for re-review, I resolved all the Copilot issues and added the improvement you suggested for finding the depth via a query rather than depending on the constant

[bradenmacdonald]

We can "just use the existing implementation exactly as it is in this PR"

What I meant was that we should change the PR to provide the desired tag count data via a separate endpoint. But using more or less the exact same code as you have now if you don't want to refactor it.

So instead of /taxonomy/:n/tags/?include_counts which we can leave alone for now or even remove the "counts" functionality from, add a new endpoint called /taxonomy/:n/usage_counts.

But I guess that's going to require some major changes on the frontend side to combine those pieces of information, so maybe that's not going to work with your timeline.

[bradenmacdonald]

I guess before we consider merging this as is, I'd like to know if the slowness mostly scales with taxonomy size or object tag count or both? If the slowness is only a factor on large taxonomies and it's just ~1s, I think that's OK for now. But if it's slow as the # of object tags increases or it's O(n_tags * n_object_tags) or anything like that, then it'll seem fine now and slow to a crawl in prod once people start using thousands of these things and re-running tagged courses.

[ormsbee]

@bradenmacdonald:

I guess before we consider merging this as is, I'd like to know if the slowness mostly scales with taxonomy size or object tag count or both? If the slowness is only a factor on large taxonomies and it's just ~1s, I think that's OK for now. But if it's slow as the # of object tags increases or it's O(n_tags * n_object_tags) or anything like that, then it'll seem fine now and slow to a crawl in prod once people start using thousands of these things and re-running tagged courses.

I agree with this. If it's ~1s for an outlier taxonomy owing to the number of tags, it's acceptable for now, and we can figure out how to optimize later. If the time scales with the number of things tagged, this will rapidly become unusable.

@mgwozdz-unicon:

I think the rest of the big picture for this is that for the folks who create new course runs every term and have very short terms, the usage count is probably meaningless for them and not very helpful anyways "Was it used 250 times or 275 times? How do I know how much usage I should be expecting compared to what I'm seeing?" I think there could also be an option to just hide the usage count column altogether if we detect that their usages are so high that this info is irrelevant for the instance. Or to hide the usage count for people who primarily use course runs each term instead of continuously running courses.

I'd be cautious about assuming people don't care. I've been told that there's sometimes grant money riding on proving how much things get used. In any case, we'd definitely need product folks to weigh in on it.

[bradenmacdonald]

FWIW Claude analyzed the query and says it could be slow. I have not had time to validate this analysis, so take with a grain of salt.

The generated SQL (at depth=3)

SELECT ...,
  COALESCE(
    (SELECT COUNT(DISTINCT U0."object_id") AS "total_usage"
     FROM "oel_tagging_objecttag" U0
       INNER JOIN "oel_tagging_tag" U2 ON (U0."tag_id" = U2."id")
       LEFT OUTER JOIN "oel_tagging_tag" U3 ON (U2."parent_id" = U3."id")
       LEFT OUTER JOIN "oel_tagging_tag" U4 ON (U3."parent_id" = U4."id")
     WHERE U0."taxonomy_id" = 1
       AND (U0."tag_id" = outer."id"
            OR U2."parent_id" = outer."id"
            OR U3."parent_id" = outer."id"
            OR U4."parent_id" = outer."id")
    ), 0) AS "usage_count"
FROM "oel_tagging_tag"
WHERE "oel_tagging_tag"."taxonomy_id" = 1

The scaling problem: it will get meaningfully slower

The old query filtered by tag_id = OuterRef("pk") — it used the FK index on tag_id and touched only the ~2-3 matching ObjectTag rows per tag. Cost per tag: O(direct_uses).

The new query is a correlated subquery that, for each tag in the result set, does this:

1. Range-scans all ObjectTags for the taxonomy (using the (taxonomy, object_id) index)
2. JOINs each to the tag table + parent chain (D PK lookups per row — fast)
3. Evaluates the OR across 4 different table aliases (can't use any index for this filter)
4. Counts distinct object_id values

Cost per tag: O(all_ObjectTags_in_taxonomy × D)

So the total work is roughly T × O × D where:

• T = tags in the result set
• O = total ObjectTags for this taxonomy
• D = max depth (small, ≤5)

Scenario	Tags (T)	ObjectTags (O)	Old cost (T × ~3)	New cost (T × O)
Small	100	50	300	5,000
Medium	500	1,000	1,500	500,000
Large	1,000	10,000	3,000	10,000,000
Very large	1,000	100,000	3,000	100,000,000

It scales linearly with ObjectTag count, but since it's inside a correlated subquery that runs per-tag, the multiplier is the number of tags displayed. This will be painfully slow once a popular taxonomy gets applied to thousands of courses/modules/sections.

Why the OR kills performance

The condition tag_id = X OR parent_id = X OR parent.parent_id = X spans different table aliases. The DB optimizer can't use a single index path — it has to evaluate all conditions per row after the JOINs. There's no composite index that helps here.

[ormsbee]

Okay. So it sounds like the most straightforward thing is to do the up-front query for counts and stitch together the hierarchy counts in Python as @bradenmacdonald outlined in:

1. Build the object counts in python. Basically, in the /taxonomy/:n/tags/ REST API endpoint, once we've evaluated the query to load the tags (along with whatever filtering and pagination etc. may be in place), then you can do a second query to load all related ObjectTags, including tag__lineage (1 simple query, no aggregation at the query level). Then, in python, you can group by object tags, split the lineage up into individual tags, de-duplicate with a set(), and then annotate the original query objects with the counts. This also lets you separate implicit counts from explicit counts in the API, which I think would be even better then combining them.

Does that sound right to everyone?

[bradenmacdonald]

That sounds good to me, and has the advantage of requiring no further changes to the frontend PR.

[jesperhodge]

@ormsbee @bradenmacdonald the only question I have is related to memory usage.
If I'm understanding correctly, the Python solution pulls every object tag related to the taxonomy into memory and then iterate over them. How many object tag applications are we expecting? Are we good or is there any memory problem?

[jesperhodge]

@bradenmacdonald @ormsbee just to make sure we have considered all alternatives:

AI is suggesting Recursive CTEs as the optimal solution. However, that requires MySQL >= 8. Do we need to support older MySQL versions?

I haven't been able to evaluate the AI response in-depth so it may be incorrect

AI suggestion:
"
Recursive CTE: Top-Down (The "Path Discovery" Strategy)
A recursive solution works in two distinct steps for each Tag:

1. Discovery: Start at the target Tag ID and recursively find all descendant Tag IDs.
2. Aggregation: Perform a single SELECT COUNT(...) FROM ObjectTag WHERE tag_id IN (discovered_ids).
3. Complexity: Approximately O(N * D + log M))
   D is the small cost of traversing the Tag table (usually very fast).
   log M is a single, highly optimized B-Tree Index Seek on the ObjectTag.tag_id column.
   The Benefit: Instead of checking D columns for every row in the big table, you find a small list of IDs first, then use the database's most optimized tool—the primary index—to grab the counts.
   "

N = Number of Tags in your main queryset
M = Total number of rows in ObjectTag
D = depth of tree

[tbain]

Ultimately, via a conversation/clarification over Slack (dated 2026-04-02), we decided to address the performance concerns via in-memory python based code processing rather than trying to rely on django joins and sub-queries, or a recursive SQL/CTE implementation. Since we were seeing such an egregious performance hit, the implementation leans towards minimizing performance issues and bottlenecks where possible, potentially at the slight cost of straight-forwardness of what exactly the code is doing (e.g. performance wise it was very expensive to implement the 'annotation' of the usage_count python implementation to the QuerySet, so the QuerySet is taken in and returned as a finalized list, rather than waiting a little later down the call chain to have django do so automatically, etc.) The logic works as expected according to unit tests and local testing, and pains were taken to remove any behavior that would lead to any performance issues, while still trying to make it as straightforward as possible. My AI IDE integration (Claude backing) reports, and I concur, that this implementation has a linear Big O cost based on the number of objects + tags (e.g. O(obj+tags)), which should be much more performant than the previous implementation which necessitated a much less performant multi-level join of indeterminate depth with would have been a multiplicative relationship to the number of tags for each layer (e.g. something like O(tags * obj * depth)).

[tbain]

Did some local testing with the large Lightcast taxonomy that Braden posted earlier; applied some tags from that taxonomy to an existing course on my local, and then watched the timings for the http://studio.local.openedx.io:8001/api/content_tagging/v1/taxonomies/2/tags/?full_depth_threshold=10000&include_counts=true call. They remained pretty much unchanged from the previous implementation on now-current main;

Various load times with Lightcast Taxonomy:
main: 83ms, 83ms, 76ms, 188ms, 73ms, 78ms (avg: 96ms)
this branch: 86ms, 82ms, 87ms, 82ms, 82ms, 85ms (avg: 84ms)

However, I'm not quite sure this is the best representation of the times to reproduce the same circumstances as Braden saw above with the 10x increase in call time, since I don't have the same tags applied the same way to the same depth, the same course, etc. Also I have a brand new computer that is very fast, which is kind of throwing this off as well. I only have a handful of tags applied; if I could either get some direction from Braden on how he had applied his tickets, or have Braden perform a quick check with his same setup, that would be great.

[bradenmacdonald]

Thanks! My performance concern is addressed now. I just caught a few more things but hopefully they're relatively straightforward to address.

[tbain]

As per previous comments, moved the _add_counts logic out to the API level just before the return so that the logic to add the usage_counts doesn't interfere with the QuerySet logic until after all the filtering and paging logic is performed. This was necessitated by performance concerns with using something like a QuerySet annotation Case/When adding a multiplicative Big O complexity, so we had to work around that logic and manually add the value derived via Python logic to the list form of the return data rather than the QuerySet form which doesn't support such direct manipulation.

Since we completely moved the counting logic out of the the DB layers up to the API level, this required moving all of the unit testing to the appropriate level and removing it from the lower level unit testing suites. I used AI help to build out as extensive a unit test suite as I could think of in addition to maintaining parity with the tests I had to remove from the lower levels.

[jesperhodge]

Hey @tbain this is amazing work. I tested it for correctness and performance, and I love the very extensive tests - that's just great.

Please regard every comment from me as a nit, they are very minor, and I think should not block the PR from being merged at all. That said, it would be nice to have the improvements for the comments in here; but merging this very soon has priority due to the Verawood release.

Can you make sure to bump the version (just a patch I think, not a minor or major version bump)?

Probably squashing can be done while merging.

I'll approve but final approve should come from Braden or Dave

[jesperhodge]

I love these comments that explain exactly what is expected.

[jesperhodge]

Awesome explanation.

[jesperhodge]

This description sounds good to me, very understandable.

[bradenmacdonald]

Looks great! I'll approve and merge as soon you've addressed the include_counts comment and as many of Jesper's suggestions as you'd like.

[jesperhodge]

Wonderful work!
Assuming the version is bumped to the right version now, this looks ready to me.