Prioritize followed locations when getting new labeling tasks

[josh-chamberlain]

Context

We have a logic tree when we Get next annotation—we favor a user's selected batch. We also have a bunch of followed locations which don't directly factor into which annotation task is served next.

• Blocked by Dev data-sources-app#885

Proposal

• Let's prioritize locations in the GET logic for labeling tasks
  • nice to have: especially prioritize the task if the current user has followed a location
• Requires Add "location" and "agency" to batch creation #260
  • This is what allows us to have a location to sort on in the first place

Sorting

1. URLs in a selected batch
2. URLs which have existing human- or robo- labels for a location followed by the user, regardless of the location of the batch.
3. URLs in batches which have a location followed by the user.
4. Other URLs which already have human labels.
5. Oldest URLs.

I'd call this a type of "sorting". In an ideal world, we'd be able to display this sorting in-app...in the meantime, we can just tell people about it when they ask by documenting it on the get next annotation task endpoint.

[maxachis]

So the wrinkle in this case is the two-database structure biting us once again -- users and follows of locations currently exist only in the data sources app/database, and not in the source manager app/database.

I have to keep reminding myself that the two-database structure seemed like a good idea at the time, even if it's becoming increasingly questionable and we're migrating more and more logic to SM.

My suspicion is that we'll need to figure out some sort of solution for our multi-database structure, but that's quite a task, and I'd like to delay that transition as long as we can, and hence deprioritize things like this.

What do you think, @josh-chamberlain?

[josh-chamberlain]

@maxachis I think we could GET and cache (even hourly or daily would be wonderful) a list of followed locations like we display on the create batches view; we don't need the followed locations in the SM database. As a matter of fact, I think info about which locations have followers would stay in the data sources database / app after the fact; maybe not, but since we can easily query it now, I don't think it's a blocker.

In any case I will note this in our issue about the databases: Police-Data-Accessibility-Project/data-sources-app#868

[maxachis]

@josh-chamberlain The other thing I'll say is that, at present, there's a few cases where we could reasonably prioritize followed locations:

1. A collector like AutoGoogler started a collection that was explicitly associated with a location, so the URLs from that batch are associated with a location.
2. One of our auto labelers or users associated a given URL with that particular location.
3. One of our auto labelers or users associated a given URL with a particular agency that is associated with that particular location.

The more interesting problem is that we now have multiple means of determining which URL to prioritize for a user. To wit:

1. At a base level, we prioritize URLs in the order they were added to the database.
2. For URLs that have a higher number of annotations, we prioritize those to incentivize them getting to agreement.
3. Now we're looking at adding a third prioritization, which is conditional on the user following certain locations.

This is a novel problem. We've previously added logic to filter on batches, but we're not, strictly speaking, filtering here as much as changing how the URLs presented are sorted, and I'll need to do some noodling to think about how I can do that in a manner that doesn't slow things down for the end user.

But there's slightly more hinky problem which is, when determining between URLs with a high number of annotations and URLs associated with locations that the user follows, which do we prioritize?

[josh-chamberlain]

@maxachis this is a case where we have a logic tree that should be documented! Let's sort them like in the updated issue

[maxachis]

This is blocked pending the dev->main PR for data sources app