add pagation to landing page

[tacaswell]

The landing page should be able to display older requests via browsing.

I have a first pass at this working at : https://github.com/tacaswell/srtracker/tree/main_page_pagation

It needs some designer attention

[Mr0grog]

I just pulled down the last 24 hours of requests (as of 3:15pm PST on December 6th) to see what happens, and it doesn't seem like the API actually returns SRs in order. Out of 1189 SRs, 1179 are out of order w/r/t requested_datetime and 14 are out of order w/r/t updated_datetime (but that's actually just one that's off and 13 that are shifted because of it).

Since we are currently sorting the landing page by requested_datetime, we might want to flip that around and sort by updated_datetime, since that seems relatively in order-ish. We should probably change the title of that section of the page from "Recent Service Requests" to "Recently Updated Service Requests" or just "Updated Service Requests," too.

[danxoneil]

So if I'm reading this issue correctly, two things will happen if this change is implemented:

-- pagination, so that users can browse through pages of service requests
-- all service requests will be displayed on the landing page according to the last time it was updated (rather than the current display, which is according to the date/time it was created)

This seems like a really great change, with a number of benefits:

-- will float the items that have "FedEx view" changes associated with them to the top of the interface as they happen. This would be cool. I have actually never seen an item with a multiple steps in the wild (and I've clicked on a few service requests since launch) :-)
-- will make the landing page more interesting, allowing for users to browse through time and increase the chance they will see interesting things on http://servicetracker.cityofchicago.org/

I am going to see if @aaronsalmon has an interest in prettying up the pagination links...

[Mr0grog]

Keep in mind that, if we do this, we are relying on a "just happens to work" feature of Chicago's API. There is no guarantee that results continue to come in order in the future. That said, it's probably reasonable to assume this is the case as long as API operates on the same foundations it currently does (though those foundations could be changing sometime soon, given last year's RFP about 311).

[tacaswell]

I am not sure what you mean by "FedEx view". The page is still static (once it is served) so you will have to reload the page to see the changes.

There is the code in here (I think) to cache a local copy of the data for use with the email notification. Maybe the main page should be changed to pull from that instead of directly from the api so we can guarantee the order, but I don't have a good idea of how much work that would be. This also seems like a match to nosql databases, but again I am not sure how much work that would be/how hard it would be to get into production. (The issues that I immediately see with this is we would have to work out how to make sure updates to the cities DB are correctly applied to the cached copy, and we would ended up mirroring the full history which may get huge. On the other hand, we could just keep a rolling cache of the last 10 days or so for the main page which could probably be done in memory.)

Built on top of this branch is https://github.com/tacaswell/srtracker/tree/sr_code_filter which adds a drop down to filter to a single type.

[Mr0grog]

I am not sure what you mean by "FedEx view".

The term "FedEx View" was how we referred to the part of the SR details page where all the "notes," or, basically, the detailed history of the SR is listed. The idea was that you should have an idea of the status of an SR that is as detailed as what you get when tracking a FedEx package.

There is the code in here (I think) to cache a local copy of the data for use with the email notification.

Nope, there is no caching. The only thing stored in a database are the date of the last update and a table of subscriptions. Subscriptions are basically just sets of:

1. a notification method (always email ATM, but other methods are pluggable) +
2. notification address (i.e. e-mail address, but could be something different for other methods, like phone # for SMS) +
3. the ID of the SR being subscribed to
4. a unique key for the subscription (used for unsubscribe links)

So there's not really any caching that we can take advantage of. It would be great if there was, but that would be a HUGE cache (even if we only cached, say, the last ten days). We did a lot of this kind of stuff earlier in the project; IIRC the city receives about 3000 SRs/day on weekdays and 1000 SRs a day on weekends. That said, the numbers might be a fair bit smaller for Open311 since only a very limited subset of SRs are exposed.

[Mr0grog]

FYI you can see all the stuff in the database in update/models.py: https://github.com/codeforamerica/srtracker/blob/master/updater/models.py

[Mr0grog]

I've let this one lie for a bit and I'm sorry about that. As @danxoneil noted, this'd be incredibly awesome to have, so I want to put it in. However, since it relies on non-standard behavior that's somewhat unique, I think we should put it behind a configuration flag… maybe PAGINATE_FRONT_PAGE or similar?

@tacaswell, if you want to make those changes, that's great. If not, I'll go ahead and add the configuration and merge it in. Let me know.

[tacaswell]

@Mr0grog I am super swamped at the moment. I will try to get to it tonight, but I have no problem with you doing it if you get sick of waiting on me.

[Mr0grog]

No worries. I won't have time to get to it until tonight, either.

[tacaswell]

@Mr0grog Now that I look at this, I am not quite sure how to do it cleanly.

[Mr0grog]

No worries; I'll try and do it today.

[Mr0grog]

Just pushed these in the main_page_pagination branch. https://github.com/codeforamerica/srtracker/tree/main_page_pagination

As noted earlier, still needs a little bit of work on the UI so it looks nicer.

[danxoneil]

Thanks for doing this, @Mr0grog. So I'm clear, when you say, "since it relies on non-standard behavior that's somewhat unique", you mean the fix for that is "needs a little bit of work on the UI so it looks nicer", right?

Is there anything else involved? I just want to be clear, because we are doing a project where we might be able to address that...

[Mr0grog]

when you say, "since it relies on non-standard behavior that's somewhat unique", you mean the fix for that is "needs a little bit of work on the UI so it looks nicer", right?

No, sorry, those are entirely orthogonal issues.

The "fix" for the bit about Chicago's "non-standard behavior that's somewhat unique" is making paging an opt-in feature, which is what I did in the last commit. Basically, if you stick with the old configuration option called MAX_RECENT_SRS, you won't get paging, but if you use SRS_PAGE_SIZE, you will get paging. Both the behavior and UI of the front page will change slightly as a result.

The other issue is that the UI needs a little bit of cleanup (i.e. it's not very pretty right now), which @tacaswell noted earlier in the thread.