🐛📄 – Is `vehicle_locations.pattern_id` supposed to be the planned/scheduled pattern ID or an observed pattern ID? How does it map to GTFS?

[lauriemerrell]

Describe the problem

This is a little theoretical but I was working on mapping GTFS RT vehicle positions to TIDES vehicle locations and I realized that there is not technically a value for pattern_id in GTFS RT (the closest is presumably route_id)

But -- this made me realize that I am not 100% sure of the intent of pattern ID in TIDES vehicle locations. Do we think it's supposed to be the scheduled pattern ID (determined in advance) or is it supposed to be inferred based on observed behavior, or perhaps a mix of both?

I'm specifically wondering about the following cases:

• Trip is a "normal" scheduled trip but in practice it takes a detour -- should pattern ID be the scheduled pattern ID or the observed pattern ID?
• Trip is observed but not scheduled. Should pattern ID be null (because not scheduled) or inferred?

The docs for this field reference GTFS shape_id which feels not right -- shape ID does not determine a stop pattern, shape ID is for display purposes.

Possible Solutions

Clarify intent of this field:

• GTFS mapping: The concept of a distinct stop pattern ID does not technically exist in GTFS schedule or in GTFS RT (vehicle positions) -- perhaps consider just saying that this field does not map to GTFS and/or consider adding route_id to vehicle locations as optional (because currently if the source data has this data -- like GTFS RT vehicle positions does -- there technically isn't a place to put it in TIDES)
• Observed vs. scheduled: Clarify whether this is supposed to be the planned/scheduled pattern ID for this trip or if it as post-hoc observed pattern ID (other fields in this table have scheduled vs. observed designations)

Additional context

Mapping from GTFS RT is probably more relevant for demo or example use cases, and I don't personally have a ton of experience with what identifier (route vs. pattern) would be more common or expected in an AVL stream. So perhaps the answers are clearer for people with more experience looking at raw AVL data

[gabriel-korbato]

The field vehicle_locations.pattern_id is supposed to be the observed pattern ID. Most AVL systems include this in their data model, as it relates to and identifies the ordered list of geocoded stops the AVL hardware is tracking. GTFS does not include the concept of pattern, although some scheduling software and many agencies adopt the convention of mapping shape_id to pattern_id for convenience. (Technically, you could have different patterns serving a different list of stops along the same exact shape, but we often see that two overlapping shapes are made, one per pattern, when this convention is followed.)

To know the scheduled pattern of a trip, the observed trip should be joined to the scheduled trip using trip_id_scheduled. If this links to a proper vehicle schedule, you will have the pattern in that system. If this links to GTFS (which is a scheduling format meant to communicate timetables to the public, not for vehicle or crew scheduling), you unfortunately won't have the scheduled pattern, unless you know that the agency follows the shape_id convention.

If trip is observed but not scheduled (sometimes referred to as a run-as-directed trip), it should still have a pattern_id, provided it has been given by the AVL system or inferred from raw data by some data processor.

[lauriemerrell]

Appreciate the response @gabriel-korbato, with that all in mind I think it might be useful to:

• Remove the reference to shape_id (or clarify that that is often observed in practice but is not technically the same concept)
• Consider adding route_id which is populated in GTFS and seems likely to be desirable to have here (but I guess it's theoretically available via trip_id at least for scheduled trips and I know sometimes there are concerns about the volume/size of AVL data so maybe this is intentionally omitted, it just felt a bit odd to drop data that is present on GTFS-RT vehicle positions)
• Clarify what we expect this to be a foreign key to -- since pattern ID does not exist in GTFS schedule, I think this is technically kind of a dangling reference? I.e. in TIDES we have vehicles, devices, operators as supporting tables since those concepts are referenced elsewhere -- do we need to have a supporting patterns table?

^ And on that 3rd point, @chrisyamas it might be worth doing a double-check review to see if there are any other identifiers referenced in TIDES spec that don't technically refer to anything in GTFS or the other existing supporting tables?

[gabriel-korbato]

* Remove the reference to `shape_id` (or clarify that that is often observed in practice but is not _technically_ the same concept)

I agree, and I think implementers would benefit from some guidance/caveat about GTFS's shape_id.

* Consider adding `route_id` which is populated in GTFS and seems likely to be desirable to have here (but I guess it's theoretically available via `trip_id` at least for scheduled trips and I know sometimes there are concerns about the volume/size of AVL data so maybe this is intentionally omitted, it just felt a bit odd to drop data that is present on GTFS-RT vehicle positions)

It's not in stop_visits because it's the same for all stop visits of the operated trip, and you can obtain the route by looking it up in trips_performed. Taking a step back, pattern_id and vehicle_id should maybe be dropped from the stop_visits table for the same reason that route_id isn't already there. @jaygordon and @botanize , what do you think?

* Clarify what we expect this to be a foreign key to -- since pattern ID does not exist in GTFS schedule, I think this is technically kind of a dangling reference? I.e. in TIDES we have vehicles, devices, operators as supporting tables since those concepts are referenced elsewhere -- do we need to have a supporting patterns table?

I agree that this should be clarified. It's a foreign key to the pattern from the schedule, in whatever format it exists, e.g. Hastus, Trapeze, Cleverworks, etc. GTFS stands out as the format that doesn't have this (because it was designed to communicate timetables to the public rather than for operations).

A lookup table of patterns has been discussed in calls from the early days of TIDES, for the reason @lauriemerrell gives. It hasn't been included because it pertains to schedules rather than operational observations. If we added it, it would really be just to extend GTFS for those TIDES implementers using that as the scheduling source. Maybe that's out of scope for TIDES and should be GTFS extension. Some agencies have taken the initiative of adding patterns to their GTFS, e.g. the MBTA.

[lauriemerrell]

Re: the second point above, this was referring to vehicle_locations not stop_visits. I think it is an interesting question though about how normalized we want things to be.... I would be curious what % of anticipated use cases for stop_visits would require a lookup of route_id or pattern_id. If we think they'll be needed I would advocate for keeping them even if it's technically redundant.

(Also -- since route_id is present on trips performed, I guess as-written the spec anticipates that you will look up route ID from the schedule using trip ID if you're creating trips performed based on vehicle locations....? I also wonder if sometimes people would want to produce trips_performed based on stop_visits on the logic that trips are performed if they have X% of relevant stop visits? In which case it might be helpful to have route_id on the upstream tables...)

[botanize]

I think that route_id is not in stop_visits because it's a property of the trip, and the availability of comprehensive schedule data (GTFS or any other source) is a core assumption of TIDES—I don't think trips_performed is supposed to play that role.

I'm a bit uncomfortable with requiring a join from stop_visits to trips_performed for observed (not scheduled) details like vehicle_id. While technically trips_performed is not an aggregation table, a version of it can be easily produced from an aggregation of stop_visits, and I'd like to keep it that way.

[lauriemerrell]

I guess @botanize I am confused about your comment re: trips not in the schedule -- how would their route be populated, especially if trips performed is a summary of stop visits?

[botanize]

I didn't mean trips not in the schedule, I meant observed features of a trip as opposed to scheduled features, e.g., vehicle assignment is an observed feature of a trip, whereas trip_id_scheduled or route_id is a scheduled feature of a trip.

[lauriemerrell]

Ah ok, but regardless, I think a sensible aggregation pattern would be vehicle_locations --> stop_visits --> trips_performed. vehicle_locations has pattern_id, stop_visits has pattern_id, trips_performed has both pattern_id and route_id. I'm curious what the source for route_id would be on trips_performed in this case, particularly for trips that were not scheduled. Like is this a dangling reference of some kind, as noted in the ticket it seems like pattern_id is sort of a dangling reference already.