build inspection app for known on-board API systems

[hannsadrian]

As I was traveling with the train recently, I remembered this awesome repo and also noticed the wifi-on-ice-portal-client as well as digital-im-regio-portal-client and a few others. From what I can tell they are more or less all using the same icomera api with a few exceptions.

What I think would be helpful is firstly, how these endpoints were reverse engineered or what kind of documentation was used to interact with them. And secondly, it'd be great to merge all of the different libraries into one if they're more or less do the same thing. This would prevent the sort of chaos, that we're noticing regarding public transport information apis and could act as a central place for documentation of all the available on-board information systems. Anyone who is potentially interested in using this data in the future could just go ahead and use the onboard information library to get the most out of every compatible vehicle without using many different libraries.

Furthermore, I would be interested in finding out if there's more than position to this api: https://www.ombord.info/api/jsonp/position/

[derhuerst]

As I was traveling with the train recently, I remembered this awesome repo and also noticed the wifi-on-ice-portal-client as well as digital-im-regio-portal-client and a few others. From what I can tell they are more or less all using the same icomera api with a few exceptions.

Maybe that's the case, I can't tell. I know that there are several Icomera products with different network & software setups in the vehicle.

What I think would be helpful is firstly, how these endpoints were reverse engineered or what kind of documentation was used to interact with them.

I just used the Browser devtools for webpages or mitmproxy for mobile apps to inspect the traffic. Once I got the API endpoints, I observed how the responses change both over time and with different request parameters.

And secondly, it'd be great to merge all of the different libraries into one if they're more or less do the same thing. This would prevent the sort of chaos, that we're noticing regarding public transport information apis and could act as a central place for documentation of all the available on-board information systems.

I agree that this could become a problem, and that overly distributing the knowledge about on-board APIs doesn't help.

But we also need to weigh that against disadvantages: Centralized "toolbelt repos" are more work to maintain (especially long-term), tend to be more heavy-weight, and require a more complex technical setup for testing, publishing, etc.

At least for documentation, discussions and reverse engineering efforts with these APIs, it would be useful to have a central place, much like what WoBike is for bike/scooter sharing APIs, or transport-apis. Are you interested in starting such a project?

For specific implementations of API clients, I'm not so sure, there are just different scopes, numbers of dependencies and programming languages involved.

Anyone who is potentially interested in using this data in the future could just go ahead and use the onboard information library to get the most out of every compatible vehicle without using many different libraries.

Yeah, there's certainly value in having a "Swiss Army knife of on-board APIs". It could, by the way, also augment the geolocation services built into most operating systems with the position data from such APIs. But we're not that far yet.

[hannsadrian]

In my opinion onboard data really has the potential to add value to public transportation on longer journeys. Imagine one app or webapp where you could view all the information (position, wifi status, real time delay, speed) that is available regarding your current journey, no matter if you are in a DB Regio, the ICE or really any vehicle that provides a wifi network.

Certainly, that's a lot of work and an ambitious goal at the moment. However, I think central documentation of those apis is a great first step. Maybe a small node script or webapp attached to the documentation could help contributors submit the kind of information that is available on a vehicle. Someone would just open the app and see data from live-icomera-position, digital-im-regio-portal-client, etc. if available. They could then just create an issue with the data they observed and the according vehicle. I would happily help building the docs and the described tool to expand them.

[derhuerst]

In my opinion onboard data really has the potential to add value to public transportation on longer journeys. Imagine one app or webapp where you could view all the information (position, wifi status, real time delay, speed) that is available regarding your current journey, no matter if you are in a DB Regio, the ICE or really any vehicle that provides a wifi network.

It's not available for Android and iOS (yet), but KDE Itinerary by @vkrause is one working towards that dream. It also unifies my other public transport APIs in order to be your travel assistant.

Certainly, that's a lot of work and an ambitious goal at the moment. However, I think central documentation of those apis is a great first step.

Yeah, let's start such an effort! Maybe we could do this as part of public-transport/transport-apis?

Maybe a small node script or webapp attached to the documentation could help contributors submit the kind of information that is available on a vehicle. [...] They could then just create an issue with the data they observed and the according vehicle. I would happily help building the docs and the described tool to expand them.

Sounds very labor-intensive.

Also: I assume that almost all vehicles of a specific operator (and a certain type, e.g. all buses) have the same on-board system, so it's not worth it checking every single vehicle, but rather discovering all the operators out there. And if there's a new type of on-board system, you can't automatically collect all necessary info to build an API client from it.

But: IMO it would be worth it to have a more streamlined way (an app would be one way) to find such an API endpoint in a wifi; It would automatically check the standard gateway (or a local server mentioned by the local DNS server) for an HTTP API.

[vkrause]

Certainly, that's a lot of work and an ambitious goal at the moment. However, I think central documentation of those apis is a great first step.

Yeah, let's start such an effort! Maybe we could do this as part of public-transport/transport-apis?

Making use of on-board APIs in KDE Itinerary has been lingering on my todo list, so collecting knowledge around that would certainly be interesting for us as well! Doing this alongside (or even inside) the transport APIs repository would seem like the obvious place for this to me too.

I haven't systematically looked into this topic yet, but I should have a few pre-pandemic ICE and Thalys captures to contribute.

[hannsadrian]

Alright, I'll soon start working on a tool that is able to discover (or at least recognize) onboard APIs. Should the endpoint structure eventually just fit in with the public-transport/transport-apis/data directory or a separate one bundled together with the tool?

btw. for the mentioned tool I am planning to create a react app that maybe could be available among with the Hafas clients on onboard-discover.transport.rest or something similar

[hannsadrian]

you can't automatically collect all necessary info to build an API client from it

I fully agree with that. But if a potential documentation contributor is on a new vehicle and found (like you mentioned) the requests in the dev tools, we could offer a small rest client that generates a log file with informations about general data availability (e.g. is there a basic icomera endpoint?) and then also could contain the url which the contributor found among with it's response. This would make it easy for us to at least get some basic information about what a potentially new onboard system contains and maybe do further research.

[hannsadrian]

It would automatically check the standard gateway (or a local server mentioned by the local DNS server) for an HTTP API.

I do not know whether the onboard systems emphasize their api or gateway endpoints and if we have any chance of scanning them. However, checking for a local captive portal (as can be found in almost all public wifis in order to login) should be somewhat possible. At least entering the IP-Address of an onboard system router can most of the time give a hint about the available data and service portal.

[derhuerst]

Alright, I'll soon start working on a tool that is able to discover (or at least recognize) onboard APIs. [...]

btw. for the mentioned tool I am planning to create a react app that maybe could be available among with the Hafas clients on onboard-discover.transport.rest or something similar

I don't think a web app will be able to do that, for the following reasons:

• It can't query the standard gateway, local DNS server, etc. of the WiFi network.
• It can only detect that there is an HTTP server on a certain IP, but it can't inspect what it responds with because of the CORS rules.

I think in practice it won't be very effective TBH.

[derhuerst]

you can't automatically collect all necessary info to build an API client from it

I fully agree with that. But if a potential documentation contributor is on a new vehicle and found (like you mentioned) the requests in the dev tools, we could offer a small rest client that generates a log file with informations about general data availability (e.g. is there a basic icomera endpoint?) and then also could contain the url which the contributor found among with it's response. [...]

Yes, if the API

• has CORS enabled
• is a REST API (as in: when just GET-ing a certain URL, the responses already contain reasonable data)
• returns relevant info without passing parameters

From my experience, what's much more valuable when reverse-engineering these APIs: A network recording, like a pcap file, mitmproxy recording HTTP requests as plain text (example). A nice guide on how to obtain these would IMHO be more helpful.

[derhuerst]

[...] Should the endpoint structure eventually just fit in with the public-transport/transport-apis/data directory or a separate one bundled together with the tool?

Within transport-apis, we have currently documented 3 different types of query-based journey/route planning APIs, but we're planning to document more. I think machine-readable info about on-board APIs would fit in there if we adjust the repo structure a bit! As @vkrause said above:

Doing this alongside (or even inside) the transport APIs repository would seem like the obvious place for this to me too.

I have created public-transport/transport-apis#31 to discuss how to add this info to transport-apis.

[hannsadrian]

Awesome! Having proxy logs seems indeed like the right solution to discover new apis.

To end the discussion here, I think that a kind of tool that tells which known apis it detected in a given network could help us expand the docs even if it does not discover new endpoints. At least knowing that in transportation type x is a basic icomera system or regio-portal etc. with data on x (known) endpoints would be worth it for me. This could enable anyone on a journey to quickly (even from their mobile phone) check what is available and maybe further investigate and add more and more confirmed routes and vehicle types with known systems to the documentation – trying to get the best possible information about coverage and preventing uncertainty like with other public transport apis and systems.

CORS is a real bummer but, at least from my experience of using the icomera api there was no request preventing CORS policy. Maybe that's the case for other systems as well.

[hannsadrian]

Furthermore, we should look into the train station wifis, as described here. Maybe there is also some sort of information available on the access points.

[derhuerst]

CORS is a real bummer but, at least from my experience of using the icomera api there was no request preventing CORS policy. Maybe that's the case for other systems as well.

That would be a serious privacy issue, as any website could track the train you're on.

AFAIK Deutsche Bahn had to disable CORS on their ICE-Portal because of this.

[hannsadrian]

Ah well, that makes a lot of sense. Will look into this once an opportunity arises. We should definitely add CORS availability to the docs.

[hannsadrian]

Shouldn't we consider creating a separate repository for the tool? I think integrating it into live-icomera-position could cause unnecessary confusion 🤔

[derhuerst]

Shouldn't we consider creating a separate repository for the tool? I think integrating it into live-icomera-position could cause unnecessary confusion 🤔

Yes, definitely!

[hannsadrian]

I've started building a tool that queries known onboard apis (adwirawien/onboard-api-discovery) through npm packages published by @derhuerst. The discovery can be expanded by any js code snippets.
Next, I am going to document a way for contributors to submit their output of onboard-api-discovery and create mitmproxy logs for new api discoveries.

Would you (@derhuerst) mind merging two pull requests which add a timeout to the api wrappers? This prevents infinite waiting for a response.

• live-icomera-position#4
• sncf-wifi-portal-client#1