Add option for polygon/linestring in results #823

red-fenix · 2024-07-11T07:42:16Z

At the moment, photon only returns the point of a location, and not the polygon (see #259). This PR will add the option to add the polygon (i.e. geometry) to the Elasticsearch Index and a API parameter polygon to return said polygon. If no polygon exists, the point is returned.

WARNING: This will increase the Elasticsearch Index size! (~575GB for a Planet import).

To enable: add the command line argument -use-geometry-column whilst importing and add &polygon=true to the API call.

lonvia · 2024-07-14T13:42:22Z

I haven't done a full review yet but I do have some general thoughts on the implementation:

This really needs to be implemented for the OpenSearch version because the ES version of Photon is on its way out. Note that the OpenSearch variant does not use mappings.json. It defines its mapping in https://github.com/komoot/photon/blob/master/app/opensearch/src/main/java/de/komoot/photon/opensearch/IndexMapping.java.
As long as the geometry isn't used for lookup, indexes should be disabled on the new field to save a bit of disk space.
We already have a field extent which contains the bounding box. The geometry should replace this field, i.e. once full geometries are enabled, do not save extent and derive the extent from the geometry field when returning a result. We need to keep the centroid because it is not necessarily the geometric centroid of the geometry. (Note that extent was missing from the mapping specification so far. So likely it was in fact saved as a text field, which really is an oversight but can't really be fixed right now without creating an incompatible database version.)
This would be the second optional database feature after add support for structured queries (opensearch only) #815. Before we become too prolific with command-line arguments, I'd lean towards adding a single parameter -extra-db-features which takes a list of features (right now: structured, geometries).
I agree on the introduction of the polygon parameter but would prefer to disable it when the extended geometries are not available in the database. Otherwise there will be an endless stream of bug reports on photon.komoot.io, why the results return a point instead of polygon. You can save the state of the feature in the property table and load it from there on start, see add support for structured queries (opensearch only) #815 for an example on how to do it. This property also comes in handy during updates of the database. It would be useful to have exactly the same behaviour as on import then.
This needs some tests for the import.

Two other considerations come in mind but they are easily deferred to follow-up PRs:

Once we have full geometries, we'd want to use them for reverse lookup. See Discussion for PR to improve accuracy of reverse geocoding #357.
It might be worth to slightly simplify the geometries before importing them, or at least make that an option. Nominatim always keeps the original OSM geometries which sometimes can have a lot more support points than necessary. Simplification might help to further reduce database size.

red-fenix · 2024-07-17T18:40:13Z

I haven't done a full review yet but I do have some general thoughts on the implementation:

This really needs to be implemented for the OpenSearch version because the ES version of Photon is on its way out. Note that the OpenSearch variant does not use mappings.json. It defines its mapping in https://github.com/komoot/photon/blob/master/app/opensearch/src/main/java/de/komoot/photon/opensearch/IndexMapping.java.

Thanks. I will update the PR with the changes in this file soon.

One question though about the 'Elasticsearch is on it's way out': I've been planning to update the Elastic client to the Java API so you can use an existing Elasticsearch cluster instead of the internal one (newer versions of Elasticsearch don't support the Transport client). Is my new PR still a good idea?

As long as the geometry isn't used for lookup, indexes should be disabled on the new field to save a bit of disk space.

Will take this along as well.

This would be the second optional database feature after add support for structured queries (opensearch only) #815. Before we become too prolific with command-line arguments, I'd lean towards adding a single parameter -extra-db-features which takes a list of features (right now: structured, geometries).

Agreed

I agree on the introduction of the polygon parameter but would prefer to disable it when the extended geometries are not available in the database. Otherwise there will be an endless stream of bug reports on photon.komoot.io, why the results return a point instead of polygon. You can save the state of the feature in the property table and load it from there on start, see add support for structured queries (opensearch only) #815 for an example on how to do it. This property also comes in handy during updates of the database. It would be useful to have exactly the same behavior as on import then.

OK. I will make the default to return the polygon when it's available in the index. The option 'polygon=false' will return the centroid instead.

This needs some tests for the import.

Will do.

Two other considerations come in mind but they are easily deferred to follow-up PRs:

Once we have full geometries, we'd want to use them for reverse lookup. See Discussion for PR to improve accuracy of reverse geocoding #357.

I have to look into this issue.

It might be worth to slightly simplify the geometries before importing them, or at least make that an option. Nominatim always keeps the original OSM geometries which sometimes can have a lot more support points than necessary. Simplification might help to further reduce database size.

I will look into this.
Another thing related to this: when someone is searching for a street Nominatim (and thus Photon) returns a street in parts because they are separate OSM id's. I'm still looking for a method to merge multiple ways (i.e. linestrings) into 1 linestring to make sure the whole street is displayed instead of a part (example)

lonvia · 2024-07-21T13:46:14Z

One question though about the 'Elasticsearch is on it's way out': I've been planning to update the Elastic client to the Java API so you can use an existing Elasticsearch cluster instead of the internal one (newer versions of Elasticsearch don't support the Transport client). Is my new PR still a good idea?

We'll drop ES support completely and go with OpenSearch. Note that the OS version already supports an external OpenSearch cluster. The support is just somewhat rudimentary and HTTP-only.

karussell · 2024-08-10T15:18:26Z

Would this PR return a Multipolygon in case for queries like "hamburg"? See the nominatim response.

Currently photon returns a slightly confusing, but correct extent as only a single extent is allowed https://photon.komoot.io/api/?q=hamburg ... or maybe we add a new field extents in the response?

red-fenix · 2024-09-05T10:24:27Z

Would this PR return a Multipolygon in case for queries like "hamburg"? See the nominatim response.

Currently photon returns a slightly confusing, but correct extent as only a single extent is allowed https://photon.komoot.io/api/?q=hamburg ... or maybe we add a new field extents in the response?

Yes, it would return a Polygon like Nominatim does.

red-fenix · 2024-09-05T18:54:14Z

I haven't done a full review yet but I do have some general thoughts on the implementation:

This really needs to be implemented for the OpenSearch version because the ES version of Photon is on its way out. Note that the OpenSearch variant does not use mappings.json. It defines its mapping in https://github.com/komoot/photon/blob/master/app/opensearch/src/main/java/de/komoot/photon/opensearch/IndexMapping.java.

Done

I agree on the introduction of the polygon parameter but would prefer to disable it when the extended geometries are not available in the database. Otherwise there will be an endless stream of bug reports on photon.komoot.io, why the results return a point instead of polygon. You can save the state of the feature in the property table and load it from there on start, see add support for structured queries (opensearch only) #815 for an example on how to do it. This property also comes in handy during updates of the database. It would be useful to have exactly the same behaviour as on import then.

Done

This needs some tests for the import.

Done

I'm looking forward to your response :)

lonvia · 2024-10-29T18:39:58Z

Sorry for the long delay in responding.

I've decided to get a release out first before looking into this again. Because of this there is a minor merge conflict now regarding the dependencies. Also, the opensearch variant doesn't compile because setUpESWithPolygons() is only defined for the elasticsearch variant. Could you quickly fix both issues?

lonvia

Couple of more comments after trying this out. The OpenSearch version currently doesn't work at all. It already fails to import.

The output for the ElasticSearch version displays a wrong projection:

...
"geometry": {
  "coordinates":[9.5227103,47.1395576],
  "type":"Point",
  "crs":{"type":"name","properties":{"name":"EPSG:0"}}
}
...

Photon doesn't supply a CRS at all with the geometry. I'd leave it at that.

lonvia · 2024-10-29T19:39:47Z

src/main/java/de/komoot/photon/SearchRequestHandler.java

+        }
+
+        if (!supportPolygons && (photonRequest.isPolygonRequest() && photonRequest.getReturnPolygon())) {
+            throw new BadRequestException(400, "You're requesting a polygon, but polygons are not imported!");


You need to throw a halt at this point and make sure the content is json formatted, see line 43 above.

lonvia · 2024-10-29T20:44:54Z

src/main/java/de/komoot/photon/SearchRequestHandler.java

+            returnPolygon = photonRequest.getReturnPolygon();
+        }
+
+        if (!supportPolygons && (photonRequest.isPolygonRequest() && photonRequest.getReturnPolygon())) {


This check comes to late. Parameter need to be tested and rejected before the actual search is done (around line 45).

lonvia · 2024-10-29T20:46:55Z

src/main/java/de/komoot/photon/StructuredSearchRequestHandler.java

@@ -51,6 +51,6 @@ public String handle(Request request, Response response) {
            debugInfo = requestHandler.dumpQuery(photonRequest);
        }
 */
-        return new GeocodeJsonFormatter(photonRequest.getDebug(), photonRequest.getLanguage()).convert(results, debugInfo);


I'd expect the same code here as in the SearchRequestHandler (check if available etc.)

lonvia · 2024-10-29T20:47:50Z

src/main/java/de/komoot/photon/ReverseSearchRequestHandler.java

@@ -53,6 +53,6 @@ public String handle(Request request, Response response) {
            debugInfo = requestHandler.dumpQuery(photonRequest);
        }

-        return new GeocodeJsonFormatter(false, photonRequest.getLanguage()).convert(results, debugInfo);
+        return new GeocodeJsonFormatter(false, photonRequest.getLanguage(), photonRequest.getPolygon()).convert(results, debugInfo);


I'd expect the same code here as in the SearchRequestHandler (check if available etc.)

lonvia · 2024-10-29T20:48:53Z

src/main/java/de/komoot/photon/nominatim/NominatimConnector.java

@@ -24,7 +24,7 @@
 public class NominatimConnector {
    private static final Logger LOGGER = org.slf4j.LoggerFactory.getLogger(NominatimConnector.class);

-    private static final String SELECT_COLS_PLACEX = "SELECT place_id, osm_type, osm_id, class, type, name, postcode, address, extratags, ST_Envelope(geometry) AS bbox, parent_place_id, linked_place_id, rank_address, rank_search, importance, country_code, centroid";
+    private static final String SELECT_COLS_PLACEX = "SELECT place_id, osm_type, osm_id, class, type, name, postcode, address, extratags, ST_Envelope(geometry) AS bbox, parent_place_id, linked_place_id, rank_address, rank_search, importance, country_code, centroid, geometry";


Geometries can become very large, so should only be added to the returned columns when really needed.

lonvia · 2024-10-29T20:54:54Z

src/main/java/de/komoot/photon/query/PhotonRequestFactory.java

+
+        if (webRequest.queryParams("polygon") != null) {
+            request.setPolygonRequest(true);
+            request.setReturnPolygon(Boolean.parseBoolean(webRequest.queryParams("polygon")));


We'd need a full parsing function here that throws on unknown values. Question is if we should allow 0/1, yes/no besides true/false.

Your Name added 6 commits June 18, 2024 17:24

feat: initial support for geometry column

9ec20d4

feat: add unit test

5a7fb1a

feat: add geometry column as an commanline argument

31e2111

Merge branch 'master' into feature/geometry_support

97899e8

bugfix: undo intellij magic

031d962

bugfix: minor build bugs

152c2d1

red-fenix changed the title ~~Add option for polygons in results~~ Add option for polygon/linestring in results Jul 11, 2024

feat: add polygon as an api request instead of a default

5d1b854

feat: add unit tests and process remarks of @lonvia

d544bb3

Your Name added 2 commits September 5, 2024 21:03

bugfix: minor fixes

41c20c4

bugfix: typo

cef09d7

lonvia reviewed Oct 29, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add option for polygon/linestring in results #823

Add option for polygon/linestring in results #823

red-fenix commented Jul 11, 2024 •

edited

Loading

lonvia commented Jul 14, 2024

red-fenix commented Jul 17, 2024 •

edited

Loading

lonvia commented Jul 21, 2024

karussell commented Aug 10, 2024

red-fenix commented Sep 5, 2024

red-fenix commented Sep 5, 2024 •

edited

Loading

lonvia commented Oct 29, 2024

lonvia left a comment

lonvia Oct 29, 2024

lonvia Oct 29, 2024

lonvia Oct 29, 2024

lonvia Oct 29, 2024

lonvia Oct 29, 2024

lonvia Oct 29, 2024

Add option for polygon/linestring in results #823

Are you sure you want to change the base?

Add option for polygon/linestring in results #823

Conversation

red-fenix commented Jul 11, 2024 • edited Loading

lonvia commented Jul 14, 2024

red-fenix commented Jul 17, 2024 • edited Loading

lonvia commented Jul 21, 2024

karussell commented Aug 10, 2024

red-fenix commented Sep 5, 2024

red-fenix commented Sep 5, 2024 • edited Loading

lonvia commented Oct 29, 2024

lonvia left a comment

Choose a reason for hiding this comment

lonvia Oct 29, 2024

Choose a reason for hiding this comment

lonvia Oct 29, 2024

Choose a reason for hiding this comment

lonvia Oct 29, 2024

Choose a reason for hiding this comment

lonvia Oct 29, 2024

Choose a reason for hiding this comment

lonvia Oct 29, 2024

Choose a reason for hiding this comment

lonvia Oct 29, 2024

Choose a reason for hiding this comment

red-fenix commented Jul 11, 2024 •

edited

Loading

red-fenix commented Jul 17, 2024 •

edited

Loading

red-fenix commented Sep 5, 2024 •

edited

Loading