@nyurik I noticed this was causing absolutely horrific tail latency for tiles. We were using .mbtiles with no index on map or tiles and this was causing >1s query times on and off. Surprisingly SQLite was still able to deliver some good results on-and-off. But there was a LARGE variation in query times.

I'd be happy to take this on!

Thanks @Auspicus , that would be awesome!

It looks like this feature more or less exists inside the mbtiles validate tool already. But the martin server does not warn the user when their MBTiles do not contain indices which could lead to sub-par performance.

I'm thinking the following could be useful:

• warnings when this is the case: easily ignored and not a breaking change but will be useful for alerting users of potential performance issues: "Using .mbtiles without an index on the underlying data can result in slower queries in most cases. It's encouraged to index either the map or tiles table."
• a way to not serve MBTiles that are lacking indices via config (ie. config.mbtiles.ignore_tiles_without_index) which defaults to false in case the user is dynamically uploading tiles

This could slow down the startup slightly and might require another config flag to turn off in the case the user wants to skip this check. Is it OK to make concessions on the startup time in the interest of ensuring a good experience with the server?

So to summarize:

• config config.mbtiles.skip_mbtiles_index_check (default: false) - avoids running this check on startup
• config config.mbtiles.ignore_mbtiles_without_index (default: false) - ignores .mbtiles without an index
• warning on startup when .mbtiles file detected with no index (unless it's skipped via previous config)

Thoughts? @nyurik

Thx, I agree that something like this is needed. I am not a big fan of tons of bools, so perhaps something like this?

validate =
  - skip -- do not validate
  - fast -- quickly check the file (default)
  - thorough -- do a slow check of everything

on_invalid =
  - warn -- print warning message, and abort if the error is critical  (default)
  - ignore_source -- skip this source
  - abort -- do not start Martin on any warnings

TODO

• Perhaps find a better name for on_invalid because the file might be valid but inefficient (no index)

Config structure

We currently have this structure:

mbtiles:
  paths:
    # scan this whole dir, matching all *.mbtiles files
    - /dir-path
    # specific mbtiles file will be published as mbtiles2 source
    - /path/to/mbtiles.mbtiles
  sources:
    # named source matching source name to a single file
    mb-src1: /path/to/mbtiles1.mbtiles

So we may need to be able to add both validate and on_invalid keys to 3 places - to the "root" (overrides the default ones), and to each place a path string is used at the moment:

mbtiles:
  # overrides built in defaults
  validate: skip
  on_invalid: abort
  paths:
    # scan this whole dir, matching all *.mbtiles files
    - path: /dir-path
      # applies to all files in the dir
      validate: skip
      on_invalid: abort
    # specific mbtiles file will be published as mbtiles2 source
    - path: /path/to/mbtiles.mbtiles
      # applies to the given file
      validate: skip
      on_invalid: abort
  sources:
    # named source matching source name to a single file
    mb-src1:
      path: /path/to/mbtiles1.mbtiles
      # applies to the named source
      validate: skip
      on_invalid: abort

CLI arguments

The same values can be used from the CLI: --validate-mbtiles skip --on-invalid-mbtiles abort

@nyurik I've made a start here, haven't gotten around to adding the config to each individual source / path yet, but the integration tests seem to have caught an issue with the tests/fixtures/webp.mbtiles file? It appears the tile_data column doesn't have the expected BLOB type: https://github.com/maplibre/martin/actions/runs/13342149377/job/37268254989#step:10:217

If you get a chance to review the PR so far, I'd love to know if I'm going in the right direction.

Cheers!

If you get a chance to review the PR so far, I'd love to know if I'm going in the right direction.

Not who you asked but chiming in anyway.
Seems pretty solid. 👍🏻

What would be really helpful for this is to add an integration test here.
Adding integration tests prevents features from breaking/being broken in the future.

Slightly offtopic (read not a reuqirement, but closely related work):
Apparently, the shown error message could use a slight pass (Invalid data format for MBTile file tests/fixtures/mbtiles/webp.mbtiles does not really tell our users what is wrong..).