Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "extract" subcommand to mbtiles tool #1725

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

NicoManderySWM
Copy link

This PR adds a "extract" subcommand to the mbtiles tool, to write the tiles contained in a mbtiles archive to the filesystem or to a object store.

While I am not sure if this feature is still in scope for the mbtiles tool, we where in need of a feature like this to switch our rendering process to martin while still being able to deploy the rendered tiles directly to a object store. Maybe the naming of this subcommand is not perfect as it does a bit more than just extracting, but due to lack of a better naming idea I went with this name for now ;)

The implementation is based on the object_store crate of the Apache Arrow project and brings support for Amazon S3 (and compatible alternatives), Azure BlobStore and Google Cloud Store. Local file access is also done through the abstraction of this crate.

As configuration options for the various object store types differs, the CLI interface of the new subcommand allows passing options to the object_store crate using key-value pairs. I tried to document this in the commands help texts and to provide links to upstream documentation.

When the object-store type supports Content-Type and Content-Encoding metadata, these are set accordingly to the contents of the mbtiles archive.

The key / filename of the single tiles is specified using a string template to provide a maximum of flexibility for the user.

@nyurik
Copy link
Member

nyurik commented Mar 7, 2025

Thanks! I'll look at it in the morning, but this is definitely something we should support... but "how" is slightly nuanced.

There are several use cases, each somewhat different:

  • copy a lot of tiles from one backend to another. For this, we have 'martin-cp' - I think it is much better suited for various bulk operations, while supporting all backends and merging. At the moment, it only supports mbtiles as the destination, but can be extended
  • debugging - dump a single tile from mbtiles, possibly also decoding it - this is more in line with mbtiles util
  • Perhaps a new debugging interface or a cli util to get a tile from any backend... this is less certain, but it could be good to support all

@CommanderStorm

This comment was marked as off-topic.

@NicoManderySWM
Copy link
Author

NicoManderySWM commented Mar 10, 2025

I placed this in the mbtiles tool as this functionality only operates on pre-rendered mbtiles archives. As I understood martin-cp operates on a martin configuration file.

PMTiles is certainly a better way to deploy static tiles, but the billing argument can be less severe depending on the usecase - for example when tiles are only rendered for smaller regions (like our original usecase), or when non-cloud services are used. ... and yes, PMTiles is the better solution to deploy to the big cloud providers ;)

@nyurik
Copy link
Member

nyurik commented Mar 10, 2025

martin-cp should appear very similar to martin itself as far as CLI goes -- see docs. In other words, martin-cp adds two required params: --source and --output-file -- the rest are the same params as in martin, like postgres://... would auto-add PG sources, and foo.mbtiles would add an mbtiles source. If you want an ability to extract all the tiles from mbtiles, we may want something like this:

martin-cp  \
           --output-dir /path/to/dir     \  # <-- instead of --output-file
           --mbtiles-type normalized     \  # optional
           "--bbox=-180,-90,180,90"      \  # optional
           --min-zoom 0                  \ 
           --max-zoom 10                 \
           --source source_name          \  # <-- same name as the mbtiles file next line
           /path/to/source_name.mbtiles

This will keep the overall structure the same, give ability to combine multiple sources as neeeded, and will not introduce any mbtiles-specific interface

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants