Support for pmtiles S3 urls #1477
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
This seems to be almost ready! The biggest challenge is ... sadly... how to test it? I wonder if we should set up some public s3 bucket with some small dummy pmtiles and use it in all the testing? @birkskyum @louwers any thoughts on where we can place it? P.S. please rebase |
|
https://pmtilestest.s3.eu-central-1.amazonaws.com/cb_2018_us_zcta510_500k.pmtiles This is in a MapLibre owned bucket. |
|
@louwers why is a public readonly s3 bucket with only a small downloadable is a bad idea? Seems like it would be no different than exposing that same bucket with the http interface, and it would let us test s3 API |
|
@nyurik I misunderstood and removed my earlier comment, I thought we need a S3 bucket for various kinds of test assets. Please use the link I provided! :) |
|
@louwers we do want to upload a bunch of test assets to an s3 bucket (by an admin), but that bucket would be used as a readonly source when running CI tests |
|
@louwers the link you posted above - do you have a full |
|
That would be |
added 8 commits
November 12, 2024 11:12
alamminsalo
force-pushed
the
pmtiles_s3
branch
from
4b5196d to
5a18b12
Compare
|
I migrated to using pmtiles aws-s3-async feature, which uses aws-sdk-s3 and aws-config libraries. Edit: also added tests for the S3 backed based on your suggestions |
|
The testcase for
You likely need the same environment variables here: martin/.github/workflows/grcov.yml Line 62 in 321af56 |
This comment was marked as resolved.
This comment was marked as resolved.
# Conflicts: # Cargo.lock # martin/src/pmtiles/mod.rs # martin/tests/pmt_server_test.rs
|
Seems that even after adressing 94f8d43, there is one other issue that prevents this being merged. I or @alamminsalo will need to investigate where this dying is coming from. |
for more information, see https://pre-commit.ci
Co-authored-by: Frank Elsinga <[email protected]>
…bre#1706) Document `null` as less confusing than `~`
…aplibre#1704) Bumps the all-cargo-version-updates group with 2 updates: [clap](https://github.com/clap-rs/clap) and [flate2](https://github.com/rust-lang/flate2-rs). Updates `clap` from 4.5.30 to 4.5.31 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/releases">clap's releases</a>.</em></p> <blockquote> <h2>v4.5.31</h2> <h2>[4.5.31] - 2025-02-24</h2> <h3>Features</h3> <ul> <li>Add <code>ValueParserFactory</code> for <code>Saturating<T></code></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/blob/master/CHANGELOG.md">clap's changelog</a>.</em></p> <blockquote> <h2>[4.5.31] - 2025-02-24</h2> <h3>Features</h3> <ul> <li>Add <code>ValueParserFactory</code> for <code>Saturating<T></code></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/clap-rs/clap/commit/acf9abbc1b564cac24ed45b1b5906c90d688f4f0"><code>acf9abb</code></a> chore: Release</li> <li><a href="https://github.com/clap-rs/clap/commit/9186a1872c64cc912e553adbcc5f693206e4f013"><code>9186a18</code></a> docs: Update changelog</li> <li><a href="https://github.com/clap-rs/clap/commit/233c316d1145be12c95eb05306575d42a44e154c"><code>233c316</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/5926">#5926</a> from sorairolake/feature/value-parser-factory-for-sa...</li> <li><a href="https://github.com/clap-rs/clap/commit/13931a25e704fadcaa33cda45e12f6544a023588"><code>13931a2</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/5923">#5923</a> from Reverier-Xu/master</li> <li><a href="https://github.com/clap-rs/clap/commit/536e29fb85919f23e8551135ae448fd92326159f"><code>536e29f</code></a> feat(builder): Add <code>ValueParserFactory</code> for <code>Saturating\<T></code></li> <li><a href="https://github.com/clap-rs/clap/commit/45ed71c0f235feb65d23568775e16b1f34c9f8d4"><code>45ed71c</code></a> chore: Avoid using gen for rust 2024 preserved keyword</li> <li><a href="https://github.com/clap-rs/clap/commit/5029bb3d11809403cb9c1b37f970a8625aee56cd"><code>5029bb3</code></a> chore: Avoid using <code>gen</code> for rust 2024 preserved keyword</li> <li><a href="https://github.com/clap-rs/clap/commit/8a1d59bbb068b82fe9ae55ff613a3a6927d3b964"><code>8a1d59b</code></a> chore(deps): Update Rust Stable to v1.85 (<a href="https://redirect.github.com/clap-rs/clap/issues/5921">#5921</a>)</li> <li><a href="https://github.com/clap-rs/clap/commit/9caee534e05a16b223d4d4d7f94e5c1717c0b6dc"><code>9caee53</code></a> docs(changelog): Clarify 5.0.0</li> <li><a href="https://github.com/clap-rs/clap/commit/cb2352f84a7663f32a89e70f01ad24446d5fa1e2"><code>cb2352f</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/5918">#5918</a> from epage/test</li> <li>Additional commits viewable in <a href="https://github.com/clap-rs/clap/compare/clap_complete-v4.5.30...v4.5.31">compare view</a></li> </ul> </details> <br /> Updates `flate2` from 1.0.35 to 1.1.0 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/rust-lang/flate2-rs/releases">flate2's releases</a>.</em></p> <blockquote> <h2>1.1.0</h2> <h2>What's Changed</h2> <ul> <li>Fix cfgs by <a href="https://github.com/kornelski"><code>@kornelski</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/441">rust-lang/flate2-rs#441</a></li> <li>update CI to use new wasi target by <a href="https://github.com/oyvindln"><code>@oyvindln</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/444">rust-lang/flate2-rs#444</a></li> <li>Implement <code>Clone</code> for <code>CompressError</code> and <code>DecompressError</code> by <a href="https://github.com/mkrasnitski"><code>@mkrasnitski</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/445">rust-lang/flate2-rs#445</a></li> <li>Update LICENSE-MIT by <a href="https://github.com/maximevtush"><code>@maximevtush</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/448">rust-lang/flate2-rs#448</a></li> <li>feat: replace custom u16 le parser with existent rust method by <a href="https://github.com/CosminPerRam"><code>@CosminPerRam</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/450">rust-lang/flate2-rs#450</a></li> <li>Fix CI by <a href="https://github.com/Byron"><code>@Byron</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/449">rust-lang/flate2-rs#449</a></li> <li>Do not use cloudflare-zlib-sys 0.3.4 by <a href="https://github.com/jongiddy"><code>@jongiddy</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/451">rust-lang/flate2-rs#451</a></li> <li>Increase minimum compiler version to 1.67 by <a href="https://github.com/jongiddy"><code>@jongiddy</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/452">rust-lang/flate2-rs#452</a></li> <li>deps: bump miniz_oxide to 0.8.4 by <a href="https://github.com/CosminPerRam"><code>@CosminPerRam</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/459">rust-lang/flate2-rs#459</a></li> <li>deps(dev): update rand to 0.9 by <a href="https://github.com/CosminPerRam"><code>@CosminPerRam</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/458">rust-lang/flate2-rs#458</a></li> <li>docs: fix spelling mistake in flate2::zlib::write::ZlibDecoder by <a href="https://github.com/CosminPerRam"><code>@CosminPerRam</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/455">rust-lang/flate2-rs#455</a></li> <li>feat: remove explicit default impls by <a href="https://github.com/CosminPerRam"><code>@CosminPerRam</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/457">rust-lang/flate2-rs#457</a></li> <li>Change private bounds from <code>R: Read</code> to <code>R: BufRead</code> by <a href="https://github.com/JonathanBrouwer"><code>@JonathanBrouwer</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/453">rust-lang/flate2-rs#453</a></li> <li>feat: replace manual copy loop with rust-provided function by <a href="https://github.com/CosminPerRam"><code>@CosminPerRam</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/456">rust-lang/flate2-rs#456</a></li> <li>feat: reduce CrcReader::sum calls in GzEncoder::read_footer by <a href="https://github.com/CosminPerRam"><code>@CosminPerRam</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/454">rust-lang/flate2-rs#454</a></li> <li>feat: remove redundant if guard on option value match by <a href="https://github.com/CosminPerRam"><code>@CosminPerRam</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/464">rust-lang/flate2-rs#464</a></li> <li>feat: add Error associated type in zio::Ops to handle multiple errors by <a href="https://github.com/CosminPerRam"><code>@CosminPerRam</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/461">rust-lang/flate2-rs#461</a></li> <li>feat: remove explicit redundant lifetime by <a href="https://github.com/CosminPerRam"><code>@CosminPerRam</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/465">rust-lang/flate2-rs#465</a></li> <li>feat: impl From<!-- raw HTML omitted --> to MZFlush by <a href="https://github.com/CosminPerRam"><code>@CosminPerRam</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/462">rust-lang/flate2-rs#462</a></li> <li>upgrade zlib-rs to version <code>0.4.2</code> by <a href="https://github.com/folkertdev"><code>@folkertdev</code></a> in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/466">rust-lang/flate2-rs#466</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/mkrasnitski"><code>@mkrasnitski</code></a> made their first contribution in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/445">rust-lang/flate2-rs#445</a></li> <li><a href="https://github.com/maximevtush"><code>@maximevtush</code></a> made their first contribution in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/448">rust-lang/flate2-rs#448</a></li> <li><a href="https://github.com/CosminPerRam"><code>@CosminPerRam</code></a> made their first contribution in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/450">rust-lang/flate2-rs#450</a></li> <li><a href="https://github.com/JonathanBrouwer"><code>@JonathanBrouwer</code></a> made their first contribution in <a href="https://redirect.github.com/rust-lang/flate2-rs/pull/453">rust-lang/flate2-rs#453</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/rust-lang/flate2-rs/compare/1.0.35...1.1.0">https://github.com/rust-lang/flate2-rs/compare/1.0.35...1.1.0</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/rust-lang/flate2-rs/blob/main/CHANGELOG.md">flate2's changelog</a>.</em></p> <blockquote> <h1>Changelog</h1> <p>All notable changes to this project will be documented in this file.</p> <p>The format is based on <a href="https://keepachangelog.com/en/1.0.0/">Keep a Changelog</a>, and this project adheres to <a href="https://semver.org/spec/v2.0.0.html">Semantic Versioning</a>.</p> <h2>1.0.31 (2024-08-03)</h2> <p>This release allows using <code>libz-rs</code> in the latest version, v0.2.1.</p> <h3>Commit Statistics</h3> <!-- raw HTML omitted --> <ul> <li>7 commits contributed to the release over the course of 95 calendar days.</li> <li>95 days passed between releases.</li> <li>0 commits were understood as <a href="https://www.conventionalcommits.org">conventional</a>.</li> <li>0 issues like '(#ID)' were seen in commit messages</li> </ul> <h3>Commit Details</h3> <!-- raw HTML omitted --> <!-- raw HTML omitted --> <ul> <li><strong>Uncategorized</strong> <ul> <li>Add exclusion rule to not package github or git specific files with crate. (<a href="https://github.com/Byron/flate2-rs/commit/25541bd2aa4fd24fbb2b370eb3c2742724f956ac"><code>25541bd</code></a>)</li> <li>Crate and update changelog in preparation for release. (<a href="https://github.com/Byron/flate2-rs/commit/1dbed76ca0f63b215faa5ad321a42bb629686456"><code>1dbed76</code></a>)</li> <li>Merge pull request <a href="https://redirect.github.com/rust-lang/flate2-rs/issues/415">#415</a> from folkertdev/bump-version-zlib-rs-0.2.1 (<a href="https://github.com/Byron/flate2-rs/commit/a7853c0f803abb45858baf82d504ed14d77cb8c0"><code>a7853c0</code></a>)</li> <li>Release version 1.0.31: bump libz-rs-sys version (<a href="https://github.com/Byron/flate2-rs/commit/e6f694918b3237175e3729e6365e83f9a66518a9"><code>e6f6949</code></a>)</li> <li>Merge pull request <a href="https://redirect.github.com/rust-lang/flate2-rs/issues/414">#414</a> from yestyle/main (<a href="https://github.com/Byron/flate2-rs/commit/9e6af00a8bd9593f9e1c6421f9d27ccdb13a03b7"><code>9e6af00</code></a>)</li> <li>Remove duplicate word in top-most doc (<a href="https://github.com/Byron/flate2-rs/commit/411d6414398099c85e1b8fc568ec8929d208777d"><code>411d641</code></a>)</li> <li>Merge pull request <a href="https://redirect.github.com/rust-lang/flate2-rs/issues/408">#408</a> from marxin/document-read-after-end (<a href="https://github.com/Byron/flate2-rs/commit/1a0daec607455f30651674675abb01715586f4d1"><code>1a0daec</code></a>)</li> </ul> </li> </ul> <!-- raw HTML omitted --> <h2>v1.0.30 (2024-04-29)</h2> <h3>Documentation</h3> <ul> <li><!-- raw HTML omitted --> Document expected behavior when Read is done for ZLIB and DEFLATE decoders</li> </ul> <h3>Commit Statistics</h3> <!-- raw HTML omitted --> <ul> <li>9 commits contributed to the release over the course of 3 calendar days.</li> <li>3 days passed between releases.</li> <li>1 commit was understood as <a href="https://www.conventionalcommits.org">conventional</a>.</li> <li>1 unique issue was worked on: <a href="https://redirect.github.com/Byron/flate2-rs/issues/404">#404</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/rust-lang/flate2-rs/commit/70de9df4f09f9505624a02e7440656293fc2de2c"><code>70de9df</code></a> Merge pull request <a href="https://redirect.github.com/rust-lang/flate2-rs/issues/466">#466</a> from folkertdev/update-zlib-rs-0.4.2</li> <li><a href="https://github.com/rust-lang/flate2-rs/commit/13104a545503cca8e08cc0751e55702a7e1a8232"><code>13104a5</code></a> upgrade zlib-rs to version <code>0.4.2</code></li> <li><a href="https://github.com/rust-lang/flate2-rs/commit/5a2fd04132faf83efbb3f694868fb3e32fa85f4a"><code>5a2fd04</code></a> Merge pull request <a href="https://redirect.github.com/rust-lang/flate2-rs/issues/462">#462</a> from CosminPerRam/feat/flush_mzflush</li> <li><a href="https://github.com/rust-lang/flate2-rs/commit/bf5bf566980f549e2042229a0e6495a6d79137c0"><code>bf5bf56</code></a> feat: remove explicit redundant lifetime</li> <li><a href="https://github.com/rust-lang/flate2-rs/commit/40c2e0f56a4be991d43ed5ec91a1c43b1ba50ec4"><code>40c2e0f</code></a> docs: fix spelling mistake in FlushCompress::Partial</li> <li><a href="https://github.com/rust-lang/flate2-rs/commit/6146cf04fe18aa5adc0cfd128b8379fd9b8db532"><code>6146cf0</code></a> Merge pull request <a href="https://redirect.github.com/rust-lang/flate2-rs/issues/461">#461</a> from CosminPerRam/experimental/traits</li> <li><a href="https://github.com/rust-lang/flate2-rs/commit/ee1571b0d3d07b40ffa50f4ec31f3312be7b5c12"><code>ee1571b</code></a> Merge pull request <a href="https://redirect.github.com/rust-lang/flate2-rs/issues/464">#464</a> from CosminPerRam/feat/remove_redundant_guard</li> <li><a href="https://github.com/rust-lang/flate2-rs/commit/1be8e37c32c7f02b9a95d938f37cb6d8a7efa101"><code>1be8e37</code></a> feat: remove redundant if guard on option value match</li> <li><a href="https://github.com/rust-lang/flate2-rs/commit/be59b78575afc55b0be7c0ba2bb09fdc1ea3aee2"><code>be59b78</code></a> docs: fix spelling mistake</li> <li><a href="https://github.com/rust-lang/flate2-rs/commit/d4aebe1f7648d728b386d4ffa326700c9f2420d2"><code>d4aebe1</code></a> feat: reorder FlushCompress variants to be in order with value</li> <li>Additional commits viewable in <a href="https://github.com/rust-lang/flate2-rs/compare/1.0.35...1.1.0">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added support for s3-urls to config.
This is a bit of a rough sketch currently.
The aws config struct does not seem to support sso-login credentials out of the box, I needed to add the credentials to environment variables along with AWS_REGION to get it working.