release-4449

What's Changed

• simplify endpoint limiter by @conradludgate in #6122
• tests: make test_bulk_insert recreate tenant in same generation by @jcsp in #6113
• refactor: calculate_synthetic_size_worker, remove PRE::NeedsDownload by @koivunej in #6111
• refactor: remove eviction batching by @koivunej in #6060
• Increase pgbouncer pool size to 64 for VMs by @vadim2404 in #6124
• Add Neon-Batch- headers to OPTIONS response for SQL-over-HTTP requests by @jawj in #6116
• Update Postgres to 14.10, 15.5, and 16.1 by @tristan957 in #5835
• proxy: add more rates to endpoint limiter by @conradludgate in #6130
• pageserver: heatmap uploads by @jcsp in #6050
• utils::logging: TracingEventCountLayer: don't use with_label_values() on hot path by @problame in #6129
• gauge guards by @conradludgate in #6138
• feat(layer): pageserver_layer_redownloaded_after histogram by @koivunej in #6132
• pageserver: improved handling of concurrent timeline creations on the same ID by @jcsp in #6139
• Fix python list_segments of sk. by @arssher in #6142
• fix: better Json parsing errors by @koivunej in #6135
• fix: Timeline deletion during busy startup by @koivunej in #6133
• Make initdb upload retries cancellable and seek to beginning by @arpad-m in #6147
• improve proxy code cov by @conradludgate in #6141
• tests: update allow list for negative_env by @jcsp in #6144
• A few comments on rust walproposer build. by @arssher in #6110
• s3_scrubber: basic support for sharding by @jcsp in #6119
• add a Rust client for Pageserver management API by @problame in #6127
• pageserver: basic cancel/timeout for remote storage operations by @jcsp in #6097
• pageserver: prioritize activation of tenants with client requests by @jcsp in #6112
• update zerocopy by @conradludgate in #6148
• CI: build build-tools image by @abhigets in #6082
• pageserver: add method to dump keyspace in mgmt api client by @problame in #6145
• Revert "CI: build build-tools image" by @bayandin in #6156
• Release 2023-12-18 by @vipvap in #6158

New Contributors

• @abhigets made their first contribution in #6082

Full Changelog: release-4414...release-4449

release-4414

What's Changed

• build: update azure-* to 0.17 by @koivunej in #6081
• test_runner: use toml instead of formatted strings by @bayandin in #6088
• pageserver: further refactoring from TenantId to TenantShardId by @jcsp in #6059
• tests: start adding tests for secondary mode, live migration by @jcsp in #5842
• Fix git ownership issue in check-codestyle-rust-arm by @tristan957 in #6100
• Add initdb_lsn to TimelineInfo by @arpad-m in #6104
• fix: accidential return Ok by @koivunej in #6106
• pageserver: tweaks to slow/hung task logging by @jcsp in #6098
• swagger: remove 'format: hex' from tenant IDs by @jcsp in #6099
• Propagate information about the protocol to console by @khanova in #6102
• Do not materialize null images in PS by @knizhnik in #5979
• Do not inherite replication slots in branch by @knizhnik in #5898
• test_runner: add from_repo_dir method by @bayandin in #6087
• scrubber: handle initdb files, fix an issue with prefixes by @jcsp in #6079
• Use extend instead of groups of append calls in tests by @arpad-m in #6109
• Update azure SDK to 0.18 and use open range support by @arpad-m in #6103
• Add per-endpoint rate limiter to proxy by @kelvich in #6120
• Release 2023-12-13 by @vipvap in #6121

Full Changelog: release-4394...release-4414

release-4394

What's Changed

• pageserver: tweak logs for index_part loading by @jcsp in #6005
• test_pg_clients: fix test that reads from stdout by @bayandin in #6021
• Perform L0 compaction before creating new image layers by @ivaxer in #5950
• Safekeepers: persist state before timeline deactivation. by @arssher in #5994
• Enable dynamic rate limiter by @khanova in #6029
• walredo: stderr cleanup & make explicitly cancel safe by @problame in #6031
• concurrency-limit initial logical size calculation [v2] by @problame in #6000
• [compute_ctl] Introduce feature flags in the compute spec by @ololobus in #6016
• Remove trusted from wal2json by @save-buffer in #6035
• Rename metric by @khanova in #6030
• proxy: fix panic in startup packet by @conradludgate in #6032
• pageserver: add Key->Shard mapping logic & use it in page service by @jcsp in #5980
• [compute_ctl] Notify waiters when Postgres failed to start by @ololobus in #6034
• pageserver: remove redundant unsafe_create_dir_all by @jcsp in #6040
• pageserver: add a /reset API for tenants by @jcsp in #6014
• pageserver: logging tweaks by @jcsp in #6039
• Track size of FSM fork while applying records at replica by @knizhnik in #5901
• virtual_file: fix the metric for close() calls done by VirtualFile::drop by @problame in #6051
• Revert timescaledb for pg14 and pg15 by @vadim2404 in #6056
• fix: avoid allocations with logging a slug by @koivunej in #6047
• Make simple_rcu::RcuWaitList::wait() async by @hlinnaka in #6046
• temp: increase the wait tenant activation timeout by @koivunej in #6058
• fix(layer): metric splitting, span rename by @koivunej in #5902
• build: update tokio to 1.34.0, tokio-utils 0.7.10 by @koivunej in #6061
• proxy: fix channel binding error messages by @conradludgate in #6054
• tests: update for tenant generations by @jcsp in #5449
• tests: use tenant_create() helper in test_bulk_insert by @jcsp in #6064
• fix(remote_storage): buffering, by using streams for upload and download by @koivunej in #5446
• virtual_file metrics: distinguish first and subsequent open() syscalls by @problame in #6066
• Add support for Python 3.12 by @tristan957 in #6067
• fix: use larger buffers for remote storage by @koivunej in #6069
• Buffer initdb.tar.zst to a temporary file before upload by @arpad-m in #5944
• test_tenant_reattach: fix reattach mode names by @bayandin in #6070
• pgindent and target for it of neon and core code by @arssher in #5618
• pageserver: filtered WAL ingest for sharding by @jcsp in #6024
• proxy: make auth more type safe by @conradludgate in #5689
• tests: update tests that used local_fs&mock_s3 to use one or the other by @jcsp in #6015
• page cache: improve eviction-related metrics by @problame in #6077
• proxy simplify cancellation by @conradludgate in #5916
• virtual_file metrics: expose max size of the fd cache by @problame in #6078
• tests: allow-lists for occasional failures by @jcsp in #6074
• proxy: pass neon options in deep object format by @prepor in #6068
• Fix RLS/REPLICATION granting by @save-buffer in #6083
• Release 2023-12-11 by @vipvap in #6090

Full Changelog: release-4344...release-4394

release-4344

What's Changed

• Revert timescaledb for pg14 and pg15 (#6056) by @vadim2404 in #6057

Full Changelog: release-4342...release-4344

release-4342

What's Changed

• pageserver: remove defunct local timeline delete markers by @jcsp in #5699
• Refactor remote extensions feature to request extensions from proxy by @lubennikovaav in #5836
• remove confusing no-op .take() of init_tenant_load_remote by @problame in #5923
• deflake: test_live_reconfig_get_evictions_low_residence_... by @koivunej in #5926
• Use test specific directory in test_remote_extensions by @lubennikovaav in #5938
• bootstrap_timeline: rename initdb_path to pgdata_path by @arpad-m in #5931
• channel binding by @conradludgate in #5683
• Make neon extension relocatable to allow SET SCHEMA by @lubennikovaav in #5942
• Update neon extension relocatable for existing installations by @lubennikovaav in #5943
• Compile wal2json by @save-buffer in #5893
• Fix check for if shmem is valid to take into account detached shmem by @save-buffer in #5937
• test suite: add method for generation-aware detachment of a tenant by @problame in #5939
• pageserver: add shard indices to layer metadata by @jcsp in #5928
• pageserver: add sharding metadata to LocationConf by @jcsp in #5932
• Move run_initdb to be async and guarded by max of 8 running tasks. Fixes #5895. Use tenant.cancel for cancellation by @shanyp in #5921
• fix: remove layer_removal_cs by @koivunej in #5108
• perf readme fix by @bojanserafimov in #5956
• build(deps): bump aiohttp from 3.8.6 to 3.9.0 by @dependabot in #5946
• Add walingest test by @bojanserafimov in #5892
• neonvm: add pgbouncer patch to support DEALLOCATD/DISCARD ALL by @bayandin in #5958
• build(deps): bump openssl from 0.10.55 to 0.10.60 by @dependabot in #5965
• build(deps): bump cryptography from 41.0.4 to 41.0.6 by @dependabot in #5970
• build(deps): bump openssl from 0.10.57 to 0.10.60 in /test_runner/pg_clients/rust/tokio-postgres by @dependabot in #5966
• pageserver: fix race between deletion completion and incoming requests by @jcsp in #5941
• Notify safekeeper readiness with systemd. by @arssher in #5972
• Optimize storing of null page in WAL by @knizhnik in #5910
• pageserver: remove redundant serialization helpers on DeletionList by @jcsp in #5960
• pageserver: refactor TenantId to TenantShardId in Tenant & Timeline by @jcsp in #5957
• neon_local: fix endpoint api to prevent two primary endpoints by @rmodpur in #5520
• walredo: latency histogram for spawn duration by @problame in #5925
• proxy: add more db error info by @conradludgate in #5951
• proxy: add flag to enable http pool for all users by @conradludgate in #5959
• refactor: use serde for TenantConf deserialization Fixes: #5300 by @rmodpur in #5310
• pagectl: add subcommand to rewrite layer file summary by @problame in #5933
• update aws sdk to 1.0.x by @ivaxer in #5976
• fix: flush loop should also keep the gate open by @koivunej in #5987
• IP allowlist on the proxy side by @khanova in #5906
• Pageserver disaster recovery RFC by @arpad-m in #5248
• proxy: fix wake_compute error prop by @conradludgate in #5989
• tests: add a log allow list entry in test_timeline_deletion_with_files_stuck_in_upload_queue by @jcsp in #5981
• walingest: refactor if-cascade on decoded.xl_rmid into match statement by @problame in #5974
• pageserver: add flush_ms and document /location_config API by @jcsp in #5860
• test: python needs thread safety as well by @koivunej in #5992
• Proxy fix metrics record by @khanova in #5996
• test: include cmdline in captured output by @koivunej in #5977
• proxy: chore: replace strings with SmolStr by @conradludgate in #5786
• Correct RFC number for Pageserver WAL DR RFC by @arpad-m in #5997
• Add existing_initdb_timeline_id param to timeline creation by @arpad-m in #5912
• Update timescaledb to 2.13.0 by @vadim2404 in #5975
• Clean up local state if index_part.json request gives 404 by @arpad-m in #6009
• initial logical size calculation: add a bunch of metrics by @problame in #5995
• init logical size: infallible accessor + type-level cleanups by @problame in #5999
• test_runner: use separate directory for each rerun by @bayandin in #6004
• add sql_exporter to vm-image by @lubennikovaav in #5949
• Speed up rel extend by @bojanserafimov in #5983
• Use TEST_OUTPUT envvar in pageserver by @bojanserafimov in #5984
• fix: use create_new instead of create for mutex file by @koivunej in #6012
• [compute_ctl] User correct HTTP code in the /configure errors by @ololobus in #6017
• initial logical size: remove CALLS metric from hot path by @problame in #6018
• Override neon.max_cluster_size for the time of compute_ctl config ope… by @lubennikovaav in #5998
• Release 2023-12-04 by @vipvap in #6022

Full Changelog: release-4277...release-4342

release-4277

What's Changed

• fix: use create_new instead of create for mutex file by @koivunej in #6013

Full Changelog: release-4275...release-4277

release-4275

What's Changed

• Notify safekeeper readiness with systemd. by @arssher in #5986

Full Changelog: release-4273...release-4275

release-4273

What's Changed

• Release 2023 11 28 hotfix by @lubennikovaav in #5945

Full Changelog: release-4270...release-4273

release-4270

What's Changed

• chore(background_tasks): missed allowed_error change, logging change by @koivunej in #5883
• build: back to opt-level=0 in debug builds, for faster compile times by @problame in #5751
• Disallow CREATE DATABASE WITH OWNER neon_superuser by @save-buffer in #5887
• Update Rust to 1.74.0 by @arpad-m in #5873
• Cleanup compact_level0_phase1 fsyncing by @koivunej in #5852
• refer to our rust-postgres.git fork by branch name by @problame in #5894
• logging: support output to stderr by @problame in #5896
• Add documentation for our sample pre-commit hook by @lubennikovaav in #5868
• LFC fixes + statistics by @knizhnik in #5727
• fix: cleanup of layers from the future can race with their re-creation by @problame in #5890
• Support cancellation for find_lsn_for_timestamp API by @shanyp in #5904
• Move weekly release PR trigger to Monday morning by @shanyp in #5908
• Upload initdb results to S3 by @arpad-m in #5390
• Create extension neon by @lubennikovaav in #5741
• proxy: fix memory leak again by @conradludgate in #5909
• Check if LFC was intialized in local_cache_pages function by @knizhnik in #5911
• test: support validating allowed_errors against a logfile by @koivunej in #5905
• fix: log cancelled request handler errors by @koivunej in #5915
• tests: fix allowed_error for compaction detecting a shutdown by @koivunej in #5919
• fix: long timeline create cancelled by tenant delete by @koivunej in #5917
• Only create neon extension in postgres database; by @lubennikovaav in #5918
• Release 2023-11-27 by @vipvap in #5922

Full Changelog: release-4245...release-4270

release-4245

What's Changed

• Logical replication startup fixes by @knizhnik in #5750
• s3_scrubber: prepare for scrubbing buckets with generation-aware content by @jcsp in #5700
• refactor: use streaming in safekeeper /v1/debug_dump http response by @duguorong009 in #5731
• Update control plane API path for getting compute spec. by @hlinnaka in #5357
• feat: improve the serde impl for several types(Lsn, TenantId, TimelineId ...) by @duguorong009 in #5335
• pageserver: add Gate as a partner to CancellationToken for safe shutdown of Tenant & Timeline by @jcsp in #5711
• Layer flush minor speedup by @bojanserafimov in #5765
• Add API description for safekeeper copy endpoint by @arpad-m in #5770
• pageserver: add InProgress tenant map state, use a sync lock for the map by @jcsp in #5367
• Make azure blob storage not do extra metadata requests by @arpad-m in #5777
• proxy: remove unsafe by @conradludgate in #5805
• Fix comments in 'receive_wal.rs'. by @wangxuqi in #5807
• pageserver: start refactoring into TenantManager by @jcsp in #5797
• pageserver: avoid converting an error to anyhow::Error by @jcsp in #5803
• refactor: remove unnecessary unsafe by @koivunej in #5802
• Use self.tenants instead of TENANTS by @shanyp in #5811
• build: Add warning when missing postgres submodule during the build by @fluz in #5614
• Dockerfile: Set BUILD_TAG for storage services by @bayandin in #5812
• pageserver: move deletion failpoint inside backoff by @jcsp in #5814
• Support JWT key reload in pageserver by @arpad-m in #5594
• Passing neon options to the console by @prepor in #5781
• pageserver: improve the shutdown log error by @duguorong009 in #5792
• vm-monitor: Remove dependency on workspace_hack by @sharnoff in #5752
• extend test_change_pageserver for failure case, rework changing pageserver by @jcsp in #5693
• test: de-flake test_deletion_queue_recovery by @jcsp in #5822
• pageserver: use Gate for stronger safety check in SlotGuard by @jcsp in #5793
• cleanup unused RemoteStorage fields by @problame in #5830
• JWT logging improvements by @arpad-m in #5823
• Add test simulating bad connection between pageserver and compute by @save-buffer in #5728
• Fix walproposer build on aarch64 by @petuhovskiy in #5827
• remote_storage: use S3 SDK's adaptive retry policy by @jcsp in #5813
• proxy: limit concurrent wake_compute requests per endpoint by @conradludgate in #5799
• Update epic-template.md by @stepashka in #5709
• fix(metric): remove pageserver_wal_redo_wait_seconds by @koivunej in #5791
• Fix blob_io test by @bojanserafimov in #5818
• pageserver: refined Timeline shutdown by @jcsp in #5833
• Lsn for get_timestamp_of_lsn should be string, not integer by @arpad-m in #5840
• temp: log all layer loading errors while we lose them by @koivunej in #5816
• fix(pageserver): spawn all kinds of tenant shutdowns by @koivunej in #5841
• find_lsn_for_timestamp fixes by @arpad-m in #5844
• workflows/neon_extra_builds: run check-codestyle-rust & build-neon on arm64 by @bayandin in #5832
• Set BUILD_TAG for build-neon job by @bayandin in #5847
• metric: add started and killed walredo processes counter by @rmodpur in #5809
• pageserver: error type for collect_keyspace by @jcsp in #5846
• build: remove async-std dependency by @koivunej in #5848
• build: deny async-std and friends by @koivunej in #5849
• build: upgrade ahash by @koivunej in #5851
• Fix comment in find_lsn_for_timestamp by @arpad-m in #5855
• neon_local: use remote storage by default, add cargo neon tenant migrate by @jcsp in #5760
• Migrate links from wiki to notion by @arpad-m in #5862
• Smaller test addition and change by @koivunej in #5858
• Proxy control plane rate limiter by @khanova in #5785
• build(deps): bump aiohttp from 3.8.5 to 3.8.6 by @dependabot in #5864
• Fix baseline commit and branch for code coverage by @bayandin in #5769
• pageserver: make TenantsMap shard-aware by @jcsp in #5819
• Collect number of connections by sni type by @khanova in #5867
• Update vm-builder to v0.19.0 and move its customization here by @sharnoff in #5783
• Proxy log pid hack by @khanova in #5869
• Improve error message when neon.max_cluster_size reached by @sharnoff in #4173
• compute_ctl: Remove deprecated flag --file-cache-on-disk by @sharnoff in #5622
• fix(background_tasks): first backoff, compaction error stacktraces by @koivunej in #5881
• pageserver: shut down idle walredo processes by @jcsp in #5877
• fix(layer): VirtualFile opening and read errors can be caused by contention by @koivunej in #5880
• Release 2023-11-17 by @vipvap in #5876

New Contributors

• @wangxuqi made their first contribution in #5807
• @fluz made their first contribution in #5614
• @prepor made their first contribution in #5781

Full Changelog: release-4179...release-4245