release-5373

What's Changed

• Make test less sensitive to exact WAL positions by @hlinnaka in #7436
• Make another walcraft test more robust by @hlinnaka in #7439
• refactor(Timeline): simpler metadata updates by @koivunej in #7422
• relaxation: allow using layers across timelines by @koivunej in #7453
• proxy: Make retries configurable and record it. by @khanova in #7438
• pageserver: fix unlogged relations with sharding by @jcsp in #7454
• pagectl draw-timeline-dir: include layer file name as an SVG comment by @problame in #7455
• chore(pageserver): separate missing key error by @skyzh in #7393
• Refactor updating relation size cache on reads by @hlinnaka in #7376
• pageserver: remove import/export script previously used for breaking format changes by @VladLazar in #7458
• Don't pass tenant_id in location_config requests from storage controller by @arpad-m in #7476
• pageserver: fix vectored read aux key handling by @VladLazar in #7404
• pageserver: add read depth metrics and test by @VladLazar in #7464
• storage_controller: wait for db on startup by @jcsp in #7479
• feat(pageserver): add metadata key range and aux key encoding by @skyzh in #7401
• remote_storage: fix prefix handling in remote storage & clean up by @jcsp in #7431
• pageserver: change pitr_interval=0 behavior by @jcsp in #7423
• get_lsn_by_timestamp: clamp commit_lsn to be >= min_lsn by @arpad-m in #7488
• fix: avoid starving background task permits in eviction task by @koivunej in #7471
• Fix a bug with retries by @khanova in #7494
• pageserver: publish the same metrics from both read paths by @VladLazar in #7486
• pageserver: finish vectored get early by @VladLazar in #7490
• fix(metrics): correct maxrss metrics on macos by @skyzh in #7487
• Ability to specify the upload_storage_class in S3 bucket configuration by @arpad-m in #7461
• controller: limit Reconciler concurrency by @jcsp in #7493
• reduce complexity of proxy protocol parse by @conradludgate in #7078
• Update connect to compute and wake compute retry configs by @khanova in #7509
• proxy: Fix cancellations by @khanova in #7510
• pageserver: coalesce read paths by @VladLazar in #7477
• proxy: Workaround switch to the regional redis by @khanova in #7513
• chore(pageserver): temporary metrics on ingestion time by @skyzh in #7515
• Compute <-> pageserver protocol version 2 by @hlinnaka in #7377
• storage controller: log hygiene & better error type by @jcsp in #7508
• refactor(owned_buffers_io::BufferedWriter): be generic over the type of buffer by @problame in #7482
• refactor(owned_buffer_io::util::size_tracking_writer): make generic over underlying writer by @problame in #7483
• refactor(ephemeral_file): reuse owned_buffers_io::BufferedWriter by @problame in #7484
• Remove unused import by @jcsp in #7519
• perf!: use larger buffers for blob_io and ephemeral_file by @problame in #7485
• pageserver: fix image layer creation check that inhibited compaction by @VladLazar in #7420
• Return Past instead of Present or Future when commit_lsn < min_lsn by @arpad-m in #7520
• feat(pageserver): add aux-file-v2 flag on tenant level by @skyzh in #7505
• chore(pageserver): shrink aux keyspace to 0x60-0x7F by @skyzh in #7502
• fix(virtual_file): compile warnings on macos by @skyzh in #7525
• metrics: record time to update gc info as a per timeline metric by @koivunej in #7473
• Release 2024-04-29 by @vipvap in #7530

Full Changelog: release-5324...release-5373

release-5324

What's Changed

• pageserver: improve readability of shard.rs by @jcsp in #7330
• storage release: stop using no-op deployProxy / deployPgSniRouter by @problame in #7382
• proxy: Exclude private ip errors from recorded metrics by @khanova in #7389
• add async walredo mode (disabled-by-default, opt-in via config) by @problame in #6548
• pageserver: return ACCEPTED when deletion already in flight by @jcsp in #7384
• storage_controller: suppress misleading log by @jcsp in #7395
• proxy: rate limit authentication by masked IPv6. by @conradludgate in #7316
• pageserver/http: remove status code boilerplate from swagger spec by @VladLazar in #7385
• feat(neon_local): support listen addr for safekeeper by @skyzh in #7328
• chore(compute_ctl): add error context to apply_spec by @skyzh in #7374
• CI(check-build-tools-image): fix getting build-tools image tag by @bayandin in #7402
• proxy: Cleanup unused rate limiter by @khanova in #7400
• Add for add-help-for-timeline-arg for timeline command by @jbajic in #7361
• pageserver: demote range end fallback log by @VladLazar in #7403
• proxy: Improve logging by @khanova in #7405
• proxy: Record role to span by @khanova in #7407
• proxy: move endpoint rate limiter by @conradludgate in #7413
• proxy: Record only valid rejected events by @khanova in #7415
• feat: copy delta layer prefix or "truncate" by @koivunej in #7228
• pageserver: fix secondary download scheduling by @jcsp in #7396
• Add a docs page for storage controller by @jcsp in #7392
• fix(Layer): metric regression with too many canceled evictions by @koivunej in #7363
• build(deps): bump aiohttp from 3.9.2 to 3.9.4 by @dependabot in #7429
• pageserver: fix cont lsn jump on vectored read path by @VladLazar in #7412
• storcon_cli: tenant-warmup command by @jcsp in #7432
• CI: fix docker/setup-buildx-action & update rustls by @bayandin in #7445
• Bump vm-builder v0.23.2 -> v0.28.1 by @sharnoff in #7433
• Release 2024-04-22 by @vipvap in #7447

Full Changelog: release-5294...release-5324

release-5294

What's Changed

• refactor(pageserver): use tokio::signal instead of spawn_blocking by @problame in #7332
• Remove async_trait from CompactionDeltaLayer by @arpad-m in #7342
• pageserver: option to run with just one tokio runtime by @problame in #7331
• implement Serialize/Deserialize for SystemTime with RFC3339 format by @kevinmingtarja in #7203
• update measured with some more convenient features by @conradludgate in #7334
• CI(flaky-tests): remove outdated restriction by @bayandin in #7345
• Update staging hostname by @bayandin in #7347
• Proxy read ids from redis by @khanova in #7205
• pageserver: time based rolling based on the first write timestamp by @VladLazar in #7346
• Add command to time travel recover prefixes by @arpad-m in #7322
• proxy: hyper1 for only proxy by @conradludgate in #7073
• proxy: fix credentials cache lookup by @khanova in #7349
• Revert "Proxy read ids from redis (#7205)" by @khanova in #7350
• compute_ctl: Auto-set dynamic_shared_memory_type by @sharnoff in #7348
• Remember last written LSN when it is first requested by @knizhnik in #7343
• Reenable test_forward_compatibility by @petuhovskiy in #7358
• compute hook: use shared client with explicit timeout by @jcsp in #7359
• CI(build-build-tools-image): Do not cancel concurrent workflows by @bayandin in #7226
• proxy: replace prometheus with measured by @conradludgate in #6717
• Read cplane events from regional redis by @khanova in #7352
• proxy: fix overloaded db connection closure by @conradludgate in #7364
• CI: speed up Allure reports upload by @bayandin in #7362
• Fix typo in pg_ctl shutdown mode by @tristan957 in #7365
• build(deps): bump idna from 3.3 to 3.7 by @dependabot in #7367
• Release 2024-04-15 by @vipvap in #7378

New Contributors

• @kevinmingtarja made their first contribution in #7203

Full Changelog: release-5269...release-5294

release-5269

What's Changed

• Release 2024-04-11(ish) cherry-pick #7348 by @sharnoff in #7356

Full Changelog: release-5267...release-5269

release-5267

What's Changed

• update tokio 1.37 by @conradludgate in #7276
• Standardizing Logging Format for Improved Middleware Integration by @macdoos in #7194
• CI(benchmarking): Add Sharded Tenant for pgbench by @bayandin in #7186
• tests: clean up compat test workarounds by @jcsp in #7097
• pageserver: drop the layer map lock after planning reads by @VladLazar in #7215
• proxy: upload consumption events to S3 by @khanova in #7213
• controller: add storcon-cli by @jcsp in #7114
• remove http timeout by @conradludgate in #7291
• pageserver: even more debug for test_secondary_downloads by @jcsp in #7295
• refactor(walreceiver): eliminate task_mgr usage by @problame in #7260
• wal_ingest: global counter for bytes received by @koivunej in #7240
• Schedule image layer uploads in tiered compaction by @arpad-m in #7282
• pageserver: on-demand activation of tenant on GET tenant status by @jcsp in #7250
• Upload partial segments by @petuhovskiy in #6530
• pageserver: replace the locked tenant config with arcsawps by @VladLazar in #7292
• refactor(Timeline::shutdown): rely more on Timeline::cancel; use it from deletion code path by @problame in #7233
• update ordered-multimap by @conradludgate in #7306
• tests: reinstate some syntethic size tests by @VladLazar in #7294
• Fix bug in prefetch cleanup by @knizhnik in #7277
• Fix length by @khanova in #7308
• storage controller: grant 'admin' access to all APIs by @jcsp in #7307
• storage_controller: setup sentry reporting by @VladLazar in #7311
• fix(walredo spawn): coalescing stalls other executors std::sync::RwLock by @problame in #7310
• pageserver: timeout on deletion queue flush in timeline deletion by @jcsp in #7315
• pageserver: handle WAL gaps on sharded tenants by @jcsp in #6788
• pageserver: update mgmt_api to use TenantShardId by @jcsp in #7313
• storage controller: fix repeated location_conf returning no shards by @jcsp in #7314
• tests: extend log allow list in test_storcon_cli by @jcsp in #7321
• Fix zero segment upload by @petuhovskiy in #7318
• Update test-context dependency to 0.3 by @arpad-m in #7303
• proxy: report metrics based on cold start info by @conradludgate in #7324
• Clean up 'attachment service' names to storage controller by @jcsp in #7326
• storage controller: rename TenantState to TenantShard by @jcsp in #7329
• tests: improve stability of test_deletion_queue_recovery by @jcsp in #7325
• pageserver: apply shard filtering to blocks ingested during initdb by @jcsp in #7319
• fixup: PR #7319 defined workload.py def stop() twice by @problame in #7333
• control_plane: revise compute_hook locking (don't serialise all calls) by @jcsp in #7088
• tests: further stabilize test_deletion_queue_recovery by @jcsp in #7335
• fix(utilization): return used as does df by @koivunej in #7337
• Release 2024-04-08 by @vipvap in #7338

New Contributors

• @macdoos made their first contribution in #7194

Full Changelog: release-5227...release-5267

release-5227

What's Changed

• Drop connections with all shards invoplved in prefetch in case of error by @knizhnik in #7249
• Support compaction_threshold=1 for tiered compaction by @arpad-m in #7257
• storage controller: tenant scheduling policy by @jcsp in #7262
• pageserver: fix oversized key on vectored read by @VladLazar in #7259
• fix: drop replication slot causes postgres stuck on exit by @skyzh in #7192
• Specify caller in 'unexpected response from page server' error by @lubennikovaav in #7272
• CI(deploy): Do not deploy storage controller to preprod for proxy releases by @bayandin in #7269
• spec: allow neon extension auto-upgrade + softfail upgrade by @skyzh in #7231
• storage controller: be more tolerant of control plane blocking notifications by @jcsp in #7268
• pageserver: check for new image layers based on ingested WAL by @VladLazar in #7230
• storage controller: scheduling optimization for sharded tenants by @jcsp in #7181
• neonvm: add LFC approximate working set size to metrics by @skyzh in #7252
• fixup(#7278/compute_ctl): remote extension download permission by @skyzh in #7280
• Release 2024-04-01 by @vipvap in #7284

Full Changelog: release-5213...release-5227

release-proxy-5201

What's Changed

• tests: log hygiene checks for storage controller by @jcsp in #6710
• fixup(#7168): neon_local: use pageserver defaults for known but unspecified config overrides by @skyzh in #7166
• Move functions for creating/extracting tarballs into utils by @tristan957 in #7132
• storage controller: tech debt by @jcsp in #7165
• test: fix on demand activation test flakyness by @VladLazar in #7180
• proxy: enable sha2 asm support by @conradludgate in #7184
• fix: ResidentLayer::load_keys should not create INFO level span by @koivunej in #7174
• refactor(layer): use detached init by @koivunej in #7152
• pageserver: extra debug for test_secondary_downloads failures by @jcsp in #7183
• fix(Layer): always init after downloading in the spawned task by @koivunej in #7175
• Dump layer map json in test_gc_feedback.py by @arpad-m in #7179
• fix: add safekeeper team to pgxn codeowners by @skyzh in #7170
• safekeeper: correctly handle signals by @skyzh in #7167
• Add state diagram for compute by @tristan957 in #7172
• fix(heavier_once_cell): take_and_deinit should take ownership by @koivunej in #7185
• fix(layer): remove the need to repair internal state by @koivunej in #7030
• Force LSN order with wrapper structure by @jbajic in #7071
• proxy: async aware password validation by @conradludgate in #7176
• storage_controller: add metrics by @VladLazar in #7178
• storage controller: fixes to secondary location handling by @jcsp in #7169
• pageserver: quieten log on shutdown-while-attaching by @jcsp in #7177
• pageserver: extend /re-attach response to include tenant mode by @jcsp in #6941
• proxy: simplify password validation by @conradludgate in #7188
• walredo benchmark: throughput-oriented rewrite by @problame in #7190
• Update Rust to 1.77.0 by @arpad-m in #7198
• proxy: connect redis with AWS IAM by @khanova in #7189
• Fix ephemeral file warning on secondaries by @jcsp in #7201
• proxy: unit tests for auth_quirks by @conradludgate in #7199
• CI: deploy release version to a preprod region by @bayandin in #6811
• pageserver: write consumption metrics to S3 by @jcsp in #7200
• Remove Get/SetZenithCurrentClusterSize from Postgres core by @knizhnik in #7196
• fixup(#7204 / postgres): revert IsPrimaryAlive checks by @skyzh in #7209
• proxy: fix stack overflow in cancel publisher by @conradludgate in #7212
• pageserver: use a single tokio runtime by @problame in #6555
• test: fix test_pageserver_recovery flakyness by @VladLazar in #7207
• storage controller: tighten up secrets handling by @jcsp in #7105
• pageserver: track total ephemeral layer bytes by @jcsp in #7182
• Try to fix test_crafted_wal_end flakiness. by @arssher in #7143
• tests: stabilize compat tests by @jcsp in #7227
• chore: remove repetitive words by @availhang in #7206
• refactor(remote_timeline_client): infallible stop() and shutdown() by @problame in #7234
• tests: tolerate transient unavailability in test_sharding_split_failures by @jcsp in #7223
• tests: wait for uploads in test_secondary_downloads by @jcsp in #7220
• proxy: reuse sess_id as request_id for the cplane requests by @khanova in #7245
• Revert "pageserver: use a single tokio runtime (#6555)" by @problame in #7246
• pageserver: drop layer map lock in Timeline::get by @VladLazar in #7217
• test_runner/performance: skip test_bulk_insert by @bayandin in #7238
• pageserver: return 429 on timeline creation in progress by @jcsp in #7225
• pageserver: limit total ephemeral layer bytes by @jcsp in #7218
• storage controller: make direct tenant creation more robust by @jcsp in #7247
• tests: add a log allow list to stabilize benchmarks by @jcsp in #7251
• pageserver: remove bare mgr::get_tenant, mgr::list_tenants by @jcsp in #7237
• add authentication rate limiting by @conradludgate in #6865
• Proxy release 2024-03-27 by @vipvap in #7254

New Contributors

• @availhang made their first contribution in #7206

Full Changelog: release-proxy-5147...release-proxy-5201

release-compute-only-5213

What's Changed

• pageserver: drop layer map lock in Timeline::get by @VladLazar in #7217
• test_runner/performance: skip test_bulk_insert by @bayandin in #7238
• pageserver: return 429 on timeline creation in progress by @jcsp in #7225
• pageserver: limit total ephemeral layer bytes by @jcsp in #7218
• storage controller: make direct tenant creation more robust by @jcsp in #7247
• tests: add a log allow list to stabilize benchmarks by @jcsp in #7251
• pageserver: remove bare mgr::get_tenant, mgr::list_tenants by @jcsp in #7237
• add authentication rate limiting by @conradludgate in #6865
• fix(walreceiver): Timeline::shutdown can leave a dangling handle_walreceiver_connection tokio task by @problame in #7235
• CI(trigger-e2e-tests): set e2e-platforms by @bayandin in #7229
• Revert "Revoke REPLICATION" by @save-buffer in #7261
• Release 2024-03-27 - compute only release by @vipvap in #7263

Full Changelog: release-5201...release-5213

release-5201

What's Changed

• test: fix test_pageserver_recovery flakyness by @VladLazar in #7207
• storage controller: tighten up secrets handling by @jcsp in #7105
• pageserver: track total ephemeral layer bytes by @jcsp in #7182
• Try to fix test_crafted_wal_end flakiness. by @arssher in #7143
• tests: stabilize compat tests by @jcsp in #7227
• chore: remove repetitive words by @availhang in #7206
• refactor(remote_timeline_client): infallible stop() and shutdown() by @problame in #7234
• tests: tolerate transient unavailability in test_sharding_split_failures by @jcsp in #7223
• tests: wait for uploads in test_secondary_downloads by @jcsp in #7220
• proxy: reuse sess_id as request_id for the cplane requests by @khanova in #7245
• Revert "pageserver: use a single tokio runtime (#6555)" by @problame in #7246
• Release 2024-03-26 by @vipvap in #7248

New Contributors

• @availhang made their first contribution in #7206

Full Changelog: release-5189...release-5201

release-5189

What's Changed

• build: make procfs linux only dependency by @koivunej in #7156
• pageserver: refactoring in TenantManager to reduce duplication by @jcsp in #6732
• fixup(#7141 / tokio_epoll_uring_ext): high frequency log message by @problame in #7160
• fix: spanless log message by @koivunej in #7155
• fixup(#7160 / tokio_epoll_uring_ext): double-panic caused by info! in thread-local's drop() by @problame in #7164
• Support backpressure for sharding by @petuhovskiy in #7100
• async password validation by @conradludgate in #7171
• tests: log hygiene checks for storage controller by @jcsp in #6710
• fixup(#7168): neon_local: use pageserver defaults for known but unspecified config overrides by @skyzh in #7166
• Move functions for creating/extracting tarballs into utils by @tristan957 in #7132
• storage controller: tech debt by @jcsp in #7165
• test: fix on demand activation test flakyness by @VladLazar in #7180
• proxy: enable sha2 asm support by @conradludgate in #7184
• fix: ResidentLayer::load_keys should not create INFO level span by @koivunej in #7174
• refactor(layer): use detached init by @koivunej in #7152
• pageserver: extra debug for test_secondary_downloads failures by @jcsp in #7183
• fix(Layer): always init after downloading in the spawned task by @koivunej in #7175
• Dump layer map json in test_gc_feedback.py by @arpad-m in #7179
• fix: add safekeeper team to pgxn codeowners by @skyzh in #7170
• safekeeper: correctly handle signals by @skyzh in #7167
• Add state diagram for compute by @tristan957 in #7172
• fix(heavier_once_cell): take_and_deinit should take ownership by @koivunej in #7185
• fix(layer): remove the need to repair internal state by @koivunej in #7030
• Force LSN order with wrapper structure by @jbajic in #7071
• proxy: async aware password validation by @conradludgate in #7176
• storage_controller: add metrics by @VladLazar in #7178
• storage controller: fixes to secondary location handling by @jcsp in #7169
• pageserver: quieten log on shutdown-while-attaching by @jcsp in #7177
• pageserver: extend /re-attach response to include tenant mode by @jcsp in #6941
• proxy: simplify password validation by @conradludgate in #7188
• walredo benchmark: throughput-oriented rewrite by @problame in #7190
• Update Rust to 1.77.0 by @arpad-m in #7198
• proxy: connect redis with AWS IAM by @khanova in #7189
• Fix ephemeral file warning on secondaries by @jcsp in #7201
• proxy: unit tests for auth_quirks by @conradludgate in #7199
• CI: deploy release version to a preprod region by @bayandin in #6811
• pageserver: write consumption metrics to S3 by @jcsp in #7200
• Remove Get/SetZenithCurrentClusterSize from Postgres core by @knizhnik in #7196
• fixup(#7204 / postgres): revert IsPrimaryAlive checks by @skyzh in #7209
• proxy: fix stack overflow in cancel publisher by @conradludgate in #7212
• pageserver: use a single tokio runtime by @problame in #6555
• Release 2024-03-25 by @vipvap in #7219

Full Changelog: release-5147...release-5189