Flaky TestIngester_PrepareInstanceRingDownscaleHandler

[dimitarvdimitrov]

--- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler (3.00s)
    --- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only (0.00s)
        downscale_test.go:59: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/downscale_test.go:59
            	            				/__w/mimir/mimir/vendor/github.com/grafana/dskit/test/poll.go:17
            	            				/__w/mimir/mimir/pkg/ingester/downscale_test.go:57
            	Error:      	Received unexpected error:
            	            	instance not found in the ring
            	            	github.com/grafana/dskit/ring.init
            	            		<autogenerated>:1
            	            	runtime.doInit1
            	            		/usr/local/go/src/runtime/proc.go:[72](https://github.com/grafana/mimir/actions/runs/11141470952/job/30962417169?pr=9493#step:8:73)90
            	            	runtime.doInit
            	            		/usr/local/go/src/runtime/proc.go:7257
            	            	runtime.main
            	            		/usr/local/go/src/runtime/proc.go:254
            	            	runtime.goexit
            	            		/usr/local/go/src/runtime/asm_amd64.s:1700
            	Test:       	TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only
level=info msg="uploading new block to long-term storage" block=00000000010000000000000000
level=debug msg="uploaded file" from=/tmp/TestShipper_DeceivingUploadErrors2045435997/001/00000000010000000000000000/index dst=00000000010000000000000000/index bucket="fs: /tmp/TestShipper_DeceivingUploadErrors2045435997/002"
level=error msg="uploading new block to long-term storage failed" block=00000000010000000000000000 err="upload meta file: base name matches, will fail upload"
FAIL

[dimitarvdimitrov]

@jhalterman in case this is still fresh after you wrote it. Does anything come to mind?

[dimitarvdimitrov]

another flake

--- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler (3.00s)
    --- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only (0.01s)
        downscale_test.go:59: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/downscale_test.go:59
            	            				/__w/mimir/mimir/vendor/github.com/grafana/dskit/test/poll.go:17
            	            				/__w/mimir/mimir/pkg/ingester/downscale_test.go:57
            	Error:      	Received unexpected error:
            	            	instance not found in the ring
            	            	github.com/grafana/dskit/ring.init
            	            		<autogenerated>:1
            	            	runtime.doInit1
            	            		/usr/local/go/src/runtime/proc.go:7290
            	            	runtime.doInit
            	            		/usr/local/go/src/runtime/proc.go:7257
            	            	runtime.main
            	            		/usr/local/go/src/runtime/proc.go:254
            	            	runtime.goexit
            	            		/usr/local/go/src/runtime/asm_amd64.s:1700
            	Test:       	TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only
level=info msg="uploading new block to long-term storage" block=00000000010000000000000000
level=debug msg="uploaded file" from=/tmp/TestShipper_DeceivingUploadErrors1375604059/001/00000000010000000000000000/index dst=00000000010000000000000000/index bucket="fs: /tmp/TestShipper_DeceivingUploadErrors1375604059/002"
level=error msg="uploading new block to long-term storage failed" block=00000000010000000000000000 err="upload meta file: base name matches, will fail upload"
FAIL

[zenador]

https://github.com/grafana/mimir/actions/runs/11383313024/job/31668680547?pr=9655

--- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler (3.00s)
    --- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only (0.00s)
        downscale_test.go:59: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/downscale_test.go:59
            	            				/__w/mimir/mimir/vendor/github.com/grafana/dskit/test/poll.go:17
            	            				/__w/mimir/mimir/pkg/ingester/downscale_test.go:57
            	Error:      	Received unexpected error:
            	            	instance not found in the ring
            	            	github.com/grafana/dskit/ring.init
            	            		<autogenerated>:1
            	            	runtime.doInit1
            	            		/usr/local/go/src/runtime/proc.go:[72](https://github.com/grafana/mimir/actions/runs/11383313024/job/31668680547?pr=9655#step:8:73)90
            	            	runtime.doInit
            	            		/usr/local/go/src/runtime/proc.go:7257
            	            	runtime.main
            	            		/usr/local/go/src/runtime/proc.go:254
            	            	runtime.goexit
            	            		/usr/local/go/src/runtime/asm_amd64.s:1700
            	Test:       	TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only
level=info msg="uploading new block to long-term storage" block=00000000010000000000000000
level=debug msg="uploaded file" from=/tmp/TestShipper_DeceivingUploadErrors3861206[84](https://github.com/grafana/mimir/actions/runs/11383313024/job/31668680547?pr=9655#step:8:85)4/001/00000000010000000000000000/index dst=00000000010000000000000000/index bucket="fs: /tmp/TestShipper_DeceivingUploadErrors3[86](https://github.com/grafana/mimir/actions/runs/11383313024/job/31668680547?pr=9655#step:8:87)1206844/002"
level=error msg="uploading new block to long-term storage failed" block=00000000010000000000000000 err="upload meta file: base name matches, will fail upload"

[jhalterman]

This type of test where we're waiting for ring state to converge can unfortunately take a while. We might consider mocking the ring communication, but since other tests just perform long polling, we could try increasing the polling duration which should help with flakiness in CI.

[narqo]

Seems that it wasn't fixed. Just saw a flake in this build:

--- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler (3.00s)
    --- FAIL: TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only (0.01s)
        downscale_test.go:59: 
            	Error Trace:	/__w/mimir/mimir/pkg/ingester/downscale_test.go:59
            	            				/__w/mimir/mimir/vendor/github.com/grafana/dskit/test/poll.go:17
            	            				/__w/mimir/mimir/pkg/ingester/downscale_test.go:57
            	Error:      	Received unexpected error:
            	            	instance not found in the ring
            	            	github.com/grafana/dskit/ring.init
            	            		<autogenerated>:1
            	            	runtime.doInit1
            	            		/usr/local/go/src/runtime/proc.go:7290
            	            	runtime.doInit
            	            		/usr/local/go/src/runtime/proc.go:7257
            	            	runtime.main
            	            		/usr/local/go/src/runtime/proc.go:254
            	            	runtime.goexit
            	            		/usr/local/go/src/runtime/asm_amd64.s:1700
            	Test:       	TestIngester_PrepareInstanceRingDownscaleHandler/POST_request_should_switch_the_instance_ring_entry_to_read-only
level=info msg="uploading new block to long-term storage" block=00000000010000000000000000
level=debug msg="uploaded file" from=/tmp/TestShipper_DeceivingUploadErrors1968523047/001/00000000010000000000000000/index dst=00000000010000000000000000/index bucket="fs: /tmp/TestShipper_DeceivingUploadErrors1968523047/002"
level=error msg="uploading new block to long-term storage failed" block=00000000010000000000000000 err="upload meta file: base name matches, will fail upload"
FAIL
FAIL	github.com/grafana/mimir/pkg/ingester	590.242s