Provisioning of new volumes is not working if remote is not available

[kvaps]

logs:

W0109 08:32:52.666743       1 controller.go:934] Retrying syncing claim "484e23e7-f6a5-4d16-a46e-41620f8c4a2d", failure 6
E0109 08:32:52.666771       1 controller.go:957] error syncing claim "484e23e7-f6a5-4d16-a46e-41620f8c4a2d": failed to provision volume with StorageClass "linstor-thindata-r2": rpc error: code = Internal desc = failed to check on potential left-over source snapshot: failed to check remote 'minio' for backups of id 'for-pvc-484e23e7-f6a5-4d16-a46e-41620f8c4a2d': Message: 'Exception thrown.'; Details: 'Client execution did not complete before the specified timeout configuration.'; Reports: '[63BBCE26-00000-000006]'
I0109 08:32:52.666785       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"disk-mydata", UID:"484e23e7-f6a5-4d16-a46e-41620f8c4a2d", APIVersion:"v1", ResourceVersion:"110412648", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "linstor-thindata-r2": rpc error: code = Internal desc = failed to check on potential left-over source snapshot: failed to check remote 'minio' for backups of id 'for-pvc-484e23e7-f6a5-4d16-a46e-41620f8c4a2d': Message: 'Exception thrown.'; Details: 'Client execution did not complete before the specified timeout configuration.'; Reports: '[63BBCE26-00000-000006]'

report:

ERROR REPORT 63BBCE26-00000-000006

============================================================

Application:                        LINBIT�� LINSTOR
Module:                             Controller
Version:                            1.20.0
Build ID:                           9c6f7fad48521899f7a99c564b1d33aeacfdbfa8
Build time:                         2022-11-07T16:37:38+00:00
Error time:                         2023-01-09 08:32:52
Node:                               linstor-controller-587ccf8cdd-vj2x4

============================================================

Reported error:
===============

Category:                           RuntimeException
Class name:                         ClientExecutionTimeoutException
Class canonical name:               com.amazonaws.http.timers.client.ClientExecutionTimeoutException
Generated at:                       Method 'handleInterruptedException', Source file 'AmazonHttpClient.java', Line #872

Error message:                      Client execution did not complete before the specified timeout configuration.

Call backtrace:

    Method                                   Native Class:Line number
    handleInterruptedException               N      com.amazonaws.http.AmazonHttpClient$RequestExecutor:872
    execute                                  N      com.amazonaws.http.AmazonHttpClient$RequestExecutor:755
    access$500                               N      com.amazonaws.http.AmazonHttpClient$RequestExecutor:713
    execute                                  N      com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl:695
    execute                                  N      com.amazonaws.http.AmazonHttpClient:559
    execute                                  N      com.amazonaws.http.AmazonHttpClient:539
    invoke                                   N      com.amazonaws.services.s3.AmazonS3Client:5453
    invoke                                   N      com.amazonaws.services.s3.AmazonS3Client:5400
    invoke                                   N      com.amazonaws.services.s3.AmazonS3Client:5394
    listObjectsV2                            N      com.amazonaws.services.s3.AmazonS3Client:971
    listObjects                              N      com.linbit.linstor.api.BackupToS3:388
    getAllS3Keys                             N      com.linbit.linstor.core.apicallhandler.controller.backup.CtrlBackupApiHelper:259
    listBackups                              N      com.linbit.linstor.core.apicallhandler.controller.backup.CtrlBackupApiCallHandler:661
    lambda$listBackups$0                     N      com.linbit.linstor.api.rest.v1.Backups:305
    doInScope                                N      com.linbit.linstor.api.rest.v1.RequestHelper:231
    doInScope                                N      com.linbit.linstor.api.rest.v1.RequestHelper:203
    listBackups                              N      com.linbit.linstor.api.rest.v1.Backups:299
    invoke0                                  Y      jdk.internal.reflect.NativeMethodAccessorImpl:unknown
    invoke                                   N      jdk.internal.reflect.NativeMethodAccessorImpl:62
    invoke                                   N      jdk.internal.reflect.DelegatingMethodAccessorImpl:43
    invoke                                   N      java.lang.reflect.Method:566
    lambda$static$0                          N      org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory:52
    run                                      N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1:124
    invoke                                   N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:167
    doDispatch                               N      org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker:176
    dispatch                                 N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:79
    invoke                                   N      org.glassfish.jersey.server.model.ResourceMethodInvoker:469
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:391
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:80
    run                                      N      org.glassfish.jersey.server.ServerRuntime$1:253
    call                                     N      org.glassfish.jersey.internal.Errors$1:248
    call                                     N      org.glassfish.jersey.internal.Errors$1:244
    process                                  N      org.glassfish.jersey.internal.Errors:292
    process                                  N      org.glassfish.jersey.internal.Errors:274
    process                                  N      org.glassfish.jersey.internal.Errors:244
    runInScope                               N      org.glassfish.jersey.process.internal.RequestScope:265
    process                                  N      org.glassfish.jersey.server.ServerRuntime:232
    handle                                   N      org.glassfish.jersey.server.ApplicationHandler:680
    service                                  N      org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer:356
    run                                      N      org.glassfish.grizzly.http.server.HttpHandler$1:200
    doWork                                   N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:569
    run                                      N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:549
    run                                      N      java.lang.Thread:829


END OF ERROR REPORT.

[WanzenBug]

I can probably fix this specific case for new volumes, but a lot of other calls involving snapshots probably won't work until the remote is available.