Retry logic for externals

[Donaim]

Kive is designed to run on clusters, and as such, it should acknowledge occasional connectivity failures.

One specific failure, that we found to be common in our deployment, is when one of the "externals" filesystems report files as missing. This results in Kive job crashing with:

========
Internal Kive Error
========
BlockingIOError: [Errno 11] Resource temporarily unavailable: '/path/to/fastq.gz'

To solve this, we can implement a retry logic.
Whenever there is an error caused by external hardware, pause for some time, and repeat the action again.

We can take inspiration from MiCall that already implements this.

[Donaim]

Another instance of it affected micall watcher.

2025-01-21 01:00:04[ERROR]micall.monitor.kive_watcher.wait_for_retry(): Waiting 0:00:05 before retrying.
Traceback (most recent call last):
  File "/usr/local/share/MiCall/micall/monitor/kive_watcher.py", line 531, in poll_runs
    self.check_completed_folders()
  File "/usr/local/share/MiCall/micall/monitor/kive_watcher.py", line 547, in check_completed_folders
    pipeline_group)
  File "/usr/local/share/MiCall/micall/monitor/kive_watcher.py", line 588, in collate_folder
    self.copy_outputs(folder_watcher, scratch_path, target_path)
  File "/usr/local/share/MiCall/micall/monitor/kive_watcher.py", line 602, in copy_outputs
    results_path)
  File "/usr/local/share/MiCall/micall/monitor/kive_watcher.py", line 690, in extract_coverage_maps
    open(target_path, 'wb') as target:
BlockingIOError: [Errno 11] Resource temporarily unavailable: '/Results/version_7.17/denovo/coverage_maps/1943P47H-HIV_S24.HIVB.V3LOOP.details.png'