[diffmap-GUI] diffmap crashes using openCL integration and GUI option

[EdgarGF93]

I think we already talked about this, but I didn't open the issue

(ewoks) edgar1993a@linada01:~/work/pyFAI$ pyFAI-diffmap --config /data/scisoft/edgar/pilx/config_ocl_v3.json -t 6 -r 6 /data/scisoft/edgar/pilx/RAW_DATA/scan0001/eiger/eiger_0000.h5 -o /tmp/hh.h5
WARNING:py.warnings:/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/pyopencl/invoker.py:374: UserWarning: pytools.persistent_dict 'pyopencl-invoker-cache-v41': enabling safe_sync as default. This provides strong protection against data loss, but can be unnecessarily expensive for use cases such as caches.Pass 'safe_sync=False' if occasional data loss is tolerable. Pass 'safe_sync=True' to suppress this warning.
  invoker_cache = WriteOncePersistentDict(

INFO:numexpr.utils:Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
INFO:numexpr.utils:NumExpr defaulting to 8 threads.
INFO:pyFAI.diffmap:Reading Mask file from: /mnt/multipath-shares/data/scisoft/edgar/pilx/mask.npy
INFO:pyFAI.gui.diffmap_widget:in start_processing
INFO:pyFAI.gui.diffmap_widget:process
INFO:pyFAI.worker:Detector Eiger2 CdTe 9M	 PixelSize= 75µm, 75µm	 BottomRight (3)
Wavelength= 3.738000e-11 m
SampleDetDist= 1.835200e-01 m	PONI= 1.303259e-01, 1.112753e-01 m	rot1=0.014562  rot2=-0.010992  rot3=0.000000 rad
DirectBeamDist= 183.551 mm	Center: x=1448.037, y=1710.777 pix	Tilt= 1.045° tiltPlanRotation= -142.949° 𝛌= 0.374Å
INFO:pyFAI.diffmap:Initialization of HDF5 file
shape for dataset: 6, 6, 3000
INFO:pyFAI.diffmap:Initialization of the Azimuthal Integrator using method Method(dim=1, split='bbox', algo='csr', impl='opencl', target=(0, 0))
Detector Eiger2 CdTe 9M	 PixelSize= 75µm, 75µm	 BottomRight (3)
Wavelength= 3.738000e-11 m
SampleDetDist= 1.835200e-01 m	PONI= 1.303259e-01, 1.112753e-01 m	rot1=0.014562  rot2=-0.010992  rot3=0.000000 rad
DirectBeamDist= 183.551 mm	Center: x=1448.037, y=1710.777 pix	Tilt= 1.045° tiltPlanRotation= -142.949° 𝛌= 0.374Å
INFO:pyFAI.azimuthalIntegrator:AI.integrate1d_ng: Resetting Cython integrator because of first initialization
****************************************************************************************************
mask.shape=(3262, 3108), shape=(3262, 3108)
INFO:pyFAI.azimuthalIntegrator:ai.integrate1d_ng: Resetting ocl_csr integrator because of first initialization
INFO:silx.opencl.processing:852.947MB are needed on device: NVIDIA RTX A2000 12GB,  which has 12632.654MB
INFO:silx.opencl.processing:Compiling file ['silx:opencl/doubleword.cl', 'pyfai:openCL/preprocess.cl', 'pyfai:openCL/memset.cl', 'pyfai:openCL/ocl_azim_CSR.cl'] with options -D NBINS=3000 -D NIMAGE=10138296
ERROR:pyFAI.worker:error in integration do_2d: False
ProgrammingError
SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140542890168704 and this is thread id 140539147310848.
data.shape: (3262, 3108)
data.size: 10138296
ai:
Detector Eiger2 CdTe 9M	 PixelSize= 75µm, 75µm	 BottomRight (3)
Wavelength= 3.738000e-11 m
SampleDetDist= 1.835200e-01 m	PONI= 1.303259e-01, 1.112753e-01 m	rot1=0.014562  rot2=-0.010992  rot3=0.000000 rad
DirectBeamDist= 183.551 mm	Center: x=1448.037, y=1710.777 pix	Tilt= 1.045° tiltPlanRotation= -142.949° 𝛌= 0.374Å
method:
IntegrationMethod(1d int, bbox split, CSR, OpenCL, NVIDIA CUDA / NVIDIA RTX A2000 12GB)
Exception in thread process:
Traceback (most recent call last):
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/pyFAI/gui/diffmap_widget.py", line 548, in process
    self.radial_data, self.azimuthal_data = diffmap.init_ai()
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/pyFAI/diffmap.py", line 514, in init_ai
    res = self.worker.process(data)
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/pyFAI/worker.py", line 411, in process
    raise err
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/pyFAI/worker.py", line 386, in process
    integrated_result = self._processor(**kwarg)
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/pyFAI/azimuthalIntegrator.py", line 1316, in integrate1d_ng
    integr = method.class_funct_ng.klass(csr_integr.lut,
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/pyFAI/opencl/azim_csr.py", line 169, in __init__
    self.compile_kernels()
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/pyFAI/opencl/azim_csr.py", line 282, in compile_kernels
    OpenclProcessing.compile_kernels(self, kernels, compile_options)
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/silx/opencl/processing.py", line 312, in compile_kernels
    self.kernels = KernelContainer(self.program)
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/silx/opencl/processing.py", line 79, in __init__
    for kernel in program.all_kernels():
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/pyopencl/__init__.py", line 442, in all_kernels
    knl._setup(self)
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/pyopencl/__init__.py", line 817, in kernel__setup
    self._enqueue, self._set_args = generate_enqueue_and_set_args(
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/pyopencl/invoker.py", line 393, in generate_enqueue_and_set_args
    pmod, enqueue_name = invoker_cache[cache_key]
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/pytools/persistent_dict.py", line 565, in __getitem__
    return self.fetch(key)
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/pytools/persistent_dict.py", line 698, in fetch
    stored_key, value = self._fetch(keyhash)
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/pytools/persistent_dict.py", line 687, in _fetch
    c = self._exec_sql("SELECT key_value FROM dict WHERE keyhash=?",
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/pytools/persistent_dict.py", line 526, in _exec_sql
    return self._exec_sql_fn(lambda: self.conn.execute(*args))
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/pytools/persistent_dict.py", line 534, in _exec_sql_fn
    return fn()
  File "/users/edgar1993a/anaconda3/envs/ewoks/lib/python3.9/site-packages/pytools/persistent_dict.py", line 526, in <lambda>
    return self._exec_sql_fn(lambda: self.conn.execute(*args))
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140542890168704 and this is thread id 140539147310848.

[EdgarGF93]

I was using here pytools 2024.1.5
It works if I use 2024.1.13 (or the main branch of pytools)

Related to inducer/pytools#233

[kif]

Please let it open until a new release of pytools comes out...