Bug: fstat on path with embedded SQL?

[joshmoore]

From: https://forum.image.sc/t/issues-with-nfs-mount-of-omero-data-dir/95763/10

Related to this we ran a strace for the Blitz-0 process and saw quite often funny lines like this:
1603 newfstatat(AT_FDCWD, "/opt/omero/server/OMERO.server-5.6.10-ice36/etc/SELECT image/series FROM Pixels WHERE id = :id.class", 0x7f39badfae60, 0) = -1 ENOENT (No such file or directory)
The mixed in SQL query into the path seems very odd and certainly do not help with reducing I/O load and errors.

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/issues-with-nfs-mount-of-omero-data-dir/95763/11

[JensWendt]

I will gather more data, both on our 5.6.10 server and our "old" live-instance which runs on 5.6.7
running strace for the Blitz-0 process

[joshmoore]

The query likely comes from omero-romio/src/main/java/ome/io/nio/PixelsService.java

 696   │     /**
 697   │      * Retrieves the series for a given set of pixels.
 698   │      * @param pixels Set of pixels to return the series for.
 699   │      * @return The series as specified by the pixels parameters or
 700   │      * <code>0</code> (the first series).
 701   │      */
 702   │     protected int getSeries(Pixels pixels)
 703   │     {
 704   │         try
 705   │         {
 706   │             final String query = "SELECT image.series FROM Pixels WHERE id = :id";
 707   │             final List<Object[]> results = iQuery.projection(query, new Parameters().addId(pixels.getId()));
 708   │             return (Integer) results.get(0)[0];
 709   │         }
 710   │         catch (Exception e)  // NumberFormatException, NullPointerException
 711   │         {
 712   │             return 0;
 713   │         }
 714   │     }

but unclear why it's be checked as a file (unless perhaps the query service is looking for a different source??)

[JensWendt]

the query-fragments I have seen differ though.
It might be hard to pinpoint them to a specific function

[joshmoore]

So I'm guessing that Class.forName looks the string on-disk regardless: https://github.com/ome/omero-server/blob/master/src/main/java/ome/services/query/ClassQuerySource.java#L53

I imagine it would be a fairly easy fix to apply a regex on the string before looking up the file. For example, something like

([\p{Letter}_$][\p{Letter}\p{Number}_$]*\.)*[\p{Letter}_$][\p{Letter}\p{Number}_$]*

from https://stackoverflow.com/questions/5205339/regular-expression-matching-fully-qualified-class-names

[JensWendt]

Okay,
did some testing.

setup:

• system 1:
  CentOS 7, OMERO.server 5.6.7, OMERO.iviewr 0.11.3
• system 2:
  Ubuntu 22.04, OMERO.server 5.6.10, OMERO.iviewer 0.13

command

strace -f -e desc -q -p <Blitz-0 PID> -o strace.log
I just copied this command from our IT guys how did the original debugging for the NFS issue.

test

I opened two large (65536 x 65536 and 69632 x 75264) tile-scan images one after the other in the OMERO.iviewer and zoomed and paned around in them for a bit.

result

centos_strace.log

ubuntu_strace.log

my thoughts

the strace seems to differ between the CentOS and Ubuntu and I am unsure if it is a different OS implementation or things got changed in the .iviewer/.server parts.
No SQL paths on the CentOS server, although I have no idea why.
It seems that between kernel 3.10.0 (the CentOS 7) and 5.15.0 (the Ubuntu 22.04) they changed from fstat to newfstatat, although I have no idea what implication this has.

[joshmoore]

Any difference between Java versions?

[JensWendt]

system 1 (old CentOS):

openjdk version "11.0.23" 2024-04-16 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.23.0.9-2.el7_9) (build 11.0.23+9-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.23.0.9-2.el7_9) (build 11.0.23+9-LTS, mixed mode, sharing)

system 2 (new Ubuntu):

openjdk version "11.0.22" 2024-01-16
OpenJDK Runtime Environment (build 11.0.22+7-post-Ubuntu-0ubuntu222.04.1)
OpenJDK 64-Bit Server VM (build 11.0.22+7-post-Ubuntu-0ubuntu222.04.1, mixed mode, sharing)