Add test for conn.getObjects('Annotations', opts={...})

[will-moore]

What this PR does

Adding tests for ome/omero-py#489

[pwalczysko]

The testGetObjectsAnnotation(self) added in this PR is consistently failing on merge-ci. It might be a case of the migration (of merge-ci) to NFS, but I have not seen it passing on the old merge-ci either. Could you please check @will-moore ? Especialy please comment on workflows which need I/O on the disk -> these might be influenced by the NFS migration.

Edit: The NFS migration has nothing to do with this. The test is failing locally for me as well (non-FS local test setup).

[pwalczysko]

Please comment on the failing https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/lastCompletedBuild/testReport/OmeroPy.test.integration.gatewaytest.test_get_objects/TestGetObject/testGetObjectsAnnotation/

[pwalczysko]

Note, the test is failing for me locally (I have only added some printouts so that I know that indeed an new user, group and 0 annotations are the starting point) with following error:

pytest test/integration/gatewaytest/test_get_objects.py::TestGetObject::testGetObjectsAnnotation -vv -s
================================================================================ test session starts ================================================================================
platform linux -- Python 3.9.18, pytest-8.4.2, pluggy-1.6.0 -- /opt/omero/server/venv3/bin/python3
cachedir: .pytest_cache
rootdir: /openmicroscopy/components/tools
configfile: pytest.ini
collected 1 item                                                                                                                                                                    

test/integration/gatewaytest/test_get_objects.py::TestGetObject::testGetObjectsAnnotation User ID: 9
Username: 1106a046-9b37-430a-b3eb-1f5fc6e3ed1c
Group ID: 10
Initial annotations: 0
FAILED

===================================================================================== FAILURES ======================================================================================
______________________________________________________________________ TestGetObject.testGetObjectsAnnotation _______________________________________________________________________

self = <test.integration.gatewaytest.test_get_objects.TestGetObject object at 0xffffb9a8ba30>

    def testGetObjectsAnnotation(self):
    
        # create new group and user - don't rely on existing test user/group
    
        group = self.new_group(perms='rwra--')
        client, user = self.new_client_and_user(group=group)
    
        conn = BlitzGateway(client_obj=client)
        update = conn.getUpdateService()
    
        print("User ID:", conn.getUserId())
        print("Username:", conn.getUser().getName())
        print("Group ID:", conn.getGroupFromContext().id)
    
        anns = list(conn.getObjects("Annotation"))
    
        print("Initial annotations:", len(anns))
        for a in anns:
            print(
                a.id,
                type(a).__name__,
                a.getOwner().id,
                a.getNs() if hasattr(a, "getNs") else None,
            )
    
        ns = "test_get_objects_annotation_comment"
        ns_tag = "test_get_objects_annotation_tag"
        ns_file = "test_get_objects_annotation_file"
    
        def create_dataset_with_annotations(name, dtype="Dataset"):
            if dtype == "Dataset":
                obj = DatasetI()
                obj.name = rstring(name)
                obj = update.saveAndReturnObject(obj)
                wrapper = omero.gateway.DatasetWrapper(conn, obj)
            elif dtype == "Project":
                obj = ProjectI()
                obj.name = rstring(name)
                obj = update.saveAndReturnObject(obj)
                wrapper = omero.gateway.ProjectWrapper(conn, obj)
    
            # create Comment
            ann = omero.gateway.CommentAnnotationWrapper(conn)
            ann.setNs(ns)
            ann.setValue("Test Comment: " + name)
            ann = wrapper.linkAnnotation(ann)
            # create Tag
            tag = omero.gateway.TagAnnotationWrapper(conn)
            tag.setNs(ns_tag)
            tag.setValue("Test Tag: " + name)
            wrapper.linkAnnotation(tag)
            # create FileAnnotation
            fileAnn = omero.gateway.FileAnnotationWrapper(conn)
            fileObj = omero.model.OriginalFileI()
            fileObj = omero.gateway.OriginalFileWrapper(conn, fileObj)
            fileObj.setName(omero.rtypes.rstring('fileName'))
            fileObj.setPath(omero.rtypes.rstring('path/to/file'))
            fileObj.setHash(omero.rtypes.rstring('a'))
            fileObj.setSize(omero.rtypes.rlong(0))
            fileObj.save()
            fileAnn.setFile(fileObj)
            fileAnn.setNs(ns_file)
            fileAnn.save()
            wrapper.linkAnnotation(fileAnn)
    
            return wrapper
    
        dataset1 = create_dataset_with_annotations("Dataset 1")
        dataset2 = create_dataset_with_annotations("Dataset 2")
        create_dataset_with_annotations("Project 1", dtype="Project")
    
        # get all annotations on one Dataset
        annGen = conn.getObjects("Annotation",
                                 opts={'parent_type': "dataset",
                                       'parent_ids': [dataset1.id]})
>       assert len(list(annGen)) == 3
E       assert 9 == 3
E        +  where 9 = len([<FileAnnotationWrapper id=36>, <TagAnnotationWrapper id=35>, <CommentAnnotationWrapper id=34>, <FileAnnotationWrapper id=33>, <TagAnnotationWrapper id=32>, <CommentAnnotationWrapper id=31>, <FileAnnotationWrapper id=30>, <TagAnnotationWrapper id=29>, <CommentAnnotationWrapper id=28>])
E        +    where [<FileAnnotationWrapper id=36>, <TagAnnotationWrapper id=35>, <CommentAnnotationWrapper id=34>, <FileAnnotationWrapper id=33>, <TagAnnotationWrapper id=32>, <CommentAnnotationWrapper id=31>, <FileAnnotationWrapper id=30>, <TagAnnotationWrapper id=29>, <CommentAnnotationWrapper id=28>] = list(<generator object _BlitzGateway.getObjects at 0xffffb9a76e40>)

test/integration/gatewaytest/test_get_objects.py:715: AssertionError

The test is not ready imho. Adding Exclude label.

[will-moore]

@pwalczysko Thanks for looking at this...

This PR adds tests for ome/omero-py#489 so without that PR these tests will fail.

I don't know if that PR got merged when the tests were failing above (not sure which build that was) but the PR seems to be merged in latest at https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-python-superbuild-push/12/console

So hopefully the exclude isn't needed, but fine to leave it till I'm back.

[pwalczysko]

don't know if that PR got merged when the tests were failing above (not sure which build that was) but the PR seems to be merged in latest at https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-python-superbuild-push/12/console

This is easy to see - that test never passed (on the new merge-ci). These are 12 runs which are encompassing the one which you and I am citing :), namely https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/12/#showFailuresLink - this is the testintegration build 12 which matches the python-superbuild 12 which you are citing. There IS a problem.

[will-moore]

Ah, OK, I see that in that failure (run 12), the test fails at

        # get ALL annotations
        anns = list(conn.getObjects("Annotation"))
>       assert len(anns) == 9
E       assert 11 == 9
E        +  where 11 = len([<FileAnnotationWrapper id=365>, <FileAnnotationWrapper id=366>, <CommentAnnotationWrapper id=5328>, <TagAnnotationWrapper id=5329>, <FileAnnotationWrapper id=5330>, <CommentAnnotationWrapper id=5331>, ...])

and this seems to be due to a couple of older FileAnnotations (IDs 365 and 366) that are created by earlier tests, possibly in the user group, and these are getting retrieved even in the new group and user created in this test.
I can add a fix for that.

[pwalczysko]

Ah, OK, I see that in that failure (run 12), the test fails at

        # get ALL annotations
        anns = list(conn.getObjects("Annotation"))
>       assert len(anns) == 9
E       assert 11 == 9
E        +  where 11 = len([<FileAnnotationWrapper id=365>, <FileAnnotationWrapper id=366>, <CommentAnnotationWrapper id=5328>, <TagAnnotationWrapper id=5329>, <FileAnnotationWrapper id=5330>, <CommentAnnotationWrapper id=5331>, ...])

and this seems to be due to a couple of older FileAnnotations (IDs 365 and 366) that are created by earlier tests, possibly in the user group, and these are getting retrieved even in the new group and user created in this test. I can add a fix for that.

Yes, @will-moore and please when you are at it, could you have a look also at https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/16/testReport/ ? I have summarized the problem in #6468.
This is similar case imho. I investigated a bit, and now I am inclined to rather think that there is a race. In the case which I am pinging here, there is a getAllObjects() call which is calling whilst the server-side chgrp is still in progress - am I right ? So the test only relies on the fact that the chgrp will be fast ? That will have following phenotype then: (and this is what is happening in the case I am talking about): the test will sometimes pass, sometimes fail in the same apparent conditions, as long as the conditions are not optimal (such as running it on slower NFS, with many other tests etc.)
I would like to have your thoughts on this please, thanks.

[pwalczysko]

@will-moore Please see also #6470

[will-moore]

These tests are passing now:
https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/19/testReport/junit/OmeroPy.test.integration.gatewaytest.test_get_objects/TestGetObject/testGetObjectsAnnotation/