Update omero-grid to the latest server #24

manics · 2015-11-22T18:59:14Z

See the README.md for instructions

Ignore the rest of this description, it should be fixed by ome/openmicroscopy#4422

Unfortunately this doesn't work, see https://trello.com/c/3fY2taKP/88-omeor-grid-omero-node-bugs-issues

Attempting to start the slave (Processor-0) leads to an initial Ice connection, but Processor-0.log contains

2015-11-22 19:03:33,716 INFO  [                       omero.util.Server] (MainThread) ********************************************************************************
2015-11-22 19:03:33,716 INFO  [                       omero.util.Server] (MainThread) Waiting 10000 ms on startup
2015-11-22 19:03:43,717 INFO  [                       omero.util.Server] (MainThread) Starting
2015-11-22 19:03:43,719 INFO  [                    omero.util.Resources] (Thread-2  ) Starting
2015-11-22 19:03:43,791 ERROR [                       omero.util.Server] (MainThread) Failed activation
Traceback (most recent call last):
  File "/home/omero/OMERO.server-5.2.0-ice35-b12/lib/python/omero/util/__init__.py", line 502, in run
    self.adapter_name)
  File "/usr/lib64/python2.7/site-packages/Ice/Ice.py", line 557, in createObjectAdapter
    adapter = self._impl.createObjectAdapter(name)
SocketException: Ice.SocketException:
Cannot assign requested address
2015-11-22 19:03:43,794 INFO  [                       omero.util.Server] (MainThread) Cleanup
2015-11-22 19:03:43,794 INFO  [              omero.processor.ProcessorI] (MainThread) Cleaning up
2015-11-22 19:03:43,794 INFO  [              omero.processor.ProcessorI] (MainThread) Done
2015-11-22 19:03:43,821 WARNI [                omero.util.ServerContext] (MainThread) Connection failure: exception ::Ice::ObjectNotExistException
{
    id = 
    {
        name = 3505efce-99d6-4f5f-b827-42e87f4278da
        category = session-eb45540f-1e4c-4d4c-b02a-447908ebc69b
    }
    facet = 
    operation = keepAlive
}
2015-11-22 19:03:43,821 INFO  [                       omero.util.Server] (MainThread) Stopped
2015-11-22 19:03:43,821 INFO  [                       omero.util.Server] (MainThread) ********************************************************************************
2015-11-22 19:03:43,822 WARNI [                                  stderr] (MainThread) 2015-11-22 19:03:43.821998 Processor-0: error: Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/Ice/Ice.py", line 1127, in main
    status = self.doMain(args, initData)
  File "/usr/lib64/python2.7/site-packages/Ice/Ice.py", line 1174, in doMain
    return self.run(args)
  File "/home/omero/OMERO.server-5.2.0-ice35-b12/lib/python/omero/util/__init__.py", line 512, in run
    sys.exit(200)
SystemExit: 200
2015-11-22 19:03:43,841 INFO  [                    omero.util.Resources] (Thread-2  ) Halted

TODO:

Remove old db image
Remove old omero-server and omero-slave images (omero image does both)
Update omero-web
Cleanup

This should now work in a multi-node configuration

manics · 2015-11-24T16:28:02Z

I think this is done, apart from some cleanups e.g. removing web dependencies from omero/Dockerfile (I could do this here or in a later PR). See the README: https://github.com/manics/ome-docker/tree/omero-grid-new/omero-grid

Note this also contains a fix for the incorrect Ice.Default.Node causing omero node xxx start to fail. In theory you can use the master configuration arg to create as many slaves and servers as you want.

manics · 2015-11-24T18:15:42Z

This might be an interesting test:

# Glacier only
docker run -d --link postgres:db \
    -e DBUSER=postgres -e DBPASS=postgres -e DBNAME=postgres \
    -p 4063:4063 -p 4064:4064 \
    --name omero-master omero-grid master \
    slave-b0:Blitz-0 slave-b1:Blitz-1 \
    slave-i0:Indexer-0 slave-i1:Indexer-1 \
    slave-d:DropBox,MonitorServer,FileServer slave-s:Storm \
    slave-pi0:PixelData-0 slave-pi1:PixelData-1 \
    slave-t0:Tables-0 slave-t1:Tables-1 \
    slave-pr0:Processor-0 slave-pr1:Processor-1

for slave in slave-b0 slave-b1 slave-i0 slave-i1 slave-d slave-pi0 slave-pi1 \
        slave-t0 slave-t1 slave-pr0 slave-pr1; do
    docker run -d --link omero-master:master \
        --volumes-from omero-master \
        --name $slave omero-grid $slave
done

docker run -d --link omero-master:master -p 80:8080 --name omero-web omero-grid-web

joshmoore · 2015-11-24T21:17:06Z

omero-grid/omero/Dockerfile

-    ln -s OMERO.server-5.0.1-ice35-b21 OMERO.server"
+USER omero
+WORKDIR /home/omero
+RUN omego download server --release $OMERO_VERSION && \


This will likely be the first thing that I'd need to change. Can we make all the arguments passable? Or unpack to a well-known location so an unzipped server can be mounted?

How would you feel about a build-arg for the ci-server and branch?

Would work for me. Could be part of the tag that we use: e.g. docker build -t omero-deploy:b148 --build-arg ...

That's exactly what I was thinking... have a script which takes the version args, sets build-args and somehow munges them into a tag

joshmoore · 2015-11-24T21:38:01Z

Cherry-pick joshmoore@d77120a ?

The directory names now match the suggested image name.

Add build script to help build and tag images

manics · 2015-11-26T19:35:17Z

Cherry-picked, ./build.py added to pass the CI server and branch to the build, README updated.

Discovered a mysterious bug in omego download --ci ..., the ci argument appears to be ignored (but it is respected in omego upgrade), which means you can't currently use it in this image.
Travis Docker is too old and doesn't support --build-arg.

joshmoore · 2015-11-26T19:54:50Z

From https://blog.travis-ci.com/2015-08-19-using-docker-on-travis-ci/ :

The default Docker version installed is currently 1.7.0. We'll keep it up-to-date as we roll out new build environments, but you're also free to install different versions of Docker as part your builds, the APT repository is added by default.

manics · 2016-01-07T09:25:02Z

@joshmoore If you want to play with 5.1 this week you'll need to comment out the rewrite workarounds in https://github.com/ome/ome-docker/pull/24/files#diff-7fe777f48c77321016945317c474b5df

manics · 2016-01-25T16:21:13Z

That's my plan, trying to do too many other things at the same time.

Should be fixed in ome/openmicroscopy#4422

manics · 2016-01-25T19:54:16Z

@joshmoore I think this is ready for testing.

joshmoore · 2016-01-26T11:58:12Z

Blocker (or PR doc issue?):

./build.py omero-grid
docker tag openmicroscopy/omero-grid:latest omero-grid:josh-test-24
docker run omero-grid:josh-test-24 exits

[jamoore@orca-3 omero-grid]$ docker run omero-grid:josh-test-24
Entering a shell
[jamoore@orca-3 omero-grid]$ docker ps

The readme says to add "master" which works.

Few non-blocker comments (i.e. they don't prevent us getting to production with this but affect usability):

build.py doesn't provide a way to name the image meaning that it's fairly easy to overwrite someone else's run
Use of build-args for CI look prevents use of links to download from a container on the same host (ideally I'd like to have the build in a docker-compose)

Testing continues after lunch.

manics · 2016-01-26T18:09:55Z

Blocker (or PR doc issue?):

Originally the default was to drop into a bash shell which in hindsight is confusing if you're not running interactively. Changed to master now , so no args or just master should start a full server (minus web).

build.py doesn't provide a way to name the image

Added ./build --tag to override the full name/tag

Use of build-args for CI look prevents use of links to download from a container on the same host

Do you mean you're unable to link directly to a zip? If so then that's because I'm using omego download instead of omego install, otherwise OMERO.server automatically starts as a daemon. See ome/omego#69

joshmoore · 2016-01-26T20:25:28Z

Currently testing the 2 new commits. Thanks.

Do you mean you're unable to link directly to a zip? If so then that's because I'm using omego download instead of omego install, otherwise OMERO.server automatically starts as a daemon. See ome/omego#69

No, I mean due to DNS, I don't have an IP (in general) for my CI, but I can easily link it into a container for us. But linking isn't available at build time.

manics · 2016-01-27T10:22:03Z

Last three commits should help with testing. Runs a master-slave config, and you can modify params at the top of test.sh, e.g. see ome/openmicroscopy#4422

manics · 2016-01-27T11:31:04Z

No, I mean due to DNS, I don't have an IP (in general) for my CI, but I can easily link it into a container for us. But linking isn't available at build time.

So that's going back to the omero-deploy workflow of installing at runtime? How about a separate PR? I could add a --build-arg devel flag to specify the server is setup during run not build

joshmoore · 2016-01-28T08:40:50Z

Summary:

works based on Rewrite templates in omero node start, add --foreground openmicroscopy#4422 (comment) (but see the minor RFE there)
for the moment, this is certainly working, so yes, let's handle build-arg later. For the moment, I can likely hard-code the IP somewhere to get my builds to work.
there might be room for simplification in the little language you're using for service configuration but that's basically coming up with a new API, so definitely down the road.

joshmoore · 2016-01-28T08:41:21Z

https://travis-ci.org/ome/ome-docker/builds/105129380 is also red.

…ravis.

manics · 2016-01-28T15:55:29Z

https://s3.amazonaws.com/archive.travis-ci.org/jobs/105445281/log.txt

build.py checks env-vars (I've defaulted to empty so that it matches the default behaviour of build.py, and overridden the env-var in .travis.yml, though if you disagree I'll hard-code it in test.sh
Added smoke tests for web login, dropbox and processor

manics · 2016-01-28T16:29:21Z

I think I really am done now.

joshmoore · 2016-01-29T14:08:37Z

I can successfully get a 5.2 server up and running now. 👍 (That doesn't help me with the metadata51 build, though, but perhaps that's just motivation for getting us migrated)

Update omero-grid to the latest server

manics · 2016-01-29T14:24:53Z

This was stretching out so I decided to focus on 5.2 develop. Now that it's in I could work on metadata51 support if the rebase is going to take a while, add a trello card if you want me to (and find a column for it...).

joshmoore · 2016-01-29T14:26:36Z

My primary need is to test @mtbc's DB, but I may be able to do that with an upgraded version. I'll try that first since we'll need to upgrade anyway.

manics added 8 commits November 22, 2015 18:39

Update to 5.2.0, start icegridnode in the foreground

f4cce02

Update README

4b3b825

Patch server to deal with incorrect Ice.Default.Host property

c82cbee

This should now work in a multi-node configuration

Remove old Dockerfiles (omero-server, omero-slave)

61d9189

Update omero-grid omero-web Dockerfile

3021f70

Update README

4a8baf8

Call omero load on /config/*, regenerate nginx config

c7803d3

Add note about nginx config generation to README

e68c485

manics force-pushed the omero-grid-new branch from 8245652 to e68c485 Compare November 24, 2015 16:26

Fix maintainer lines

ed738d0

joshmoore reviewed Nov 24, 2015
View reviewed changes

joshmoore and others added 2 commits November 26, 2015 17:42

Add omero-grid test and a few fixes

52d8d0c

Include -grid in Dockerfile dir names

c7b7d8a

The directory names now match the suggested image name.

manics force-pushed the omero-grid-new branch from bca7830 to d5d8e2a Compare November 26, 2015 19:07

Add Dockerfile build-args to allow OMERO branch/version to be overriden.

3cb83d9

Add build script to help build and tag images

manics force-pushed the omero-grid-new branch from d5d8e2a to 3cb83d9 Compare November 26, 2015 19:11

manics added 3 commits November 26, 2015 19:15

Explain tag formulation for custom builds

7a97b31

Improve README formatting

c578d5f

Fix tag (should be lowercase)

edd09c6

manics force-pushed the omero-grid-new branch from 0f3124e to edd09c6 Compare November 26, 2015 19:31

manics added 2 commits November 26, 2015 20:16

Attempt to upgrade docker in travis

19420e2

Use openmicroscopy/ image prefix in README and test.sh

e206aa8

manics added 2 commits January 15, 2016 12:31

Allow env-var MASTER_ADDR as an alternative to link for web

2ff4b99

Move nginx var to /home/omero/nginx, clean-up nginx.conf

a324e3f

manics added 2 commits January 25, 2016 19:23

Remove workarounds in omero node

f8b363d

Should be fixed in ome/openmicroscopy#4422

Add volumes and ports to readme

87616ba

manics added 2 commits January 26, 2016 17:40

allow built name to be overriden with --tag

53e7ea6

Defaulta (no args): start master with all servers

7a3b9f8

manics added 3 commits January 27, 2016 08:51

Add params to test.sh to make manual testing easier

8070935

Add single slave to test.sh

531d940

Add --no-cache option to build.py

c9ac98b

manics added 3 commits January 28, 2016 11:19

test.sh vars can be overwritten externally. Use OMERO-DEV-latest in t…

3f7de0b

…ravis.

Add simple web login test

0401b1a

Add smoke tests for dropbox and processor

118347c

manics added 3 commits January 28, 2016 16:01

Minor cleanups

78e47f5

Forgot to mention PREFIX required in test_login.sh

ac16633

Cleanup volumes associated with container

ac7f870

joshmoore added a commit that referenced this pull request Jan 29, 2016

Merge pull request #24 from manics/omero-grid-new

5b12c22

Update omero-grid to the latest server

joshmoore merged commit 5b12c22 into ome:master Jan 29, 2016

manics deleted the omero-grid-new branch January 29, 2016 14:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update omero-grid to the latest server #24

Update omero-grid to the latest server #24

manics commented Nov 22, 2015

manics commented Nov 24, 2015

manics commented Nov 24, 2015

joshmoore Nov 24, 2015

manics Nov 25, 2015

joshmoore Nov 25, 2015

manics Nov 25, 2015

joshmoore commented Nov 24, 2015

manics commented Nov 26, 2015

joshmoore commented Nov 26, 2015

manics commented Jan 7, 2016

manics commented Jan 25, 2016

manics commented Jan 25, 2016

joshmoore commented Jan 26, 2016

manics commented Jan 26, 2016

joshmoore commented Jan 26, 2016

manics commented Jan 27, 2016

manics commented Jan 27, 2016

joshmoore commented Jan 28, 2016

joshmoore commented Jan 28, 2016

manics commented Jan 28, 2016

manics commented Jan 28, 2016

joshmoore commented Jan 29, 2016

manics commented Jan 29, 2016

joshmoore commented Jan 29, 2016

Update omero-grid to the latest server #24

Update omero-grid to the latest server #24

Conversation

manics commented Nov 22, 2015

manics commented Nov 24, 2015

manics commented Nov 24, 2015

joshmoore Nov 24, 2015

Choose a reason for hiding this comment

manics Nov 25, 2015

Choose a reason for hiding this comment

joshmoore Nov 25, 2015

Choose a reason for hiding this comment

manics Nov 25, 2015

Choose a reason for hiding this comment

joshmoore commented Nov 24, 2015

manics commented Nov 26, 2015

joshmoore commented Nov 26, 2015

manics commented Jan 7, 2016

manics commented Jan 25, 2016

manics commented Jan 25, 2016

joshmoore commented Jan 26, 2016

manics commented Jan 26, 2016

joshmoore commented Jan 26, 2016

manics commented Jan 27, 2016

manics commented Jan 27, 2016

joshmoore commented Jan 28, 2016

joshmoore commented Jan 28, 2016

manics commented Jan 28, 2016

manics commented Jan 28, 2016

joshmoore commented Jan 29, 2016

manics commented Jan 29, 2016

joshmoore commented Jan 29, 2016