omego db upgrade

[manics]

Originally I used the psycopg2 python module to connect to Postgres (https://github.com/manics/omero-setup/compare/f42ba2d42f092e3e78930248a295285965816785), but this is an additional dependency which may require the postgres-devel package and a compiler. 8195422 changes this to use the psql binary instead. The PGPASSWORD envvar is used internally to pass the password to psql. If DB backups are implemented the pg_dump binary will need to be run anyway.

• omego db upgrade --dbhost host --dbname name --dbuser user --dbpass pass --serverdir path/to/server
• omego db init --dbhost host --dbname name --dbuser user --dbpass pass --serverdir path/to/server
• omego upgrade --dbhost host --dbname name --dbuser user --dbpass pass --upgradedb

OMERO4.4 to OMERO5.0 and OMERO5.0 to OMERO5.1DEV should work.

Note credentials aren't read from config.xml so if you've got multiple DBs ensure you pass the connection details.

Current issues:

• Credentials. Currently passed on the command line, should they be read from etc/config.xml, or by calling bin/omero config get omero.db.xxx
• Initialisation. If this is a new DB then perhaps it makes sense to take the credentials from the command line and use those to set the omero config, perhaps along with other server initialisation tasks.
• Tests. Currently there are none.
• Should this go in omero db instead? Or will we end up tying ourselves in knots?
• omego db upgrade --dry-run tells you if an upgrade would be performed, omego upgrade --upgradedb --dry-run doesn't
• Travis. It should be failing without this db upgrade, it's not.

[manics]

In theory travis should currently fail due to the db change from 5.0.0 to 5.1-latest (TestUpgrade.testUpgrade, TestUpgrade.testUpgradeMatrixBuild).

[joshmoore]

Thoughts:

• if you're passing serverdir, I'd vote 👍 for omego being able to get the values it needs from bin/omero config or etc/grid/config.xml directly, but that can also be a later PR.
• the idea of having init also set config.xml is interesting but I don't have a feeling for if that should be the default.
• should the parameters be closer to those of psql?
• would be interesting to think about how a omego db backup and restore command could fit into this workflow
• as for whether this should go in omero/plugins/db.py : perhaps eventually, but at the moment none of our released versions would have that logic. Instead, omego could install a new extra plugin (or a new version of db.py?) if we think we can keep support for all the various versions.

[sbesson]

• re db.py, I would vote 👍 for driving this effort via omego initially, test it widely via our CI infrastructure and then reinclude it into the core CLI plugins. Same idea if we decide to develop omego db backup.
• seconding @joshmoore, reading directly from server config may simplify things in the end but we can refactor this in a second step
• re init, an alternative would be to have omego db init as well as omego init (including DB + server initialization)
• re tests, I will give this some thoughts when I find a bit more time. A real limit of our current testing infrastructure is that the full server needs to be downloaded to run most omego tests. For pure omego db commands, we should be able to get some quick tests though.

[manics]

Only one of the upgrades in TestUpgrade was working, the other just restarted the same server since the Python CLI module from the second upgraded server wasn't re-imported. I've changed this to only test a single upgrade with a breaking DB change, so Travis should really fail this time.

[manics]

should the parameters be closer to those of psql?

I wanted them to be as unambiguous as possible

would be interesting to think about how a omego db backup and restore command could fit into this workflow

Implementing backups by running pg_dump should be straightforward, the danger with doing it automatically is that it could take longer than expected. Restore would depend on the format- if it's custom pg_dump -Fc then it requires pg_restore, if it's in plain SQL format ...

re init, an alternative would be to have omego db init as well as omego init (including DB + server initialization)

I've already implemented omego db init :). It runs omero db script unless the file to be generated already exists (so the DB could be initialised with existing data).

[manics]

The upgrade test on Travis passes on my fork (though it fails later because there are no tags, so there is no version defined): https://travis-ci.org/manics/omero-setup/builds/20954998
Whereas this PR build fails with no indication of why: https://travis-ci.org/ome/omero-setup/builds/20955000

[sbesson]

This now needs merging of origin/master.

[manics]

I think this is ready for some testing.

[pwalczysko]

Atm I have problems with my old omego. I just do not get the linking between the OMERO-CURRENT and the folder with the unzipped server right.
The first mistake might have been to execute omego upgrade from inside the OMERO-CURRENT folder itself. Then I had some other problems with the omego upgrade reporting it cannot find something in the server-folder which I have downloaded via the first misplaced omego command. So I deleted all the folders except the one relevant OMERO.server folder. But then OMERO-CURRENT was not linked to it properly. So I deleted OMERO-CURRENT and recreated it. Then I tried to link it to the folder, but failed.

petr@ls25165 ~/Work/omero5-server]$ ln OMERO-CURRENT OMERO.server-5.0.1-rc1-155-d1d79c1-ice33-b203
ln: OMERO-CURRENT: Is a directory
[petr@ls25165 ~/Work/omero5-server]$ ln /OMERO-CURRENT /OMERO.server-5.0.1-rc1-155-d1d79c1-ice33-b203
ln: /OMERO-CURRENT: No such file or directory
[petr@ls25165 ~/Work/omero5-server]$ ln -s /OMERO-CURRENT /OMERO.server-5.0.1-rc1-155-d1d79c1-ice33-b203
ln: /OMERO.server-5.0.1-rc1-155-d1d79c1-ice33-b203: Permission denied
[petr@ls25165 ~/Work/omero5-server]$ 

Any ideas please ?

[pwalczysko]

Maybe some progress:
I have merged this PR and then did
pip install omego
Then I went into my omero5-server folder and did

[petr@ls25165 ~/Work/omero5-server]$ rm -rf OMERO*
[petr@ls25165 ~/Work/omero5-server]$ ls
ICE=3.5                ci.openmicroscopy.org  matlab                 unzip
[petr@ls25165 ~/Work/omero5-server]$ omego upgrade --branch OMERO-5.0-merge-ice33 --skipweb=true 

followed by the very long output (probably unpacking the server, no errors there)
but at the end:

  inflating: OMERO.server-5.0.1-rc1-155-d1d79c1-ice33-b203/sql/misc/current.sql  
  inflating: OMERO.server-5.0.1-rc1-155-d1d79c1-ice33-b203/sql/misc/Makefile  
2014-04-03 19:37:01,812 [omego.upgrad] INFO  UnixUpgrade: Upgrading OMERO.server-5.0.1-rc1-155-d1d79c1-ice33-b203 (OMERO-CURRENT)...
2014-04-03 19:37:03,339 [        root] INFO  Processing custom settings for module omeroweb.settings
2014-04-03 19:37:03,615 [omego.upgrad] ERROR Failed to load environment variables from OMERO-CURRENT/omero.envvars: [Errno 2] No such file or directory: 'OMERO-CURRENT/omero.envvars'
2014-04-03 19:37:03,615 [omego.upgrad] INFO  Stopping server
2014-04-03 19:37:03,615 [omego.upgrad] INFO  Running [old environment]: omero admin status --nodeonly
2014-04-03 19:37:04,439 [omego.upgrad] ERROR Error whilst stopping server: Non-zero return code: 1
2014-04-03 19:37:04,440 [omego.upgrad] INFO  /Users/petr/config.xml not found. Copying old files
Traceback (most recent call last):
  File "/usr/local/bin/omego", line 8, in <module>
    load_entry_point('omego==0.1.2', 'console_scripts', 'omego')()
  File "/Library/Python/2.7/site-packages/omego/main.py", line 46, in entry_point
    (Version.NAME, Version)])
  File "/Library/Python/2.7/site-packages/omego/framework.py", line 185, in main
    ns.func(ns)
  File "/Library/Python/2.7/site-packages/omego/upgrade.py", line 446, in __call__
    UnixUpgrade(dir, args)
  File "/Library/Python/2.7/site-packages/omego/upgrade.py", line 80, in __init__
    self.configure(noconfigure)
  File "/Library/Python/2.7/site-packages/omego/upgrade.py", line 115, in configure
    if os.path.samefile(old_cfg, target):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 153, in samefile
    s1 = os.stat(f1)
OSError: [Errno 2] No such file or directory: 'OMERO-CURRENT/etc/grid/config.xml'
[petr@ls25165 ~/Work/omero5-server]$ 

But OMERO-CURRENT was magically created, see below:

[petr@ls25165 ~/Work/omero5-server]$ ls -l
total 367176
-rw-r--r--  1 petr  staff          0 22 Jan 12:55 ICE=3.5
lrwxr-xr-x  1 petr  staff         45  3 Apr 19:37 OMERO-CURRENT@ -> OMERO.server-5.0.1-rc1-155-d1d79c1-ice33-b203
drwxr-xr-x  9 petr  staff        306  3 Apr 06:07 OMERO.server-5.0.1-rc1-155-d1d79c1-ice33-b203/
-rw-r--r--  1 petr  staff  187987075  3 Apr 19:36 OMERO.server-5.0.1-rc1-155-d1d79c1-ice33-b203.zip
-rw-r--r--  1 petr  staff          0 22 Jan 12:55 ci.openmicroscopy.org
-rw-r--r--  1 petr  staff          0 22 Jan 12:55 matlab
-rw-r--r--  1 petr  staff          0 22 Jan 12:55 unzip
[petr@ls25165 ~/Work/omero5-server]$ 

[manics]

omego upgrade in general isn't design to handle initialilising a new server. In practice it works if you're using the default omero config because of a combination of commit e6aff5f which creates a symlink if none is present, and the automatic creation of etc/grid/config.xml by import omero.cli in Upgrade.setup_script_environment, now External.setup_omero_cli() (bug or feature?).

To add to the confusion I'm not sure if ome/openmicroscopy#2184 was in your build, and whether it would affect things.

When I tried this just now it seemed to work:

$ omego upgrade --branch OMERO-5.0-merge-ice35

2014-04-07 17:08:46,378 [omego.extern] INFO  Executing [custom environment]: ome
ro admin status --nodeonly
2014-04-07 17:08:47,001 [omego.upgrad] ERROR Error whilst stopping server: Non-zero return code
command: omero admin status --nodeonly
return code: 1
stdout:
stderr:
2014-04-07 17:08:47,001 [omego.upgrad] INFO  Stopping web
2014-04-07 17:08:47,002 [omego.extern] INFO  Running [old environment]: web stop
2014-04-07 17:08:47,002 [omego.extern] INFO  Executing [custom environment]: omero web stop
2014-04-07 17:08:47,579 [omego.upgrad] INFO  /Users/simon/config.xml not found. Copying old files
2014-04-07 17:08:47,582 [omego.extern] INFO  Invoking CLI [current environment]: admin ports --skipcheck --registry 4061 --tcp 4063 --ssl 4064

Can you try this again exeactly as before: delete everything and re-run. Also pass -v to get some debug logs.

The plan is to have a proper omego init command, hopefully soon, along with an overhaul of the configuration system. We should decide on the scope of this PR.

[manics]

Also pip install omego will install the latest release, not the one you're meant to be testing.

[pwalczysko]

Works fine for me. Thanks to @manics for explanation and handholding. Ready to merge.

[sbesson]

@manics: next steps? Merge this, tag, upload a new version of omego and use it in the breaking suite?

[manics]

@sbesson Could do, though bear in mind I'm planning some further breaking changes e.g. refactoring command line arguments #27. If it's just one or two jobs then it shouldn't be too much work to modify them again later.