Parse config.xml for db parameters, add omero db dump #74

manics · 2015-11-27T17:06:56Z

Adds a new --use-db-config flag to omego db ... and omego upgrade --upgradedb .... If omero.db.* properties are already in the OMERO.server config using this flag means you don't need the --db{host,name,user} arguments

e.g. Instead of:
omego upgrade -v --sym OMERO-TEST --upgradedb --dbname name --dbuser user --dbpass pass ...
You can do:
omego upgrade -v --sym OMERO-TEST --upgradedb --use-db-config ...

To backup your existing omero DB (or replace --use-db-config with the old --db* arguments):
omego db dump --serverdir OMERO-TEST --use-db-config -v
If you want to override the dumped filename pass --dumpfile ...
Use -n for a dry-run`

joshmoore · 2015-11-27T21:12:13Z

Under what conditions will you not want to use omero.db.* properties if they are in config.xml? i.e would reversing the flag to --no-db-config be the more common case?

manics · 2015-11-30T12:57:04Z

@joshmoore Changed as you suggested. Suggestions on defaults if they're not specified in config.xml or commandline, especially if we eventually encourage externals to use it? At the moment:

dbhost: localhost,
dbuser: omero
dbpass: omero
dbname: no default, must be specified (thinking was to reduce chance of accidentally modifying an existing DB)

joshmoore · 2015-11-30T12:59:51Z

Defaults make sense since that's what will be used if no bin/omero config set is called. Under what conditions will an existing db be modified, only on upgrade, right?

manics · 2015-11-30T13:25:59Z

Yes, just that it'd be annoying if you inadvertently upgraded to a DEV schema since you couldn't revert to the last release.

joshmoore · 2015-11-30T13:38:58Z

Agreed. It's likely too big a change for here, but it just occurs to me (even for our own use) if it would be safer to do:

    $ psql -c "CREATE DATABASE omero52 WITH TEMPLATE omero51 OWNER dbuser;"
    $ bin/omero config set omero.db.name omero52
    $ psql -f sql/psql/OMERO5.2__0/OMERO5.1__0.sql

i.e. non-destructively upgrade unless a flag like --overwrite is passed.

manics · 2015-11-30T13:49:05Z

Issue added #75

manics · 2016-01-15T17:15:09Z

Fixed a bug reported in #69 (comment)

joshmoore · 2016-01-21T10:49:26Z

Time to retest?

manics · 2016-01-21T11:04:00Z

It's on @pwalczysko's todo list

pwalczysko · 2016-01-21T12:12:48Z

ToDo today.

pwalczysko · 2016-01-21T16:52:45Z

It seems that #69 (comment) is fixed.
Nevertheless, I am afraid that on the whole, I got into very confused state about how to use omego and mainly how to construct the commands.
~/Work/omego/omego/main.py install --upgradedb --dbname old_db to my surprise just started to do something, although I thought it will end up immediately with an error.
@manics : Would you please be able to advise a workflow about how to reasonably execute the command from the header or is it okay to conclude here that #69 (comment) is fixed ?

manics · 2016-01-24T18:45:54Z

@pwalczysko The flag was changed (#74 (comment)) so that the omero config file will be used during upgrades unless --no-db-config is given.

omego upgrade -v --sym OMERO-TEST --upgradedb should upgrade your db and server providing OMERO-TEST points to your existing server.

omego install -v --sym OMERO-TEST --upgradedb --dbname old_db will install a new server, but use an existing OMERO database called old_db and will upgrade it if necessary. In this case since you don't have an existing server or config file you'll need to explicitly provide --dbuser, --dbpass, and --dbhost if they are different from the defaults.

pwalczysko · 2016-01-26T16:53:17Z

step 1: I have created an old_db (a 5.0 DB) from scratch which I have connected successfully to an 5.0.0 server. Now I suppose that this old_db has the required config file (as per header of this PR).
In next step, I will

create the OMERO-TEST directory and link it to a new 5.2 server
execute the omego upgrade -v --sym OMERO-TEST --upgradedb --use-db-config
expect that the old_db will be upgraded to the level of this new server

manics · 2016-01-26T16:55:35Z

OMERO-TEST should point to your old server before you start (since you are doing an upgrade). omego upgrade will switch the symlink to point to the new server as part of the upgrade process.

pwalczysko · 2016-01-27T10:34:46Z

.[pwalczysko@ls31619 ~/Work/omego]$ ./omego/main.py db dump --serverdir OMERO-TEST --dumpfile dump_of_old_db
2016-01-27 10:28:39,177 [        root] INFO  Processing custom settings for module omeroweb.settings
2016-01-27 10:28:39,295 [    omego.db] INFO  DbAdmin: DbAdmin OMERO-TEST ...
Traceback (most recent call last):
  File "./omego/main.py", line 64, in <module>
    entry_point()
  File "./omego/main.py", line 54, in entry_point
    (Version.NAME, Version)])
  File "/usr/local/lib/python2.7/site-packages/yaclifw/framework.py", line 197, in main
    ns.func(ns)
  File "/Users/pwalczysko/Work/omego/omego/db.py", line 283, in __call__
    DbAdmin(d, args.dbcommand, args, ext)
  File "/Users/pwalczysko/Work/omego/omego/db.py", line 35, in __init__
    psqlv = self.psql('--version')
  File "/Users/pwalczysko/Work/omego/omego/db.py", line 210, in psql
    db, env = self.get_db_args_env()
  File "/Users/pwalczysko/Work/omego/omego/db.py", line 190, in get_db_args_env
    c = self.external.get_config()
  File "/Users/pwalczysko/Work/omego/omego/external.py", line 85, in get_config
    exclusive=False, read_only=True)
TypeError: __init__() got an unexpected keyword argument 'read_only'
[pwalczysko@ls31619 ~/Work/omego]$

Trying to make the dump of the DB which is 5.0 connected to the 5.0 server.

pwalczysko · 2016-01-27T11:20:49Z

The command
./omego/main.py upgrade -v --sym OMERO-TEST --upgradedb failed with

2016-01-27 10:47:40,077 [    omego.db] INFO  DbAdmin: DbAdmin ./OMERO.server-5.2.1-ice35-b15 ...
Traceback (most recent call last):
  File "./omego/main.py", line 64, in <module>
    entry_point()
  File "./omego/main.py", line 54, in entry_point
    (Version.NAME, Version)])
  File "/usr/local/lib/python2.7/site-packages/yaclifw/framework.py", line 197, in main
    ns.func(ns)
  File "/Users/pwalczysko/Work/omego/omego/upgrade.py", line 415, in __call__
    UnixInstall(self.NAME, args)
  File "/Users/pwalczysko/Work/omego/omego/upgrade.py", line 70, in __init__
    self.upgrade_db()
  File "/Users/pwalczysko/Work/omego/omego/upgrade.py", line 205, in upgrade_db
    DbAdmin(self.dir, 'upgrade', self.args, self.external)
  File "/Users/pwalczysko/Work/omego/omego/db.py", line 35, in __init__
    psqlv = self.psql('--version')
  File "/Users/pwalczysko/Work/omego/omego/db.py", line 210, in psql
    db, env = self.get_db_args_env()
  File "/Users/pwalczysko/Work/omego/omego/db.py", line 200, in get_db_args_env
    raise Exception('Database name required')
Exception: Database name required
2016-01-27 10:47:40,080 [omero.util.T] DEBUG Removing tree: /Users/pwalczysko/omero/tmp/omero_pwalczysko/55577
[pwalczysko@ls31619 ~/Work/omego]$ ./omego/main.py upgrade -v --sym OMERO-TEST --upgradedb

Again, 5.0 db and 5.0 server.

Older versions of OMERO don't support this flag

manics · 2016-02-02T11:01:23Z

omego db dump --serverdir OMERO.server-5.0.0-ice35-b19 -v should work with the last commit

pwalczysko · 2016-02-04T15:40:04Z

./omego/main.py db dump --serverdir OMERO-TEST --dumpfile dump_of_old_db executed without error and created a file dump_of_old_db as expected.

pwalczysko · 2016-02-04T15:45:30Z

Nevertheless the command
./omego/main.py upgrade -v --sym OMERO-TEST --upgradedb
failed with

2016-02-04 15:44:01,930 [omego.upgrad] DEBUG Upgrading database
2016-02-04 15:44:01,931 [    omego.db] INFO  DbAdmin: DbAdmin ./OMERO.server-5.2.1-ice35-b15 ...
Traceback (most recent call last):
  File "./omego/main.py", line 64, in <module>
    entry_point()
  File "./omego/main.py", line 54, in entry_point
    (Version.NAME, Version)])
  File "/usr/local/lib/python2.7/site-packages/yaclifw/framework.py", line 197, in main
    ns.func(ns)
  File "/Users/pwalczysko/Work/omego/omego/upgrade.py", line 415, in __call__
    UnixInstall(self.NAME, args)
  File "/Users/pwalczysko/Work/omego/omego/upgrade.py", line 70, in __init__
    self.upgrade_db()
  File "/Users/pwalczysko/Work/omego/omego/upgrade.py", line 205, in upgrade_db
    DbAdmin(self.dir, 'upgrade', self.args, self.external)
  File "/Users/pwalczysko/Work/omego/omego/db.py", line 35, in __init__
    psqlv = self.psql('--version')
  File "/Users/pwalczysko/Work/omego/omego/db.py", line 210, in psql
    db, env = self.get_db_args_env()
  File "/Users/pwalczysko/Work/omego/omego/db.py", line 200, in get_db_args_env
    raise Exception('Database name required')
Exception: Database name required
2016-02-04 15:44:01,932 [omero.util.T] DEBUG Removing tree: /Users/pwalczysko/omero/tmp/omero_pwalczysko/31344
[pwalczysko@ls31619 ~/Work/omego]$

Again, the attempted upgrade was from 5.0.0 server (downloads page).

THe presence/absence of config.xml is recorded at initialisation since importing omero.cli may lead to the creation of an empty config.xml when previosuly there was none. However for a server upgrade we want to re-read the config.xml after it is copied.

manics · 2016-02-05T17:18:33Z

@pwalczysko Sorry about that, I thought it was tested by travis but it wasn't (though it is now). Should all be working now.

pwalczysko · 2016-02-08T15:40:35Z

./omego/main.py upgrade -v --sym OMERO-TEST --upgradedb works fine now.
The DB was verified as upgraded from 5.0 to 5.2

old_db=# select * from dbpatch;
 id | currentpatch | currentversion | permissions |          finished          |                                           message                                            | previouspatch | previousversion | external_id 
----+--------------+----------------+-------------+----------------------------+----------------------------------------------------------------------------------------------+---------------+-----------------+-------------
  1 |            0 | OMERO5.0       |         -52 | 2016-01-21 15:32:20.333478 | Database ready.                                                                              |             0 | OMERO5.0        |            
  2 |            1 | OMERO5.1       |         -52 | 2016-02-08 15:16:40.356604 | Database updated.                                                                            |             0 | OMERO5.0        |            
  3 |            0 | OMERO5.2       |         -52 | 2016-02-08 15:16:40.395968 | Database updated.                                                                            |             1 | OMERO5.1        |            
  4 |            0 | OMERO5.2       |         -52 | 2016-02-08 15:16:51.084905 | DB check DBEnumCheck: done for Bio-Formats revision 8ef015dc4b2a5f2035e49486c9da9a9f301b53cc |             0 | OMERO5.2        |            
  5 |            0 | OMERO5.2       |         -52 | 2016-02-08 15:16:51.098742 | DB check DBUnicodeUnitsCheck: done                                                           |             0 | OMERO5.2        |            
(5 rows)

The server has started successfully.
Ready to merge.

Parse config.xml for db parameters, add omero db dump

manics added 3 commits November 27, 2015 13:20

Add omego dump, optionally parse db properties from config.xml

1314a2e

Add unit tests for pgdump()

ce5d95a

Fix name of omego db init command

f33e5ad

manics added 2 commits November 30, 2015 11:30

Invert use-db-config: no-db-config

46133e7

ConfigXml read_only flag was added in 5.0.2

41f540c

manics added 2 commits January 15, 2016 16:49

Check config.xml exists before reading properties

ffde65f

Update db config tests

28b9014

manics mentioned this pull request Jan 15, 2016

Remove ports config, add no-start option, convert boolean-string-args to flags #69

Merged

manics mentioned this pull request Jan 27, 2016

omero db should auto-detect serverdir if possible #77

Open

Don't fail if ConfigXml read_only arg not supported.

c8aaa3f

Older versions of OMERO don't support this flag

Force re-reading of config.xml

de23851

THe presence/absence of config.xml is recorded at initialisation since importing omero.cli may lead to the creation of an empty config.xml when previosuly there was none. However for a server upgrade we want to re-read the config.xml after it is copied.

manics added 2 commits February 5, 2016 16:38

Add a few more tests to travis-build

f00b066

Set omero.db.name in .travis.yml, use config.xml db params for upgrade

d3a83a3

joshmoore added a commit that referenced this pull request Feb 9, 2016

Merge pull request #74 from manics/pgdump

41693e1

Parse config.xml for db parameters, add omero db dump

joshmoore merged commit 41693e1 into ome:master Feb 9, 2016

manics deleted the pgdump branch February 9, 2016 10:39

joshmoore added this to the 0.3.0 milestone Feb 10, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parse config.xml for db parameters, add omero db dump #74

Parse config.xml for db parameters, add omero db dump #74

manics commented Nov 27, 2015

joshmoore commented Nov 27, 2015

manics commented Nov 30, 2015

joshmoore commented Nov 30, 2015

manics commented Nov 30, 2015

joshmoore commented Nov 30, 2015

manics commented Nov 30, 2015

manics commented Jan 15, 2016

joshmoore commented Jan 21, 2016

manics commented Jan 21, 2016

pwalczysko commented Jan 21, 2016

pwalczysko commented Jan 21, 2016

manics commented Jan 24, 2016

pwalczysko commented Jan 26, 2016

manics commented Jan 26, 2016

pwalczysko commented Jan 27, 2016

pwalczysko commented Jan 27, 2016

manics commented Feb 2, 2016

pwalczysko commented Feb 4, 2016

pwalczysko commented Feb 4, 2016

manics commented Feb 5, 2016

pwalczysko commented Feb 8, 2016

Parse config.xml for db parameters, add omero db dump #74

Parse config.xml for db parameters, add omero db dump #74

Conversation

manics commented Nov 27, 2015

joshmoore commented Nov 27, 2015

manics commented Nov 30, 2015

joshmoore commented Nov 30, 2015

manics commented Nov 30, 2015

joshmoore commented Nov 30, 2015

manics commented Nov 30, 2015

manics commented Jan 15, 2016

joshmoore commented Jan 21, 2016

manics commented Jan 21, 2016

pwalczysko commented Jan 21, 2016

pwalczysko commented Jan 21, 2016

manics commented Jan 24, 2016

pwalczysko commented Jan 26, 2016

manics commented Jan 26, 2016

pwalczysko commented Jan 27, 2016

pwalczysko commented Jan 27, 2016

manics commented Feb 2, 2016

pwalczysko commented Feb 4, 2016

pwalczysko commented Feb 4, 2016

manics commented Feb 5, 2016

pwalczysko commented Feb 8, 2016