Skip to content

Commit fca9ebf

Browse files
committed
METRON-2239 Metron Automated backup and restore (mmiklavc) closes #1546
1 parent 12e16df commit fca9ebf

File tree

4 files changed

+191
-0
lines changed

4 files changed

+191
-0
lines changed

Upgrade_steps.md

+40
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing, software
13+
distributed under the License is distributed on an "AS IS" BASIS,
14+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
See the License for the specific language governing permissions and
16+
limitations under the License.
17+
-->
18+
# Upgrade Steps
19+
General guidance for upgrading Metron
20+
21+
1. Cut-off all inputs to Metron
22+
1. Monitor processing until all in-flight data is indexed and stored
23+
1. Stop Metron service
24+
1. Run the [Metron Upgrade Helper](./metron-platform/metron-common#metron-upgrade-helper) script in backup mode - does these 2 tasks
25+
1. Export zookeeper based metron configs
26+
1. Export ambari-based metron configs
27+
1. Delete the metron service via Ambari
28+
1. Upgrade OS, if applicable
29+
1. Major Hadoop platform upgrade steps (example. using HDP 2.6 to 3.x) - https://docs.cloudera.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-upgrade-major/content/ambari_upgrade_guide.html
30+
1. Update Ambari to latest version
31+
1. Update to HDP 3.1 using existing Ambari/HDP update documentation (including Solr/ES etc).
32+
1. Reinstall Metron mpack using "--force" option
33+
1. Reinstall Metron service
34+
1. Turn off Metron service
35+
1. Tweak any configs required to support new Metron version
36+
1. Minimally, you should update `metron.home` in `Ambari -> Metron -> Configs -> Advanced metron-env -> Metron home` to point to the new Metron home dir.
37+
1. Run upgrade helper script in restore mode - does these 2 tasks
38+
1. Re-import zookeeper based metron configs
39+
1. Re-import ambari-based zookeeper configs
40+
1. Turn on Metron service

metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec

+1
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,7 @@ This package installs the Metron common files %{metron_home}
143143
%{metron_home}/bin/stellar
144144
%{metron_home}/bin/cluster_info.py
145145
%{metron_home}/bin/tgt_renew.py
146+
%{metron_home}/bin/upgrade_helper.sh
146147
%{metron_home}/config/zookeeper/global.json
147148
%attr(0644,root,root) %{metron_home}/lib/metron-common-%{full_version}-uber.jar
148149

metron-platform/metron-common/README.md

+30
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ limitations under the License.
2525
* [Topology Errors](topology-errors)
2626
* [Performance Logging](#performance-logging)
2727
* [Metron Debugging](#metron-debugging)
28+
* [Metron Upgrade Helper](#metron-upgrade-helper)
2829

2930
# Stellar Language
3031

@@ -471,3 +472,32 @@ Options:
471472
-p DIRECTORY, --hdp_home=DIRECTORY
472473
HDP home directory
473474
```
475+
476+
# Metron Upgrade Helper
477+
478+
A bash script is provided to assist in performing backup and restore operations for Metron Ambari configurations and configurations stored in Zookeeper.
479+
480+
If your Ambari Server is installed on a separate host from Metron, you may need to scp the upgrade_helper.sh script to the Ambari host along with the file `/etc/default/metron`.
481+
There is an optional argument, `directory_base`, that allows you to specify where you would like backups to be written to and restored from. Be aware that while it's optional, the
482+
default is to write the data to the directory from which you're executing the script, i.e. `./metron-backup`.
483+
484+
```
485+
# $METRON_HOME/bin/upgrade_helper.sh -h
486+
5 args required
487+
Usage:
488+
mode: [backup|restore] - backup will save configs to a directory named "metron-backup". Restore will take those same configs and restore them to Ambari.
489+
ambari_address: host and port for Ambari server, e.g. "node1:8080"
490+
username: Ambari admin username
491+
password: Ambari admin user password
492+
cluster_name: hadoop cluster name. Can be found in Ambari under "Admin > Manage Ambari"
493+
directory_base: (Optional) root directory location where the backup will be written to and read from. Default is the executing directory, ".", with backup data stored to a subdirectory named "metron-backup"
494+
```
495+
496+
```
497+
Examples:
498+
# backup
499+
$METRON_HOME/bin/upgrade_helper.sh backup node1:8080 admin admin metron_cluster
500+
# restore
501+
$METRON_HOME/bin/upgrade_helper.sh restore node1:8080 admin admin metron_cluster
502+
```
503+
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
#!/bin/bash
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing, software
14+
# distributed under the License is distributed on an "AS IS" BASIS,
15+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16+
# See the License for the specific language governing permissions and
17+
# limitations under the License.
18+
#
19+
20+
if [ "$#" -ne 5 ] || [ "$1" == "-h" ] || [ "$1" == "--help" ]; then
21+
echo 5 args required
22+
echo Usage:
23+
echo " mode: [backup|restore] - backup will save configs to a directory named \"metron-backup\". Restore will take those same configs and restore them to Ambari."
24+
echo " ambari_address: host and port for Ambari server, e.g. \"node1:8080\""
25+
echo " username: Ambari admin username"
26+
echo " password: Ambari admin user password"
27+
echo " cluster_name: hadoop cluster name. Can be found in Ambari under \"Admin > Manage Ambari\""
28+
echo " directory_base: (Optional) root directory location where the backup will be written to and read from. Default is the executing directory, \".\", with backup data stored to a subdirectory named \"metron-backup\""
29+
exit 1
30+
fi
31+
32+
mode=$1
33+
ambari_address=$2
34+
username=$3
35+
password=$4
36+
cluster_name=$5
37+
# optional base directory
38+
outdir_base=${6:-.}
39+
40+
if [ -f "/etc/default/metron" ]; then
41+
source /etc/default/metron
42+
fi
43+
44+
OUT_DIR=$outdir_base/metron-backup
45+
AMBARI_CONFIG_DIR=$OUT_DIR/ambari-configs
46+
ZK_CONFIG_DIR=$OUT_DIR/zk-configs
47+
48+
if [ "$mode" == "backup" ]; then
49+
if [ ! -d "$OUT_DIR" ]; then
50+
mkdir $OUT_DIR
51+
fi
52+
if [ ! -d "$AMBARI_CONFIG_DIR" ]; then
53+
mkdir $AMBARI_CONFIG_DIR
54+
fi
55+
if [ ! -d "$ZK_CONFIG_DIR" ]; then
56+
mkdir $ZK_CONFIG_DIR
57+
fi
58+
if [ -f "/var/lib/ambari-server/resources/scripts/configs.py" ]; then
59+
echo Backing up Ambari config...
60+
for config_type in $(curl -u $username:$password -H "X-Requested-By: ambari" -X GET http://$ambari_address/api/v1/clusters/$cluster_name?fields=Clusters/desired_configs | grep '" : {' | grep -v Clusters | grep -v desired_configs | cut -d'"' -f2 | grep metron);
61+
do
62+
echo Saving $config_type
63+
/var/lib/ambari-server/resources/scripts/configs.py -u $username -p $password -a get -l ${ambari_address%:*} -n $cluster_name -c $config_type -f $AMBARI_CONFIG_DIR/${config_type}.json
64+
done
65+
echo Done backing up Ambari config...
66+
else
67+
echo Skipping Ambari config backup - Ambari not found on this host
68+
fi
69+
70+
if [ -f "$METRON_HOME/bin/zk_load_configs.sh" ]; then
71+
echo Backing up Metron config
72+
$METRON_HOME/bin/zk_load_configs.sh -m PULL -o $ZK_CONFIG_DIR -z $ZOOKEEPER -f
73+
echo Done backing up Metron config
74+
else
75+
echo Skipping Metron config backup - Metron not found on this host
76+
fi
77+
elif [ "$mode" == "restore" ]; then
78+
if [ ! -d "$OUT_DIR" ]; then
79+
echo Backup directory not found, aborting
80+
exit 1
81+
fi
82+
if [ -f "/var/lib/ambari-server/resources/scripts/configs.py" ]; then
83+
if [ -d "$AMBARI_CONFIG_DIR" ]; then
84+
echo Restoring metron config from files in $AMBARI_CONFIG_DIR
85+
i=0
86+
for filename in $AMBARI_CONFIG_DIR/*;
87+
do
88+
[ -e "$filename" ] || continue
89+
((i=i+1))
90+
filename=${filename##*/}
91+
echo $i. Found config: $filename
92+
config_type=${filename%.json}
93+
echo " Setting config_type to $config_type"
94+
/var/lib/ambari-server/resources/scripts/configs.py -u $username -p $password -a set -l ${ambari_address%:*} -n $cluster_name -c $config_type -f $AMBARI_CONFIG_DIR/${config_type}.json
95+
echo " Done restoring $config_type"
96+
done
97+
echo Done restoring $i metron config files from $OUT_DIR
98+
else
99+
echo Ambari backup directory not found, skipping
100+
fi
101+
else
102+
echo Skipping Ambari config restore - Ambari not found on this host
103+
fi
104+
105+
if [ -f "$METRON_HOME/bin/zk_load_configs.sh" ]; then
106+
if [ -d "$ZK_CONFIG_DIR" ]; then
107+
echo Restoring Metron zookeeper config from files in $ZK_CONFIG_DIR
108+
$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i $ZK_CONFIG_DIR -z $ZOOKEEPER -f
109+
echo Pulling config locally into Metron home config dir
110+
$METRON_HOME/bin/zk_load_configs.sh -m PULL -o ${METRON_HOME}/config/zookeeper -z $ZOOKEEPER -f
111+
echo Done restoring Metron zookeeper config from files in $ZK_CONFIG_DIR
112+
else
113+
echo Metron config backup directory not found, skipping
114+
fi
115+
else
116+
echo Skipping Metron config restore - Metron not found on this host
117+
fi
118+
else
119+
echo Mode "$mode" not recognized. Exiting.
120+
fi

0 commit comments

Comments
 (0)