Skip to content

Commit 07cdb9f

Browse files
exajobsexajobs
authored
Add files via upload
1 parent b2781ca commit 07cdb9f

37 files changed

+944
-0
lines changed

‎elasticsearch-admin-scripts/README.md

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# elasticsearch-admin-scripts
2+
3+
This repo stores the Cerebro team's scripts that make it easier to manage Elasticsearch. Detailed descriptions, including required arguments and example usages, are in the scripts themselves.
4+
5+
## Bash scripts
6+
7+
### indexUtil.sh
8+
Every script makes use of `indexUtil.sh` for argument handling and url building. Possible arguments and examples follow.
9+
10+
### Required args
11+
Every script requires two arguments:
12+
* `env`: the environment of the cluster the script is run against. This is used to build the URL of the cluster the script will be run against. Options are:
13+
* `local`
14+
* `oscf-dev`
15+
* `oscf-stage`
16+
* `oscf-prod`
17+
* `user`: a user name with access to the cluster in `env`. You will be prompted for the password when the script is run.
18+
19+
### Other arguments
20+
21+
#### `index`
22+
Name of the index the command will run against. You can see what indexes are on your cluster by running `./indexList.sh` for the desired environment.
23+
24+
#### `docType`
25+
The name of the type of document that the command will run against, e.g. `constituent`. You define document types on your index with a name and field mappings. An index may have multiple types.
26+
27+
#### `alias`
28+
The index alias the script will be run against, e.g. `lonxt-current`. You can see what aliases have been created on your cluster with `./getIndexAliases.sh`.
29+
30+
#### `docMappingFile`
31+
The path to a file with the json mapping for a document type. For example:
32+
```json
33+
{
34+
"settings" : {
35+
"number_of_shards" : 1
36+
},
37+
"mappings" : {
38+
"type1" : {
39+
"properties" : {
40+
"field1" : { "type" : "text" }
41+
}
42+
}
43+
}
44+
}
45+
```
46+
47+
#### `query`
48+
The path to a file containing the body of the search query request that will be made against an index. Note: must be valid json. Example:
49+
```json
50+
{
51+
"query": {
52+
"term": { "contact_id": "782cd1a5-a560-4c77-a9ed-904d8d1a582b" }
53+
}
54+
}
55+
```
56+
57+
#### `update`
58+
The path of a file containing the body of the update request that will be made against an index. Note: must be valid json. Example:
59+
```json
60+
{
61+
"script": {
62+
"lang": "painless",
63+
"file": "delete_individual_gift",
64+
"params": {
65+
"gift_id": "1"
66+
}
67+
},
68+
"query": {
69+
"term": { "contact_id": "783cd1a5-a260-4c67-aaed-904d8d1a582b" }
70+
}
71+
}
72+
```
73+
74+
#### `file`
75+
The path of a file containing some json used to add or update some aspect of the cluster or index. Script-dependent.
76+
77+
#### `slices`
78+
A positive integer value indicating the number of slices Elasticsearch should use in an update by query. Note: this is only applicable to the `updateByQueryLongRunning` script. Note also that Elasticsearch suggests using a multiple of the number of shards on the index, with the best performance seen using the exact number of shards on the index.
79+
80+
## Prettifying output
81+
Output of the scripts being run is in unformatted json. We highly recommend you install and use `jq`.
82+
83+
## Task monitoring script
84+
We have a handy dandy little python script to report out Elasticsearch task info, which is most useful like so:
85+
```bash
86+
watch -n 3 "curl -# --user 'elastic:PASSWORD' 'http://es-master.oscf-prod.blackbaudcloud.com:9200/_tasks?detailed=true&actions=*byquery' | python3 taskStatus.py
87+
```
88+
89+
Things to note:
90+
* `-n 3` indicates the watch will update every 3 seconds.
91+
* `PASSWORD` will need to change.
92+
* `actions=*byquery` specifically targets Update By Query tasks. This can be changed to monitor other tasks (e.g.`*reindex`).

‎elasticsearch-admin-scripts/addField.sh

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
#! /bin/bash
2+
#
3+
# Add a new field to an existing index.
4+
# Field mapping is defined in specified file. Must be valid json.
5+
# Required args: user, env, index, docType, file
6+
#
7+
# Example usage:
8+
# ./addField --user "elastic" --env "oscf-dev" --index "lonxt-dev" --docType "constituent" --file "./field1.json"
9+
#
10+
11+
source "./indexUtil.sh"
12+
13+
echo "Environment $environment"
14+
echo "Index name $indexName"
15+
16+
buildElasticSearchUrl elasticSearchUrl $environment $indexName
17+
echo $elasticSearchUrl
18+
19+
curl -k --user $username -X PUT $elasticSearchUrl/_mapping/$documentType -d "@$file"

‎elasticsearch-admin-scripts/createIndex.sh

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
#! /bin/bash
2+
#
3+
# Create new index with the given name.
4+
# Index schema defined in specified file. Must be valid json.
5+
# Required args: user, env, index, docMappingFile
6+
#
7+
# Example usage:
8+
# ./createIndex.sh --user "elastic" --env "oscf-dev" --index "lonxt-dev" --docMappingFile "./indexMapping.json"
9+
#
10+
11+
source "./indexUtil.sh"
12+
13+
echo "Environment $environment"
14+
echo "Index name $indexName"
15+
echo "Doc mapping file $docMappingFile"
16+
17+
buildElasticSearchUrl elasticSearchUrl $environment $indexName
18+
19+
curl -k --user $username -X POST $elasticSearchUrl -d "@$docMappingFile"

‎elasticsearch-admin-scripts/createIndexAlias.sh

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#! /bin/bash
2+
#
3+
# Create new index alias associated with the specified index.
4+
# Required args: user, env, index, alias
5+
#
6+
# Example usage:
7+
# ./createIndexAlias.sh --user "elastic" --env "oscf-dev" --index "lonxt-dev" --alias "lonxt-dev-current"
8+
#
9+
10+
source "./indexUtil.sh"
11+
12+
echo "Environment $environment"
13+
14+
buildElasticSearchUrl elasticSearchUrl $environment $indexName
15+
16+
curl -k --user $username -X PUT $elasticSearchUrl/_alias/$alias

‎elasticsearch-admin-scripts/deleteIndex.sh

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#! /bin/bash
2+
#
3+
# Delete existing index with the specified name.
4+
# Required args: user, env, index
5+
#
6+
# Example usage:
7+
# ./deleteIndex --user "elastic" --env "oscf-dev" --index "lonxt-v1"
8+
#
9+
10+
source "./indexUtil.sh"
11+
12+
echo "Environment $environment"
13+
echo "Index name $indexName"
14+
15+
buildElasticSearchUrl elasticSearchUrl $environment $indexName
16+
17+
curl -k --user $username -X DELETE $elasticSearchUrl

‎elasticsearch-admin-scripts/deleteIndexAlias.sh

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#! /bin/bash
2+
#
3+
# Delete alias with the specified name for the given index.
4+
# Required args: user, env, index, alias
5+
#
6+
# Example usage:
7+
# ./deleteIndexAlias --user "elastic" --env "oscf-dev" --index "lonxt-dev" --alias "lonxt-mistake"
8+
#
9+
10+
source "./indexUtil.sh"
11+
12+
echo "Environment $environment"
13+
14+
buildElasticSearchUrl elasticSearchUrl $environment $indexName
15+
16+
curl -k --user $username -X DELETE $elasticSearchUrl/_alias/$alias

‎elasticsearch-admin-scripts/fields/administrative_area_normalized.json

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"properties": {
3+
"address" : {
4+
"type" : "nested",
5+
"properties" : {
6+
"administrative_area_normalized" : {
7+
"type" : "keyword",
8+
"index" : "true"
9+
}
10+
}
11+
}
12+
}
13+
}

‎elasticsearch-admin-scripts/fields/country_normalized.json

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"properties": {
3+
"address" : {
4+
"type" : "nested",
5+
"properties" : {
6+
"country_normalized" : {
7+
"type" : "keyword",
8+
"index" : "true"
9+
}
10+
}
11+
}
12+
}
13+
}

‎elasticsearch-admin-scripts/fields/date_of_birth.json

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"properties": {
3+
"date_of_birth": {
4+
"type": "date",
5+
"format": "yyyy/MM/dd||epoch_millis"
6+
}
7+
}
8+
}

‎elasticsearch-admin-scripts/fields/gift.json

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
{
2+
"properties": {
3+
"gifts": {
4+
"type": "nested",
5+
"properties": {
6+
"id": {
7+
"type": "keyword",
8+
"index": "true"
9+
},
10+
"amount": {
11+
"type": "long"
12+
},
13+
"gift_type": {
14+
"type": "keyword",
15+
"index": "true"
16+
},
17+
"frequency":{
18+
"type": "keyword",
19+
"index": "true"
20+
},
21+
"scheduled": {
22+
"type": "date",
23+
"format": "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd HH:mm||yyyy-MM-dd||epoch_millis"
24+
},
25+
"gift_date": {
26+
"type": "date",
27+
"format": "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd HH:mm||yyyy-MM-dd||epoch_millis"
28+
},
29+
"campaign_id": {
30+
"type": "long"
31+
},
32+
"form_id": {
33+
"type": "long"
34+
}
35+
}
36+
}
37+
}
38+
}

‎elasticsearch-admin-scripts/fields/group_membership.json

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"properties": {
3+
"group_membership": {
4+
"type": "long"
5+
}
6+
}
7+
}

‎elasticsearch-admin-scripts/fields/locality.lowercase.json

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
{
2+
"properties": {
3+
"address" : {
4+
"type" : "nested",
5+
"properties" : {
6+
"locality" : {
7+
"type" : "keyword",
8+
"index" : "true",
9+
"fields" : {
10+
"lowercase" : {
11+
"type" : "keyword",
12+
"normalizer" : "case_insensitive"
13+
}
14+
}
15+
}
16+
}
17+
}
18+
}
19+
}

‎elasticsearch-admin-scripts/fields/postal_code.lowercase.json

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
{
2+
"properties": {
3+
"address" : {
4+
"type" : "nested",
5+
"properties" : {
6+
"postal_code_normalized" : {
7+
"type" : "keyword",
8+
"index" : "true",
9+
"fields" : {
10+
"lowercase" : {
11+
"type" : "keyword",
12+
"normalizer" : "case_insensitive"
13+
}
14+
}
15+
}
16+
}
17+
}
18+
}
19+
}

‎elasticsearch-admin-scripts/fields/sub_adminiatrative_area.lowercase.json

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
{
2+
"properties": {
3+
"address" : {
4+
"type" : "nested",
5+
"properties" : {
6+
"sub_administrative_area" : {
7+
"type" : "keyword",
8+
"index" : "true",
9+
"fields" : {
10+
"lowercase" : {
11+
"type" : "keyword",
12+
"normalizer" : "case_insensitive"
13+
}
14+
}
15+
}
16+
}
17+
}
18+
}
19+
}

‎elasticsearch-admin-scripts/fields/wealth.json

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
{
2+
"properties": {
3+
"wealth": {
4+
"type": "object",
5+
"properties": {
6+
"score": {
7+
"type": "integer"
8+
},
9+
"date_scored": {
10+
"type" : "date",
11+
"format" : "yyyy/MM/dd HH:mm:ss||yyyy-MM-dd||yyyy/MM/dd||epoch_millis",
12+
"index" : "not_analyzed"
13+
}
14+
}
15+
}
16+
}
17+
}

‎elasticsearch-admin-scripts/fields/y2k_birthday.json

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"properties": {
3+
"y2k_birthday": {
4+
"type": "date",
5+
"format": "yyyy/MM/dd||epoch_millis"
6+
}
7+
}
8+
}

‎elasticsearch-admin-scripts/getAggregations.sh

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
#! /bin/bash
2+
#
3+
# Run the given aggregation query for the given index and docType.
4+
# Required args: user, env, index, docType, query
5+
#
6+
# Example usage:
7+
# ./getAggregations --user "elastic" --env "oscf-dev" --index "lonxt-dev" --docType "constituent" --query "aggregationQuery.json"
8+
#
9+
10+
source "./indexUtil.sh"
11+
12+
buildSearchUrl searchUrl $environment $indexName $documentType
13+
14+
curl -k --user $username -X GET $searchUrl -d "@$query"

‎elasticsearch-admin-scripts/getDocumentCount.sh

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
#! /bin/bash
2+
#
3+
# Get a count of the documents on the given index.
4+
# Required args: user, env, index
5+
#
6+
# Example usage:
7+
# ./getDocumentCount --user "elastic" --env "oscf-dev" --index "lonxt-dev"
8+
#
9+
10+
source "./indexUtil.sh"
11+
12+
buildElasticSearchCatUrl elasticSearchCatUrl $environment
13+
curl -k --user $username -X GET $elasticSearchCatUrl/count/$indexName?v

0 commit comments

Comments
 (0)