Add selectable distribution support for cloudera.cluster (#35)

* FIX: parcel os distribution
- When dynamically creating the cluster template, the OS distribution is not taken into account.
- For instance, Ubuntu tries to load the KEYTRUSTEE_SERVER parcel but fails because it's only available for `el7` and `el8`

Signed-off-by: Cody Fincher <[email protected]>

* FIX: add ansible_distribution_release to product parsing.

Signed-off-by: Cody Fincher <[email protected]>

* Centos 8 support

WIP
Disable dnf AppStream module for Postgresql when centos 8 in use
Setup system Python2 for CM to use

Signed-off-by: Daniel Chaffelson <[email protected]>

* Add EL8 support to Cloudera Collections

Set default CM version to 7.4.4 in cloudera.cluster
Derive parcel version of el7 or el8 from inventory target OS family in cloudera.cluster.cloudera_manager.repo
Ensure that Python2/3 are present on nodes when deploying on Rhel8 family OS
Change cloudera-deploy defaults to el7, but allow el8 to be specified and recognised
Update utility VM used for download mirror to function correctly with el7 or el8 OS
Add default_cluster tag to cluster.yml playbook CM Agent heartbeat test to avoid obscure failures

Signed-off-by: Daniel Chaffelson <[email protected]>

* Add selectable distribution support for cloudera.cluster

Enable by setting parcel_distro in definition to [el7, el8, or bionic] per dynamic OS options in cloudera.exe.infrastructure.vars
Change default dynamic inventory selection strings from 'centos7' to match the distribution identity strings of el7 etc. - user can update defaults to a different el7 distro etc. in cloudera.exe.infrastructure.vars
Add Ubuntu 18.04 'bionic' as option to dynamic inventory
Determine preferred parcel distribution in cloudera-deploy init
Add uniqueness to generated dynamic inventory VM name to reflect selection of distribution in case multiple clusters are deployed in the same account
Move dynamic inventory OS selection to globals, update appropriate reference docs to reflect change
Add filtering by distro to download mirror support, and ensure that manifest is still always collected
Enforce no_log always when working with Paywall credentials
Increase initial paywall download timeout to 7200s due to present CDN speed issues when deploying on EC2 outside of us-east-1
Modify ansible.builtin.package lock_timeout to only be used on RedHat, as it is not a Debian option
Pass selected parcel distribution to repo analysis during intial deployment when target cluster OS is not yet determined by deployment
Move cloudera.cluster plays which require knowledge of the cluster distribution to run on the cloudera_manager host instead of the Ansible controller so the correct distribution actions are applied
Fix extract_products_from_manifests filter in cloudera.cluster to correctly reference self and process os_distribution value
Fix import ordering in cloudera.cluster filters.py to not break under recent versions of Python3
Add distribution specific tasks for cloudera.cluster.deployment.repometa so it can identify the cluster distribution using the strings recognised by Cloudera Manager deployment
Force refresh of apt package cache on Debian distributions during OS prereqs setup in cloudera.cluster as the package cache in the image is sometimes missing packages

Signed-off-by: Daniel Chaffelson <[email protected]>

* Improve for CDH5 and Centos7

Pin boto3 version <1.18 when using Python2 for s3sync to work
Set GPG to not be checked by default when deploying cm5
Set variants for cm5 and cm6/7 paths for cloudera manager URL

Signed-off-by: Daniel Chaffelson <[email protected]>

* Further multi-os improvements for clusters

Improve readability of cloudera manager database user tasks
Simplify variable inclusion for Debian OS by standardising for both Ubuntu18 and Ubuntu20
Replace hardcoded Ubuntu1804 version with derived version string where necessary
Improve ca_server setup to recognise python differences between el7, el8 and debian
Set rdbms setup to use psycopg2-binary instead of slightly less reliable python package install in Debian
Move os-specific configurations to top of os setup tasks to allow Python to be fixed first as dependency for other things
Add fix for Ubuntu20.04 to allow root to edit any file on the OS by default, which otherwise breaks cloudera manager database setup
Ensure python2 is symlinked to /usr/bin/python on Redhat where /usr/bin/python is not already symlinked. This fixes Ranger startup issues.
tls_signing now explicitly uses /bin/bash as executable to also work on Ubuntu20
Updated readme to reflect additionally tested OSs

Signed-off-by: Daniel Chaffelson <[email protected]>

Co-authored-by: Cody Fincher <[email protected]>

Fixes #32

Author: Chaffelson

README.md

@@ -23,7 +23,9 @@ This Ansible Collection is designed to work hand-in-hand with [Cloudera Deploy](
 ### Operating Systems
 
 - Red Hat / CentOS 7.x
-- Ubuntu 18.04.04 LTS (Bionic Beaver)
+- Red Hat / CentOS 8.x
+- Ubuntu 18.04 LTS (Bionic Beaver)
+- Ubuntu 20.04 LTS (Focal Fossa)
 
 Active development is focused on **CDP Private Cloud Base** (formerly CDP-DC) deployments and their respective platform compatibility matrices.
 

plugins/filter/filters.py

@@ -11,8 +11,8 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
-from __future__ import (absolute_import, division, print_function)
+from __future__ import absolute_import, division, print_function
+from typing import Optional
 import re
 
 __metaclass__ = type
@@ -74,15 +74,24 @@ def _flatten_dict_list(i, l, parents):
 
         return state
 
-    def extract_products_from_manifests(self, manifests):
+    def extract_products_from_manifests(self, manifests, os_distribution: Optional[str] = None):
         products = dict()
         for manifest in manifests:
-            for parcel in manifest['parcels']:
+            for parcel in manifest["parcels"]:
+                # fetch the full parcel name from the manifest
+                full_parcel_name = str(parcel["parcelName"])
+                # the parcel OS distribution is between the last "-" and the ".parcel" extension
+                parcel_os_distribution = full_parcel_name[
+                    full_parcel_name.rindex("-")
+                    + 1: full_parcel_name.rindex(".parcel")
+                ]
                 # take first parcel, strip off OS name and file extension
-                parcel_name = re.sub(r"-[a-z0-9]+\.parcel$", "", str(parcel['parcelName']))
+                parcel_name = re.sub(r"-[a-z0-9]+\.parcel$", "", full_parcel_name)
                 # the product name is before the first dash
-                product = parcel_name[:parcel_name.index("-")]
-                if product not in products:
+                product = parcel_name[: parcel_name.index("-")]
+                if product not in products and (
+                    os_distribution == parcel_os_distribution or os_distribution is None
+                ):
                     # the version string is everything after the first dash
                     version = parcel_name[parcel_name.index("-") + 1:]
                     products[product] = version

roles/cloudera_manager/agent/tasks/main.yml

@@ -37,7 +37,7 @@
 # Note on the CM server this will perform an upgrade of the server if it hasn't already been upgraded
 - name: Install or Upgrade Cloudera Manager agent packages
   ansible.builtin.package:
-    lock_timeout: 60
+    lock_timeout: "{{ (ansible_os_family == 'RedHat') | ternary(60, omit) }}"
     name: cloudera-manager-agent
     update_cache: yes
     state: latest

roles/cloudera_manager/daemons/tasks/main.yml

@@ -16,7 +16,7 @@
 
 - name: Install Cloudera Manager daemons package
   ansible.builtin.package:
-    lock_timeout: 180
+    lock_timeout: "{{ (ansible_os_family == 'RedHat') | ternary(180, omit) }}"
     name: cloudera-manager-daemons
     update_cache: yes
     state: present

roles/cloudera_manager/database/tasks/embedded.yml

@@ -16,7 +16,7 @@
 
 - name: Install Cloudera Manager embedded database
   ansible.builtin.package:
-    lock_timeout: 60
+    lock_timeout: "{{ (ansible_os_family == 'RedHat') | ternary(60, omit) }}"
     name:
     - cloudera-manager-server-db-2
     state: present

roles/cloudera_manager/database/tasks/external.yml

@@ -14,7 +14,7 @@
 
 ---
 
-- name: Create Cloudera Manager database user
+- name: Create Cloudera Manager database user for Postgres
   postgresql_user:
     name: "{{ cloudera_manager_database_user }}"
     password: "{{ cloudera_manager_database_password }}"
@@ -39,7 +39,7 @@
   - cloudera_manager_database_type == 'postgresql'
   - cloudera_manager_database_host in groups.db_server | default([])
 
-- name: Create Cloudera Manager database user
+- name: Create Cloudera Manager database user for MySQL
   mysql_user:
     name: "{{ cloudera_manager_database_user }}"
     password: "{{ cloudera_manager_database_password }}"
@@ -66,7 +66,7 @@
 
 - name: Prepare Cloudera Manager Server external database
   command: |
-    {{ cloudera_manager_database_prepare_script }}  -f --host {{ cloudera_manager_database_host }} --port {{ cloudera_manager_database_port }}
+    {{ cloudera_manager_database_prepare_script }} -f --host {{ cloudera_manager_database_host }} --port {{ cloudera_manager_database_port }}
     {{ cloudera_manager_database_type | regex_replace('mariadb', 'mysql') }}
     {{ cloudera_manager_database_name }}
     {{ cloudera_manager_database_user }}

roles/cloudera_manager/repo/defaults/main.yml

@@ -14,6 +14,6 @@
 
 ---
 cloudera_archive_base_url: https://archive.cloudera.com
-cloudera_manager_version: 7.1.4
+cloudera_manager_version: 7.4.4
 
 install_repo_on_host: yes

roles/cloudera_manager/repo/tasks/main-Debian.yml

@@ -18,18 +18,18 @@
   set_fact:
     cloudera_manager_repo_url_with_creds: "{{ cloudera_manager_repo_url | regex_replace('^(?P<proto>http[s]?://)','\\g<proto>' + cloudera_manager_repo_username|string + ':' + cloudera_manager_repo_password|string + '@')}}"
     cloudera_manager_repo_key_with_creds: "{{ cloudera_manager_repo_key | regex_replace('^(?P<proto>http[s]?://)','\\g<proto>' + cloudera_manager_repo_username|string + ':' + cloudera_manager_repo_password|string + '@')}}"
-  no_log: cloudera_manager_repo_username is defined
+  no_log: yes
   when: cloudera_manager_repo_username is defined
 
 - name: Add Cloudera Manager apt repository key
   apt_key:
     url: "{{ cloudera_manager_repo_key_with_creds | default(cloudera_manager_repo_key) }}"
     state: present
-  no_log: cloudera_manager_repo_username is defined
+  no_log: yes
 
 - name: Add Cloudera Manager apt repository
   apt_repository:
     repo: "deb [arch=amd64] {{ cloudera_manager_repo_url_with_creds | default(cloudera_manager_repo_url) }} {{ cloudera_manager_repo_apt_codename }} {{ cloudera_manager_repo_apt_component }}"
     filename: cloudera-manager
-  no_log: cloudera_manager_repo_username is defined
+  no_log: yes
 

roles/cloudera_manager/repo/tasks/main.yml

@@ -14,10 +14,9 @@
 
 ---
 
-- name: Include variables for not-Debian in cm6/7
+- name: Include variables
   include_vars:
     file: "{{ ansible_os_family }}.yml"
-  when: ansible_os_family != "Debian"
 
 - name: Correct repo URL for Redhat with cm5
   ansible.builtin.set_fact:
@@ -26,11 +25,6 @@
     - ansible_os_family != "Debian"
     - cloudera_manager_version.split('.')[0] == "5"
 
-- name: Include variables
-  include_vars:
-    file: "{{ ansible_distribution }}{{ ansible_distribution_major_version }}.yml"
-  when: ansible_os_family == "Debian"
-
 - name: Define Cloudera Manager trial repository details
   set_fact:
     cloudera_manager_repo_url: "{{ __cloudera_manager_repo_url_trial }}"

roles/cloudera_manager/repo/vars/Ubuntu18.yml renamed to roles/cloudera_manager/repo/vars/Debian.yml

@@ -14,8 +14,8 @@
 
 ---
 __cloudera_manager_major_version: "{{ cloudera_manager_version.split('.')[0] }}"
-__cloudera_manager_repo_url_trial: "{{ cloudera_archive_base_url }}/cm{{ __cloudera_manager_major_version }}/{{ cloudera_manager_version }}/ubuntu1804/apt"
-__cloudera_manager_repo_url_paywall: "{{ cloudera_archive_base_url }}/p/cm{{ __cloudera_manager_major_version }}/{{ cloudera_manager_version }}/ubuntu1804/apt"
+__cloudera_manager_repo_url_trial: "{{ cloudera_archive_base_url }}/cm{{ __cloudera_manager_major_version }}/{{ cloudera_manager_version }}/{{ ansible_distribution | lower }}{{ ansible_distribution_version | replace('.','') }}/apt"
+__cloudera_manager_repo_url_paywall: "{{ cloudera_archive_base_url }}/p/cm{{ __cloudera_manager_major_version }}/{{ cloudera_manager_version }}/{{ ansible_distribution | lower }}{{ ansible_distribution_version | replace('.','') }}/apt"
 __cloudera_manager_repo_key_filename: archive.key
 __cloudera_manager_repo_key_trial: "{{ __cloudera_manager_repo_url_trial }}/{{ __cloudera_manager_repo_key_filename }}"
 __cloudera_manager_repo_key_paywall: "{{ __cloudera_manager_repo_url_paywall }}/{{ __cloudera_manager_repo_key_filename }}"