Red Hat Ansible Tower Upgrade from 3.5 to 3.8 – when running setup.sh is not enough – or: I have made fire!
I a customer project I had to update an existing Red Hat Ansible Tower setup from version 3.5.1 to the newest available version 3.8. The upgrade scenario as described in 8. Upgrading an Existing Tower Installation — Ansible Tower Installation and Reference Guide v3.8.0 does not work here. For example: the delivered setup.sh is not able to restore data exported from Postgres 9.6 into a new Postgres 10 database. This upgrade scenario as described is a result of a long discussion with Red Hat support (Thanks to Swati – he did a great job!) and an intensive test period on local virtual machines until the live system was in the row.
Remark: The servers in this blog post are listed without domain names to make it easier readable.
The running Ansible Tower Setup
It contains three Red Hat Ansible Tower servers and the repository. All servers can connect among each other. Firewall ports are opened.
| Server | Tower Version | Operating System | PostgreSQL Version | Remarks |
| ansible-tower-01 | 3.5.1 | RHEL 7.8 | no Internet access | |
| ansible-tower-02 | 3.5.1 | RHEL 7.8 | no Internet access | |
| ansible-tower-03 | 3.5.1 | RHEL 7.8 | no Internet access | |
| ansible-db03 | RHEL 7.8 | 9.6 | no Internet access, runs on port 5432 |
Upgrade Path
This documents describes the upgrade path, a release cannot be more than two release numbers behind to upgrade it. In our case, Tower 3.5.1 needs to be updated to 3.7.1 first before 3.8 can be applied. Ansible Tower 3.8 requires Postgres 10.
What are the Recommended Upgrade Paths for Ansible Tower? – Red Hat Customer Portal
Bundles: We use bundles (aka offline installers) for version 3.7.4 and 3.8 – the Ansible Tower servers don’t require internet access to upgrade and install relevant packages. They are included in the bundle. The bundles are accessible by the Ansible Tower server where the setup processes are executed.
| Bundle | URL | Note |
| 3.7.4.1 | https://releases.ansible.com/ansible-tower/setup-bundle/ansible-tower-setup-bundle-3.7.4-1.tar.gz | free download |
| 3.8 | https://access.redhat.com/downloads/content/480/ver=1.2/rhel—7/1.2/x86_64/product-software | requires a Red Hat subscription to download |
Prerequisites
- The software and the inventory file of the current Ansible Tower installation
- Bundle of the future releases are downloaded
- Postgres 10 installed on database server
- A Red Hat Subscription Manifest File which contains the license information for Tower 3.8 – the Manifest file can only be created in the Red Hat account when a valid subscription is available
Upgrade Overview
- Shutdown all involved virtual machines properly, create a snapshot for fallback, restart all involved servers properly
- Install Postgres 10 on database server and create a new Postgres 10 database
- Manual export/import tower data into the new database with pg_dump/pg_restore
- Re-run Ansible Tower 3.5.1 setup against new database
- Upgrade to Ansible Tower 3.7.1
- Upgrade to Ansible Tower 3.8
- Add Red Hat Subscription Manifest
- I have made fire!
1. Shutdown Machines and create Snapshot of the Hosts
It is highly recommended to create a backup/snapshot of the existing environment in case of upgrade troubles. In my case, we did VMWare vRealize Automation snaphots first after shutting down tower and database properly.
Tower and Database Shutdown
ansible-tower-01 / ansible-tower-02 / ansible-tower-03: stop the tower service:
|
1 |
# ansible-tower-service stop |
db03: Stop the database service:
|
1 |
# systemctl stop postgresql-9.6.service |
Create a backup or snapshot
In my case, we did VMWare vRealize Automation snaphots first after shutting down tower and database properly.
Restart machines
Restart all involved servers and verify in the Tower UI that everything works properly. Tower and database are configured as OS service, they are started automatically after server power on.
2. Install Postgres 10 on database server and create a new Postgres 10 database
Install Postgres 10
Connection details of the new repository database:
- Host: ansible-db03
- Port: 5433 (old Tower repository was 5432)
There is no internet access to the official PostgreSQL yum repository. Therefore, download the following files from https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7.8-x86_64/ and copy them to /tmp to the databse server. Ansible Tower 3.7.x requires PostgreSQL 10. It does not work with a newer version! See here: https://docs.ansible.com/ansible-tower/latest/html/installandreference/requirements_refguide.html.
|
1 2 3 4 5 |
$ sudo su - # yum localinstall ./postgresql10-server-10.14-1PGDG.rhel7.x86_64.rpm \ postgresql10-libs-10.14-1PGDG.rhel7.x86_64.rpm \ postgresql10-10.14-1PGDG.rhel7.x86_64.rpm \ postgresql10-contrib-10.14-1PGDG.rhel7.x86_64.rpm |
Create a new PostgreSQL Cluster with Trivadis pgOperate
On the database server ansible-db03, a new cluster database is required with PostgreSQL version 10. Therefore I used pgOperate. pgOperate is part of the Open Source tool pgBasEnv. My Trivadis colleagues have developed a really cool framework to manage PostgreSQL clusters. Both tools with the manuals and examples are available on GitHub:
GitHub – Trivadis/pgbasenv: pgBasEnv – PostgreSQL Base Environment Tool
GitHub – Trivadis/pgoperate: pgOperate – PostgreSQL Operation Tool
Example to create a new cluster when the tools are installed in base directory /var/lib/pgsql/tvdtoolbox. You will find more information how to configure and how to use the tool on the GitHub pages.
|
1 2 3 |
$ cd /var/lib/pgsql/tvdtoolbox/pgoperate/etc $ cp parameters_mycls.conf.tpl parameters_tower01.conf $ vi parameters_tower01.conf |
Set the cluster parameters to create a new cluster which is running on port 5433:
|
1 2 3 4 |
TVD_PGHOME_ALIAS=pgh1014 PGSQL_BASE=/u01/app/postgres/tower01 PG_PORT=5433 PG_SUPERUSER_PWD=<see keepass> |
Set the alias and execute cluster creation script.
|
1 2 |
$ pgh1014 $ pgoperate --create-cluster -a tower01 |
Run root.sh to enable automated startup and allow user postgres to start/stop the service.
|
1 2 |
$ sudo su - # /u01/app/postgres/tower01/scripts/root.sh |
Login as user postgres and verify the new created cluster which is started automatically. Here you can see the old PostgreSQL database version 9.6 running together with version 10 as output from the status screen after login.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
$ sudo su - postgres pgBasEnv v1.1 by Trivadis AG Installation homes: ==================================================== ALIAS | VER | OPTIONS | HOME DIR ==================================================== pgh966 | 9.6.6 | ssl:1G:8K | /usr/pgsql-9.6 pgh1014 | 10.14 | ssl:1G:8K | /usr/pgsql-10 ==================================================== Cluster data directories: ===================================================================================================================== ALIAS | VER | STAT | PORT | PID | SIZE | PGDATA | LAST START | LAST START HOME ===================================================================================================================== pgd96 | 9.6 | UP | 5432 | 1241 | 8.8G | /u01/pgdata/tower | 2020-06-05 07:19 | /usr/pgsql-9.6 tower01 | 10 | UP | 5433 | 110468 | 43M | /u01/app/postgres/tower01/data | 2020-12-20 12:03 | /usr/pgsql-10 ===================================================================================================================== |
Verify that the file pg_hba.conf allows traffic to the database, for example allow database connects from each server:
|
1 2 3 4 5 6 7 8 9 10 11 |
$ vi /u01/app/postgres/tower01/etc/pg_hba.conf # Database administrative login by UNIX sockets # note: you may wish to restrict this further later local all postgres peer # TYPE DATABASE USER CIDR-ADDRESS METHOD local replication all trust local all all md5 host all all 0.0.0.0/0 md5 host all all ::/0 md5 |
0.0.0.0/0 ensures, that all Ansible Tower servers can connect to the new database. This does not implicit allow everybody to connect. For example you can control the access on OS firewall level.
Create a new Database for Tower Repository
Login as user postgres and start psql against the new cluster. In this case, we create a new database called tower10 with same username and password as the existing database.
|
1 2 3 |
$ psql -p 5433 postgres=# CREATE USER tower WITH ENCRYPTED PASSWORD 'thisismypassword'; postgres=# CREATE DATABASE tower10 OWNER tower; |
Verification
Test the database connection from all Tower servers to avoid firewall or configuration issues and verify if the new created database tower10 is listed.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
$ psql -U tower -p 5433 -h ansible-db03 -d tower10 psql (10.14) Type "help" for help. tower10=> \l List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges -----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres tower10 | tower | UTF8 | en_US.UTF-8 | en_US.UTF-8 | (4 rows) |
3. Manual Export / Import of Ansible Tower Repository Data
In this step we migrate the tower database to the new PostgreSQL 10 database. It’s recommended to stop the Ansible Tower before starting the export.
ansible-tower-01 / ansible-tower-02 / ansible-tower-03: stop the tower service:
|
1 |
# ansible-tower-service stop |
Export Data
|
1 2 |
# Backup pg_dump -h ansible-db03 -p 5432 -U tower -F c -b -v -f "/var/lib/pgsql/dbexport_tower_3.5.1.backup" tower |
Import Data
|
1 2 |
# Restore pg_restore -h ansible-db03 -p 5433 -U tower -d tower10 -v "/var/lib/pgsql/dbexport_tower_3.5.1.backup" |
Stop and disable existing Postgres 9.6 Database
|
1 |
# systemctl stop postgresql-9.6.service |
4. Re-run Ansible Tower 3.5.1 setup against new database
Adapt Inventory
In this step, the existing Tower setup is registered again against the new database. We have to change the inventory file of the existing 3.5.1 installation.
|
1 2 3 4 5 6 7 8 9 |
# vi /root/install/ansible-tower-setup-3.5.3-1/inventory -- old: pg_port='5432' pg_database='tower' -- new: pg_port='5433' pg_database='tower10' |
Example inventory file of the existing 3.5.1 installation with three Ansible Tower servers. Note: the [database] section is empty, so the setup procedure will not try to install a new database and uses the new one what we have created above.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
[tower] ansible-towert01 ansible-towert02 ansible-towert03 [database] [all:vars] admin_password='thisismypassword' pg_host='ansible-db03' pg_port='5433' pg_database='tower10' pg_username='tower' pg_password='thisismypassword' rabbitmq_username=tower rabbitmq_port=5672 rabbitmq_vhost=tower rabbitmq_use_long_name=true rabbitmq_password='thisismypassword' rabbitmq_cookie=cookiemonster |
Re-run setup.sh
As user root, execute the setup.sh script on ansible-towert01.
|
1 |
# /root/install/ansible-tower-setup-3.5.3-1/setup.sh |
Verify the setup playbook result – no failed tasks should occur. The Ansible Tower 3.5.1 runs now with the PostgreSQL database version 10 and is ready to upgrade. Verify if all Tower are running properly and log in in the user interface.
|
1 2 3 4 5 6 7 |
PLAY RECAP ******************************************************************************************************************************************************************************** ansible-towert01 : ok=125 changed=27 unreachable=0 failed=0 skipped=64 rescued=0 ignored=1 ansible-towert02 : ok=119 changed=28 unreachable=0 failed=0 skipped=68 rescued=0 ignored=1 ansible-towert03 : ok=119 changed=28 unreachable=0 failed=0 skipped=68 rescued=0 ignored=1 The setup process completed successfully. Setup log saved to /var/log/tower/setup-2020-12-21-21:08:46.log |
Verification
Log in into Ansible Tower Servers and verify if the version is still 3.5.1.
5. Upgrade to Ansible Tower 3.7.1
The software bundle is transferred to the target server. As I don’t have much free space, I moved the bundle on a NFS which is attached on all Ansible Tower servers. It’s recommended to stop the Ansible Tower before starting the upgrade process.
ansible-tower-01 / ansible-tower-02 / ansible-tower-03: stop the tower service:
|
1 |
# ansible-tower-service stop |
As user root, go to the install directory and extract the bundle.
|
1 2 |
# cd /nfs/ansible/tower/install # tar xvfz ansible-tower-setup-bundle-3.7.4-1.tar.gz |
For the inventory file, the same settings as used in 3.5.1 can be used. The RabbitMQ settings are not required anymore and can be removed. This component is removed from Ansible Tower during the upgrade process. Example inventory file:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# vi /nfs/ansible/tower/install/ansible-tower-setup-bundle-3.7.4-1/inventory [tower] ansible-towert01 ansible-towert02 ansible-towert03 [database] [all:vars] admin_password='thisismypassword' pg_host='ansible-db03' pg_port='5433' pg_database='tower10' pg_username='tower' pg_password='thisismypassword' |
Run setup.sh
As user root, execute the setup.sh script on ansible-towert01.
|
1 |
# /nfs/ansible/tower/install/ansible-tower-setup-bundle-3.7.4-1/setup.sh |
Verify the setup playbook result – no failed tasks should occur.
|
1 2 3 4 |
PLAY RECAP ********************************************************************************************************************************************************* ansible-towert01 : ok=132 changed=55 unreachable=0 failed=0 skipped=65 rescued=0 ignored=1 ansible-towert02 : ok=121 changed=50 unreachable=0 failed=0 skipped=61 rescued=0 ignored=1 ansible-towert03 : ok=121 changed=50 unreachable=0 failed=0 skipped=61 rescued=0 ignored=1 |
Verification
Log in into Ansible Tower Servers and verify if the version is now 3.7.4.
Package verification on Ansible Tower servers:
|
1 2 3 4 5 6 |
# rpm -qa | grep tower ansible-tower-venv-ansible-3.7.4-1.el7at.x86_64 ansible-tower-ui-3.7.4-1.el7at.x86_64 ansible-tower-venv-tower-3.7.4-1.el7at.x86_64 ansible-tower-server-3.7.4-1.el7at.x86_64 ansible-tower-3.7.4-1.el7at.x86_64 |
Output from the play where RabbitMQ is removed:
|
1 2 3 4 |
TASK [awx_install : Remove Tower rabbitmq settings] **************************************************************************************************************** changed: [ansible-towert01] => {"changed": true, "path": "/etc/tower/conf.d/rabbitmq.py", "state": "absent"} changed: [ansible-towert02] => {"changed": true, "path": "/etc/tower/conf.d/rabbitmq.py", "state": "absent"} changed: [ansible-towert03] => {"changed": true, "path": "/etc/tower/conf.d/rabbitmq.py", "state": "absent"} |
6. Upgrade to Ansible Tower 3.8
The software bundle is transferred to the target server. As I don’t have much free space, I moved the bundle on a NFS which is attached on all Ansible Tower servers. It’s recommended to stop the Ansible Tower before starting the upgrade process.
ansible-tower-01 / ansible-tower-02 / ansible-tower-03: stop the tower service:
|
1 |
# ansible-tower-service stop |
As user root, go to the install directory and extract the bundle.
|
1 2 |
# cd /nfs/ansible/tower/install # tar xvfz ansible-tower-setup-bundle-3.7.4-1.tar.gz |
For the inventory file, the same settings as used in 3.7.4 can be used. Example inventory file:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# vi /nfs/ansible/tower/install/ansible-automation-platform-setup-bundle-1.2.0-1/inventory [tower] ansible-towert01 ansible-towert02 ansible-towert03 [database] [all:vars] admin_password='thisismypassword' pg_host='ansible-db03' pg_port='5433' pg_database='tower10' pg_username='tower' pg_password='thisismypassword' |
Run setup.sh
As user root, execute the setup.sh script on ansible-towert01.
|
1 |
# /nfs/ansible/tower/install/ansible-automation-platform-setup-bundle-1.2.0-1/setup.sh |
Note: Ansible verifies if the ansible RPM version is 2.4 or higher. Only on the node where the installer runs, the ansible package is updated. If you want to upadte the package on the oder Tower servers too, you can force the update of the package to 2.9.15 with the parameter upgrade_ansible_with_tower=1. A manual upgrade on by rpm -Uhv is possible too, the ansible package is available in the bundle.
|
1 |
# /nfs/ansible/tower/install/ansible-automation-platform-setup-bundle-1.2.0-1/setup.sh -e upgrade_ansible_with_tower=1 |
Verify the setup playbook result – no failed tasks should occur.
|
1 2 3 4 |
PLAY RECAP ******************************************************************************************************************************************************************************** ansible-towert01 : ok=15 changed=0 unreachable=0 failed=1 skipped=32 rescued=0 ignored=1 ansible-towert02 : ok=14 changed=0 unreachable=0 failed=1 skipped=16 rescued=0 ignored=1 ansible-towert03 : ok=14 changed=0 unreachable=0 failed=1 skipped=16 rescued=0 ignored=1 |
Verification
Log in into Ansible Tower Servers and verify if the version is now 3.7.4.
Package verification on Ansible Tower servers:
|
1 2 3 4 5 6 |
# rpm -qa | grep tower ansible-tower-server-3.8.0-1.el7at.x86_64 ansible-tower-venv-ansible-3.8.0-1.el7at.x86_64 ansible-tower-venv-tower-3.8.0-1.el7at.x86_64 ansible-tower-ui-3.8.0-1.el7at.x86_64 ansible-tower-3.8.0-1.el7at.x86_64 |
7. Add Red Hat Subscription Manifest
In former versions, a license file was required. Now it has changed to a subscription manifest. This file was generated in the Red Hat customer portal. After the first login into the 3.8 servers, you have to add the manifest. That’s all folks.
8. I have made fire!
After spending a lot of time to figure out the correct way how to upgrade, doing a lot of tests and finally the implementation on the live system, I did it. After version 3.8 was showing up, I felt like Tom Hanks in the movie Cast Away – I have made fire!
Summary
Upgrading the Ansible Tower with the existing online documentation? No chance! I have opened a support case at Red Had to clarify a lot of this like changing the repository database, updating the ansible packages etc. The procedure with setup.sh -b / setup.sh -r as described in the upgrade documentation did not work. It needs a manual data transfer. I really like the method to install the new Ansible Tower versions with a bundle, so no internet connection is required to keep the environment up to date. Hopefully Red Hat will update the documentation in the near future, for example with an upgrade cookbook section or however they want to call it.