Latest Posts

OCI Cloud Performance Management for On-Premises Databases – Part 1 – Management Agent Installation

Author: Martin Berger 22. February 2021

The OCI Management Agent service collects data from services and sources for monitoring and management in Oracle Cloud Infrastructure. In this blog post series I will show you how you can monitor and manage an on-premises Oracle databases in OCI. The communication between an agent and OCI requires an Agent Install Key and is based on HTTPS. Service Plugins extend a Management Agent for example for Oracle database performance monitoring and management or log analytics.

This is a small blog post series

Part 1 shows the installation of the Management Agent on an on-premises system
Part 2 describes how to manage this Oracle database in OCI
Part 3 watches behind the scenes for billing, job execution etc.

My Setup

An OCI Tenant in datacenter EU-FRANKFURT-1
An OCI compartment called datacenter-kestenholz
An on-premises Container Database called CDB114, running on Oracle Linux 7
Three on-premises Pluggable Databases

The goal is to manage the on-premises database in Oracle Cloud Infrastructure OCI. Output from the Trivadis TVD-Basenv(TM) framework which show the database up and running:

TYPE (Cluster|DG)  : SID/PROCESS  STATUS      HOME      [2021-02-22 08:05:03]
-----------------------------------------------------------------------------
Dummy rdbms_ee     : rdbms19      n/a         /u01/app/oracle/product/19.0.0/dbhome_1

DB-instance (N|N)  : CDB144       open        /u01/app/oracle/product/19.0.0/dbhome_1

Listener           : LISTENER     up          /u01/app/oracle/product/19.0.0/dbhome_1

TYPE (Cluster|DG) : SID/PROCESS STATUS HOME [2021-02-22 08:05:03]

-----------------------------------------------------------------------------

Dummy rdbms_ee : rdbms19 n/a /u01/app/oracle/product/19.0.0/dbhome_1

DB-instance (N|N) : CDB144 open /u01/app/oracle/product/19.0.0/dbhome_1

Listener : LISTENER up /u01/app/oracle/product/19.0.0/dbhome_1

Prerequisites for Management Agent Installation

Oracle Linux 6 or higher
Red Hat Enterprise Linux 6 or higher
CentOS 6 or 7
SUSE Linux Enterprise Server 12 or 15
Windows Server 2012 R2, 2016 or 2019

There are other prerequisites on the target server like the correct Java version (e.g. version 11 does not work) and sudo permissions. For the complete list of prerequisites, see here:

https://docs.oracle.com/en-us/iaas/management-agents/doc/perform-prerequisites-deploying-management-agents.html#GUID-BC5862F0-3E68-4096-B18E-C4462BC76271

Setup for OCI Management Agent

It is recommended to handle the agents in a separate user group and with policies. This allows us to define the Management Agent management on an fine granular level.

Group AGENT_ADMINS

According the documentation, I have created an user group called AGENT_ADMINS.

Policy Datacenter_Kestenholz_Agent_Policy

A new policy is created that allows the admin group to interact with the management agents, handle keys etc.

ALLOW GROUP AGENT_ADMINS TO MANAGE management-agents IN COMPARTMENT datacenter-kestenholz
ALLOW GROUP AGENT_ADMINS TO MANAGE management-agent-install-keys IN COMPARTMENT datacenter-kestenholz
ALLOW GROUP AGENT_ADMINS TO READ METRICS IN COMPARTMENT datacenter-kestenholz
ALLOW GROUP AGENT_ADMINS TO READ USERS IN TENANCY

ALLOW GROUP AGENT_ADMINS TO MANAGE management-agents IN COMPARTMENT datacenter-kestenholz

ALLOW GROUP AGENT_ADMINS TO MANAGE management-agent-install-keys IN COMPARTMENT datacenter-kestenholz

ALLOW GROUP AGENT_ADMINS TO READ METRICS IN COMPARTMENT datacenter-kestenholz

ALLOW GROUP AGENT_ADMINS TO READ USERS IN TENANCY

Dynamic Group Management_Agent_Dynamic_Group

New added agents in the compartment belong automatically to this group. Replace the OCID with the OCID for your compartment.

Policy Datacenter_Kestenholz_Agent_Communication_Policy

A policy is required that allows the agents to communicate with the OCI endpoints. This policy is important, otherwise you run in an communication error (see below in section troubleshooting).

ALLOW DYNAMIC-GROUP Management_Agent_Dynamic_Group TO MANAGE management-agents IN COMPARTMENT datacenter-kestenholz
ALLOW DYNAMIC-GROUP Management_Agent_Dynamic_Group TO USE METRICS IN COMPARTMENT datacenter-kestenholz

1 2	ALLOW DYNAMIC-GROUP Management_Agent_Dynamic_Group TO MANAGE management-agents IN COMPARTMENT datacenter-kestenholz ALLOW DYNAMIC-GROUP Management_Agent_Dynamic_Group TO USE METRICS IN COMPARTMENT datacenter-kestenholz

Install On-Premises Management Agent

Create Agent Install Key

Go to Management Agent Menu / Downloads and Keys, create a new Agent Install Key. Set the compartment and the time how long the key is valid. In this example, I need to replace the key after one month.

When click on Download Key to File, a textfile is created with the ManagementAgentInstallKey and all other (optional) parameters which can be used for install. You can use this file as responsefile template later.

Download the Software and Transfer it to the Target Server

I use the Agent for Linux and transferred it to the target on-premises server into a stage directory as OS user root.

[root@bifangstrasse stage]# ll
total 8128
-rw-r--r--. 1 root root 8323072 Feb 21 20:38 oracle.mgmt_agent.rpm

[root@bifangstrasse stage]# ll

total 8128

-rw-r--r--. 1 root root 8323072 Feb 21 20:38 oracle.mgmt_agent.rpm

Create a Local Response File

This is an example of a simple two-lines response file for agent installation in the same folder where the rpm is located, called input.rsp. The parameter managementAgentInstallKey is visible in the OCI web interface, the CredentialWalletPassword is your password for the wallet.

managementAgentInstallKey = MS4wLHVzLWFzaGJ1cm4tMSxvY2lkMS50ZW5hbmN5-<YOUR_AGENT_KEY_HERE>
CredentialWalletPassword = W3lcome#1

1 2	managementAgentInstallKey = MS4wLHVzLWFzaGJ1cm4tMSxvY2lkMS50ZW5hbmN5-<YOUR_AGENT_KEY_HERE> CredentialWalletPassword = W3lcome#1

RPM Installation

As OS user root (or a user with sudo permissions) – install the rpm file. Here you can see that the minimum required Java version is not met.

[root@bifangstrasse stage]# 
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Checking pre-requisites
        Checking if any previous agent service exists
        Checking if OS has systemd or initd
        Checking available disk space for agent install
        Checking if /opt/oracle/mgmt_agent directory exists
        Checking if 'mgmt_agent' user exists
        Checking agent version
        Checking Java version
                JAVA_HOME is not set or not readable to root
                Trying default path /usr/bin/java
                Agent only supports JDK 8 with a miniumum upgrade version JDK 8u162. Please set your preferred path in JAVA_HOME
error: %prein(oracle.mgmt_agent-210114.0548-1.x86_64) scriptlet failed, exit status 1
error: oracle.mgmt_agent-210114.0548-1.x86_64: install failed

[root@bifangstrasse stage]#

Verifying... ################################# [100%]

Preparing... ################################# [100%]

Checking pre-requisites

Checking if any previous agent service exists

Checking if OS has systemd or initd

Checking available disk space for agent install

Checking if /opt/oracle/mgmt_agent directory exists

Checking if 'mgmt_agent' user exists

Checking agent version

Checking Java version

JAVA_HOME is not set or not readable to root

Trying default path /usr/bin/java

Agent only supports JDK 8 with a miniumum upgrade version JDK 8u162. Please set your preferred path in JAVA_HOME

error: %prein(oracle.mgmt_agent-210114.0548-1.x86_64) scriptlet failed, exit status 1

error: oracle.mgmt_agent-210114.0548-1.x86_64: install failed

After installing the jdk-8u281-linux-x64.rpm to update the server Java version, the installer runs fine.

[root@bifangstrasse stage]# rpm -Uhv jdk-8u281-linux-x64.rpm
warning: jdk-8u281-linux-x64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:jdk1.8-2000:1.8.0_281-fcs        ################################# [100%]
Unpacking JAR files...
        tools.jar...
        plugin.jar...
        javaws.jar...
        deploy.jar...
        rt.jar...
        jsse.jar...
        charsets.jar...
        localedata.jar...

[root@bifangstrasse stage]# java -version
java version "1.8.0_281"
Java(TM) SE Runtime Environment (build 1.8.0_281-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.281-b09, mixed mode)

[root@bifangstrasse stage]# rpm -Uhv jdk-8u281-linux-x64.rpm

warning: jdk-8u281-linux-x64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY

Verifying... ################################# [100%]

Preparing... ################################# [100%]

Updating / installing...

1:jdk1.8-2000:1.8.0_281-fcs ################################# [100%]

Unpacking JAR files...

tools.jar...

plugin.jar...

javaws.jar...

deploy.jar...

rt.jar...

jsse.jar...

charsets.jar...

localedata.jar...

[root@bifangstrasse stage]# java -version

java version "1.8.0_281"

Java(TM) SE Runtime Environment (build 1.8.0_281-b09)

Java HotSpot(TM) 64-Bit Server VM (build 25.281-b09, mixed mode)

2nd try – successful

[root@bifangstrasse stage]# rpm -ihv oracle.mgmt_agent.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Checking pre-requisites
        Checking if any previous agent service exists
        Checking if OS has systemd or initd
        Checking available disk space for agent install
        Checking if /opt/oracle/mgmt_agent directory exists
        Checking if 'mgmt_agent' user exists
        Checking agent version
        Checking Java version
                JAVA_HOME is not set or not readable to root
                Trying default path /usr/bin/java
                Java version: 1.8.0_281 found at /usr/bin/java
Updating / installing...
   1:oracle.mgmt_agent-210114.0548-1  ################################# [100%]

Executing install
        Unpacking software zip
        Copying files to destination dir (/opt/oracle/mgmt_agent)
        Initializing software from template
        Creating 'mgmt_agent' daemon
        Agent Install Logs: /opt/oracle/mgmt_agent/installer-logs/installer.log.0

        Setup agent using input response file (run as any user with 'sudo' privileges)
        Usage:
                sudo /opt/oracle/mgmt_agent/agent_inst/bin/setup.sh opts=[FULL_PATH_TO_INPUT.RSP]

Agent install successful

[root@bifangstrasse stage]# rpm -ihv oracle.mgmt_agent.rpm

Verifying... ################################# [100%]

Preparing... ################################# [100%]

Checking pre-requisites

Checking if any previous agent service exists

Checking if OS has systemd or initd

Checking available disk space for agent install

Checking if /opt/oracle/mgmt_agent directory exists

Checking if 'mgmt_agent' user exists

Checking agent version

Checking Java version

JAVA_HOME is not set or not readable to root

Trying default path /usr/bin/java

Java version: 1.8.0_281 found at /usr/bin/java

Updating / installing...

1:oracle.mgmt_agent-210114.0548-1 ################################# [100%]

Executing install

Unpacking software zip

Copying files to destination dir (/opt/oracle/mgmt_agent)

Initializing software from template

Creating 'mgmt_agent' daemon

Agent Install Logs: /opt/oracle/mgmt_agent/installer-logs/installer.log.0

Setup agent using input response file (run as any user with 'sudo' privileges)

Usage:

sudo /opt/oracle/mgmt_agent/agent_inst/bin/setup.sh opts=[FULL_PATH_TO_INPUT.RSP]

Agent install successful

Agent Configuration

Run the install script with the created response file as additional parameter. The agent will be started automatically.

[root@bifangstrasse stage]# /opt/oracle/mgmt_agent/agent_inst/bin/setup.sh opts=/stage/input.rsp

Executing configure

        Parsing input response file
        Generating communication wallet
        Validating install key
        Generating security artifacts
        Registering Management Agent

Starting agent...
Agent started successfully


Agent setup completed and the agent is running.
In the future agent can be started by directly running: sudo systemctl start mgmt_agent

Please make sure that you delete /stage/input.rsp or store it in secure location.

[root@bifangstrasse stage]# /opt/oracle/mgmt_agent/agent_inst/bin/setup.sh opts=/stage/input.rsp

Executing configure

Parsing input response file

Generating communication wallet

Validating install key

Generating security artifacts

Registering Management Agent

Starting agent...

Agent started successfully

Agent setup completed and the agent is running.

In the future agent can be started by directly running: sudo systemctl start mgmt_agent

Please make sure that you delete /stage/input.rsp or store it in secure location.

A systemd service is created.

[root@bifangstrasse stage]# systemctl list-units --type=service --state=active | grep mgmt_agent
mgmt_agent.service                 loaded active running mgmt_agent

1 2	[root@bifangstrasse stage]# systemctl list-units --type=service --state=active \| grep mgmt_agent mgmt_agent.service loaded active running mgmt_agent

Verify the Management Agent in Oracle Cloud Infrastructure

Immediately after the setup, the Management Agent is visible with status Active in OCI and starts uploading data.

Agent details:

Troubleshooting

Management Agent logfiles are located in directory /opt/oracle/mgmt_agent/agent_inst/log.

[root@bifangstrasse log]# ls -latr
total 64
drwxr-x---. 7 mgmt_agent mgmt_agent    68 Feb 21 21:23 ..
-rw-r-----. 1 mgmt_agent mgmt_agent     5 Feb 21 21:24 mgmt_agent.pid
-rw-r-----. 1 mgmt_agent mgmt_agent  3631 Feb 21 21:24 wrapper.log
-rw-r-----. 1 mgmt_agent mgmt_agent     0 Feb 21 21:24 mgmt_agent_errors.log
-rw-r-----. 1 mgmt_agent mgmt_agent     0 Feb 21 21:24 mgmt_agent_work.log
-rw-r-----. 1 mgmt_agent mgmt_agent    75 Feb 21 21:24 agent.pid
-rw-r-----. 1 mgmt_agent mgmt_agent  1679 Feb 21 21:24 startup.info
drwxr-x---. 2 mgmt_agent mgmt_agent   246 Feb 21 21:24 .
-rw-r-----. 1 mgmt_agent mgmt_agent     8 Feb 21 21:24 mgmt_agent.status
-rw-r-----. 1 mgmt_agent mgmt_agent     8 Feb 21 21:24 mgmt_agent.java.status
-rw-r-----. 1 mgmt_agent mgmt_agent 31966 Feb 21 21:25 mgmt_agent.log
-rw-r-----. 1 mgmt_agent mgmt_agent  5091 Feb 21 21:25 mgmt_agent_client.log

[root@bifangstrasse log]# ls -latr

total 64

drwxr-x---. 7 mgmt_agent mgmt_agent 68 Feb 21 21:23 ..

-rw-r-----. 1 mgmt_agent mgmt_agent 5 Feb 21 21:24 mgmt_agent.pid

-rw-r-----. 1 mgmt_agent mgmt_agent 3631 Feb 21 21:24 wrapper.log

-rw-r-----. 1 mgmt_agent mgmt_agent 0 Feb 21 21:24 mgmt_agent_errors.log

-rw-r-----. 1 mgmt_agent mgmt_agent 0 Feb 21 21:24 mgmt_agent_work.log

-rw-r-----. 1 mgmt_agent mgmt_agent 75 Feb 21 21:24 agent.pid

-rw-r-----. 1 mgmt_agent mgmt_agent 1679 Feb 21 21:24 startup.info

drwxr-x---. 2 mgmt_agent mgmt_agent 246 Feb 21 21:24 .

-rw-r-----. 1 mgmt_agent mgmt_agent 8 Feb 21 21:24 mgmt_agent.status

-rw-r-----. 1 mgmt_agent mgmt_agent 8 Feb 21 21:24 mgmt_agent.java.status

-rw-r-----. 1 mgmt_agent mgmt_agent 31966 Feb 21 21:25 mgmt_agent.log

-rw-r-----. 1 mgmt_agent mgmt_agent 5091 Feb 21 21:25 mgmt_agent_client.log

Example error when the policy for agent communication is not set properly:

Error:
<--> Endpoint:       management-agent.eu-frankfurt-1.oci.oraclecloud.com:-1
     opc-request-id: US3E76CLWXAJYY5YH6Y37BOULWGHCTT1
     StartTime:      2021-02-21 20:12:44,529 GMT
     Status:         404 Not Found
     Headers:        Content-Length=111
                     opc-request-id=US3E76CLWXAJYY5YH6Y37BOULWGHCTT1/01E0A8DF939076673DF971035E99E335/5BB6AA66C5811DC0CA68288F16327F47
                     Date=Sun, 21 Feb 2021 20:12:44 GMT
                     Content-Type=application/json
     ErrorBody:
{
  "code" : "NotAuthorizedOrNotFound",
  "message" : "Authorization failed or requested resource not found."
}
     EndTime:        2021-02-21 20:12:44,795 GMT

Error:

<--> Endpoint: management-agent.eu-frankfurt-1.oci.oraclecloud.com:-1

opc-request-id: US3E76CLWXAJYY5YH6Y37BOULWGHCTT1

StartTime: 2021-02-21 20:12:44,529 GMT

Status: 404 Not Found

Headers: Content-Length=111

opc-request-id=US3E76CLWXAJYY5YH6Y37BOULWGHCTT1/01E0A8DF939076673DF971035E99E335/5BB6AA66C5811DC0CA68288F16327F47

Date=Sun, 21 Feb 2021 20:12:44 GMT

Content-Type=application/json

ErrorBody:

{

"code" : "NotAuthorizedOrNotFound",

"message" : "Authorization failed or requested resource not found."

}

EndTime: 2021-02-21 20:12:44,795 GMT

From My Oracle Support: OCI : Management Agent Status Reporting As “Not Available” Post Installation (Doc ID 2745566.1)

Summary Part 1

The Management Agent installation and integration is easy to setup when all prerequisites are met. For troubleshooting you have full access on the agent logs. See you for blog post 2, where I try to integrate the on-premises Oracle databases into OCI.

Uncategorized

Oracle Cloud Infrastructure – A short Blog Post about a secure and small Development Setup

Author: Martin Berger 14. January 2021

For an internal project I had the pleasure to setup a new Oracle Cloud Infrastructure environment for an APEX development team. Here is a short overview about the setup.

Requirements

VPN Access from everywhere – 2 people are working maximal at same time on the environment
Oracle Standard Edition 2 – no license available in project
Small monitoring to verify server stats
Instances can be started and stopped from the developers to save costs for example over night, weekend, holiday etc.

Architecture Diagram

Resource	Network	Usage	Remarks
Open VPN Access Server	Public Subnet	VPN client access and traffic routing	OCI Cloud Marketplace Image – OpenVPN Access Server (2 FREE VPN Connections) – OpenVPN Inc. – Oracle Cloud Marketplace
Management Server	Private Subnet	OCI-CLI, Monitoring	Application server and database node start/stop with OCI-CLI, Grafana and Prometheus for monitoring
Application Server	Private Subnet	Tomcat ORDS, APEX	–
Database System	Private Subnet	–	OCI Database Standard Edition 2, Backup to Object Store enabled

Network Components

Regional private and public subnet
Security lists and network security groups
Private and public routing table
NAT gateway for regional private subnet

Monitoring

Grafana and Prometheus, running on the management server. The free shape VM.Standard.E2.1.Micro fits perfect for this small setup! The Prometheus node exporter runs on the database and the application server. I used this Grafana dashboard here: Prometheus Node Exporter Full dashboard for Grafana | Grafana Labs

Next Steps

Adding Influx DB for persistence
Adding the Oracle database to Grafana monitoring
Optimizing shape size for the database server according usage

Other Ideas

Create a blueprint for internal developer environments
Automate the setup with Terraform and Ansible

Summary

Setting up this infrastructure in Oracle Cloud Infrastructure was fun. All developer requirements are fulfilled. Started with the Network and OpenVPN configuration – I really like their Marketplace instance – and the moved on to application and database server, step-by-step. There are many other ideas what we can do more based on this setup, the work will not run out. #ilike

Oracle Cloud Infrastructure, Oracle Tools, Security, Testing, TrivadisContent

Oracle Cloud Infrastructure Data Safe – How to burn down 201.44 Swiss Francs in 30 Seconds…

Author: Martin Berger 11. January 2021

Is Data Safe really for free?

In the last autumn, the new Oracle Cloud Infrastructure feature called Data Safe was released. For sure, new features has to be tested. I have tested the Data Safe feature too and added a cloud database to Data Safe. But in my enthusiasm about this cool feature – or maybe it was just too late in the evening – I did a mistake by adding the database target. Four days later, I recognized that Data Safe is charged in my account. Mmm, but should it not be for free? First reaction: I raised an SR and described the case. The nice guy from My Oracle Support realized the situation quickly:

Dear Mister Berger, you have used the wrong target type when adding the Oracle Cloud Infrastructure database as a new Data Safe target.

From the Service Request:

B91632 – Oracle Cloud Infrastructure – Data Safe for Database Cloud Service – Each (Includes 1 million audit records per target per month) – Free
B91631 – Oracle Cloud Infrastructure – Data Safe for Database Cloud Service – Audit Record Collection Over 1 Million Records (over 1 million audit records per target per month) – 0.0800 / 10,000 Audit Records Per Target Per Month
B92733 – Oracle Cloud Infrastructure – Data Safe for On-Premises Databases – Target Database Per Month – 200.00 Target Database Per Month + Includes 1 million audit records per target per month (pre-requisite under B91632)

Indeed, indeed. According My Oracle Support I have used the wrong target type. Instead Oracle Cloud Database, I used Oracle Database on Compute. And did not realized, the mistake and ignored the text below to the dropdown box. Shame on me 😉 – here is the small, but important difference:

So far so good, the mistake was recognized. I deleted the target and added it from scratch with the correct target type. But this didn’t help, the charging went on.

Oracle Cloud Infrastructure Price List

Adding an other target type than Oracle Cloud Database is charged on monthly fee base as described here: Cloud Price List | Oracle

Cost and Usage Report

In the detailed cost and usage report, the target is marked as deleted (suffix DELETED + deletion date), and charged.

All you can do is getting angry about that mistake and wait. After a month, the money was burned down, and there were no more Oracle Cloud Infrastructure Data Safe costs charged. As you can see, there are 201.44 CHF charged for a month.

I don’t know what Oracle has for a currency converter, but actual 200 USD are less that 180 CHF 😉

Lessons learned

Pity about the beautiful money – and for my next test run: RTFM.

Oracle Cloud Infrastructure, Oracle Tools, Security, TrivadisContent

Red Hat Ansible Tower Upgrade from 3.5 to 3.8 – when running setup.sh is not enough – or: I have made fire!

Author: Martin Berger 22. December 2020

I a customer project I had to update an existing Red Hat Ansible Tower setup from version 3.5.1 to the newest available version 3.8. The upgrade scenario as described in 8. Upgrading an Existing Tower Installation — Ansible Tower Installation and Reference Guide v3.8.0 does not work here. For example: the delivered setup.sh is not able to restore data exported from Postgres 9.6 into a new Postgres 10 database. This upgrade scenario as described is a result of a long discussion with Red Hat support (Thanks to Swati – he did a great job!) and an intensive test period on local virtual machines until the live system was in the row.

Remark: The servers in this blog post are listed without domain names to make it easier readable.

The running Ansible Tower Setup

It contains three Red Hat Ansible Tower servers and the repository. All servers can connect among each other. Firewall ports are opened.

Server	Tower Version	Operating System	PostgreSQL Version	Remarks
ansible-tower-01	3.5.1	RHEL 7.8		no Internet access
ansible-tower-02	3.5.1	RHEL 7.8		no Internet access
ansible-tower-03	3.5.1	RHEL 7.8		no Internet access
ansible-db03		RHEL 7.8	9.6	no Internet access, runs on port 5432

Upgrade Path

This documents describes the upgrade path, a release cannot be more than two release numbers behind to upgrade it. In our case, Tower 3.5.1 needs to be updated to 3.7.1 first before 3.8 can be applied. Ansible Tower 3.8 requires Postgres 10.

What are the Recommended Upgrade Paths for Ansible Tower? – Red Hat Customer Portal

Bundles: We use bundles (aka offline installers) for version 3.7.4 and 3.8 – the Ansible Tower servers don’t require internet access to upgrade and install relevant packages. They are included in the bundle. The bundles are accessible by the Ansible Tower server where the setup processes are executed.

Bundle	URL	Note
3.7.4.1	https://releases.ansible.com/ansible-tower/setup-bundle/ansible-tower-setup-bundle-3.7.4-1.tar.gz	free download
3.8	https://access.redhat.com/downloads/content/480/ver=1.2/rhel—7/1.2/x86_64/product-software	requires a Red Hat subscription to download

Prerequisites

The software and the inventory file of the current Ansible Tower installation
Bundle of the future releases are downloaded
Postgres 10 installed on database server
A Red Hat Subscription Manifest File which contains the license information for Tower 3.8 – the Manifest file can only be created in the Red Hat account when a valid subscription is available

Upgrade Overview

Shutdown all involved virtual machines properly, create a snapshot for fallback, restart all involved servers properly
Install Postgres 10 on database server and create a new Postgres 10 database
Manual export/import tower data into the new database with pg_dump/pg_restore
Re-run Ansible Tower 3.5.1 setup against new database
Upgrade to Ansible Tower 3.7.1
Upgrade to Ansible Tower 3.8
Add Red Hat Subscription Manifest
I have made fire!

1. Shutdown Machines and create Snapshot of the Hosts

It is highly recommended to create a backup/snapshot of the existing environment in case of upgrade troubles. In my case, we did VMWare vRealize Automation snaphots first after shutting down tower and database properly.

Tower and Database Shutdown

ansible-tower-01 / ansible-tower-02 / ansible-tower-03: stop the tower service:

# ansible-tower-service stop

1	# ansible-tower-service stop

db03: Stop the database service:

# systemctl stop postgresql-9.6.service

1	# systemctl stop postgresql-9.6.service

Create a backup or snapshot

In my case, we did VMWare vRealize Automation snaphots first after shutting down tower and database properly.

Restart machines

Restart all involved servers and verify in the Tower UI that everything works properly. Tower and database are configured as OS service, they are started automatically after server power on.

2. Install Postgres 10 on database server and create a new Postgres 10 database

Install Postgres 10

Connection details of the new repository database:

Host: ansible-db03
Port: 5433 (old Tower repository was 5432)

There is no internet access to the official PostgreSQL yum repository. Therefore, download the following files from https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7.8-x86_64/ and copy them to /tmp to the databse server. Ansible Tower 3.7.x requires PostgreSQL 10. It does not work with a newer version! See here: https://docs.ansible.com/ansible-tower/latest/html/installandreference/requirements_refguide.html.

$ sudo su -
# yum localinstall ./postgresql10-server-10.14-1PGDG.rhel7.x86_64.rpm \
postgresql10-libs-10.14-1PGDG.rhel7.x86_64.rpm \
postgresql10-10.14-1PGDG.rhel7.x86_64.rpm \
postgresql10-contrib-10.14-1PGDG.rhel7.x86_64.rpm

$ sudo su -

# yum localinstall ./postgresql10-server-10.14-1PGDG.rhel7.x86_64.rpm \

postgresql10-libs-10.14-1PGDG.rhel7.x86_64.rpm \

postgresql10-10.14-1PGDG.rhel7.x86_64.rpm \

postgresql10-contrib-10.14-1PGDG.rhel7.x86_64.rpm

Create a new PostgreSQL Cluster with Trivadis pgOperate

On the database server ansible-db03, a new cluster database is required with PostgreSQL version 10. Therefore I used pgOperate. pgOperate is part of the Open Source tool pgBasEnv. My Trivadis colleagues have developed a really cool framework to manage PostgreSQL clusters. Both tools with the manuals and examples are available on GitHub:

GitHub – Trivadis/pgbasenv: pgBasEnv – PostgreSQL Base Environment Tool

GitHub – Trivadis/pgoperate: pgOperate – PostgreSQL Operation Tool

Example to create a new cluster when the tools are installed in base directory /var/lib/pgsql/tvdtoolbox. You will find more information how to configure and how to use the tool on the GitHub pages.

$ cd /var/lib/pgsql/tvdtoolbox/pgoperate/etc
$ cp parameters_mycls.conf.tpl parameters_tower01.conf
$ vi parameters_tower01.conf

$ cd /var/lib/pgsql/tvdtoolbox/pgoperate/etc

$ cp parameters_mycls.conf.tpl parameters_tower01.conf

$ vi parameters_tower01.conf

Set the cluster parameters to create a new cluster which is running on port 5433:

TVD_PGHOME_ALIAS=pgh1014
PGSQL_BASE=/u01/app/postgres/tower01
PG_PORT=5433
PG_SUPERUSER_PWD=<see keepass>

TVD_PGHOME_ALIAS=pgh1014

PGSQL_BASE=/u01/app/postgres/tower01

PG_PORT=5433

PG_SUPERUSER_PWD=<see keepass>

Set the alias and execute cluster creation script.

$ pgh1014
$ pgoperate --create-cluster -a tower01

1 2	$ pgh1014 $ pgoperate --create-cluster -a tower01

Run root.sh to enable automated startup and allow user postgres to start/stop the service.

$ sudo su -
# /u01/app/postgres/tower01/scripts/root.sh

1 2	$ sudo su - # /u01/app/postgres/tower01/scripts/root.sh

Login as user postgres and verify the new created cluster which is started automatically. Here you can see the old PostgreSQL database version 9.6 running together with version 10 as output from the status screen after login.

$ sudo su - postgres
pgBasEnv v1.1 by Trivadis AG

Installation homes:
====================================================
ALIAS | VER | OPTIONS | HOME DIR
====================================================
pgh966 | 9.6.6 | ssl:1G:8K | /usr/pgsql-9.6
pgh1014 | 10.14 | ssl:1G:8K | /usr/pgsql-10
====================================================

Cluster data directories:
=====================================================================================================================
ALIAS | VER | STAT | PORT | PID | SIZE | PGDATA | LAST START | LAST START HOME
=====================================================================================================================
pgd96 | 9.6 | UP | 5432 | 1241 | 8.8G | /u01/pgdata/tower | 2020-06-05 07:19 | /usr/pgsql-9.6
tower01 | 10 | UP | 5433 | 110468 | 43M | /u01/app/postgres/tower01/data | 2020-12-20 12:03 | /usr/pgsql-10
=====================================================================================================================

$ sudo su - postgres

pgBasEnv v1.1 by Trivadis AG

Installation homes:

====================================================

ALIAS | VER | OPTIONS | HOME DIR

====================================================

pgh966 | 9.6.6 | ssl:1G:8K | /usr/pgsql-9.6

pgh1014 | 10.14 | ssl:1G:8K | /usr/pgsql-10

====================================================

Cluster data directories:

=====================================================================================================================

=====================================================================================================================

pgd96 | 9.6 | UP | 5432 | 1241 | 8.8G | /u01/pgdata/tower | 2020-06-05 07:19 | /usr/pgsql-9.6

tower01 | 10 | UP | 5433 | 110468 | 43M | /u01/app/postgres/tower01/data | 2020-12-20 12:03 | /usr/pgsql-10

=====================================================================================================================

Verify that the file pg_hba.conf allows traffic to the database, for example allow database connects from each server:

$ vi /u01/app/postgres/tower01/etc/pg_hba.conf

# Database administrative login by UNIX sockets
# note: you may wish to restrict this further later
local all postgres peer

# TYPE DATABASE USER CIDR-ADDRESS METHOD
local replication all trust
local all all md5
host all all 0.0.0.0/0 md5
host all all ::/0 md5

$ vi /u01/app/postgres/tower01/etc/pg_hba.conf

# Database administrative login by UNIX sockets

# note: you may wish to restrict this further later

local all postgres peer

# TYPE DATABASE USER CIDR-ADDRESS METHOD

local replication all trust

local all all md5

host all all 0.0.0.0/0 md5

host all all ::/0 md5

0.0.0.0/0 ensures, that all Ansible Tower servers can connect to the new database. This does not implicit allow everybody to connect. For example you can control the access on OS firewall level.

Create a new Database for Tower Repository

Login as user postgres and start psql against the new cluster. In this case, we create a new database called tower10 with same username and password as the existing database.

$ psql -p 5433
postgres=# CREATE USER tower WITH ENCRYPTED PASSWORD 'thisismypassword';
postgres=# CREATE DATABASE tower10 OWNER tower;

$ psql -p 5433

postgres=# CREATE USER tower WITH ENCRYPTED PASSWORD 'thisismypassword';

postgres=# CREATE DATABASE tower10 OWNER tower;

Verification

Test the database connection from all Tower servers to avoid firewall or configuration issues and verify if the new created database tower10 is listed.

$ psql -U tower -p 5433 -h ansible-db03 -d tower10

psql (10.14)
Type "help" for help.

tower10=> \l
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
 tower10   | tower    | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
(4 rows)

$ psql -U tower -p 5433 -h ansible-db03 -d tower10

psql (10.14)

Type "help" for help.

tower10=> \l

List of databases

-----------+----------+----------+-------------+-------------+-----------------------

(4 rows)

3. Manual Export / Import of Ansible Tower Repository Data

In this step we migrate the tower database to the new PostgreSQL 10 database. It’s recommended to stop the Ansible Tower before starting the export.

ansible-tower-01 / ansible-tower-02 / ansible-tower-03: stop the tower service:

# ansible-tower-service stop

1	# ansible-tower-service stop

Export Data

# Backup
pg_dump -h ansible-db03 -p 5432 -U tower -F c -b -v -f "/var/lib/pgsql/dbexport_tower_3.5.1.backup" tower

1 2	# Backup pg_dump -h ansible-db03 -p 5432 -U tower -F c -b -v -f "/var/lib/pgsql/dbexport_tower_3.5.1.backup" tower

Import Data

# Restore
pg_restore -h ansible-db03 -p 5433 -U tower -d tower10 -v "/var/lib/pgsql/dbexport_tower_3.5.1.backup"

1 2	# Restore pg_restore -h ansible-db03 -p 5433 -U tower -d tower10 -v "/var/lib/pgsql/dbexport_tower_3.5.1.backup"

Stop and disable existing Postgres 9.6 Database

# systemctl stop postgresql-9.6.service

1	# systemctl stop postgresql-9.6.service

4. Re-run Ansible Tower 3.5.1 setup against new database

Adapt Inventory

In this step, the existing Tower setup is registered again against the new database. We have to change the inventory file of the existing 3.5.1 installation.

# vi /root/install/ansible-tower-setup-3.5.3-1/inventory

-- old:
pg_port='5432'
pg_database='tower'

-- new: 
pg_port='5433'
pg_database='tower10'

# vi /root/install/ansible-tower-setup-3.5.3-1/inventory

-- old:

pg_port='5432'

pg_database='tower'

-- new:

pg_port='5433'

pg_database='tower10'

Example inventory file of the existing 3.5.1 installation with three Ansible Tower servers. Note: the [database] section is empty, so the setup procedure will not try to install a new database and uses the new one what we have created above.

[tower]
ansible-towert01
ansible-towert02
ansible-towert03

[database]

[all:vars]
admin_password='thisismypassword'

pg_host='ansible-db03'
pg_port='5433'
pg_database='tower10'
pg_username='tower'
pg_password='thisismypassword'

rabbitmq_username=tower
rabbitmq_port=5672
rabbitmq_vhost=tower
rabbitmq_use_long_name=true
rabbitmq_password='thisismypassword'
rabbitmq_cookie=cookiemonster

[tower]

ansible-towert01

ansible-towert02

ansible-towert03

[database]

[all:vars]

admin_password='thisismypassword'

pg_host='ansible-db03'

pg_port='5433'

pg_database='tower10'

pg_username='tower'

pg_password='thisismypassword'

rabbitmq_username=tower

rabbitmq_port=5672

rabbitmq_vhost=tower

rabbitmq_use_long_name=true

rabbitmq_password='thisismypassword'

rabbitmq_cookie=cookiemonster

Re-run setup.sh

As user root, execute the setup.sh script on ansible-towert01.

# /root/install/ansible-tower-setup-3.5.3-1/setup.sh

1	# /root/install/ansible-tower-setup-3.5.3-1/setup.sh

Verify the setup playbook result – no failed tasks should occur. The Ansible Tower 3.5.1 runs now with the PostgreSQL database version 10 and is ready to upgrade. Verify if all Tower are running properly and log in in the user interface.

PLAY RECAP ********************************************************************************************************************************************************************************
ansible-towert01 : ok=125 changed=27 unreachable=0 failed=0 skipped=64 rescued=0 ignored=1
ansible-towert02 : ok=119 changed=28 unreachable=0 failed=0 skipped=68 rescued=0 ignored=1
ansible-towert03 : ok=119 changed=28 unreachable=0 failed=0 skipped=68 rescued=0 ignored=1

The setup process completed successfully.
Setup log saved to /var/log/tower/setup-2020-12-21-21:08:46.log

PLAY RECAP ********************************************************************************************************************************************************************************

ansible-towert01 : ok=125 changed=27 unreachable=0 failed=0 skipped=64 rescued=0 ignored=1

ansible-towert02 : ok=119 changed=28 unreachable=0 failed=0 skipped=68 rescued=0 ignored=1

ansible-towert03 : ok=119 changed=28 unreachable=0 failed=0 skipped=68 rescued=0 ignored=1

The setup process completed successfully.

Setup log saved to /var/log/tower/setup-2020-12-21-21:08:46.log

Verification

5. Upgrade to Ansible Tower 3.7.1

The software bundle is transferred to the target server. As I don’t have much free space, I moved the bundle on a NFS which is attached on all Ansible Tower servers. It’s recommended to stop the Ansible Tower before starting the upgrade process.

ansible-tower-01 / ansible-tower-02 / ansible-tower-03: stop the tower service:

# ansible-tower-service stop

1	# ansible-tower-service stop

As user root, go to the install directory and extract the bundle.

# cd /nfs/ansible/tower/install
# tar xvfz ansible-tower-setup-bundle-3.7.4-1.tar.gz

1 2	# cd /nfs/ansible/tower/install # tar xvfz ansible-tower-setup-bundle-3.7.4-1.tar.gz

For the inventory file, the same settings as used in 3.5.1 can be used. The RabbitMQ settings are not required anymore and can be removed. This component is removed from Ansible Tower during the upgrade process. Example inventory file:

# vi /nfs/ansible/tower/install/ansible-tower-setup-bundle-3.7.4-1/inventory

[tower]
ansible-towert01
ansible-towert02
ansible-towert03

[database]

[all:vars]
admin_password='thisismypassword'

pg_host='ansible-db03'
pg_port='5433'
pg_database='tower10'
pg_username='tower'
pg_password='thisismypassword'

# vi /nfs/ansible/tower/install/ansible-tower-setup-bundle-3.7.4-1/inventory

[tower]

ansible-towert01

ansible-towert02

ansible-towert03

[database]

[all:vars]

admin_password='thisismypassword'

pg_host='ansible-db03'

pg_port='5433'

pg_database='tower10'

pg_username='tower'

pg_password='thisismypassword'

Run setup.sh

As user root, execute the setup.sh script on ansible-towert01.

# /nfs/ansible/tower/install/ansible-tower-setup-bundle-3.7.4-1/setup.sh

1	# /nfs/ansible/tower/install/ansible-tower-setup-bundle-3.7.4-1/setup.sh

Verify the setup playbook result – no failed tasks should occur.

PLAY RECAP *********************************************************************************************************************************************************
ansible-towert01 : ok=132 changed=55 unreachable=0 failed=0 skipped=65 rescued=0 ignored=1
ansible-towert02 : ok=121 changed=50 unreachable=0 failed=0 skipped=61 rescued=0 ignored=1
ansible-towert03 : ok=121 changed=50 unreachable=0 failed=0 skipped=61 rescued=0 ignored=1

PLAY RECAP *********************************************************************************************************************************************************

ansible-towert01 : ok=132 changed=55 unreachable=0 failed=0 skipped=65 rescued=0 ignored=1

ansible-towert02 : ok=121 changed=50 unreachable=0 failed=0 skipped=61 rescued=0 ignored=1

ansible-towert03 : ok=121 changed=50 unreachable=0 failed=0 skipped=61 rescued=0 ignored=1

Verification

Package verification on Ansible Tower servers:

# rpm -qa | grep tower
ansible-tower-venv-ansible-3.7.4-1.el7at.x86_64
ansible-tower-ui-3.7.4-1.el7at.x86_64
ansible-tower-venv-tower-3.7.4-1.el7at.x86_64
ansible-tower-server-3.7.4-1.el7at.x86_64
ansible-tower-3.7.4-1.el7at.x86_64

# rpm -qa | grep tower

ansible-tower-venv-ansible-3.7.4-1.el7at.x86_64

ansible-tower-ui-3.7.4-1.el7at.x86_64

ansible-tower-venv-tower-3.7.4-1.el7at.x86_64

ansible-tower-server-3.7.4-1.el7at.x86_64

ansible-tower-3.7.4-1.el7at.x86_64

Output from the play where RabbitMQ is removed:

TASK [awx_install : Remove Tower rabbitmq settings] ****************************************************************************************************************
changed: [ansible-towert01] => {"changed": true, "path": "/etc/tower/conf.d/rabbitmq.py", "state": "absent"}
changed: [ansible-towert02] => {"changed": true, "path": "/etc/tower/conf.d/rabbitmq.py", "state": "absent"}
changed: [ansible-towert03] => {"changed": true, "path": "/etc/tower/conf.d/rabbitmq.py", "state": "absent"}

TASK [awx_install : Remove Tower rabbitmq settings] ****************************************************************************************************************

changed: [ansible-towert01] => {"changed": true, "path": "/etc/tower/conf.d/rabbitmq.py", "state": "absent"}

changed: [ansible-towert02] => {"changed": true, "path": "/etc/tower/conf.d/rabbitmq.py", "state": "absent"}

changed: [ansible-towert03] => {"changed": true, "path": "/etc/tower/conf.d/rabbitmq.py", "state": "absent"}

6. Upgrade to Ansible Tower 3.8

ansible-tower-01 / ansible-tower-02 / ansible-tower-03: stop the tower service:

# ansible-tower-service stop

1	# ansible-tower-service stop

As user root, go to the install directory and extract the bundle.

# cd /nfs/ansible/tower/install
# tar xvfz ansible-tower-setup-bundle-3.7.4-1.tar.gz

1 2	# cd /nfs/ansible/tower/install # tar xvfz ansible-tower-setup-bundle-3.7.4-1.tar.gz

For the inventory file, the same settings as used in 3.7.4 can be used. Example inventory file:

# vi /nfs/ansible/tower/install/ansible-automation-platform-setup-bundle-1.2.0-1/inventory

[tower]
ansible-towert01
ansible-towert02
ansible-towert03

[database]

[all:vars]
admin_password='thisismypassword'

pg_host='ansible-db03'
pg_port='5433'
pg_database='tower10'
pg_username='tower'
pg_password='thisismypassword'

# vi /nfs/ansible/tower/install/ansible-automation-platform-setup-bundle-1.2.0-1/inventory

[tower]

ansible-towert01

ansible-towert02

ansible-towert03

[database]

[all:vars]

admin_password='thisismypassword'

pg_host='ansible-db03'

pg_port='5433'

pg_database='tower10'

pg_username='tower'

pg_password='thisismypassword'

Run setup.sh

As user root, execute the setup.sh script on ansible-towert01.

# /nfs/ansible/tower/install/ansible-automation-platform-setup-bundle-1.2.0-1/setup.sh

1	# /nfs/ansible/tower/install/ansible-automation-platform-setup-bundle-1.2.0-1/setup.sh

Note: Ansible verifies if the ansible RPM version is 2.4 or higher. Only on the node where the installer runs, the ansible package is updated. If you want to upadte the package on the oder Tower servers too, you can force the update of the package to 2.9.15 with the parameter upgrade_ansible_with_tower=1. A manual upgrade on by rpm -Uhv is possible too, the ansible package is available in the bundle.

# /nfs/ansible/tower/install/ansible-automation-platform-setup-bundle-1.2.0-1/setup.sh -e upgrade_ansible_with_tower=1

1	# /nfs/ansible/tower/install/ansible-automation-platform-setup-bundle-1.2.0-1/setup.sh -e upgrade_ansible_with_tower=1

Verify the setup playbook result – no failed tasks should occur.

PLAY RECAP ********************************************************************************************************************************************************************************
ansible-towert01 : ok=15 changed=0 unreachable=0 failed=1 skipped=32 rescued=0 ignored=1
ansible-towert02 : ok=14 changed=0 unreachable=0 failed=1 skipped=16 rescued=0 ignored=1
ansible-towert03 : ok=14 changed=0 unreachable=0 failed=1 skipped=16 rescued=0 ignored=1

PLAY RECAP ********************************************************************************************************************************************************************************

ansible-towert01 : ok=15 changed=0 unreachable=0 failed=1 skipped=32 rescued=0 ignored=1

ansible-towert02 : ok=14 changed=0 unreachable=0 failed=1 skipped=16 rescued=0 ignored=1

ansible-towert03 : ok=14 changed=0 unreachable=0 failed=1 skipped=16 rescued=0 ignored=1

Verification

Package verification on Ansible Tower servers:

# rpm -qa | grep tower
ansible-tower-server-3.8.0-1.el7at.x86_64
ansible-tower-venv-ansible-3.8.0-1.el7at.x86_64
ansible-tower-venv-tower-3.8.0-1.el7at.x86_64
ansible-tower-ui-3.8.0-1.el7at.x86_64
ansible-tower-3.8.0-1.el7at.x86_64

# rpm -qa | grep tower

ansible-tower-server-3.8.0-1.el7at.x86_64

ansible-tower-venv-ansible-3.8.0-1.el7at.x86_64

ansible-tower-venv-tower-3.8.0-1.el7at.x86_64

ansible-tower-ui-3.8.0-1.el7at.x86_64

ansible-tower-3.8.0-1.el7at.x86_64

7. Add Red Hat Subscription Manifest

In former versions, a license file was required. Now it has changed to a subscription manifest. This file was generated in the Red Hat customer portal. After the first login into the 3.8 servers, you have to add the manifest. That’s all folks.

8. I have made fire!

After spending a lot of time to figure out the correct way how to upgrade, doing a lot of tests and finally the implementation on the live system, I did it. After version 3.8 was showing up, I felt like Tom Hanks in the movie Cast Away – I have made fire!

Summary

Upgrading the Ansible Tower with the existing online documentation? No chance! I have opened a support case at Red Had to clarify a lot of this like changing the repository database, updating the ansible packages etc. The procedure with setup.sh -b / setup.sh -r as described in the upgrade documentation did not work. It needs a manual data transfer. I really like the method to install the new Ansible Tower versions with a bundle, so no internet connection is required to keep the environment up to date. Hopefully Red Hat will update the documentation in the near future, for example with an upgrade cookbook section or however they want to call it.

Red Hat Ansible Tower, Red Hat Linux, TrivadisContent

Oracle OCI Data Transfer Service – A journey from Kestenholz/Jurasüdfuss/Switzerland to Frankfurt and back

Author: Martin Berger 24. November 2020

The Oracle Data Transfer service is a offline data transfer method to migrate data to the Oracle Cloud Infrastructure. A transfer service is useful, when your network bandwidth and connection is not sufficient to upload your migration data in a meaningful time. Oracle offers two methods: The disk-based data transfer and the appliance-based data transfer. The service is no only one-way, data can also be exported in an Oracle Cloud Infrastructure data center and shipped to your data center.

According one of my Company Trivadis’ cultural value called curiosity, I was wondering how this service works. This is the story of a tiny USB Hard Disk Drive full of data, which was going on a long journey from Kestenholz / Jurasüdfuss / Solothurn / Switzerland to the Oracle Infrastructure data center in Frankfurt and back.

Setup

The OCI Data Transfer utility is Linux based, the USB 3.0 HDD is attached to a VMware Virtual machine where Oracle Linux is running
The virtual machine has access to the Internet
Data is available – for this example I used some open data from Swiss government (opendata.swiss)

Data Transfer Service Regions

Actually data transfer is available in Frankfurt, Ashburn, Phoenix, London and Osaka. From Switzerland, Frankfurt is the nearest location.

How is your data coming into the Oracle Cloud

Enable Data Transfer Service – Data Transfer
Prepare an Object Storage bucket
Create a Transfer Job
Attach a HDD to a Linux based host, use the Data Transfer Utility to create and encrypt the device
Copy data to the HDD
Create and upload the disk manifest
Lock the disk
Attach the Disk to the transfer label and create the package
Create a transfer package
Shipping and update shipping information
Tracking
Data Verification
Object Storage Replication Policy (optional)
Finally…

Note: Most of the jobs above can be done by the OCI CLI on command line and are very well described in the Oracle doumentation.

1. Enable Data Transfer Service – Entitlement

Before you can use this service, the Data Transfer service has to be enabled in general. Therefore you have to request it. The OCI tenant administrator gets a document, what he has to sign in a digital way. It contains for example a description how to bring data to OCI, and if you order an appliance that there will be a 45day maximum limit where the appliance has to be returned to Oracle. And a few days later, the service is ready to use. Basically now you have the permissions, to order a Data Transfer Appliance, but in this test I used the Disk Service.

2. Prepare an Object Storage Bucket

In the Frankfurt region, I created a new Object Storage bucket called data_transfer_usb. This is the bucket where the shipped data will be transferred in.

3. Create a Transfer Job

In Object Storage – Data Transfer Import, we create a new transfer job. It contains the upload bucket from above and as method the transfer device type is disk. For furtehr processing, we need the OCID of the job. As you can see, actually there is no transfer disk attached.

4. Attach a HDD to a Linux based host, use the Data Transfer Utility to create and encrypt the Device

Prerequisites for the Data Transfer Utility according the documentation:

An OCI Account which have the IAM permissions for Data Transfer
A Linux machine with Oracle Linux 6 or greater, Ubuntu 14.04 or greater, SUSE 11 or greater
Java 1.8 or 1.11
hdparm 9.0 or later
Cryptsetup 1.2.0 or later

Package Installation for my Oracle Linux 7 Machine

# yum install java-1.8.0-openjdk-devel
# yum install hdparm
# yum install cryptsetup

# yum install java-1.8.0-openjdk-devel

# yum install hdparm

# yum install cryptsetup

Download and Installation of the Data Transfer Utility

The actual link to the file is in the online documentation.

# wget https://docs.cloud.oracle.com/tools/dts-rpm/3.0.2461/dts-3.0.2461.x86_64.rpm
# yum localinstall /tmp/dts-3.0.2461.x86_64.rpm

1 2	# wget https://docs.cloud.oracle.com/tools/dts-rpm/3.0.2461/dts-3.0.2461.x86_64.rpm # yum localinstall /tmp/dts-3.0.2461.x86_64.rpm

Test

# dts --version
3.0.2461 ( DEV_NO_HASH )

1 2	# dts --version 3.0.2461 ( DEV_NO_HASH )

Configure IAM Credentials for Data Transfer Actions

The configuration is according configuring the Oracle Cloud Infrastructure CLI with user, fingerprint, key_file, tenancy and region. Example configuration file:

[DEFAULT]
user=ocid1.user.oc1..aaaaaaaadlv5vwm3zfdnyanosmdjm7cyjzdhmgawre12345671234567
fingerprint=6a:ce:8e:a7:4a:a1:70:ad:4e:d7:e6:8f:d1:23:45:g6
key_file=/root/.oci/oci_api_key.pem
tenancy=ocid1.tenancy.oc1..aaaaaaaaxuk4je4t3aorovuzmwyeaq5sftqv3nkyz64s12345671234567
region=eu-frankfurt-1

[DEFAULT]

user=ocid1.user.oc1..aaaaaaaadlv5vwm3zfdnyanosmdjm7cyjzdhmgawre12345671234567

fingerprint=6a:ce:8e:a7:4a:a1:70:ad:4e:d7:e6:8f:d1:23:45:g6

key_file=/root/.oci/oci_api_key.pem

tenancy=ocid1.tenancy.oc1..aaaaaaaaxuk4je4t3aorovuzmwyeaq5sftqv3nkyz64s12345671234567

region=eu-frankfurt-1

Verify Credentials

# dts job verify-upload-user-credentials --bucket data_transfer_usb
created object BulkDataTransferTestObject in bucket data_transfer_usb
overwrote object BulkDataTransferTestObject in bucket data_transfer_usb
inspected object BulkDataTransferTestObject in bucket data_transfer_usb
read object BulkDataTransferTestObject in bucket data_transfer_usb

# dts job verify-upload-user-credentials --bucket data_transfer_usb

created object BulkDataTransferTestObject in bucket data_transfer_usb

overwrote object BulkDataTransferTestObject in bucket data_transfer_usb

inspected object BulkDataTransferTestObject in bucket data_transfer_usb

read object BulkDataTransferTestObject in bucket data_transfer_usb

Show Data Transfer Job Details – Status is PREPARED

Here you can see the shipping address from the Oracle Infrastructure data center frankfurk and the label. Both information are used later in process.

# dts job show --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekvpmgdnnobcgrdb4awa7c5q
Transfer Job :
ID : ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekvpmgdnnobcgrdb4awa7c5q
CompartmentId : ocid1.compartment.oc1..aaaaaaaat5uo2xh77edws4huwvqorengp7x4xdv6x3giw3vryk36vyydwsdq
UploadBucket : data_transfer_usb
Name : data_transfer_usb_job_01
Label : J5Y0R5KAS
CreationDate : 2020/10/27 12:53:02 CET
Status : PREPARING
freeformTags : *** none ***
definedTags :
Oracle-Tags :
CreatedBy : oracleidentitycloudservice/martin.berger@trivadis.com
CreatedOn : 2020-10-27T11:53:02.320Z
Packages :
[1] :
Label : PDQ0QMLNT
TransferSiteShippingAddress : Telehouse c/o Oracle Data Transfer Service; Christos Panoudis; Phone: +49152 22882673 Job:J5Y0R5KAS Package:PDQ0QMLNT ; Kleyerstraße 75-87; 60326 Frankfurt am Main; Germany
DeliveryVendor : DHL
DeliveryTrackingNumber : *** none ***
ReturnDeliveryTrackingNumber : *** none ***
Status : PREPARING
Devices : [*** none ***]
UnattachedDevices : [*** none ***]
Appliances : [*** none ***]

# dts job show --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekvpmgdnnobcgrdb4awa7c5q

Transfer Job :

ID : ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekvpmgdnnobcgrdb4awa7c5q

CompartmentId : ocid1.compartment.oc1..aaaaaaaat5uo2xh77edws4huwvqorengp7x4xdv6x3giw3vryk36vyydwsdq

UploadBucket : data_transfer_usb

Name : data_transfer_usb_job_01

Label : J5Y0R5KAS

CreationDate : 2020/10/27 12:53:02 CET

Status : PREPARING

freeformTags : *** none ***

definedTags :

Oracle-Tags :

CreatedBy : oracleidentitycloudservice/martin.berger@trivadis.com

CreatedOn : 2020-10-27T11:53:02.320Z

Packages :

[1] :

Label : PDQ0QMLNT

TransferSiteShippingAddress : Telehouse c/o Oracle Data Transfer Service; Christos Panoudis; Phone: +49152 22882673 Job:J5Y0R5KAS Package:PDQ0QMLNT ; Kleyerstraße 75-87; 60326 Frankfurt am Main; Germany

DeliveryVendor : DHL

DeliveryTrackingNumber : *** none ***

ReturnDeliveryTrackingNumber : *** none ***

Status : PREPARING

Devices : [*** none ***]

UnattachedDevices : [*** none ***]

Appliances : [*** none ***]

Prepare USB Hard Disk Drive

The disk is attached as /dev/sdb – it is a Western Digital drive. Important: The disk needs no partition.

# fdisk -l /dev/sdb
Disk /dev/sdb: 931.5 GiB, 1000170586112 bytes, 1953458176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xe7ec0c10

Device Boot Start End Sectors Size Id Type
/dev/sdb1 2048 1953458175 1953456128 931.5G 7 HPFS/NTFS/exFAT

# hdparm -I /dev/sdb | grep Model
Model Number: WDC WD10JMVW-11S5XS0

# fdisk -l /dev/sdb

Disk /dev/sdb: 931.5 GiB, 1000170586112 bytes, 1953458176 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: dos

Disk identifier: 0xe7ec0c10

Device Boot Start End Sectors Size Id Type

/dev/sdb1 2048 1953458175 1953456128 931.5G 7 HPFS/NTFS/exFAT

# hdparm -I /dev/sdb | grep Model

Model Number: WDC WD10JMVW-11S5XS0

Create Transfer Disk for Data Copy

This command will setup the disk and mount it immediately. As additional information we need the disk label for further processing.

# dts disk create --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekv12345671234567 --block-device /dev/sdb

Will prepare /dev/sdb
serial number WD-WX61E82FY633 as an encrypted block device.
=> Set up disk for encryption using cryptsetup.
=> Create file system.
=> Mount file system.

Continue?? [y/n]y

---> IMPORTANT: RECORD THE ENCRYPTION PASSPHRASE: [m, o, 3, 4, +, j, w, I, 3, +, s, M, =] <---


The passphrase cannot be retrieved again. It will be used to encrypt this disk
and normally will not be needed again. However, if the system is restarted before
all files are copied to the filesystem and the disk is then finalized through this
CLI, you will need to provide the passphrase.

Continue?? [y/n]y

The disk label is DAMOED7GH
DO NOT INTERRUPT THIS COMMAND WHILE THE DISK IS BEING PREPARED.
This can take several minutes ...

initialized device for encryption
set up device for encryption
creating ext4 filesystem on /dev/mapper/WD-WX61E82FY633 ...
Found a atari partition table in /dev/mapper/WD-WX61E82FY633
created filesystem

Successfully prepared the disk.
You can now copy files to the file system.
After all files are copied, run this command to create the manifest:

dts disk manifest --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekv12345671234567 --disk-label DAMOED7GH [--object-name-prefix <object_name_prefix>]

Transfer Disk :
Label : DAMOED7GH
SerialNumber : WD-WX61E82FY633
Status : PREPARING
EncryptionPassphrase : mo34+jwI3+sUaL/NORhXOKQU9k12345671234567

# dts disk create --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekv12345671234567 --block-device /dev/sdb

Will prepare /dev/sdb

serial number WD-WX61E82FY633 as an encrypted block device.

=> Set up disk for encryption using cryptsetup.

=> Create file system.

=> Mount file system.

Continue?? [y/n]y

---> IMPORTANT: RECORD THE ENCRYPTION PASSPHRASE: [m, o, 3, 4, +, j, w, I, 3, +, s, M, =] <---

The passphrase cannot be retrieved again. It will be used to encrypt this disk

and normally will not be needed again. However, if the system is restarted before

all files are copied to the filesystem and the disk is then finalized through this

CLI, you will need to provide the passphrase.

Continue?? [y/n]y

The disk label is DAMOED7GH

DO NOT INTERRUPT THIS COMMAND WHILE THE DISK IS BEING PREPARED.

This can take several minutes ...

initialized device for encryption

set up device for encryption

creating ext4 filesystem on /dev/mapper/WD-WX61E82FY633 ...

Found a atari partition table in /dev/mapper/WD-WX61E82FY633

created filesystem

Successfully prepared the disk.

You can now copy files to the file system.

After all files are copied, run this command to create the manifest:

dts disk manifest --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekv12345671234567 --disk-label DAMOED7GH [--object-name-prefix <object_name_prefix>]

Transfer Disk :

Label : DAMOED7GH

SerialNumber : WD-WX61E82FY633

Status : PREPARING

EncryptionPassphrase : mo34+jwI3+sUaL/NORhXOKQU9k12345671234567

Mount point is /mnt/orcdts_DAMOED7GH.

# df -TH /mnt/orcdts_DAMOED7GH
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/WD-WX61E82FY633 ext4 984G 80M 934G 1% /mnt/orcdts_DAMOED7GH

# df -TH /mnt/orcdts_DAMOED7GH

Filesystem Type Size Used Avail Use% Mounted on

/dev/mapper/WD-WX61E82FY633 ext4 984G 80M 934G 1% /mnt/orcdts_DAMOED7GH

The Transfer Disk status has changed to PREPARING and the disk serial number is registered now.

5. Copy Data to HDD

For the test run I have copied some Open Data stuff, an Oracle Backup and Oracle Data Pump export files to the disk.

# ll /mnt/orcdts_DAMOED7GH
total 1540788
-rw-r--r--. 1 root root 13816663 Oct 28 11:07 basel-opendata.tgz
-rw-r--r--. 1 root root 132009 Oct 28 11:07 bernmobil-opendata.tgz
-rw-r--r--. 1 root root 28493853 Oct 28 11:07 bundesverwaltung-opendata.tgz
-rw-r--r--. 1 root root 307998059 Oct 28 11:07 expdphr-28-10-20-01.dmp.tgz
-rw-r--r--. 1 root root 307998059 Oct 28 11:07 expdphr-28-10-20-02.dmp.tgz
drwx------. 2 root root 16384 Oct 28 11:03 lost+found
-rw-r--r--. 1 root root 695625592 Oct 28 11:08 RMAN_CDB118_28102020.tgz
-rw-r--r--. 1 root root 223666017 Oct 28 11:08 sbb-opendata.tgz

# ll /mnt/orcdts_DAMOED7GH

total 1540788

-rw-r--r--. 1 root root 13816663 Oct 28 11:07 basel-opendata.tgz

-rw-r--r--. 1 root root 132009 Oct 28 11:07 bernmobil-opendata.tgz

-rw-r--r--. 1 root root 28493853 Oct 28 11:07 bundesverwaltung-opendata.tgz

-rw-r--r--. 1 root root 307998059 Oct 28 11:07 expdphr-28-10-20-01.dmp.tgz

-rw-r--r--. 1 root root 307998059 Oct 28 11:07 expdphr-28-10-20-02.dmp.tgz

drwx------. 2 root root 16384 Oct 28 11:03 lost+found

-rw-r--r--. 1 root root 695625592 Oct 28 11:08 RMAN_CDB118_28102020.tgz

-rw-r--r--. 1 root root 223666017 Oct 28 11:08 sbb-opendata.tgz

6. Generate Manifest File

It generates a local file which contains a list of the files and her MD5 checksums like an inventory file. Here the disk label is required.

# dts disk manifest --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekv12345671234567 --disk-label DAMOED7GH
Walking the filesystem to gather stats and to check for errors
Reading all checkpoint data. This might take a while depending on checkpoint log size.
2020/10/28 11:09:18 Manifest generation started

Files processed : 2 / 8
Bytes processed : 41.15 MB / 1.47 GB

Files processed : 2 / 8
Bytes processed : 103.15 MB / 1.47 GB

Files processed : 2 / 8
Bytes processed : 164.15 MB / 1.47 GB

Files processed : 2 / 8
Bytes processed : 228.15 MB / 1.47 GB

Files processed : 2 / 8
Bytes processed : 284.15 MB / 1.47 GB

Files processed : 2 / 8
Bytes processed : 353.15 MB / 1.47 GB

Files processed : 2 / 8
Bytes processed : 412.15 MB / 1.47 GB

Files processed : 2 / 8
Bytes processed : 476.15 MB / 1.47 GB

Files processed : 2 / 8
Bytes processed : 545.15 MB / 1.47 GB

Files processed : 3 / 8
Bytes processed : 616.88 MB / 1.47 GB

Files processed : 4 / 8
Bytes processed : 881.06 MB / 1.47 GB

Files processed : 6 / 8
Bytes processed : 950.54 MB / 1.47 GB

Files processed : 6 / 8
Bytes processed : 1007.54 MB / 1.47 GB

Files processed : 6 / 8
Bytes processed : 1.04 GB / 1.47 GB

Files processed : 6 / 8
Bytes processed : 1.11 GB / 1.47 GB

Files processed : 6 / 8
Bytes processed : 1.18 GB / 1.47 GB

Files processed : 6 / 8
Bytes processed : 1.25 GB / 1.47 GB

Files processed : 6 / 8
Bytes processed : 1.31 GB / 1.47 GB

Files processed : 6 / 8
Bytes processed : 1.37 GB / 1.47 GB

Files processed : 6 / 8
Bytes processed : 1.41 GB / 1.47 GB

Files processed : 8 / 8
Bytes processed : 1.47 GB / 1.47 GB

# dts disk manifest --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekv12345671234567 --disk-label DAMOED7GH

Walking the filesystem to gather stats and to check for errors

Reading all checkpoint data. This might take a while depending on checkpoint log size.

2020/10/28 11:09:18 Manifest generation started

Files processed : 2 / 8

Bytes processed : 41.15 MB / 1.47 GB

Files processed : 2 / 8

Bytes processed : 103.15 MB / 1.47 GB

Files processed : 2 / 8

Bytes processed : 164.15 MB / 1.47 GB

Files processed : 2 / 8

Bytes processed : 228.15 MB / 1.47 GB

Files processed : 2 / 8

Bytes processed : 284.15 MB / 1.47 GB

Files processed : 2 / 8

Bytes processed : 353.15 MB / 1.47 GB

Files processed : 2 / 8

Bytes processed : 412.15 MB / 1.47 GB

Files processed : 2 / 8

Bytes processed : 476.15 MB / 1.47 GB

Files processed : 2 / 8

Bytes processed : 545.15 MB / 1.47 GB

Files processed : 3 / 8

Bytes processed : 616.88 MB / 1.47 GB

Files processed : 4 / 8

Bytes processed : 881.06 MB / 1.47 GB

Files processed : 6 / 8

Bytes processed : 950.54 MB / 1.47 GB

Files processed : 6 / 8

Bytes processed : 1007.54 MB / 1.47 GB

Files processed : 6 / 8

Bytes processed : 1.04 GB / 1.47 GB

Files processed : 6 / 8

Bytes processed : 1.11 GB / 1.47 GB

Files processed : 6 / 8

Bytes processed : 1.18 GB / 1.47 GB

Files processed : 6 / 8

Bytes processed : 1.25 GB / 1.47 GB

Files processed : 6 / 8

Bytes processed : 1.31 GB / 1.47 GB

Files processed : 6 / 8

Bytes processed : 1.37 GB / 1.47 GB

Files processed : 6 / 8

Bytes processed : 1.41 GB / 1.47 GB

Files processed : 8 / 8

Bytes processed : 1.47 GB / 1.47 GB

The file:

# cat /mnt/orcdts_DAMOED7GH/.bulk_data_transfer/metadata/manifest

{"path":"logFile.log","md5":"PZdMVbAydkbIo3JuJW5HKQ==","lastModifiedTime":1603879758896,"size":26710}
{"path":"bernmobil-opendata.tgz","md5":"J6VvwZemr5ayF+hZmEMvUQ==","lastModifiedTime":1603879648205,"size":132009}
{"path":"expdphr-28-10-20-02.dmp.tgz","md5":"tgtoAyh79EaAHnG2cMhJ4g==","lastModifiedTime":1603879668772,"size":307998059}
{"path":"basel-opendata.tgz","md5":"rRKvfnfw3kjRgp+yyJd9eg==","lastModifiedTime":1603879648180,"size":13816663}
{"path":"sbb-opendata.tgz","md5":"iggsOeCuvpE+zDlwoYmXcA==","lastModifiedTime":1603879699757,"size":223666017}
{"path":"bundesverwaltung-opendata.tgz","md5":"Pbs8W5rNpqs+x48bb/wVQw==","lastModifiedTime":1603879649038,"size":28493853}
{"path":"RMAN_CDB118_28102020.tgz","md5":"6ivSfbVKHceQ6FMy2xVSNA==","lastModifiedTime":1603879692329,"size":695625592}
{"path":"expdphr-28-10-20-01.dmp.tgz","md5":"tgtoAyh79EaAHnG2cMhJ4g==","lastModifiedTime":1603879658104,"size":307998059}

# cat /mnt/orcdts_DAMOED7GH/.bulk_data_transfer/metadata/manifest

{"path":"logFile.log","md5":"PZdMVbAydkbIo3JuJW5HKQ==","lastModifiedTime":1603879758896,"size":26710}

{"path":"bernmobil-opendata.tgz","md5":"J6VvwZemr5ayF+hZmEMvUQ==","lastModifiedTime":1603879648205,"size":132009}

{"path":"expdphr-28-10-20-02.dmp.tgz","md5":"tgtoAyh79EaAHnG2cMhJ4g==","lastModifiedTime":1603879668772,"size":307998059}

{"path":"basel-opendata.tgz","md5":"rRKvfnfw3kjRgp+yyJd9eg==","lastModifiedTime":1603879648180,"size":13816663}

{"path":"sbb-opendata.tgz","md5":"iggsOeCuvpE+zDlwoYmXcA==","lastModifiedTime":1603879699757,"size":223666017}

{"path":"bundesverwaltung-opendata.tgz","md5":"Pbs8W5rNpqs+x48bb/wVQw==","lastModifiedTime":1603879649038,"size":28493853}

{"path":"RMAN_CDB118_28102020.tgz","md5":"6ivSfbVKHceQ6FMy2xVSNA==","lastModifiedTime":1603879692329,"size":695625592}

{"path":"expdphr-28-10-20-01.dmp.tgz","md5":"tgtoAyh79EaAHnG2cMhJ4g==","lastModifiedTime":1603879658104,"size":307998059}

7. Lock the Disk

# dts disk lock --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekv12345671234567 --disk-label DAMOED7GH --block-device /dev/sdb
Copying upload user credentials.
created object BulkDataTransferTestObject in bucket data_transfer_usb
overwrote object BulkDataTransferTestObject in bucket data_transfer_usb
inspected object BulkDataTransferTestObject in bucket data_transfer_usb
read object BulkDataTransferTestObject in bucket data_transfer_usb
Scanning filesystem to validate manifest. If special files are encountered, they will be listed below.
validated manifest
/dev/sdb DAMOED7GH is encrypted and locked
Locked disk.

# dts disk lock --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekv12345671234567 --disk-label DAMOED7GH --block-device /dev/sdb

Copying upload user credentials.

created object BulkDataTransferTestObject in bucket data_transfer_usb

overwrote object BulkDataTransferTestObject in bucket data_transfer_usb

inspected object BulkDataTransferTestObject in bucket data_transfer_usb

read object BulkDataTransferTestObject in bucket data_transfer_usb

Scanning filesystem to validate manifest. If special files are encountered, they will be listed below.

validated manifest

/dev/sdb DAMOED7GH is encrypted and locked

Locked disk.

8. Attach the Disk to the Transfer Label and create the Package

The status now changes to ACTIVE.

# dts disk attach --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekv12345671234567 --disk-label DAMOED7GH --package-label PDQ0QMLNT
Attached disk: DAMOED7GH to package: PDQ0QMLNT

1 2	# dts disk attach --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekv12345671234567 --disk-label DAMOED7GH --package-label PDQ0QMLNT Attached disk: DAMOED7GH to package: PDQ0QMLNT

9. Shipping and Shipping Information Update

As shipping company I used DHL Switzerland. They have a pick point near by in Langenthal. At this point, it’s important to organize the return shipping too and put the return shipping label in the box. I didn’t realize it and have forgotten to organize the return shipping. So the disk was stranded in the Frankfurt data center. And then the story began. As a private person, the delivery companies DHL and UPS doesn’t allow private persons to re-import packages from outside Switzerland without a customer number. But, private persons don’t get such a number. Finally with FedEx I was able to organize the return shipping. Thanks to Andrew and Christos from Oracle’s OCI Data Transfer team for their patience!

Note: Companies like DHL have templates to create pro-forma commercial invoices – https://www.dhl.ch/exp-de/express/zollabwicklung/zollpapiere/proforma_rechnung.html#invoice

This disk was sent to the Oracle Cloud Infrastructure data center Frankfurt.

Now the shipping information has to be updated with vendor and the tracking numbers.

# dts package ship --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekv12345671234567 --package-label PDQ0QMLNT --package-vendor DHL --tracking-number 5542918614 --return-tracking-number 12345678
Update package: PDQ0QMLNT

1 2	# dts package ship --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekv12345671234567 --package-label PDQ0QMLNT --package-vendor DHL --tracking-number 5542918614 --return-tracking-number 12345678 Update package: PDQ0QMLNT

10. Tracking

DHL required two days until delivery in Frankfurt. Oracle started one day later with the data import.

11. Data Processing

Then Oracle is uploading the data and the disk is attached, the job transfer status changes to PROCESSING.

# dts package show --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekv12345671234567 --package-label PDQ0QMLNT
Transfer Package :
Label : PDQ0QMLNT
TransferSiteShippingAddress : Telehouse c/o Oracle Data Transfer Service; Christos Panoudis; Phone: +49152 22882673 Job:J5Y0R5KAS Package:PDQ0QMLNT ; Kleyerstraße 75-87; 60326 Frankfurt am Main; Germany
DeliveryVendor : DHL
DeliveryTrackingNumber : 5542918614
ReturnDeliveryTrackingNumber : 12345678
Status : PROCESSING
Devices :
[1] :
Label : DAMOED7GH
SerialNumber : WD-WX61E82FY633
UploadStatusLogURL : J5Y0R5KAS/DAMOED7GH/upload_summary.txt
Status : PROCESSING

# dts package show --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekv12345671234567 --package-label PDQ0QMLNT

Transfer Package :

Label : PDQ0QMLNT

DeliveryVendor : DHL

DeliveryTrackingNumber : 5542918614

ReturnDeliveryTrackingNumber : 12345678

Status : PROCESSING

Devices :

[1] :

Label : DAMOED7GH

SerialNumber : WD-WX61E82FY633

UploadStatusLogURL : J5Y0R5KAS/DAMOED7GH/upload_summary.txt

Status : PROCESSING

12. Data Verification

Finally the data is arrive in the Oracle Cloud Infrastructure Object Storage and is ready for use. The file processing is logged in the new created file upload_summary.txt.

# dts package show --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekv12345671234567 --package-label PDQ0QMLNT
Transfer Package :
Label : PDQ0QMLNT
TransferSiteShippingAddress : Telehouse c/o Oracle Data Transfer Service; Christos Panoudis; Phone: +49152 22882673 Job:J5Y0R5KAS Package:PDQ0QMLNT ; Kleyerstraße 75-87; 60326 Frankfurt am Main; Germany
DeliveryVendor : DHL
DeliveryTrackingNumber : 5542918614
ReturnDeliveryTrackingNumber : 12345678
Status : PROCESSING
Devices :
[1] :
Label : DAMOED7GH
SerialNumber : WD-WX61E82FY633
UploadStatusLogURL : J5Y0R5KAS/DAMOED7GH/upload_summary.txt
Status : PROCESSING


########################################################################################################################################################################################################
######################################### SUMMARY FOR DEVICE [WD-WX61E82FY633] ############################################
Generated at 2020-11-02 18:27:35
TOTAL: 8
### P present: 8
### M missing: 0
### C name collision: 0
### U unreadable: 0
### N nameTooLong: 0
########################################################################################################################################################################################################
| STATUS | NAME | LAST_MODIFIED | SIZE(MB) | MD5 | ETag |
| present | RMAN_CDB118_28102020.tgz | Mon Nov 02 18:27:36 UTC 2020 | 663.40 | 6ivSfbVKHceQ6FMy2xVSNA== | 99baa855-b15c-4f79-bfaa-dbf16385a182 |
| present | logFile.log | Mon Nov 02 18:27:16 UTC 2020 | 0.05 | Aepiez79Y7/WZYweOWA40w== | 894a88ee-c2f2-454b-875a-0bc3339ff6ab |
| present | bernmobil-opendata.tgz | Mon Nov 02 18:27:16 UTC 2020 | 0.13 | J6VvwZemr5ayF+hZmEMvUQ== | 38aa09fa-46f7-4009-8f8d-b710a43fe74f |
| present | expdphr-28-10-20-02.dmp.tgz | Mon Nov 02 18:27:29 UTC 2020 | 293.73 | tgtoAyh79EaAHnG2cMhJ4g== | 78094b66-6366-464a-9c93-3d4a4e1be3ca |
| present | basel-opendata.tgz | Mon Nov 02 18:27:17 UTC 2020 | 13.18 | rRKvfnfw3kjRgp+yyJd9eg== | f70fb2e9-baf2-40b1-a1dc-7522e6005d66 |
| present | bundesverwaltung-opendata.tgz | Mon Nov 02 18:27:19 UTC 2020 | 27.17 | Pbs8W5rNpqs+x48bb/wVQw== | 20f46883-2e13-4655-8c63-ff455db81fba |
| present | sbb-opendata.tgz | Mon Nov 02 18:27:26 UTC 2020 | 213.30 | iggsOeCuvpE+zDlwoYmXcA== | 90b81b13-611d-440a-9915-35c53e46b10b |
| present | expdphr-28-10-20-01.dmp.tgz | Mon Nov 02 18:27:29 UTC 2020 | 293.73 | tgtoAyh79EaAHnG2cMhJ4g== | 258c26c8-6d2f-4fba-864f-b03062de699e |
########################################################################################################################################################################################################

# dts package show --job-id ocid1.datatransferjob.oc1.eu-frankfurt-1.antheljszffzlryauwznondp2zp6rztpbvzyekv12345671234567 --package-label PDQ0QMLNT

Transfer Package :

Label : PDQ0QMLNT

DeliveryVendor : DHL

DeliveryTrackingNumber : 5542918614

ReturnDeliveryTrackingNumber : 12345678

Status : PROCESSING

Devices :

[1] :

Label : DAMOED7GH

SerialNumber : WD-WX61E82FY633

UploadStatusLogURL : J5Y0R5KAS/DAMOED7GH/upload_summary.txt

Status : PROCESSING

########################################################################################################################################################################################################

######################################### SUMMARY FOR DEVICE [WD-WX61E82FY633] ############################################

Generated at 2020-11-02 18:27:35

TOTAL: 8

### P present: 8

### M missing: 0

### C name collision: 0

### U unreadable: 0

### N nameTooLong: 0

13. Object Storage Replication Policy (optional)

The files are now in the data center Frankfurt, but I want to have them in the Swiss data center region Zurich. Therefore I set a replication policy on level Object Storage. In Zurich, a new bucket called data_transfer_usb_from_FRA is created. And a few minutes later, the files were available in the Object Storage Zurich. Sure, it depends on the file size 😉

Finally…

Detach the transfer disk so the data center guys can send it back to you.

And after a few days…welcome FedEx in Kestenholz / Jurasüdfuss / Solothurn / Switzerland!

Some words about Shipping and Costs

Shipping costs from DHL and Fedex:

Vendor	From	To	Costs
DHL	Langenthal / Switzerland	Frankfurt	79.50 CHF
FedEx	Frankfurt	Kestenholz	130.45 CHF

Summary

To watch nice marketing slides and documents about cool features is not enough. To find out how it works in the real word, a real is test is required. How to migrate data into a data center of any cloud provider should be basic know-how of each consultant which is working with and on cloud themes. Moving data by a disk ro an appliance opens a lot of possibilities for data migrations into the cloud. For example a huge DWH: Transfer the RMAN backup into the cloud, restore it, close the GAP by an incremental backup and synchronize it with Oracle Golden Gate. #ilike

Oracle Cloud Infrastructure

Latest Posts

OCI Cloud Performance Management for On-Premises Databases – Part 1 – Management Agent Installation

This is a small blog post series

My Setup

Prerequisites for Management Agent Installation

Setup for OCI Management Agent

Group AGENT_ADMINS

Policy Datacenter_Kestenholz_Agent_Policy

Dynamic Group Management_Agent_Dynamic_Group

Policy Datacenter_Kestenholz_Agent_Communication_Policy

Install On-Premises Management Agent

Create Agent Install Key

Download the Software and Transfer it to the Target Server

Create a Local Response File

RPM Installation

Agent Configuration

Verify the Management Agent in Oracle Cloud Infrastructure

Troubleshooting

Summary Part 1

Oracle Cloud Infrastructure – A short Blog Post about a secure and small Development Setup

Requirements

Architecture Diagram

Network Components

Monitoring

Links

Next Steps

Other Ideas

Summary

Oracle Cloud Infrastructure Data Safe – How to burn down 201.44 Swiss Francs in 30 Seconds…

Is Data Safe really for free?

Oracle Cloud Infrastructure Price List

Cost and Usage Report

Lessons learned

Red Hat Ansible Tower Upgrade from 3.5 to 3.8 – when running setup.sh is not enough – or: I have made fire!

The running Ansible Tower Setup

Upgrade Path

Prerequisites

Upgrade Overview

1. Shutdown Machines and create Snapshot of the Hosts

Tower and Database Shutdown

Create a backup or snapshot

Restart machines

2. Install Postgres 10 on database server and create a new Postgres 10 database

Install Postgres 10

Create a new PostgreSQL Cluster with Trivadis pgOperate

Create a new Database for Tower Repository

Verification

3. Manual Export / Import of Ansible Tower Repository Data

Export Data

Import Data

Stop and disable existing Postgres 9.6 Database

4. Re-run Ansible Tower 3.5.1 setup against new database

Adapt Inventory

Re-run setup.sh

Verification

5. Upgrade to Ansible Tower 3.7.1

Run setup.sh

Verification

6. Upgrade to Ansible Tower 3.8

Run setup.sh

Verification

7. Add Red Hat Subscription Manifest

8. I have made fire!

Summary

Oracle OCI Data Transfer Service – A journey from Kestenholz/Jurasüdfuss/Switzerland to Frankfurt and back

Setup

Data Transfer Service Regions

How is your data coming into the Oracle Cloud

1. Enable Data Transfer Service – Entitlement

2. Prepare an Object Storage Bucket

3. Create a Transfer Job

4. Attach a HDD to a Linux based host, use the Data Transfer Utility to create and encrypt the Device

Package Installation for my Oracle Linux 7 Machine

Download and Installation of the Data Transfer Utility

Configure IAM Credentials for Data Transfer Actions

Verify Credentials

Show Data Transfer Job Details – Status is PREPARED

Prepare USB Hard Disk Drive

Create Transfer Disk for Data Copy

5. Copy Data to HDD