Uncategorized

Oracle OCI Data Transfer Service – A journey from Kestenholz/Jurasüdfuss/Switzerland to Frankfurt and back

The Oracle Data Transfer service is a offline data transfer method to migrate data to the Oracle Cloud Infrastructure. A transfer service is useful, when your network bandwidth and connection is not sufficient to upload your migration data  in a meaningful time. Oracle offers two methods: The disk-based data transfer and the appliance-based data transfer. The service is no only one-way, data can also be exported in an Oracle Cloud Infrastructure data center and shipped to your data center.

According one of my Company Trivadis’ cultural value called curiosity, I was wondering how this service works. This is the story of a tiny USB Hard Disk Drive full of data, which was going on a long journey from Kestenholz / Jurasüdfuss / Solothurn / Switzerland to the Oracle Infrastructure data center in Frankfurt and back.

Setup

  • The OCI Data Transfer utility is Linux based, the USB 3.0 HDD is attached to a VMware Virtual machine where Oracle Linux is running
  • The virtual machine has access to the Internet
  • Data is available – for this example I used some open data from Swiss government (opendata.swiss)

Data Transfer Service Regions

Actually data transfer is available in Frankfurt, Ashburn, Phoenix, London and Osaka. From Switzerland, Frankfurt is the nearest location.

How is your data coming into the Oracle Cloud

  1. Enable Data Transfer Service – Data Transfer
  2. Prepare an Object Storage bucket
  3. Create a Transfer Job
  4. Attach a HDD to a Linux based host, use the Data Transfer Utility to create and encrypt the device
  5. Copy data to the HDD
  6. Create and upload the disk manifest
  7. Lock the disk
  8. Attach the Disk to the transfer label and create the package
  9. Create a transfer package
  10. Shipping and update shipping information
  11. Tracking
  12. Data Verification
  13. Object Storage Replication Policy (optional)
  14. Finally…

Note: Most of the jobs above can be done by the OCI CLI on command line and are very well described in the Oracle doumentation.

1. Enable Data Transfer Service – Entitlement

Before you can use this service, the Data Transfer service has to be enabled in general. Therefore you have to request it. The OCI tenant administrator gets a document, what he has to sign in a digital way. It contains for example a description how to bring data to OCI, and if you order an appliance that there will be a 45day maximum limit where the appliance has to be returned to Oracle. And a few days later, the service is ready to use. Basically now you have the permissions, to order a Data Transfer Appliance, but in this test I used the Disk Service.

2. Prepare an Object Storage Bucket

In the Frankfurt region, I created a new Object Storage bucket called data_transfer_usb. This is the bucket where the shipped data will be transferred in.

3. Create a Transfer Job

In Object Storage – Data Transfer Import,  we create a new transfer job. It contains the upload bucket from above and as method the transfer device type is disk. For furtehr processing, we need the OCID of the job. As you can see, actually there is no transfer disk attached.

4. Attach a HDD to a Linux based host, use the Data Transfer Utility to create and encrypt the Device

Prerequisites for the Data Transfer Utility according the documentation:

  • An OCI Account which have the IAM permissions for Data Transfer
  • A Linux machine with Oracle Linux 6 or greater, Ubuntu 14.04 or greater, SUSE 11 or greater
  • Java 1.8  or 1.11
  • hdparm 9.0 or later
  • Cryptsetup 1.2.0 or later

Package Installation for my Oracle Linux 7 Machine

Download and Installation of the Data Transfer Utility

The actual link to the file is in the online documentation.

Test

Configure IAM Credentials for Data Transfer Actions

The configuration is according configuring the Oracle Cloud Infrastructure CLI with user, fingerprint, key_file, tenancy and region. Example configuration file:

Verify Credentials

Show Data Transfer Job Details – Status is PREPARED

Here you can see the shipping address from the Oracle Infrastructure data center frankfurk and the label. Both information are used later in process.

Prepare USB Hard Disk Drive

The disk is attached as /dev/sdb – it is a Western Digital drive. Important: The disk needs no partition.

Create Transfer Disk for Data Copy

This command will setup the disk and mount it immediately. As additional information we need the disk label for further processing.

Mount point is /mnt/orcdts_DAMOED7GH.

The Transfer Disk status has changed to PREPARING and the disk serial number is registered now.

5. Copy Data to HDD

For the test run I have copied some Open Data stuff, an Oracle Backup and Oracle Data Pump export files to the disk.

6. Generate Manifest File

It generates a local file which contains a list of the files and her MD5 checksums like an inventory file. Here the disk label is required.

The file:

7. Lock the Disk

8. Attach the Disk to the Transfer Label and create the Package

The status now changes to ACTIVE.

9. Shipping and Shipping Information Update

As shipping company I used DHL Switzerland. They have a pick point near by in Langenthal. At this point, it’s important to organize the return shipping too and put the return shipping label in the box. I didn’t realize it and have forgotten to organize the return shipping. So the disk was stranded in the Frankfurt data center.  And then the story began. As a private person, the delivery companies DHL and UPS doesn’t allow private persons to re-import packages from outside Switzerland  without a customer number. But, private persons don’t get such a number. Finally with FedEx I was able to organize the return shipping. Thanks to Andrew and Christos from Oracle’s OCI Data Transfer team for their patience!

Note: Companies like DHL have templates to create pro-forma commercial invoices – https://www.dhl.ch/exp-de/express/zollabwicklung/zollpapiere/proforma_rechnung.html#invoice

This disk was sent to the Oracle Cloud Infrastructure data center Frankfurt.

Now the shipping information has to be updated with vendor and the tracking numbers.

10. Tracking

DHL required two days until delivery in Frankfurt. Oracle started one day later with the data import.

11. Data Processing

Then Oracle is uploading the data and the disk is attached, the job transfer status changes to PROCESSING.

12. Data Verification

Finally the data is arrive in the Oracle Cloud Infrastructure Object Storage and is ready for use. The file processing is logged in the new created file upload_summary.txt.

13. Object Storage Replication Policy (optional)

The files are now in the data center Frankfurt, but I want to have them in the Swiss data center region Zurich. Therefore I set a replication policy on level Object Storage. In Zurich, a new bucket called data_transfer_usb_from_FRA is created. And a few minutes later, the files were available in the Object Storage Zurich. Sure, it depends on the file size 😉

Finally…

Detach the transfer disk so the data center guys can send it back to you.

And after a few days…welcome FedEx in Kestenholz / Jurasüdfuss / Solothurn / Switzerland!

Some words about Shipping and Costs

Shipping costs from DHL and Fedex:

Vendor From To Costs
DHL Langenthal / Switzerland Frankfurt 79.50 CHF
FedEx Frankfurt Kestenholz 130.45 CHF

Links

Summary

To watch nice marketing slides and documents about  cool features is not enough. To find out how it works in the real word, a real is test is required. How to migrate data into a data center of any cloud provider should be basic know-how of each consultant which is working with and on cloud themes. Moving data by a disk ro an appliance opens a lot of possibilities for data migrations into the cloud. For example a huge DWH: Transfer the RMAN backup into the cloud, restore it, close the GAP by an incremental backup and synchronize it with Oracle Golden Gate. #ilike

Patching a DBaaS Database in the Oracle Cloud

To apply a Patch on a Database which is hosted by Oracle is done with one mouse click. This service is available only if you have selected the full database service during the DBaaS order process.

To the manual: https://docs.oracle.com/cloud/latest/dbcs_dbaas/CSDBI/GUID-50BDEF7D-A30E-4B32-BAE7-486538413E2D.htm#CSDBI3422

Patchset Availability

As soon as a new Oracle patchset is available, you can see that in the database management dashboard of the selected database. The precheck function verifies if the selected database is ready to patch.

patch01

Precheck Result

After 2 until 3 minutes and a refresh of the page you see the result.

patch02

 

 

 

 

 

 

 

Command Line Interface dbpatchm

If you have no web access, all these steps can be executed manually in the shell. Therefore a command line tool called dbpatchm is available. You have to login as user root before using this tool.

List of available commands:

To the dbpatchm Documentation: https://docs.oracle.com/cloud/latest/dbcs_dbaas/CSDBI/GUID-50BDEF7D-A30E-4B32-BAE7-486538413E2D.htm#CSDBI3422

Patch Apply

If the prerequisite check is successful, we can apply the patch by selecting Patch.

patch03

If there are any Java Cloud Services (Weblogic Servers) which are using the selected database too, you can force to apply the patchset.

patch04

The patch progress is show in the dashboard.

patch05

In the background, Oracle transfers a new Oracle image where the selected PSU is already applied, extract it and changes the /u01/app location afterwards. During the patch apply process, the image is located under /u01/donwload:

Patch installed

After 45 minutes, the database is patched and up and running again.

patch07

The verification with OPatch shows the correct patch numbers. Here is an extract from the OPatch output. You can see that the apply-date is the 14th of July and that Oracle has installed bundle patch with patches for the RDBMS and the OJVM. I remember, in the background Oracle extracts a fully patched ORACLE_BASE / ORACLE_HOME directory for later usage.

The data dictionary is up to date too – the query on dba_registry_patch shows the correct PSU number:

Patching kills the EM12c Agent

Oracle is creating a new directory for the software, the old stuff which was in directory /u01/app is copied in /u01/app.ORG. So the EM12c Agent is not available anymore.

patch06

You have two possibilities to get the EM12c agent back:

  • Copy the installation directories from /u01/app.ORG back into /u01/app and re-attach the Agent Home in the inventory
  • Clean up the agent in EM12c and execute a reinstall

To avoid future problems, you can attach a new disk and use them for the agent or for other tools.

Summary

Patching a DBaaS database in the Oracle Cloud is very easy. All important steps are done automatically. In the background Oracle creates a new ORACLE_BASE directory in /u01. But you have to take care about installing your own tools or for example an EM12c agent. Also the database logs are no more available in the base directory. But you can find the in an subfolder of the direcory /u01/app.ORG where the old stuff is located.