Author Archive: Martin Berger

Never stop Learning – why I love Oracle LiveLabs

Since over one and a half year, this week I was back in an onsite training, live people, live teaching. With a motivated junior DBA class, we started with all about Oracle architecture based on our Trivadis training O-AI – Oracle Architecture and Internals. The training is a mix between slides, demos and labs. Therefore during the course we run the training environments in Oracle Cloud Infrastructure, build by Terraform (Credits to Stefan Oehrli from which has ramped up the whole stuff). After the course at the end of the month, the environments will be cleaned up. And what’s next?

Training Environments

There are a lot of possibilities to get a deeper knowledge of all this Oracle stuff like processes, data encryption, multitenancy, datapump and so on:

But my actual favorite is Oracle LiveLabs!

Oracle LiveLabs

This platform is not only for DBAs, it has a lot of workshops for Application Developers, Data Scientists and DevOps Engineers too. There are different workshop types available:

  • get your free Oracle Cloud Infrastructure training environment for free during a time period like Oracle Database 19c New Features – run on LiveLabs
  • workshops which are running in a free tenancy
  • workshops what you can do in your own paid tenancy

At the moment there are 21 workshops where you get a live environment with all components you need virtual machines or database in Oracle Cloud Infrastructure like Oracle Multitenant Fundamentals, Database 19c – Automatic Indexing, 21c New Features on Autonomous Database and many more. All workshops are very well described, from the access to the initial setup and finally for the workshop himself too.

In this case I have decided to start the Oracle Multitenant lab to gather more information how PDB Snapshot Copy works.

1st – Search your training in the available workshops  and press Launch

2nd – Define where the Workshop should run

In this case, I want to reserve an environment. This is not possible for all workshops, you can see that in the workshop details if it’s possible to the an Oracle Cloud Infrastructure setup.

3rd – Define the Start Date and propose your SSH Public Key

With the key, you can get access to the training servers by SSH. In this case I want to start the workshop immediately. Otherwise define a start and end date. If you don’t want to start now, you will get a confirmation that the workshop is reserved and an email at the day where the workshop starts with the credential information.

4rd – View my Reservation

After some minutes, the status for the workshop is updated. As you can see here, in about three minutes from now, the environment should be ready. You will receive a confirmation mail.

5th – Launch Workshop

When the workshop is ready, the workshop can be launched.

6th – Workshop Details

All information you need is in the details like:

  • User name
  • Initial password for OCI
  • Compartment
  • Instance public IP

Here you have also the chance to extend your workshop reservation time. Follow the Get Started instructions to the bottom and push the button to move on the introduction. Step by step you are guide through the login and setup process. All labs contain a manual how to connect and to do the initial setup like starting listeners or get scripts from the OCI Object Storage.

There are the connection options how you can interact with the LiveLab:

  1. Connect using Cloud Shell
  2. Connect using MAC or a Windows CYGWIN Emulator
  3. Connect using Putty

Example code for the multitenancy lab preparation:


Oracle LiveLabs is another great opportunity to learn and train new stuff. All you have to take care now is to follow the workshop instructions and take care about the limited time. Enjoy it, learn new stuff and have fun! Oracle LiveLabs are easy to join, easy to set up and well described. This is why I love it 🙂

Oracle 21c runInstaller on Oracle Linux 8 – Don’t forget to run – [FATAL] [INS-32035]

In one of my lab environments where an Oracle 19c database is up and running, it was time to upgrade it to 21c by AutoUpgrade. The new ORACLE_HOME was ready, the software extracted. But when running the OUI installer in silent mode, it stops working: [FATAL] [INS-32035].

runInstaller – 1st run

The error from the OUI logfile:

Quick Solution

An opatch lsinventory command from the existing 19c installation shows, that there is a file called oraInst.loc is located in the 19c ORACLE_HOME. But for existing RDBMS installations, the OUI has to verify the inventory location in /etc/oraInst.loc or depending on the OS in  /var/opt/oracle/oraInst.loc. But: NO FILE THERE.

The quick and simple solution was as OS user root to copy the existing file which points to the inventory directory to /etc and re-start the OUI.

runInstaller – 2nd run

Why this happened?

During the old 19c installation, I did not run /u01/app/oraInventory/ which creates this file. Extract from the 19c runInstaller log:

Alternative solution

Just run /u01/app/oraInventory/ again – it creates the file /etc/oraInst.loc for you and sets the correct permissions. Take a look here in this My Oracle Support note  for a detailed description what the script does:

  • What Are And Scripts In A Standalone RDBMS Installation? (Doc ID 1493121.1)


This was really a stupid error. Lesson learned: read the f* terminal output – even in lab environments. AutoUpgrade can start now 🙂

Oracle Cloud Infrastructure – one DRG to many VCNs – arrived in Zurich / Switzerland – Upgrade Time!

One of my OCI favorites is finally arrived in Oracle Cloud Infrastructure data center eu-zurich-1 – the possibility to attach one Dynamic Routing Gateway DRG to multiple Virtual Cloud Networks VCNs. To see how it works, Simo has written three blog posts where it’s very well described. Now it’s time to upgrade my existing DRG!

Oracle’s OCI documentation:

How to upgrade an existing DRG? See here – just click on the images to see the details.

Step 1 – verify on an existing DRG if your region is ready – if yes, press the Upgrade DRG button

As you can see here, in this version of DRG, there is the existing IPSec connection listed.

Step 2 – Do you really want to upgrade the DRG? Sure!

Step 3 – The UI View changes, new Resources are created

After some minutes, the DRG view has changed, now the Virtual Cloud Network attachments are visible. New resources like DRG Route Tables and Route Distributions are created.

Step 4 – What about the existing VPN Tunnel?

Verify if the existing tunnel is still running and try to connect to be sure that the routing work properly. It works – nice!


To attach a single DRG to multiple VCNS opens new possibilities in network design. Read the manual properly to understand how components like DRG Route Tables and DRG Attachments are working. Well done, well done, Oracle!

Oracle EM13c in Oracle Cloud Infrastructure – sending Mail Notifications by Email Delivery Service

For the Trivadis Oracle Enterprise Manager Training, we setup and Oracle EM13c in the Oracle Cloud Infrastructure. One student exercise is, to setup notification rules. Therefore a mail server is required. In Oracle Cloud Infrastructure, there is an Email Delivery service available which fits best: Overview of the Email Delivery Service (

I have tried to setup a local postfix mail gateway which used the NAT gateway address as sender’s address. But the NAT gateway addresses are blacklisted in spam services like Even Microsoft doesn’t want to allow these mails send by the NAT gateway. You can see my attempts at the bottom of this blog post.


Subnet Component Purpose
Public Virtual Machine Bastion Server, SSH Access, Apache Guacamole
Private Virtual Machine Oracle Enterprise Manager EM13c – OMS
Private Database System Oracle Enterprise Manager EM13c – Repository


Email Delivery Service Prerequisites

The configuration for this service has to be done as non-federated user.  If you want to use the service as a federated user, you get this message: Email Delivery is not available with your current permissions. Please Contact Support for further assistance.

IAM User

This is why I have created a IAM user called email-delivery-service-user. The user needs these permissions to manage the email delivery service.

email-delivery-service-policy Allow group Email-Delivery-Service-Group to manage approved-senders in compartment training:o-em
Allow group Email-Delivery-Service-Group to use email-family in compartment training:o-em


SMTP Credentials

Additional a SMTP credential for this user is created. This SMTP credentials are used for Oracle Enterprise Manager EM13c mail server configuration. Identity >> Users >> User Details >> SMTP Credentials. Copy the provided OCID and password temporary for later use.


Email Delivery Service Setup

Login in Oracle Cloud Infrastructure user interface as previous created user to configure the approved sender list. Developer Services >> Email Delivery. Add the mail address what you want to use for OEM communication to the approved sender list. Take care at the policy. Here in this case, the user is only allowed to do it in the O-EM called sub-compartment. Now we are ready to configure the Oracle Enterprise Manager EM13c.

The SMTP server is visible on the Email Configuration page and depends on your region. In my case, the SMTP endpoint in data center Zurich is used:

Oracle Enterprise Manager EM13c – Mail Servers Configuration

In Setup >> Notifications >> Mail Servers, we add a new mail server.

Host SMTP host provided by OCI
Port 587
User Name SMTP Credentials user name
Password SMTP Credentials password
Use Secure Connections TLS, if available


Set the Sender Identity, the Sender’s Email Address corresponds to the entry in the deliver approved sender list.

Oracle Enterprise Manager EM13c – Mail Servers Test

The configuration is done, we can test it. Test Mail Servers – click on the image for more details. After some seconds, you see a confirmation at the top that the test succeeded. Verify your mailbox for the test message. That’s all folks.

Addendum – Spamhaus and

Before I used the Email Delivery Service, the first try was to configure postfix as local mail agent, there is a good manual available here: Oracle Linux: Install the Postfix Email Server. But this was not successful. When I tried to send mails to my companies’ address or to an ( is one of them), I always got a SMTP error: Client host [] blocked using Spamhaus is the public IP address from the Oracle Cloud Infrastructure NAT gateway, first I did a removal request online at – the URL was provided in the error message. And 2hrs later, I got a message and confirmed my request.

Two things:

  1. my company syncs their spam filter on a regular base, so after a while, I was able to send notifications to my personal company mail address
  2. but for the Hotmail ( address, it did still not work: Unfortunately, messages from [] weren’t sent. Please contact your Internet service provider since part of their network is on our block list (S3140)

So I tried to contact Microsoft on the same way with a support request:

And some minutes later I git the answer: Not qualified for mitigation.


This is quite simple: If you are in the cloud – use the cloud services.

OCI Cloud Performance Management for On-Premises Databases – Part 2 – Database Configuration

In the part 1 of this blog post series, we have installed the Management Agent. Now it’s time to add the database. For this case an agent plug-in has to be installed first before we can configure the database.

More information about the management agent administration:

This is a small blog post series

My Setup

  • An OCI Tenant in datacenter EU-FRANKFURT-1
  • An empty compartment called datacenter-kestenholz
  • An on-premises database called CDB114, running on Oracle Linux 7
  • OCI Management Agent up and running

The goal is to handle the on-premises database in OCI. Output from the Trivadis TVD-Basenv(TM) framework which show the database up and running:

Service Plug-In Deployment

From the agent page, we select the Management Agent to see the details. Click on Deploy Plug-Ins.

Select the plug-in for the database management, press Update to deploy it on the management agent.

Now you see the confirmation that the deployment process is initiated.

After some seconds, you can see on the top right of the agent overview page, that the service plug-in is installed.

Register the External Database

In External Database menu, we register an external container database first. Be sure that the right compartment is selected.

Set the compartment and display name and click Register.

More about this process:

Connect to the External Database

After registration, the container database is added, but not connected. In this step we create a connector to setup the database in OCI.

Set display name, DNS hostname, port and service first.

The connector type cannot be changed, select the management agent from the connector agent id list, specify database connection credentials. I use the database schema SYSTEM. Click on Connect to External Container Database.

After some minutes, the work request is done and the database is connected. The database is in state Available.

We repeat the steps above to add the three external Pluggable Databases on the same way.

Enable Database Management

Before Database Management from OCI can be used, it has to be enabled. This is done on the page where the external Container Database is visible. According Oracle’s license handling, this has to be done manually. You can select between BYOL and License Included. After some minutes, the database management is enabled and the agent is updated. More about licensing and pricing in the next blog post number 3 which is coming soon.

Database Management – Database Groups and Fleet Summary

When Database Management is enabled in OCI for all Container and Pluggable Databases, the management agent starts to collect database related data. We can organized the external databases in groups.

In Database Management menu, create a new database group. In my case I named it Datacenter_Kestenholz which contains the previous added external databases.

When Database Management is enabled in OCI for all Container and Pluggable Databases, the Management Agent starts to collect database related data. Some moments later, the dashboard has the first values.

Note: The Container Database is not shown in the Fleet Summary dashboard at the moment, only Pluggable Databases are listed with performance data etc. – I will open a SR to clarify it.

Summary Part 2

When the Management Agent is up and running, adding a database is straight forward. But we have to clarify the license situation first. More about licensing and cost in the next blog post part 3.