Latest Posts

Install Oracle 19c RDBMS on Oracle Linux 8 – avoid [WARNING] [INS-08101] Unexpected error while executing the action at state: ‘supportedOSCheck’

Author: 18. May 2020

Oracle RDBMS 19c is now certified on Oracle Linux 8. Unfortunately the Oracle Universal Installer in Silent Mode fails when checking the OS version. My OS release:

Oracle Universal Installer in Silent Mode

The OUI installer in silent mode stops and shows this error message: [WARNING] [INS-08101] Unexpected error while executing the action at state: ‘supportedOSCheck’. There is no parameter available like -ignoreOS or whatever, but two workarounds.

Workaround 1

Set the OS parameter in your terminal to a previous version and re-start the Oracle Universal Installer.

Workaround 2

Edit the file cvu_config which is located in $ORACLE_HOME/cv/admin, change the line from

to

Save the file and re-start the Oracle Universal Installer.

Et voilà

Some moments later… the WARNING message about the inventory can be ignored. OUI has created there some files from the previous attempt.

Oracle Universal Installer in GUI Mode

In the OUI Graphical User Interface you can ignore the message which occurs when the installer starts:

Summary

The installation of the RDBMS software by GUI is not my favourite. I try to use the silent method whenever it’s possible. And with a small workaround, this can be successfully done with 19c on OL8.

19c, Oracle Linux

Oracle Cloud Infrastructure and SSH Keys – Jump!

Author: 27. April 2020

Jump!

In our Trivadis Oracle Cloud Infrastructure training environments, we never use direct access to an application or database server by a public IP address. For this case, we use an Oracle Linux based bastion host which acts as a jump host. For security reasons, I never put any SSH keys on a bastion host to connect from there to the target instances. If your bastion host is compromitted, your SSH keys are lost! In one of the last trainings, some participants had problems with. So I decided to blog about. This blog post shows you the different methods to connect to an Oracle Cloud Infrastructure private/public network by using a bastion host. 

SSH Keys

Oracle Cloud Infrastructure Linux based offerings like compute instances and virtual machines for databases are accessible by SSH key as per default. For working with these machines, I use these three types of SSH keys:

  • id_rsa_oci – Private key generated by ssh-keygen
  • id_rsa_oci.pub – Public key generated by ssh-keygen
  • id_rsa_oci.ppk – Puttygen-converted private key

This gives me the flexibility, to connect to running OCI instances on different ways like Putty, MobaXterm, Windows Subsystem for Linux, WinSCP etc. 

Oracle Cloud Infrastructure Sample Setup

 

Host Public IP Private IP Accessible by
Bastion Host 140.238.216.114 10.0.0.2 SSH 
Windows Application Server   10.0.1.2 RDP
Oracle Database Server   10.0.2.2 SSH

 

Reminder: In OCI only SSH port 22 is open in the subnet security lists as per default when the VCN is created by the VCN Wizard. If you want to allow connection from the public to the private subnet by RDP and Oracle Net, then port 3389 and 1521 must be added in the security list for the private subnet. Create stateful ingress rules and restrict the source connections to the bastion host private IP range.

Build your own SSH Tunnel

There different ways to build a SSH (tunnel) configuration to Oracle Cloud Infrastructure instances on a Windows based platform, my favourites:

  1. Windows Subsystem for Linux (WSL)
  2. MobaXterm
  3. Putty

Here are some connection examples how to work with instances in a private subnet via bastion host with this three methods. As a Windows 10 user, for some connections I d’ like to use WSL Ubuntu more and more – now available in version 20 🙂

1. Connect by using Windows Subsystem for Linux (WSL)

Test: Verify the Connection to the Bastion Host public IP Address

Database Server: SSH Connect via Bastion Host

This opens a session on the database server as user opc.

Database Server: Create a new SSH Tunnel to forward port 1521 as port 15210

This opens a connection to the bastion host.

Database Server: Connect to the Database by SQL Developer

Use port 15210 and localhost as hostname.

Verify the Oracle Net service name from the DBA panel menu.

Application Server: Create a new SSH Tunnel to forward port 3389 as port 33890

This opens a connection to the bastion host.

Application Server: Connect to the Windows Desktop by Remote Desktop Connection

Use port 33890 and localhost as hostname.

2. Connect by using MobaXterm

Database Server: SSH Connect via Bastion Host

This opens a session ion the database server as user opc.

Fill in Remote Host, Specify username and Port. Activate Use private key and select the local private SSH key in Putty format.

Activate Connect through SSH gateway, fill in Gateway SSH server, Port, User. Activate Use private key and select the local private SSH key in Putty format.

Start the session.

As you can see in the MobaXterm Header, X-Forwarding works too.

Database Server: Create a new SSH Tunnel to forward port 1521 as port 15210

Open MobaXterm Tunneling menu and add a New SSH tunnel. Fill in Forwarded port, Remote server, Remote port, SSH server, SSH login and SSH port. Save the tunnel settings. For an application server tunnel, just replace Remote server, Remote port and Forwared port settings.

Add the private SSH key in Putty format by click on the key icon. Start the tunnel.

Database Server: Connect to the Database by SQL Developer

Use port 15210 and localhost as hostname.

Verify the database control file settings from the DBA panel menu.

3. Connect by using Putty

Database Server: SSH Connect via Bastion Host

As prerequisite, I have created a Putty session called OCI Bastion Host for the jump host connection with the SSH private key in Putty format and user opc. This session is now used as Proxy.

Fill in database server private IP. The red one is the already existing session.

Add proxy command and save session settings. Optioanl enable proxy diagnostics.

Open the new created session to connect to database server with user opc.

Application Server: Create a new SSH Tunnel to forward port 3389 as port 33890

This opens a connection to the bastion host. Fill in bastion host public IP. 

Add private key file in Putty format and enable checkbox Allow agent forwarding.

Add a port forwarding rule for RDP. Save session.

Open the new created session to enable port forwarding for Remote Desktop Protocol.

Application Server: Connect to the Windows Desktop by Remote Desktop Connection

Use port 33890 and localhost as hostname.

Alternative Method – Start Putty from Command Line

Start Putty with the port forwarding settings by command line. This opens a Putty session and port 3389 can be used. No addtional settings are required.

Summary

A bastion host is an “easy-to-setup” alternative to a VPN connection without any huge infrastructure overhead. There are several ways how to connect & tunnel to the target servers. Use the method which are you familiar with it, but NEVER place SSH keys on a bastion host. 

And now: click here to make some noise – Jump by Van Halen

Links

Oracle Cloud Infrastructure, Oracle Linux, Oracle RDBMS, TrivadisContent

Oracle Cloud Infrastructure – Stop your 20c Preview Database Instance Node by OCI CLI now – Great MOS Experience

Author: 31. March 2020

This was really great experience with the guys from My Oracle Support and Oracle teams. Seven days ago I raised an SR that there was no possibility to stop a 20c Preview database instance node if it’s not in use.

No chance to stop a 20c Preview Instance

There was no stop action in the Oracle Cloud Infrastructure UI and not in OCI CLI. The only way was to terminate an 20c preview instance. If I tried to stop a database node in the OCI CLI, this message occurs: Operation is not allowed for Preview Database version.

Service Request Update today

Today my SR was updated with a short message: “PIease retry through API and confirm.” Sure 🙂

After some seconds, the instance node was successfully stopped.

Let’s start it again.

Node is starting…

Node is available

Summary

Thanks My Oracle Support to implement this change!

Oracle Cloud Infrastructure

Oracle Cloud Infrastructure – Network Troubleshooting with VCN Flow Logs

Author: 5. March 2020

Do have a problem with a connection from or to your private/public subnet? There is a new functionality called VCN Flow Logs available. It collects information about network traffic (source/target) in the Oracle Cloud Infrastructure VCN subnet. At the moment (05/03/2020), this functionality is not available in all regions and I did not find any command in OCI CLI, but will be rolled out. There is no documentation available at  docs.cloud.oracle.com.

Link to the OCI blog announcement and demo: https://blogs.oracle.com/cloud-infrastructure/announcing-vcn-flow-logs-for-oracle-cloud-infrastructure

LA

I have registered our company tenant for the Cloud Native Limited Availability Program to get this brand new feature available. Watch here: https://blogs.oracle.com/cloud-infrastructure/announcing-limited-availability-of-oracle-cloud-infrastructure-logging-service

Use Case

A public compute instance with private IP 10.92.10.2 is not able to connect to the private database server with IP 10.92.100.2 anymore via SSH/22 – data center is Switzerland North (Zurich).

Create a new Log Group in your Compartment

Fill in name and description for the Log Group

The Log Group is created,  Enable Log

Enable Resource Log

Define the service and resource for VCN Flow Logs and enable logging. For the private subnet investigation I used:

  • Service: Flow Logs
  • Resource: My Private Subnet Name

Flow Log

The Flow is created, now we can explore the log. You can also disable logging or indexing or edit the name.

Log Search

Basically you see all log entries, with Explore with Log Search we can add filters. For example for a source IP address or a log content text like REJECTED.

Modify Filters & Columns

Now we add a filter to find out REJECTED connections. Wildcards are allowed in search terms.

  • Log Field: msg
  • Value: *REJECT* 

Apply.

Now we see the connections with state REJECT.

The solution – Add the IP to the Security List

There was a missing entry in the private subnet security list. After adding the source IP address range to the list, the connection is ok now. There are no REJECT message entries anymore in the VCN Flow Logs by this source IP address.

Object Storage

Flow logs are stored in Object Storage too. The bucket is created automatically. Housekeeping can be configured by a Lifecycle Rule for the log file bucket or by CLI. Take a look into the documentation to avoid error when you want to create a lifecycle rule . You have to create a Service Permissions policy first for the object storage before you can create a rule.

OCI Object Storage Lifecycle Rule

You can remove them by a lifecycle rule or by CLI. Take a look at the OCI documentation section Using Object Lifecycle Management to avoid permission errors when you want to create a lifecycle rule . You have to create a service permissions policy first for the object storage before you can create a rule.

Missing permissions error message:

Example Policy Statement to allow actions on object store:

OCI CLI example command to remove old files – for example with date pattern 2020-03-05T07 – 7AM

OCI Command Line Interface starter page: https://docs.cloud.oracle.com/en-us/iaas/Content/API/Concepts/cliconcepts.htm

What’s next

Try out the new logging feature for other OCI components like Functions, Event Service and Object Storage. And why not to integrate the logs in your existing Splunk environment? There is Splunk OCI object storage plugin available. Take a look here: https://blogs.oracle.com/cloud-infrastructure/announcing-the-object-storage-plugin-for-splunk

Autonomous Database, Oracle Cloud Infrastructure

Oracle Enterprise Manager 13c Release 4 – Time to Upgrade – First Experiences

Author: 3. February 2020

The roll-out of the newest Oracle Enterprise Manager 13 Release 4 is a few days old, about time to try out the upgrade process in my ESXi lab environment.

First: This blog post about the OEM upgrade process is based on my own experience.

Software

https://www.oracle.com/enterprise-manager/downloads/cloud-control-downloads.html

Documents

https://docs.oracle.com/en/enterprise-manager/cloud-control/enterprise-manager-cloud-control/13.4/emupg/index.html

The Environment

  • Oracle Enterprise Manager 13.3 running on Oracle Linux 7.6 – OEM Patchlevel January 2020 – Non-HA
  • Oracle Enterprise Edition Repository 19.6.0 Single Tenant Database 
  • Oracle Restart / ASM 19.6.0
  • Additional Oracle Linux Server with 19.3.0 Container Databases
  • All targets up and running
  • My Oracle Support connected
  • Software staged directory in /u01/app/oracle/stage/em13cr4 and permission changed to execute bin file (chmod u+x em13400_linux64.bin).

Prerequisites

It’s important to verify the changed prerequisites – Chapter 3 – Prerequisites for Upgrading to Enterprise Manager Cloud Control 13c Release 4 – I had do disable all adaptive features in the repository pluggable database:

Restart the database after the changed settings.From my view this sentence here is wrong is the documentation (my database has version 19.6.0):

If your Management Repository is using Oracle Database 12.2 or higher, none of these parameters need to be set. 

Why? If these parameters are not set, the installer refuses working.

EMKEY Copy Requirements

The Key was copied to the repository. It well be removed after the successful upgrade.

Run Installer

Start the Oracle Universal Installer

The OMS is down now (emctl stop oms -all). Start the installer.

My Oracle Support Details

I am a registered user in My Oracle Support, I get enough information every week… I don’t enable this checkbox.

Software Updates

At the moment, there are no updates available. Maybe in future there will be any patches available for auto apply during the installation/upgrade process.

Installation Type

Upgrade an existing Enterprise Manager system, this one is my existing installation.

 

Installation Details

Enter a new Middleware Home Location.

Database Connection Details

Enter the SYS and SYSMAN password of the running repository. DDMP (Deferred Data Migration) Jobs are enabled. According the documentation, these jobs are running in the background when the OEM is starting up during the upgrade process to convert old data to the new format. If you have a huge amount of data in your earlier release, then the upgrade can take longer. You can run this job – if disabled – later as Post Upgrade Tasks.

Warnings

I will change to SHA communication later.

The repository has 19.6.0, all required patches are included.

I confirm that the JVMD engine is stopped, this has happened by stopping the OMS.

I had to stop the agent which was running on the management server.

Fix Parameter Settings by the Installer

For my environment, I let the installer fix this settings.

Plug-in Upgrade

Here we see the already installed plugins, they will be upgraded too.

Select Plug-ins

I don’t ant to install additional plug-ins.

Extend WebLogic Server Domain

Enter the password for the weblogic user and define the OMS Instance Base Location.

Enterprise Manager Shared Location Details

My Oracle Enterprise Manager doesn’t run in a high availability setup, I don’t need any shared location.

Port Configuration Details

I use the settings from the existing installation.

Review

Let’s start the upgrade!

Repository Upgrade failed

The upgrade process fails at the step where the repository has to be upgraded.

View Log – ORA-01950

When I scroll up the provided log from the installer, I see this error message here:

The schema manager logfile located in the 13.4 subdirectory $ORACLE_HOME/sysman/log/schemamanager confirms this error.

At this point, this is curious, because in 13.3, there were no objects of the SYSMAN in the USERS tablespace. All objects are located in OEM tablespaces with the MGMT prefix. Here is a list of objects in a repository where an Oracle Enterprise Manager 13c Release 3 is up and running.

The solution is simple. Just grant the permissions for the USERS tablespace to SYSMAN and retry the installation progress.But, don’t ask me why SYSMAN creates now objects in USERS…

After a while, you can see new created tables and indexes in tablespace USERS for user SYSMAN.

The repository upgrade step runs fine now, the whole upgrade process continuous.

Finish

After while, yes we did it, the root script execution is the last task. 

Script execution.

The Enterprise Manager is now running with Release 4.

About Enterprise Manager

Next steps are

  • upgrade the Oracle agents
  • uninstall the old OEM software –  btw, the installer already detaches the old ORACLE_HOME from Oracle’s central inventory for you and removes the emkey from the repository 🙂

Summary

This is a lab environment, not a huge setup with hundreds of targets. But we can see here two important points for the upgrade process. a) read the manual and disable all adaptive features, even when you have an 19c database and b) user SYSMAN needs permission on the USERS tablespace. This is very unusual and should be corrected from my side. But now, enjoy Oracle Enterprise Manager 13c Release 4!

19c, EM13c Cloud Control, Oracle RDBMS, TrivadisContent