Oracle Cloud

The Grafana Plugins for Oracle Cloud Infrastructure Monitoring are back!

In September 2019 I wrote a blog post how to monitor an Oracle Cloud Infrastructure Autonomous database with Grafana plugin oci-datasource. But some weeks after publication, the plugin was not available on the Grafana page anymore. And only Oracle and Grafana had a clue why.

Now everything will be fine now. Since the 6th of October, there are two new Grafana plugins available for download. They both don’t require a Grafana enterprise account.

The first one is a successor of the former oci-datasource plugin, the second allows to get logs from OCI resources like Compute or Storage. As an infrastructure guy, let’s install the Oracle Cloud Infrastructure Metrics on an local Oracle Enterprise Linux 8 installation!

Install and configure the OCI CLI

Link: https://docs.cloud.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm

OS User oci and Installer

As OS user root, create a new user mentioned oci, change to new created user oci.

Run the installer script.

In this demo case, I use the default settings and the tab completion. After some seconds, all packages are installed and the OCI CLI is ready to configure.

Configure the OCI CLI

If you have already a created SSH key pair from a former OCI action, then you can use it here. Otherwise this setup process creates a new private and public key for you. Take care, the public key has to be in the PEM format!

Required values to finish the setup:

config location /home/oci/.oci/config
user OCID OCI > Identity > Users > [YOUR_USER] > OCID
tenancy OCID OCI > Administration > Tenancy Details > [YOUR_TENANCY] > OCID
region choose your region, e.g. eu-zurich-1
generate a new API signing RSA key pair Y -> only if you don’t have already an existing key pair
key directory /home/oci/.oci
key name oci_api_key_07102020

 

Run the setup.

OCI Console API Key

The content of the created public key has to be added in OCI Console as API key – just copy and paste it. OCI Console >> Identity >> Users >> User Details >> API Keys >> Add Public Key.

How to: https://docs.cloud.oracle.com/Content/API/Concepts/apisigningkey.htm#How2

OCI CLI Configuration Test

Verify the configuration by execute a CLI command. Example to list images based on Oracle Linux.

OCI Console Group Policy

If your user is not part of the Administrator group, a new group and a group policy is needed which has the permissions to read tenant metrics. OCI Console >> Identity >> Groups >> Create Group.

Create the policy in the root compartment of your tenant. OCI Console >> Identity >> Policy>> Create Policy.

Install and configure Grafana and the Oracle Cloud Infrastructure Metrics Data Source Plugin

Grafana

Link: https://docs.cloud.oracle.com/en-us/iaas/Content/API/SDKDocs/grafana.htm

Start and enable the service.

Don’t forget to open the firewall port 3000 for the Grafana UI.

Oracle Cloud Infrastructure Metrics Data Source Plugin

List the available OCI Grafana plugins.


Install the metric plugin.

Restart Grafana Server.

Grafana Data Source Configuration

RSA Key Configuration

Grafana needs the configuration file and the RSA Key from the user oci. One solution: as user root, copy the files and set the ownership to OS user grafana.

Change the path to the key file in /usr/share/grafana/.oci/config.

from:

to:

Add a new Data Source

Login into the Grafana server by address <server-ip>:3000. The initial username and password is admin. It’s recommended to change the password at the first login. Add a new data source. Configuration >> Data Sources >> Add data source.

Filter by oracle and select the Oracle Cloud Infrastructure Metrics plugin.

Set Tenancy OCID, select your Default Region and set the Environment to local. Press Save & Test to verify the functionality.

Create a new Dashboard and add a new panel.

Now you can query the data, for example the VPN bandwidth for region eu-zurich1 in my personal compartment. Feel free to add new panels based on the available metrics.

Example

Summary

Great to have the Oracle Cloud Infrastructure Grafana plugins back. To get an idea, which metrics are all available, verify it in the OCI Console >> Monitoring >> Metrics Explorer. The free ADB is not available in the collected metrics. But this is a general issue.

This was a review of the first OCI plugin. In the next week I will take a deeper look into the Oracle Cloud Infrastructure Logging Data Source plugin.

Monitor your Oracle Cloud Free Tier with Grafana on Oracle Linux 8

In a previous blog post I wrote about monitoring Oracle Cloud Infrastructure components by Grafana. In the meantime, we got the Oracle Cloud Free Tier. Here is an updated version.

This blog post shows you how to install and configure the Grafana plugin based on the Oracle blog entry https://blogs.oracle.com/cloudnative/data-source-grafana on an Oracle Enterprise Linux 8 server.

Steps to monitor the Oracle Cloud Free Tier by the OCI Grafana Plugin

  1. Install and configure the Oracle Cloud Infrastructure CLI – by download or by YUM install
  2. Configure Group, User and Policy in Oracle Cloud Infrastructure Console
  3. Install Grafana and the OCI Plugin
  4. Configure the Grafana DataSource
  5. Create a new Dashboard with OCI Metrics

Machine Requirements

The server needs access to the internet.

Install and configure the Oracle Cloud Infrastructure CLI

Link: https://docs.cloud.oracle.com/iaas/Content/API/SDKDocs/cliinstall.htm

In this step, the software will be installed an configured. The new created SSH public key has to be added in the OCI console for further actions.

As OS user root we create a new user for OCI actions. 

Login as user oci, execute the CLI download and installation script. Answer questions with Y / Enter to get the default installation.

Default values:

install directory /home/oci/lib/oracle-cli
executable directory /home/oci/bin
OCI scripts /home/oci/bin/oci-cli-scripts
optional CLI packages db
shell/tab completion Y
path to rc file /home/oci/.bashrc

 

After the successful CLI installation, you have to configure it.

Based on your OCI account, these information are required – let the config and key location on default values.

config location /home/oci/.oci/config
user OCID OCI > Identity > Users > [YOUR_USER] > OCID
tenancy OCID OCI > Administration > Tenancy Details > [YOUR_TENANCY] > OCID
region choose your region, e.g. eu-zurich-1
generate a new key pair Y -> only if you don’t have already created a key pair
key directory /home/oci/.oci
key name oci_api_key

 

Add the content of the public key file in the OCI console to your user which you want to work with.

Attention: Be sure that you add the public key to the user which you have used for the CLI configuration!

Test the CLI configuration – example to list all compartments in your tenant.

Alternative Method Oracle Linux 7 – YUM Repository

Thanks to Sergio Leunissen from Oracle for his input, the Python SDK and oci utilities are is available in the YUM repository too and ready to install. Take a look at his blog post to see how to work with the Python SDK and OCI metadata:

Configure Group, modify User and add a Policy in Oracle Cloud Infrastructure Web Interface

Group

Create a new OCI group called Grafana. OCI > Identity > Groups.

Modify User

Add the selected user to the group – for example this is my user.

Add a Policy

Create a new policy called GrafanaPolicy. OCI > Identity > Policies.

allow group grafana to read metrics in tenancy
allow group grafana to read compartments in tenancy

Install Grafana and the OCI Plugin

Link: https://grafana.com/grafana/download?platform=linux

Login as user root and install Grafana.

Enable auto start and start the Grafana server manually.

Enable port 3000 (Grafana default port in firewall – the port can be changed in /etc/grafana/grafana.ini) to provide web access to Grafana.

Install the Grafana Oracle Cloud Infrastructure oci-datasource plugin.

Verify the Grafana plugin directory with the installed plugin.

Grafana needs the configuration file and the SSH Key from the user oci. As user root, copy the files and set the ownership to OS user grafana.

Change the path to the key file in /usr/share/grafana/.oci/config.

# vi /usr/share/grafana/.oci/config

From:

To:

Create a new Dashboard based on OCI Metrics

Open your browser and log in into Grafana with [SERVERNAME]:3000. Username and password are admin/admin. You have to change your initial password imme diately.

Add data source

Select Oracle Cloud Infrastructure

Configure the Data Source

Fill in your tenancy OCI, region and set Environment = Local. Test the connection. For troubleshooting see Grafana logfile in directory /var/log/grafana. If your default region like ZRH / EU-ZURICH-1 is not listed, then you have to edit the a plugin file as described below. Otherweise no metrics are shown.

Example to use Grafana for the Datacenter eu-zurich-1:

Edit the file /var/lib/grafana/plugins/oci-datasource/dist/constants.js and add your missed region – restart Grafana.

Error message in the grafana.log when your region is not added in file content.js but you select the region as data source:

Create a new Dashboard and Add Query

Create a Query to visualize Data

In this dashboard example I used the region eu-zurich-1, my compartment, the namespace oci_autonomous_database and the metric CpuUtilization.

There are a lot of other metrics available like:

  • CurrentLogons
  • ExecutionCount
  • Sessions
  • StorageUtilization (in %)
  • etc.

Available Metrics

 Learn more about metrics and monitoring in the OCI documentation here:

Summary

The OCI Grafana plugin is a nice solution to visualize your Oracle Cloud Free Tier environment based on Open Source software. Take care, Grafana needs access to the OCI CLI SSH information for the Oracle Cloud Infrastructure connection.

Oracle Database Backup Service – Encrypt your 12.2 Database Backups to the Cloud

The Oracle RMAN backup encryption is necessary if you want to backup your database into the Oracle cloud. In Oracle 12c, you have three methods available to encrypt an Oracle RMAN backup:

  • with a passphrase
  • with a master encryption key
  • hybrid with a passphrase and an encryption key

On docs.oracle.com, the basic setup is described here: https://docs.oracle.com/en/cloud/paas/db-backup-cloud/csdbb/configuring-encryption-backups.html#GUID-4A1F5CF5-7EAF-4D71-9B7F-B46412F552CE

In this blog post, I show you how to configure your database environment with a master encryption key and a keystore. I use this solution to to backup and recovery to and into the Oracle cloud. And in the cloud, I don’t like to type in passwords manually for every action or write passwords in backup and restore scripts.

There are also some issues reports like in My Oracle Support Note TDE Wallet Problem in 12c: Cannot do a Set Key operation when an auto-login wallet is present (Doc ID 1944507.1).

Here are steps to create an autologin wallet.

Configure SQLNET.ora in $TNS_ADMIN to use a Keystore

Create Keystore as SYSDBA

Open Keystore

The status is set to OPEN_NO_MASTER_KEY.

Set Master Key

Now the master key has to defined. When you have already defined a wallet earlier and deleted the keys,  you have to set the undocumented parameter to set the master key again. This works here too to set the key. Otherwise you get an ORA-28374: typed master key not found in wallet error. See Master Note For Transparent Data Encryption ( TDE ) (Doc ID 1228046.1) for further information.

Now the status is set to OPEN.

Activate Auto Login

Restart the Database

Verify if the keystore is available and WALLET_TYPE is AUTOLOGIN.

Configure RMAN for Encryption

RMAN Backup Test

A simple RMAN controlfile backup into the Oracle cloud (OPC Backup Module is already configured).

Error message if you want to backup into the Oracle cloud and the encryption is not configured correctly:

Backup Verification in V$BACKUP_PIECE – Column ENCRYPTED

Links

http://www.oracle.com/technetwork/database/security/twp-transparent-data-encryption-bes-130696.pdf

http://www.oracle.com/technetwork/database/security/index-095354.html

Get your Oracle 18c Instance in the Oracle Infrastructure Cloud OCI Classic

Do you want to work with Oracle 18c in the Oracle Cloud but the database version is not selectable in the webinterface? You can create an 18c instance in the command-line interface with the PaaS Service Manager (psm). The installation is very well described here, for example you need Python and OpenSSL. My personal installation of the psm executable runs in the Windows 10 integrated Ubuntu system.

Link to the PaaS Service Manager: https://docs.oracle.com/en/cloud/paas/java-cloud/pscli/abouit-paas-service-manager-command-line-interface.html

After the successful psm setup, you can create an DBaaS instance with this command

The file db18c-ee.json contains all the information you need to create an 18c instance. Here is my example – I have created a cloud storage container called dbcsbackup in advance because I want to use the OCI backup service.

Some minutes later you can login by terminal and user SQL*Plus. Oracle 18c: Here we are!

And in the OCI dashboard it looks fine too.

Addtional Info: If you want a Standard Edition, just replace the line “edition”: “SE”, happy 18c.

 

Oracle Cloud Infrastructure Storage Software Appliance – Installation and Configuration

The Oracle Cloud Infrastructure Storage Software Appliance – also known as Oracle Storage Cloud Software Appliance OSCSA – acts as a gateway between classic storage and the Oracle Cloud Infrastructure Object Storage Service. The appliance can be installed on an on-premises Linux system or in an Oracle Compute Cloud machine and runs in a Docker container. It offers a local cache where clients can place their files before the OSCSA moves them into the Storage Service. The communication between a client with a filesystem to the OSCSA works with NFSv4, from the OSCSA to the Object Storage Service, Oracle is using their REST interface. Traffic from the OSCSA to and from the Oracle cloud can be encrypted and compressed.

In this blog post first I will show you how you can install and configure the OSCSA in an on-premises environment. In a second step I configure an on-premises database server which uses the Object Storage Service as Oracle RMAN backup location.

Key Features

  • Compression and Exncryption
  • File Versioning
  • End-to-end Data Integrity with Checksum Verification
  • Support for Data Archival (Oracle Storage Archive Class)
  • Pin files to the appliance cache for faster access

Where to get the OSCSA and more Information

Requirements

  • Two dual-core CPUs (4-core CPUs recommended)
  • Minimum memory requirements (based on the maximum number of files that can be uploaded to the appliance filesystem):
    • 16 GB for filesystems up to 1 million files
    • 32 GB for filesystems up to 5 million files
    • 64 GB for filesystems up to 10 million file
  • Minimum disk size required to install Docker: 10 GB
  • Oracle Linux 7 with UEK Release 4 or later
  • Docker 1.12.6
  • NFS version 4.0

The installation and configuration of the required Oracle Linux components OL7 with UEK4, Docker and NFS is very well described in the “Using Oracle…” guide. Please take a look in the guide, it’s straight forward. The OSCSA installer does not start when the requirements are not fullfilled. 

My Test Environment

OSCSA breitenbach.martinberger.local Oracle Linux 7.4 100GB Storage
Database Server zuchwil.martinberger.local Oracle Linux 7.4 Oracle RDBMS 12.1.0.2
Traditional Cloud Account cloud.oracle.com Zone EM2 Oracle Cloud Infrastructure Object Storage Classic

Firewall

Port 32771 (Appliance Web Interface) ,  32772 (NFS) and 32773 (REST) have to be opened on the appliance machine. If you don’t want to use these port numbers, you can set them during the installation process. Execute as user root:

File Content

All installation steps are executed as OS user root. The Oracle Storage Cloud Software Appliance Software Release 16.3.1.3 is available on my local machine in folder /stage. The extracted OSCSA file contains a file called OSCSA_GATEWAY_README.txt where you can get more information about the installation and configuration possibilities like proxy etc.

Installation

The installation starts by executing oscsa-install.sh. I have added the parameter -a = advanced so I am able to set ports for NFS, Administrative Web Interface and REST. Oracle recommends for the cache storage a minimal size. I have ignored that for my test environment.

Appliance Start

Now the Oracle Storage Cloud Software Appliance can be started. A server reboot is not problem. The docker image will be started after server startup automatically.

 

Configure a OSCSA FileSystem in the Appliance Web Interface

In this step, the OSCSA will be connected to the Oracle Cloud. At the moment, no <OSCSA FileSystem name> is configured. An OSCSA  filesystem is like a namespace containing a set of data. Now we can log in into the Appliance Web Interface to create our first Object Storage filesystem. URL for the interface is https://<servername>:<port>. The port for the interface was set during the installation process.  

For the connection to the cloud, you need to know your Identity Domain, Username, Password and REST Storage Endpoint URL. The FileSystem name will be reused later for the NFS mount.

Create a FileSystem called OCIClassicStorage01

Enter Domain, Username, Password and REST Storage Endpoint URL. Below this screenshot you can see where you find the URL in your Traditional Cloud Account.

Here you can see the REST URL.

Click on Validate

If the account informations are verified, you are able to enable compression and encryption. I have enable encryption here. Click Save.

The OSCSA storage is now ready to synchronize with the cloud. Click Connect.

Now you can see the the connection between the Oracle Storage Cloud Software Appliance and the Oracle Cloud Infrastructure Object Storage Service is ready.

In the Traditonal Cloud Account in the Storage Classic dashboard is a new object storage filesystem available.

Connect the Database Server to the Oracle Storage Cloud Software Appliance

Let’s connect the database server to the OSCSA to store data in the Object Storage Service. First we check again on the OSCSA server if the service is running.

On the database server a new mountpoint will be created.

We mount the OSCSA with NFS v4 to the local server. This entry can be added later to autofs or whatever you use to automatically mount an NFS filesystem. Permission of the mountpoint is drwxrwxrwx – so everybody can write into it at the moment.

Check.

Execute an Oracle RMAN Backup to the Cloud

A new subdirectory on the NFS mountpoint will be created.

Start Oracle Recovery Manager RMAN database backup.

The backup files are created locally in the specified mountpoint directory.

At the moment where the backup sets are arrived on the mountpoint, the OSCSA begins to encrypt  (this was my selections during filesystem creation) and transfer them into the Oracle Cloud. This is visible in the Appliance Web Interface.

Now the files are uploaded into the Oracle Cloud. This can be verified in the Traditional Cloud Account in the specific filesystem. The files are encrypted and have file names like 10101-v1, 10103-v1 etc.

 

Anything else?

Sure, this was just a basic overview how to configure the on-premises Oracle Storage Cloud Software Appliance. There are many more features like retrieve data, cloud access via command line, preserve filesystem cache, create directory permissions, set user permissons, monitor the appliance, backup the appliance, encryption key handling, use the archive storage and so on which are worth to spend more time for investigation in the future.

Summary

The Oracle Storage Cloud Software Appliance is a nice piece of software which helps you to use the Oracle Cloud Infrastructure Object Storage Service. The appliance is easy to install and configure, local encryption is possible and the documentation is very good. Database Backups and Database Export a perfect candidates for this service. The price is hot, $0.0204 per GB for the first TB, and $0.0201 per GB for the next 49TB.

Thumbs up!