Oracle Linux

Oracle Cloud Infrastructure and SSH Keys – Jump!

Jump!

In our Trivadis Oracle Cloud Infrastructure training environments, we never use direct access to an application or database server by a public IP address. For this case, we use an Oracle Linux based bastion host which acts as a jump host. For security reasons, I never put any SSH keys on a bastion host to connect from there to the target instances. If your bastion host is compromitted, your SSH keys are lost! In one of the last trainings, some participants had problems with. So I decided to blog about. This blog post shows you the different methods to connect to an Oracle Cloud Infrastructure private/public network by using a bastion host. 

SSH Keys

Oracle Cloud Infrastructure Linux based offerings like compute instances and virtual machines for databases are accessible by SSH key as per default. For working with these machines, I use these three types of SSH keys:

  • id_rsa_oci – Private key generated by ssh-keygen
  • id_rsa_oci.pub – Public key generated by ssh-keygen
  • id_rsa_oci.ppk – Puttygen-converted private key

This gives me the flexibility, to connect to running OCI instances on different ways like Putty, MobaXterm, Windows Subsystem for Linux, WinSCP etc. 

Oracle Cloud Infrastructure Sample Setup

 

Host Public IP Private IP Accessible by
Bastion Host 140.238.216.114 10.0.0.2 SSH 
Windows Application Server   10.0.1.2 RDP
Oracle Database Server   10.0.2.2 SSH

 

Reminder: In OCI only SSH port 22 is open in the subnet security lists as per default when the VCN is created by the VCN Wizard. If you want to allow connection from the public to the private subnet by RDP and Oracle Net, then port 3389 and 1521 must be added in the security list for the private subnet. Create stateful ingress rules and restrict the source connections to the bastion host private IP range.

Build your own SSH Tunnel

There different ways to build a SSH (tunnel) configuration to Oracle Cloud Infrastructure instances on a Windows based platform, my favourites:

  1. Windows Subsystem for Linux (WSL)
  2. MobaXterm
  3. Putty

Here are some connection examples how to work with instances in a private subnet via bastion host with this three methods. As a Windows 10 user, for some connections I d’ like to use WSL Ubuntu more and more – now available in version 20 ūüôā

1. Connect by using Windows Subsystem for Linux (WSL)

Test: Verify the Connection to the Bastion Host public IP Address

Database Server: SSH Connect via Bastion Host

This opens a session on the database server as user opc.

Database Server: Create a new SSH Tunnel to forward port 1521 as port 15210

This opens a connection to the bastion host.

Database Server: Connect to the Database by SQL Developer

Use port 15210 and localhost as hostname.

Verify the Oracle Net service name from the DBA panel menu.

Application Server: Create a new SSH Tunnel to forward port 3389 as port 33890

This opens a connection to the bastion host.

Application Server: Connect to the Windows Desktop by Remote Desktop Connection

Use port 33890 and localhost as hostname.

2. Connect by using MobaXterm

Database Server: SSH Connect via Bastion Host

This opens a session ion the database server as user opc.

Fill in Remote Host, Specify username and Port. Activate Use private key and select the local private SSH key in Putty format.

Activate Connect through SSH gateway, fill in Gateway SSH server, Port, User. Activate Use private key and select the local private SSH key in Putty format.

Start the session.

As you can see in the MobaXterm Header, X-Forwarding works too.

Database Server: Create a new SSH Tunnel to forward port 1521 as port 15210

Open MobaXterm Tunneling menu and add a New SSH tunnel. Fill in Forwarded port, Remote server, Remote port, SSH server, SSH login and SSH port. Save the tunnel settings. For an application server tunnel, just replace Remote server, Remote port and Forwared port settings.

Add the private SSH key in Putty format by click on the key icon. Start the tunnel.

Database Server: Connect to the Database by SQL Developer

Use port 15210 and localhost as hostname.

Verify the database control file settings from the DBA panel menu.

3. Connect by using Putty

Database Server: SSH Connect via Bastion Host

As prerequisite, I have created a Putty session called OCI Bastion Host for the jump host connection with the SSH private key in Putty format and user opc. This session is now used as Proxy.

Fill in database server private IP. The red one is the already existing session.

Add proxy command and save session settings. Optioanl enable proxy diagnostics.

Open the new created session to connect to database server with user opc.

Application Server: Create a new SSH Tunnel to forward port 3389 as port 33890

This opens a connection to the bastion host. Fill in bastion host public IP. 

Add private key file in Putty format and enable checkbox Allow agent forwarding.

Add a port forwarding rule for RDP. Save session.

Open the new created session to enable port forwarding for Remote Desktop Protocol.

Application Server: Connect to the Windows Desktop by Remote Desktop Connection

Use port 33890 and localhost as hostname.

Alternative Method – Start Putty from Command Line

Start Putty with the port forwarding settings by command line. This opens a Putty session and port 3389 can be used. No addtional settings are required.

Summary

A bastion host is an “easy-to-setup” alternative to a VPN connection without any huge infrastructure overhead. There are several ways how to connect & tunnel to the target servers. Use the method which are you familiar with it, but NEVER place SSH keys on a bastion host. 

And now: click here to make some noise – Jump by Van Halen

Links

OCI Database Backup Service Configuration – Avoid 401 Unauthorized Error

While I a preparing new exercises for an Oracle Cloud Infrastructure training, I ran into an issue while configuring the Oracle Database Backup Service for the Object Storage. The OCI backup module installer returns an error 401.

My Environment

  • Oracle Linux 7 Virtual Box Machine
  • Oracle 19c RDBMS

Backup Service Module Installation Error

The installation was done according the documenation https://docs.oracle.com/en/cloud/paas/db-backup-cloud/csdbb/oracle-database-cloud-backup-module-oci.html

Error Message – java.io.IOException: testConnection: 401 Unauthorized

What I have verified:

  • Private key format and permissions
  • OCIDs
  • FingerPrint

But all of them were correct. There is no My Oracle Support note available about this error together with Oracle Database Backup Service. But after some more investigation, I found this note here: EBSCloudBackup.pl Failed When Performing Database Tier Upload Task (Doc ID 2588278.1) – bingo! This note described exactly my case with the cloud backup. The machine time is wrong!

My actual Machine Time and Date

The timezone CEST is correct. But wait, here in Kestenholz at the famous Juras√ľdfuss / Switzerland, we have 14:38 local time. The virtual machine was 2 hours “in the future”. Let’s install the NTP service deamon.

NTP Installation and Configuration

Now the time is right:

OCI Backup Configuration – 2nd Try

Now the oci_installer.jar runs fine and the configuration will be created. Et voilà.

Lesson learned

Take care about time and date settings when you build virtual machines for testing purposes. And aways install a time service like NTP or chrony.

Monitor your Oracle Cloud Free Tier with Grafana on Oracle Linux 8

In a previous blog post I wrote about monitoring Oracle Cloud Infrastructure components by Grafana. In the meantime, we got the Oracle Cloud Free Tier. Here is an updated version.

This blog post shows you how to install and configure the Grafana plugin based on the Oracle blog entry https://blogs.oracle.com/cloudnative/data-source-grafana on an Oracle Enterprise Linux 8 server.

Steps to monitor the Oracle Cloud Free Tier by the OCI Grafana Plugin

  1. Install and configure the Oracle Cloud Infrastructure CLI – by download or by YUM install
  2. Configure Group, User and Policy in Oracle Cloud Infrastructure Console
  3. Install Grafana and the OCI Plugin
  4. Configure the Grafana DataSource
  5. Create a new Dashboard with OCI Metrics

Machine Requirements

The server needs access to the internet.

Install and configure the Oracle Cloud Infrastructure CLI

Link: https://docs.cloud.oracle.com/iaas/Content/API/SDKDocs/cliinstall.htm

In this step, the software will be installed an configured. The new created SSH public key has to be added in the OCI console for further actions.

As OS user root we create a new user for OCI actions. 

Login as user oci, execute the CLI download and installation script. Answer questions with Y / Enter to get the default installation.

Default values:

install directory /home/oci/lib/oracle-cli
executable directory /home/oci/bin
OCI scripts /home/oci/bin/oci-cli-scripts
optional CLI packages db
shell/tab completion Y
path to rc file /home/oci/.bashrc

 

After the successful CLI installation, you have to configure it.

Based on your OCI account, these information are required – let the config and key location on default values.

config location /home/oci/.oci/config
user OCID OCI > Identity > Users > [YOUR_USER] > OCID
tenancy OCID OCI > Administration > Tenancy Details > [YOUR_TENANCY] > OCID
region choose your region, e.g. eu-zurich-1
generate a new key pair Y -> only if you don’t have already created a key pair
key directory /home/oci/.oci
key name oci_api_key

 

Add the content of the public key file in the OCI console to your user which you want to work with.

Attention: Be sure that you add the public key to the user which you have used for the CLI configuration!

Test the CLI configuration – example to list all compartments in your tenant.

Alternative Method Oracle Linux 7 – YUM Repository

Thanks to Sergio Leunissen from Oracle for his input, the Python SDK and oci utilities are is available in the YUM repository too and ready to install. Take a look at his blog post to see how to work with the Python SDK and OCI metadata:

Configure Group, modify User and add a Policy in Oracle Cloud Infrastructure Web Interface

Group

Create a new OCI group called Grafana. OCI > Identity > Groups.

Modify User

Add the selected user to the group – for example this is my user.

Add a Policy

Create a new policy called GrafanaPolicy. OCI > Identity > Policies.

allow group grafana to read metrics in tenancy
allow group grafana to read compartments in tenancy

Install Grafana and the OCI Plugin

Link: https://grafana.com/grafana/download?platform=linux

Login as user root and install Grafana.

Enable auto start and start the Grafana server manually.

Enable port 3000 (Grafana default port in firewall – the port can be changed in /etc/grafana/grafana.ini) to provide web access to Grafana.

Install the Grafana Oracle Cloud Infrastructure oci-datasource plugin.

Verify the Grafana plugin directory with the installed plugin.

Grafana needs the configuration file and the SSH Key from the user oci. As user root, copy the files and set the ownership to OS user grafana.

Change the path to the key file in /usr/share/grafana/.oci/config.

# vi /usr/share/grafana/.oci/config

From:

To:

Create a new Dashboard based on OCI Metrics

Open your browser and log in into Grafana with [SERVERNAME]:3000. Username and password are admin/admin. You have to change your initial password imme diately.

Add data source

Select Oracle Cloud Infrastructure

Configure the Data Source

Fill in your tenancy OCI, region and set Environment = Local. Test the connection. For troubleshooting see Grafana logfile in directory /var/log/grafana. If your default region like ZRH / EU-ZURICH-1 is not listed, then you have to edit the a plugin file as described below. Otherweise no metrics are shown.

Example to use Grafana for the Datacenter eu-zurich-1:

Edit the file /var/lib/grafana/plugins/oci-datasource/dist/constants.js and add your missed region – restart Grafana.

Error message in the grafana.log when your region is not added in file content.js but you select the region as data source:

Create a new Dashboard and Add Query

Create a Query to visualize Data

In this dashboard example I used the region eu-zurich-1, my compartment, the namespace oci_autonomous_database and the metric CpuUtilization.

There are a lot of other metrics available like:

  • CurrentLogons
  • ExecutionCount
  • Sessions
  • StorageUtilization (in %)
  • etc.

Available Metrics

 Learn more about metrics and monitoring in the OCI documentation here:

Summary

The OCI Grafana plugin is a nice solution to visualize your Oracle Cloud Free Tier environment based on Open Source software. Take care, Grafana needs access to the OCI CLI SSH information for the Oracle Cloud Infrastructure connection.

OCI Compute Instances – Stop SSH Brute Force Attacks with fail2ban & UseDNS

Every day and night, the SSH login by key into my public accessible Oracle Cloud Infrastructure Linux Compute Instance was permitted for hours. And sometimes, when I had luck, it worked. For me it was not clear when it works and when not. But something has blocked me. The password authentification in the OCI Linux instance is basically disabled, the key is the only way to log in.

After some investigation on the OCI instance, I found a huge amount of login trials in the /var/log/secure file. These brute force attacks were locking me out!

There is a interesting OCI documentation available called Securing Compute with steps how to secure OCI compute cloud instances – and one of this recommendation is: install fail2ban.

https://docs.cloud.oracle.com/iaas/Content/Security/Reference/compute_security.htm

fail2ban

fail2ban is an open source tool which reads several types of logfiles and creates based on rules new entries in the firewall table to block remote connections. I has default filters for ssh, apache postfix and many more. From Wikipedia: 

Fail2Ban is an intrusion prevention software framework that protects computer servers from brute-force attacks.

Link: https://www.fail2ban.org/wiki/index.php/Main_Page

Installation

All steps have to be executed as user root. FYI: I wanted to be informed when a new IP was banned, therefore I have installed sendmail too.

Configuration

For my fail2ban configuration I have created a new file called jail.local and made my settings there.

jail.local

After 2 unsuccesful logins, the source IP will be banned for 86400 seconds. And if a new IP is added to the ban list, I get an email.

/etc/fail2ban/jail.d/00-firewalld.conf

For OL7 where firewalld is used, verify if the command firewallcmd-ipset is set in /etc/fail2ban/jail.d/00-firewalld.conf. If you use iptables, the command can be changed. Please read the documentation how to change the firewall.

Start fail2ban

Verification

Status Check

Status Check with details, there is already one IP listed.

After some minutes, more entries were recognized in the /var/log/secure log file and added.

firewall-cmd

A new rule is automatically added with the match set failban-sshd.

sshd Configuration

After the fail2ban installation, there were other entries left in  /var/log/secure.

After changing the parameter UseDNS to no in /etc/ssdh/sshd_config and a restart, these entries were history.

Summary

Never let a OCI compute cloud running with a public IP without to monitor login attemps! fail2ban is one step to get more security. It is easy to configure and it helps a lot. But you have to do the basic work like software updates, SSH key enabling etc. The Oracle documentation is a good base to start! My next step will be to install and configure WAZUH – I keep you up to date!

Links

https://www.oracle.com/technetwork/articles/servers-storage-admin/tips-harden-oracle-linux-1695888.html

https://fedoraproject.org/wiki/Fail2ban_with_FirewallD

Oracle Cloud Infrastructure Storage Software Appliance – Installation and Configuration

The Oracle Cloud Infrastructure Storage Software Appliance – also known as Oracle Storage Cloud Software Appliance OSCSA – acts as a gateway between classic storage and the Oracle Cloud Infrastructure Object Storage Service. The appliance can be installed on an on-premises Linux system or in an Oracle Compute Cloud machine and runs in a Docker container. It offers a local cache where clients can place their files before the OSCSA moves them into the Storage Service. The communication between a client with a filesystem to the OSCSA works with NFSv4, from the OSCSA to the Object Storage Service, Oracle is using their REST interface. Traffic from the OSCSA to and from the Oracle cloud can be encrypted and compressed.

In this blog post first I will show you how you can install and configure the OSCSA in an on-premises environment. In a second step I configure an on-premises database server which uses the Object Storage Service as Oracle RMAN backup location.

Key Features

  • Compression and Exncryption
  • File Versioning
  • End-to-end Data Integrity with Checksum Verification
  • Support for Data Archival (Oracle Storage Archive Class)
  • Pin files to the appliance cache for faster access

Where to get the OSCSA and more Information

Requirements

  • Two dual-core CPUs (4-core CPUs recommended)
  • Minimum memory requirements (based on the maximum number of files that can be uploaded to the appliance filesystem):
    • 16 GB for filesystems up to 1 million files
    • 32 GB for filesystems up to 5 million files
    • 64 GB for filesystems up to 10 million file
  • Minimum disk size required to install Docker: 10 GB
  • Oracle Linux 7 with UEK Release 4 or later
  • Docker 1.12.6
  • NFS version 4.0

The installation and configuration of the required Oracle Linux components OL7 with UEK4, Docker and NFS is very well described in the “Using Oracle…” guide. Please take a look in the guide, it’s straight forward. The OSCSA installer does not start when the requirements are not fullfilled. 

My Test Environment

OSCSA breitenbach.martinberger.local Oracle Linux 7.4 100GB Storage
Database Server zuchwil.martinberger.local Oracle Linux 7.4 Oracle RDBMS 12.1.0.2
Traditional Cloud Account cloud.oracle.com Zone EM2 Oracle Cloud Infrastructure Object Storage Classic

Firewall

Port 32771 (Appliance Web Interface) ,  32772 (NFS) and 32773 (REST) have to be opened on the appliance machine. If you don’t want to use these port numbers, you can set them during the installation process. Execute as user root:

File Content

All installation steps are executed as OS user root. The Oracle Storage Cloud Software Appliance Software Release 16.3.1.3 is available on my local machine in folder /stage. The extracted OSCSA file contains a file called OSCSA_GATEWAY_README.txt where you can get more information about the installation and configuration possibilities like proxy etc.

Installation

The installation starts by executing oscsa-install.sh. I have added the parameter -a = advanced so I am able to set ports for NFS, Administrative Web Interface and REST. Oracle recommends for the cache storage a minimal size. I have ignored that for my test environment.

Appliance Start

Now the Oracle Storage Cloud Software Appliance can be started. A server reboot is not problem. The docker image will be started after server startup automatically.

 

Configure a OSCSA FileSystem in the Appliance Web Interface

In this step, the OSCSA will be connected to the Oracle Cloud. At the moment, no <OSCSA FileSystem name> is configured. An OSCSA  filesystem is like a namespace containing a set of data. Now we can log in into the Appliance Web Interface to create our first Object Storage filesystem. URL for the interface is https://<servername>:<port>. The port for the interface was set during the installation process.  

For the connection to the cloud, you need to know your Identity Domain, Username, Password and REST Storage Endpoint URL. The FileSystem name will be reused later for the NFS mount.

Create a FileSystem called OCIClassicStorage01

Enter Domain, Username, Password and REST Storage Endpoint URL. Below this screenshot you can see where you find the URL in your Traditional Cloud Account.

Here you can see the REST URL.

Click on Validate

If the account informations are verified, you are able to enable compression and encryption. I have enable encryption here. Click Save.

The OSCSA storage is now ready to synchronize with the cloud. Click Connect.

Now you can see the the connection between the Oracle Storage Cloud Software Appliance and the Oracle Cloud Infrastructure Object Storage Service is ready.

In the Traditonal Cloud Account in the Storage Classic dashboard is a new object storage filesystem available.

Connect the Database Server to the Oracle Storage Cloud Software Appliance

Let’s connect the database server to the OSCSA to store data in the Object Storage Service. First we check again on the OSCSA server if the service is running.

On the database server a new mountpoint will be created.

We mount the OSCSA with NFS v4 to the local server. This entry can be added later to autofs or whatever you use to automatically mount an NFS filesystem. Permission of the mountpoint is drwxrwxrwx – so everybody can write into it at the moment.

Check.

Execute an Oracle RMAN Backup to the Cloud

A new subdirectory on the NFS mountpoint will be created.

Start Oracle Recovery Manager RMAN database backup.

The backup files are created locally in the specified mountpoint directory.

At the moment where the backup sets are arrived on the mountpoint, the OSCSA begins to encrypt  (this was my selections during filesystem creation) and transfer them into the Oracle Cloud. This is visible in the Appliance Web Interface.

Now the files are uploaded into the Oracle Cloud. This can be verified in the Traditional Cloud Account in the specific filesystem. The files are encrypted and have file names like 10101-v1, 10103-v1 etc.

 

Anything else?

Sure, this was just a basic overview how to configure the on-premises Oracle Storage Cloud Software Appliance. There are many more features like retrieve data, cloud access via command line, preserve filesystem cache, create directory permissions, set user permissons, monitor the appliance, backup the appliance, encryption key handling, use the archive storage and so on which are worth to spend more time for investigation in the future.

Summary

The Oracle Storage Cloud Software Appliance is a nice piece of software which helps you to use the Oracle Cloud Infrastructure Object Storage Service. The appliance is easy to install and configure, local encryption is possible and the documentation is very good. Database Backups and Database Export a perfect candidates for this service. The price is hot, $0.0204 per GB for the first TB, and $0.0201 per GB for the next 49TB.

Thumbs up!