Oracle Cloud Infrastructure

Oracle Cloud Infrastructure – Security first: Cloud Guard and Security Zones – a first View

Author: Martin Berger 15. September 2020

Two new Oracle Cloud Infrastructure Cloud Security Services

Good news: Oracle has provided two new services for cloud security. Cloud Guard to get an overview of existing possible security breaches and Security Zones, which allows to create a full restricted compartment. In this blog, I will give you a short overview about this brand new services.

Cloud Guard

The Cloud Guard service helps you to identify security issues in your tenancy. Before the first use, it has to be enabled and a base region and for a minimum one compartment has to be selected. It needs a new policy which Cloud Guard allows to gather information in your tenancy. Oracle Cloud Guard discovers available object in compartments like Compute Instances, Object Storage and many more and checks Oracle Cloud Infrastructure security best practices.

Link to documentation: https://docs.cloud.oracle.com/en-us/iaas/cloud-guard/using/index.htm

Enable Cloud Guard first

Based on recipes, it show you security recommendations for the findings and can execute corrective actions. There are two different receipes types available:

Oracle Managed Detector Recipe – provided by Cloud Guard, doesn’t allow to disable rules
User Managed Detector Recipe – a clone of an Oracle managed recipe, allows to disable individual rules

Examples for recipes – docs.cloud.oracle.com

	Oracle Managed Recipe		User Managed Recipe
Rule	Status	Risk Level	Status	Risk Level
Bucket is public	ENABLED	HIGH	DISABLED	HIGH
Instance has public IP address	ENABLED	CRITICAL	ENABLED	HIGH
VCN has no inbound Security List	ENABLED	MEDIUM	DISABLED	MEDIUM

Based on detected findings, Cloud Guard is able to to corrective actions. This feature called Responder Rules requires a policy. Problems can be fixed on three ways:

Remediated – Fix using Cloud Guard responder
Resolved – Fixed by other process
Dismissed – Ignore and close

Example for a Cloud Guard Responder Action

Example – Cloud Guard has detected a Public IP

Cloud Guard Dashboard

The dashboard gives you an overview of the findings and actions. There are direct links to the findings and recommendations. Ok, It looks I have to review my test compartment 😉

Security Zones

A security zone is associated on a compartment and a security zone recipe. For example when in the recipe is defined, users cannot create an Internet Gateway in a defined compartement, an error message occurs when he tries to create one.

Link to documentation: https://docs.cloud.oracle.com/en-us/iaas/security-zone/using/security-zones.htm

Create a new Security Zone

Recipes

There are some basic rules in the Oracle defined recipe (at the moment you can not create a customer based recipe) – for example:

Resources can’t be moved out from a security zone to a regular compartment
Resources are not accessible by Internet
Resources must be regularly backed up

Test – Create an Internet Gateway in the new created Security Zone

A violation message occurs, the security zone recipe doesn’t allow creating Internet Gateways.

Summary

I really like these two new services. Cloud Guard which helps me to identify possible security issues and Security Zones to create secure compartments without writing manual policies. This is only a short overview, in next days I will definitely take a deeper look, especially in Cloud Guard and the corrective actions. I have a great interest to find out how it works in the background for example when a public IP is detected and so on. The Oracle Cloud Infrastructure security is definitely on track!

DBaaS, Oracle Cloud Infrastructure, Oracle RDBMS, Security, TrivadisContent

Oracle Cloud Infrastructure and SSH Keys – Jump!

Author: Martin Berger 27. April 2020

Jump!

In our Trivadis Oracle Cloud Infrastructure training environments, we never use direct access to an application or database server by a public IP address. For this case, we use an Oracle Linux based bastion host which acts as a jump host. For security reasons, I never put any SSH keys on a bastion host to connect from there to the target instances. If your bastion host is compromitted, your SSH keys are lost! In one of the last trainings, some participants had problems with. So I decided to blog about. This blog post shows you the different methods to connect to an Oracle Cloud Infrastructure private/public network by using a bastion host.

SSH Keys

Oracle Cloud Infrastructure Linux based offerings like compute instances and virtual machines for databases are accessible by SSH key as per default. For working with these machines, I use these three types of SSH keys:

id_rsa_oci – Private key generated by ssh-keygen
id_rsa_oci.pub – Public key generated by ssh-keygen
id_rsa_oci.ppk – Puttygen-converted private key

This gives me the flexibility, to connect to running OCI instances on different ways like Putty, MobaXterm, Windows Subsystem for Linux, WinSCP etc.

Oracle Cloud Infrastructure Sample Setup

Host	Public IP	Private IP	Accessible by
Bastion Host	140.238.216.114	10.0.0.2	SSH
Windows Application Server		10.0.1.2	RDP
Oracle Database Server		10.0.2.2	SSH

Reminder: In OCI only SSH port 22 is open in the subnet security lists as per default when the VCN is created by the VCN Wizard. If you want to allow connection from the public to the private subnet by RDP and Oracle Net, then port 3389 and 1521 must be added in the security list for the private subnet. Create stateful ingress rules and restrict the source connections to the bastion host private IP range.

Build your own SSH Tunnel

There different ways to build a SSH (tunnel) configuration to Oracle Cloud Infrastructure instances on a Windows based platform, my favourites:

Windows Subsystem for Linux (WSL)
MobaXterm
Putty

Here are some connection examples how to work with instances in a private subnet via bastion host with this three methods. As a Windows 10 user, for some connections I d’ like to use WSL Ubuntu more and more – now available in version 20 🙂

1. Connect by using Windows Subsystem for Linux (WSL)

Test: Verify the Connection to the Bastion Host public IP Address

$ ssh -i .ssh/id_rsa_oci opc@140.238.216.114
Last login: Mon Apr 27 15:47:54 2020 from 139.178.22.30
[opc@bastion-host ~]$ oci-metadata | grep hostname
hostname: bastion-host

Database Server: SSH Connect via Bastion Host

This opens a session on the database server as user opc.

$ ssh -i .ssh/id_rsa_oci -o ProxyCommand="ssh -i .ssh/id_rsa_oci -W %h:%p opc@140.238.216.114" opc@10.0.2.2
Last login: Mon Apr 27 15:51:32 2020 from 10.0.0.2
[opc@dbsrv01 ~]$ sudo su - oracle
Last login: Mon Apr 27 15:51:47 UTC 2020 on pts/0
[oracle@dbsrv01 ~]$ . oraenv
ORACLE_SID = [DB0427] ?
The Oracle base has been set to /u01/app/oracle
[oracle@dbsrv01 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Apr 27 15:52:14 2020
Version 19.6.0.0.0

Copyright (c) 1982, 2019, Oracle. All rights reserved.


Connected to:
Oracle Database 19c Standard Edition 2 Release 19.0.0.0.0 - Production
Version 19.6.0.0.0

SQL> show pdbs

CON_ID CON_NAME OPEN MODE RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED READ ONLY NO
3 PDB01 READ WRITE NO

Database Server: Create a new SSH Tunnel to forward port 1521 as port 15210

This opens a connection to the bastion host.

$ ssh -i .ssh/id_rsa_oci -A -L 15210:10.0.2.2:1521 opc@140.238.216.114
Last login: Mon Apr 27 15:48:23 2020 from 139.178.22.30
[opc@bastion-host ~]$

Database Server: Connect to the Database by SQL Developer

Use port 15210 and localhost as hostname.

Verify the Oracle Net service name from the DBA panel menu.

Application Server: Create a new SSH Tunnel to forward port 3389 as port 33890

This opens a connection to the bastion host.

$ ssh -i .ssh/id_rsa_oci -A -L 33890:10.0.1.2:3389 opc@140.238.216.114
Last login: Mon Apr 27 15:48:23 2020 from 139.178.22.30
[opc@bastion-host ~]$

Application Server: Connect to the Windows Desktop by Remote Desktop Connection

Use port 33890 and localhost as hostname.

2. Connect by using MobaXterm

Database Server: SSH Connect via Bastion Host

This opens a session ion the database server as user opc.

Fill in Remote Host, Specify username and Port. Activate Use private key and select the local private SSH key in Putty format.

Activate Connect through SSH gateway, fill in Gateway SSH server, Port, User. Activate Use private key and select the local private SSH key in Putty format.

Start the session.

As you can see in the MobaXterm Header, X-Forwarding works too.

Database Server: Create a new SSH Tunnel to forward port 1521 as port 15210

Open MobaXterm Tunneling menu and add a New SSH tunnel. Fill in Forwarded port, Remote server, Remote port, SSH server, SSH login and SSH port. Save the tunnel settings. For an application server tunnel, just replace Remote server, Remote port and Forwared port settings.

Add the private SSH key in Putty format by click on the key icon. Start the tunnel.

Database Server: Connect to the Database by SQL Developer

Use port 15210 and localhost as hostname.

Verify the database control file settings from the DBA panel menu.

3. Connect by using Putty

Database Server: SSH Connect via Bastion Host

As prerequisite, I have created a Putty session called OCI Bastion Host for the jump host connection with the SSH private key in Putty format and user opc. This session is now used as Proxy.

Fill in database server private IP. The red one is the already existing session.

Add proxy command and save session settings. Optioanl enable proxy diagnostics.

plink "OCI Bastion Host" -agent -nc %host:%port

Open the new created session to connect to database server with user opc.

Application Server: Create a new SSH Tunnel to forward port 3389 as port 33890

This opens a connection to the bastion host. Fill in bastion host public IP.

Add private key file in Putty format and enable checkbox Allow agent forwarding.

Add a port forwarding rule for RDP. Save session.

Open the new created session to enable port forwarding for Remote Desktop Protocol.

Application Server: Connect to the Windows Desktop by Remote Desktop Connection

Use port 33890 and localhost as hostname.

Alternative Method – Start Putty from Command Line

Start Putty with the port forwarding settings by command line. This opens a Putty session and port 3389 can be used. No addtional settings are required.

C:\> putty.exe -ssh -A -i C:\oci\ssh\id_rsa_oci.ppk -L 33890:10.0.1.2:3389 opc@140.238.216.114

Summary

A bastion host is an “easy-to-setup” alternative to a VPN connection without any huge infrastructure overhead. There are several ways how to connect & tunnel to the target servers. Use the method which are you familiar with it, but NEVER place SSH keys on a bastion host.

And now: click here to make some noise – Jump by Van Halen

Links

Oracle Cloud Infrastructure, Oracle Linux, Oracle RDBMS, TrivadisContent

Oracle Cloud Infrastructure – Stop your 20c Preview Database Instance Node by OCI CLI now – Great MOS Experience

Author: Martin Berger 31. March 2020

This was really great experience with the guys from My Oracle Support and Oracle teams. Seven days ago I raised an SR that there was no possibility to stop a 20c Preview database instance node if it’s not in use.

No chance to stop a 20c Preview Instance

There was no stop action in the Oracle Cloud Infrastructure UI and not in OCI CLI. The only way was to terminate an 20c preview instance. If I tried to stop a database node in the OCI CLI, this message occurs: Operation is not allowed for Preview Database version.

mbg@LTMBG03:~$ oci db node stop --db-node-id ocid1.dbnode.oc1.eu-zurich-1.ab5heljr6uogrcbvopqvw4ppihq3hl75nh6e1234567891234567
ServiceError:
{
"code": "NotAuthorizedOrNotFound",
"message": "Operation is not allowed for Preview Database version.",
"opc-request-id": "0627623907F948C6AB50645F4D5086B9/A5A12041A7E77DDD5551AF4146963663/2A14DAC00F90DC2A657DA47E7856F0F1",
"status": 404
}

Service Request Update today

Today my SR was updated with a short message: “PIease retry through API and confirm.” Sure 🙂

mbg@LTMBG03:~$ oci db node stop --db-node-id ocid1.dbnode.oc1.eu-zurich-1.ab5heljruj5qry2fs6l3xgk33cvolprvqs123456789123456789a
{
  "data": {
    "backup-vnic-id": null,
    "db-system-id": "ocid1.dbsystem.oc1.eu-zurich-1.ab5heljruj5qry2fs6l3xgk33cvolprvqs123456789123456789a",
    "fault-domain": "FAULT-DOMAIN-3",
    "hostname": "srvst20c01",
    "id": "ocid1.dbnode.oc1.eu-zurich-1.ab5heljrmxvsgn44lrphahrvkmcbnldukzkyw123456789123456789a",
    "lifecycle-state": "STOPPING",
    "software-storage-size-in-gb": 200,
    "time-created": "2020-02-17T08:56:22.751000+00:00",
    "vnic-id": "ocid1.vnic.oc1.eu-zurich-1.ab5heljr3niyga3b73pxitfplpo6uvmjtqqt2123456789123456789a"
  },
  "etag": "a2cfce62",
  "opc-work-request-id": "ocid1.coreservicesworkrequest.oc1.eu-zurich-1.ab5heljr7p5ez76oycfupmxwj4pvyhuekwicb5ketwa7avipq2cgpu5ipfpq,ocid1.coreservicesworkrequest.oc1.eu-zurich-1.ab5heljr7p5ez76oycfupmxwj4pvyhuekwicb5ketwa7avipq2cgpu5ipfpq"
}

After some seconds, the instance node was successfully stopped.

Let’s start it again.

mbg@LTMBG03:~$ oci db node start --db-node-id ocid1.dbnode.oc1.eu-zurich-1.ab5heljruj5qry2fs6l3xgk33cvolprvqs123456789123456789a
"data": {
"backup-vnic-id": null,
"db-system-id": "ocid1.dbsystem.oc1.eu-zurich-1.ab5heljruj5qry2fs6l3xgk33cvolprvqs123456789123456789a",
"fault-domain": "FAULT-DOMAIN-2",
"hostname": "srvst20c01",
"id": "ocid1.dbnode.oc1.eu-zurich-1.ab5heljrmxvsgn44lrphahrvkmcbnldukzkyw123456789123456789a",
"lifecycle-state": "STARTING",
"software-storage-size-in-gb": 200,
"time-created": "2020-02-17T08:22:36.336000+00:00",
"vnic-id": "ocid1.vnic.oc1.eu-zurich-1.ab5heljr3niyga3b73pxitfplpo6uvmjtqqt2123456789123456789a"
},
"etag": "69f24776",
"opc-work-request-id": "ocid1.coreservicesworkrequest.oc1.eu-zurich-1.ab5heljreak3amaiavzqgqek7zwbwnnlc47xbasmxart63putgdr7cqlo3ia"
}

Node is starting…

Node is available

Summary

Thanks My Oracle Support to implement this change!

Oracle Cloud Infrastructure

Oracle Cloud Infrastructure – Network Troubleshooting with VCN Flow Logs

Author: Martin Berger 5. March 2020

Do have a problem with a connection from or to your private/public subnet? There is a new functionality called VCN Flow Logs available. It collects information about network traffic (source/target) in the Oracle Cloud Infrastructure VCN subnet. At the moment (05/03/2020), this functionality is not available in all regions and I did not find any command in OCI CLI, but will be rolled out. There is no documentation available at docs.cloud.oracle.com.

Link to the OCI blog announcement and demo: https://blogs.oracle.com/cloud-infrastructure/announcing-vcn-flow-logs-for-oracle-cloud-infrastructure

LA

I have registered our company tenant for the Cloud Native Limited Availability Program to get this brand new feature available. Watch here: https://blogs.oracle.com/cloud-infrastructure/announcing-limited-availability-of-oracle-cloud-infrastructure-logging-service

Use Case

A public compute instance with private IP 10.92.10.2 is not able to connect to the private database server with IP 10.92.100.2 anymore via SSH/22 – data center is Switzerland North (Zurich).

Create a new Log Group in your Compartment

Fill in name and description for the Log Group

The Log Group is created, Enable Log

Enable Resource Log

Define the service and resource for VCN Flow Logs and enable logging. For the private subnet investigation I used:

Service: Flow Logs
Resource: My Private Subnet Name

Flow Log

The Flow is created, now we can explore the log. You can also disable logging or indexing or edit the name.

Log Search

Basically you see all log entries, with Explore with Log Search we can add filters. For example for a source IP address or a log content text like REJECTED.

Modify Filters & Columns

Now we add a filter to find out REJECTED connections. Wildcards are allowed in search terms.

Log Field: msg
Value: *REJECT*

Apply.

Now we see the connections with state REJECT.

The solution – Add the IP to the Security List

There was a missing entry in the private subnet security list. After adding the source IP address range to the list, the connection is ok now. There are no REJECT message entries anymore in the VCN Flow Logs by this source IP address.

Object Storage

Flow logs are stored in Object Storage too. The bucket is created automatically. Housekeeping can be configured by a Lifecycle Rule for the log file bucket or by CLI. Take a look into the documentation to avoid error when you want to create a lifecycle rule . You have to create a Service Permissions policy first for the object storage before you can create a rule.

OCI Object Storage Lifecycle Rule

You can remove them by a lifecycle rule or by CLI. Take a look at the OCI documentation section Using Object Lifecycle Management to avoid permission errors when you want to create a lifecycle rule . You have to create a service permissions policy first for the object storage before you can create a rule.

Missing permissions error message:

Example Policy Statement to allow actions on object store:

Allow service objectstorage-eu-zurich-1 to {BUCKET_INSPECT, BUCKET_READ, OBJECT_INSPECT, OBJECT_CREATE, OBJECT_DELETE} in compartment Compartment_Trivadis_MOHNWEG

OCI CLI example command to remove old files – for example with date pattern 2020-03-05T07 – 7AM

oci os object bulk-delete -ns trivadisbdsxsp -bn oci-logs._flowlogs.ocid1.compartment.oc1..aaaaaaaayc5kgqshdb5g2mjg4bnt34htnybbho3hx2exkz5pzi6kt4kunhiq --include *2020-03-05T07*

OCI Command Line Interface starter page: https://docs.cloud.oracle.com/en-us/iaas/Content/API/Concepts/cliconcepts.htm

What’s next

Try out the new logging feature for other OCI components like Functions, Event Service and Object Storage. And why not to integrate the logs in your existing Splunk environment? There is Splunk OCI object storage plugin available. Take a look here: https://blogs.oracle.com/cloud-infrastructure/announcing-the-object-storage-plugin-for-splunk

Autonomous Database, Oracle Cloud Infrastructure

Oracle Cloud Infrastructure – Change a Compute Instance Shape – Tested

Author: Martin Berger 16. January 2020

There is a new feature available in Oracle Cloud infrastructure since the 13th of January 2020, now you can change the shape of a Compute instance. It replaces all the manual steps like stopping the existing instance, create a new one, attach the block device etc. – this is a short summary how it works.

From the OCI Release Notes :

You can change the shape of a virtual machine (VM) instance without having to rebuild your instances or redeploy your applications. This lets you scale up your Compute resources for increased performance, or scale down to reduce cost.

My existing Compute Instance

The existing machine has the shape VM-Standard2.1 – one OCPU and 15GB of memory.

[opc@webserverpublic01 ~]$ cat /proc/cpuinfo | grep processor
processor : 0
processor : 1

Change the Shape

Actions – Change Shape

Select the new shape – for the test case I selected VM-Standard2.2 – press Change Shape.

On Compute instance level, you can verify the work request UpdateInstance.

In the details of the work request, the progress is visible.

After tree minutes, the machine runs with the new shape and is ready to use.

[opc@webserverpublic01 ~]$ cat /proc/cpuinfo | grep processor
processor : 0
processor : 1
processor : 2
processor : 3

Summary

Changing and existing Compute shape is a feature what I have waited for since the beginning of OCI, after a few minutes the instance is back again with the new shape. I hope Oracle will now implement it for DBaaS too in the next weeks.

Oracle Cloud Infrastructure, Oracle RDBMS, Trivadis Toolbox