Lost Disk after DBaaS Restart via Cloud Service Console

I have created a DBaaS instance three weeks ago, and now it was time to attach new storage in the Oracle Compute Cloud Service  to have additional disk space for some Trivadis tools. But after a reboot with a mouse-click in the DBaaS Cloud Service Console, the disk was lost and had to be re-attached.

The procedure to add additional storage is documented in this Oracle tutorial: http://www.oracle.com/webfolder/technetwork/tutorials/obe/cloud/dbaas/OU/IntroDBaaS/AddStorageDBaaSInstance/AddStorageDBaaSInstance.html

Creating additional  Storage in the Oracle Compute Cloud Service

In the Create Storage Volume menu, I have created a new storage volume of 10GB and attached it to my running DBaaS instance called TrivadisExPerf.

attach01

OS Verfication

After some minutes waiting, I logged in on the database server to verify the operation. The disk was attached as device xvdg.

-bash-4.1# lsblk
NAME                          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvdb                          202:16   0   21G  0 disk
├─xvdb1                       202:17   0  200M  0 part /boot
├─xvdb2                       202:18   0 15.8G  0 part /
└─xvdb3                       202:19   0    4G  0 part [SWAP]
xvdc                          202:32   0   30G  0 disk
└─xvdc1                       202:33   0   30G  0 part /u01
xvdd                          202:48   0   42G  0 disk
└─xvdd1                       202:49   0   42G  0 part
  └─fraVolGroup-lvol0 (dm-2)  252:2    0   42G  0 lvm  /u03
xvde                          202:64   0   10G  0 disk
└─xvde1                       202:65   0   10G  0 part
  └─redoVolGroup-lvol0 (dm-1) 252:1    0   10G  0 lvm  /u04
xvdf                          202:80   0   25G  0 disk
└─xvdf1                       202:81   0   25G  0 part
  └─dataVolGroup-lvol0 (dm-0) 252:0    0   25G  0 lvm  /u02
xvdg                          202:96   0   10G  0 disk

Creating a Filesystem

Then I created a new partition with fdisk, formatted the partion as Ext4 and mounted it as /u05. At this point, I did not add something to /etc/fstab. But this had no impact for later actions.

-bash-4.1# df -m
Filesystem           1M-blocks      Used Available Use% Mounted on
/dev/xvdb2               15931      8235      6887  55% /
tmpfs                     7538      3945      3593  53% /dev/shm
/dev/xvdb1                 194        91        94  49% /boot
/dev/xvdc1               30237     10755     17947  38% /u01
/dev/mapper/dataVolGroup-lvol0
                         25195     11697     12218  49% /u02
/dev/mapper/fraVolGroup-lvol0
                         42330     22682     17498  57% /u03
/dev/mapper/redoVolGroup-lvol0
                         10076      3223      6341  34% /u04
/dev/xvdg1               10077       151      9414   2% /u05

Restart via DBaaS Cloud Service Console

The DBaaS restart was executed by the Service Console. In the background the DBaaS service was restarted automatically. The process is visible in the console and you can see the progress. I don’t know why Oracle requires more than 10 minutes for a simple restart, in some cases more than 20 minutes…

attach02

The Storage is lost…

After the restart I logged in again in the terminal. And what do I see? The storage was no more attached at the instance. There was no /dev/xdg or /dev/xdg1 visible.

-bash-4.1# lsblk
NAME                          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvdb                          202:16   0   21G  0 disk
├─xvdb1                       202:17   0  200M  0 part /boot
├─xvdb2                       202:18   0 15.8G  0 part /
└─xvdb3                       202:19   0    4G  0 part [SWAP]
xvdc                          202:32   0   30G  0 disk
└─xvdc1                       202:33   0   30G  0 part /u01
xvdd                          202:48   0   42G  0 disk
└─xvdd1                       202:49   0   42G  0 part
  └─fraVolGroup-lvol0 (dm-2)  252:2    0   42G  0 lvm  /u03
xvde                          202:64   0   10G  0 disk
└─xvde1                       202:65   0   10G  0 part
  └─redoVolGroup-lvol0 (dm-1) 252:1    0   10G  0 lvm  /u04
xvdf                          202:80   0   25G  0 disk
└─xvdf1                       202:81   0   25G  0 part
  └─dataVolGroup-lvol0 (dm-0) 252:0    0   25G  0 lvm  /u02

After re-attaching the storage to the instance in the Oracle Compute Cloud Storage menu, the storage was back again. And I tried another way to restart a server…

attach03

Restart in the Terminal

First I checked again if the storage was attached. The device /dev/xdg was here.

-bash-4.1# lsblk
NAME                          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvdb                          202:16   0   21G  0 disk
├─xvdb1                       202:17   0  200M  0 part /boot
├─xvdb2                       202:18   0 15.8G  0 part /
└─xvdb3                       202:19   0    4G  0 part [SWAP]
xvdc                          202:32   0   30G  0 disk
└─xvdc1                       202:33   0   30G  0 part /u01
xvdd                          202:48   0   42G  0 disk
└─xvdd1                       202:49   0   42G  0 part
  └─fraVolGroup-lvol0 (dm-2)  252:2    0   42G  0 lvm  /u03
xvde                          202:64   0   10G  0 disk
└─xvde1                       202:65   0   10G  0 part
  └─redoVolGroup-lvol0 (dm-1) 252:1    0   10G  0 lvm  /u04
xvdf                          202:80   0   25G  0 disk
└─xvdf1                       202:81   0   25G  0 part
  └─dataVolGroup-lvol0 (dm-0) 252:0    0   25G  0 lvm  /u02
xvdg                          202:96   0   10G  0 disk
└─xvdg1                       202:97   0   10G  0 part

Then I restarted the server with the shutdown command.

-bash-4.1# shutdown -r now

Broadcast message from opc@TrivadisExPerf
        (/dev/pts/0) at 20:48 ...

The system is going down for reboot NOW!

And after the restart, the storage was still here. The device /dev/xdg was available. And it didn’t require 10 mins, after 2 mins the server was back again and ready to log-in.

-bash-4.1# lsblk
NAME                          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvdb                          202:16   0   21G  0 disk
├─xvdb1                       202:17   0  200M  0 part /boot
├─xvdb2                       202:18   0 15.8G  0 part /
└─xvdb3                       202:19   0    4G  0 part [SWAP]
xvdc                          202:32   0   30G  0 disk
└─xvdc1                       202:33   0   30G  0 part /u01
xvdd                          202:48   0   42G  0 disk
└─xvdd1                       202:49   0   42G  0 part
  └─fraVolGroup-lvol0 (dm-2)  252:2    0   42G  0 lvm  /u03
xvde                          202:64   0   10G  0 disk
└─xvde1                       202:65   0   10G  0 part
  └─redoVolGroup-lvol0 (dm-1) 252:1    0   10G  0 lvm  /u04
xvdf                          202:80   0   25G  0 disk
└─xvdf1                       202:81   0   25G  0 part
  └─dataVolGroup-lvol0 (dm-0) 252:0    0   25G  0 lvm  /u02
xvdg                          202:96   0   10G  0 disk
└─xvdg1                       202:97   0   10G  0 part

Summary

If you want to attach additional storage via Oracle Compute Cloud Service, it’s better at the moment NOT to restart your DBasS service instance via the console. Just use the good old shutdown command. I am sure, this can not be a feature, this is must be a bug.

Additional Information

When you attach a disk with the Scale Up function, then it works properly.

attach03

Now you have two ways how to attach additonal storage which are working. It’s like in the movie Indiana Jones and the Last Crusade:

You must choose. But choose wisely…