Oracle Engineered Systems

Oracle Database Appliance X6-L Memory Expansion – Overwritten HugePage Settings after Reboot

Author: Martin Berger 15. May 2019

A Customer has extended his Oracle Database Appliance X6-L with Oracle Database Appliance Memory Expansion Kit from 192GB to 384GB. After the first startup, the odacli tool has recognized the additional available memory and gave us the suggested value for the new HugePage settings.

The ODA has the newest available patchset applied, OS is Oracle Enterprise Linux 6.10.

Update the OS Configuration

Oracle suggested the new values for the HugePage settings to 135350MB.

[root@srv20 ~]# odacli list-osconfigurations 

Parameter User ConfiguredValue SuggestedValue 
--------------- --------------- ------------------------------ --------------- 
Memlock grid 386716274KB 386716168KB 
Memlock oracle 386716274KB 386716168KB 
HugePage default 67815MB 135350MB

[root@srv20 ~]# odacli list-osconfigurations

Parameter User ConfiguredValue SuggestedValue

--------------- --------------- ------------------------------ ---------------

Memlock grid 386716274KB 386716168KB

Memlock oracle 386716274KB 386716168KB

HugePage default 67815MB 135350MB

According the Oracle documentation Oracle® Database Appliance X6-2S/X6-2M/X6-2L Deployment and User’s Guide – Section 13.14 – we updated the OS configuration by odacli. The job was finished after some seconds.

odacli update-osconfigurations (page 13-70)
Use the command odacli update-osconfigurations to update the HugePage and memlock values.

[root@srv20]# odacli update-osconfigurations

1	[root@srv20]# odacli update-osconfigurations

Verfication

The HugePage Settings were updated now.

[root@srv20 ~]# odacli list-osconfigurations 

Parameter User ConfiguredValue SuggestedValue 
--------------- --------------- ------------------------------ --------------- 
Memlock grid 386716168KB 386716168KB 
Memlock oracle 386716168KB 386716168KB 
HugePage default 135350MB 135350MB

[root@srv20 ~]# odacli list-osconfigurations

Parameter User ConfiguredValue SuggestedValue

--------------- --------------- ------------------------------ ---------------

Memlock grid 386716168KB 386716168KB

Memlock oracle 386716168KB 386716168KB

HugePage default 135350MB 135350MB

Time for a hardware restart.

Old HugePage Values after ODA Restart

But, after the machine has started, the old HugePage values were back again. But who has overwritten the settings?

[root@srv20 ~]# odacli list-osconfigurations 

Parameter User ConfiguredValue SuggestedValue 
--------------- --------------- ------------------------------ --------------- 
Memlock grid 386716274KB 386716168KB 
Memlock oracle 386716274KB 386716168KB 
HugePage default 67815MB 135350MB

[root@srv20 ~]# odacli list-osconfigurations

Parameter User ConfiguredValue SuggestedValue

--------------- --------------- ------------------------------ ---------------

Memlock grid 386716274KB 386716168KB

Memlock oracle 386716274KB 386716168KB

HugePage default 67815MB 135350MB

An SR has solved the problem, the are two files in the OS which are responsible for that behavior:

/etc/sysctl.d/99-initial-sysctl.conf
/etc/sysctl.d/99-oracle-rdbms-server-12cR1-preinstall-sysctl.conf

We take a look at the values:

[root@srv20 ~]# cat /etc/sysctl.d/99-initial-sysctl.conf | grep huge
vm.nr_hugepages=67815

[root@srv20 ~]# cat /etc/sysctl.d/99-oracle-rdbms-server-12cR1-preinstall-sysctl.conf | grep huge
vm.nr_hugepages=67815

[root@srv20 ~]# cat /etc/sysctl.d/99-initial-sysctl.conf | grep huge

vm.nr_hugepages=67815

[root@srv20 ~]# cat /etc/sysctl.d/99-oracle-rdbms-server-12cR1-preinstall-sysctl.conf | grep huge

vm.nr_hugepages=67815

Solution: Remove these two Files

First, we updated the OS Configuration again.

[root@srv20 ~]# odacli update-osconfigurations

1	[root@srv20 ~]# odacli update-osconfigurations

Second, we removed the files.

[root@srv20 ~]# rm /etc/sysctl.d/99-initial-sysctl.conf
[root@srv20 ~]# rm /etc/sysctl.d/99-oracle-rdbms-server-12cR1-preinstall-sysctl.conf

1 2	[root@srv20 ~]# rm /etc/sysctl.d/99-initial-sysctl.conf [root@srv20 ~]# rm /etc/sysctl.d/99-oracle-rdbms-server-12cR1-preinstall-sysctl.conf

Third, we restarted the Oracle Database Appliance.

Finally

HugePage settings are persistent now. By the way, there is an My Oracle Support note for the OS Level 6.10 which describes such a case too:

Changes in sysctl.conf File are not Persistent after Reboot (Doc ID 2477376.1)

Oracle Engineered Systems, TrivadisContent

Oracle Database Appliance X6 – resize your /u01

Author: Martin Berger 14. June 2018

Basically on an ODA X6-L the directory /u01 has the size of 100 GB. With seven different ORACLE_HOME directories for the Oracle databases and the Grid Infrastructure software, there is no disk space left for patching the Oracle Database Appliance. ODA patches require between 13 GB and 15 GB free space on /u01 for the new software. Just FYI – during the patch process a new stage directory called /u01/patching is created where the new software is temporarily located.

Patch Apply failed – Not enough free Space

If there is not enough free space available, update processes like updating a database home fail.

[root@oda-srv2046 ~]# odacli describe-job -i f024f27d-7c9a-43fc-9ae8-4f99ef809280

Job details
----------------------------------------------------------------
ID: f024f27d-7c9a-43fc-9ae8-4f99ef809280
Description: DB Home Patching: Home Id is 92c3af4e-1ef0-4ec6-8245-57b995a045d7
Status: Failure
Created: June 12, 2018 4:42:09 PM CEST
Message: DCS-10001:Internal error encountered: Failed to apply the patch 26925263 on the database home 92c3af4e-1ef0-4ec6-8245-57b995a045d7.

Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
DB Home Patching June 12, 2018 4:42:09 PM CEST June 12, 2018 4:48:37 PM CEST Failure
DB Home Patching June 12, 2018 4:42:09 PM CEST June 12, 2018 4:48:37 PM CEST Failure
clusterware patch verification June 12, 2018 4:42:09 PM CEST June 12, 2018 4:42:13 PM CEST Success
Patch location validation June 12, 2018 4:42:13 PM CEST June 12, 2018 4:42:19 PM CEST Success
Opatch updation June 12, 2018 4:43:31 PM CEST June 12, 2018 4:43:34 PM CEST Success
Patch conflict check June 12, 2018 4:43:34 PM CEST June 12, 2018 4:45:24 PM CEST Success
task:TaskSequential_4338 June 12, 2018 4:45:24 PM CEST June 12, 2018 4:48:37 PM CEST Failure
db upgrade June 12, 2018 4:45:24 PM CEST June 12, 2018 4:48:37 PM CEST Failure

[root@oda-srv2046 ~]# odacli describe-job -i f024f27d-7c9a-43fc-9ae8-4f99ef809280

Job details

----------------------------------------------------------------

ID: f024f27d-7c9a-43fc-9ae8-4f99ef809280

Description: DB Home Patching: Home Id is 92c3af4e-1ef0-4ec6-8245-57b995a045d7

Status: Failure

Created: June 12, 2018 4:42:09 PM CEST

Message: DCS-10001:Internal error encountered: Failed to apply the patch 26925263 on the database home 92c3af4e-1ef0-4ec6-8245-57b995a045d7.

Task Name Start Time End Time Status

---------------------------------------- ----------------------------------- ----------------------------------- ----------

DB Home Patching June 12, 2018 4:42:09 PM CEST June 12, 2018 4:48:37 PM CEST Failure

clusterware patch verification June 12, 2018 4:42:09 PM CEST June 12, 2018 4:42:13 PM CEST Success

Patch location validation June 12, 2018 4:42:13 PM CEST June 12, 2018 4:42:19 PM CEST Success

Opatch updation June 12, 2018 4:43:31 PM CEST June 12, 2018 4:43:34 PM CEST Success

Patch conflict check June 12, 2018 4:43:34 PM CEST June 12, 2018 4:45:24 PM CEST Success

task:TaskSequential_4338 June 12, 2018 4:45:24 PM CEST June 12, 2018 4:48:37 PM CEST Failure

db upgrade June 12, 2018 4:45:24 PM CEST June 12, 2018 4:48:37 PM CEST Failure

Verify the actual Situation and the Disk Configuration

There is not enough free space for the update process – but we see that /u01 is a logical volume.

[root@oda-srv2046 ~]# df -m /u01
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/VolGroupSys-LogVolU01
100666 85926 9621 90% /u01

[root@oda-srv2046 ~]# df -m /u01

Filesystem 1M-blocks Used Available Use% Mounted on

/dev/mapper/VolGroupSys-LogVolU01

100666 85926 9621 90% /u01

Let’s verify the physical volume for free space, we can see 7214 free physical extents are available.

[root@oda-srv2046 ~]# pvdisplay
--- Physical volume ---
PV Name /dev/sda2
VG Name VolGroupSys
PV Size 439.45 GiB / not usable 16.00 MiB
Allocatable yes
PE Size 32.00 MiB
Total PE 14062
Free PE 7214
Allocated PE 6848
PV UUID 3NtnHJ-Cg1Z-wS8c-4ADC-7465-FMUN-df2Gu1

[root@oda-srv2046 ~]# pvdisplay

--- Physical volume ---

PV Name /dev/sda2

VG Name VolGroupSys

PV Size 439.45 GiB / not usable 16.00 MiB

Allocatable yes

PE Size 32.00 MiB

Total PE 14062

Free PE 7214

Allocated PE 6848

PV UUID 3NtnHJ-Cg1Z-wS8c-4ADC-7465-FMUN-df2Gu1

This is the actual size of the logical volume, the actual logical volume size is 100 GB.

[root@oda-srv2046 ~]# lvdisplay /dev/mapper/VolGroupSys-LogVolU01
--- Logical volume ---
LV Path /dev/VolGroupSys/LogVolU01
LV Name LogVolU01
VG Name VolGroupSys
LV UUID lKPvcP-w0xv-Gezy-v6Pj-gtdU-TMMz-xtaj7s
LV Write Access read/write
LV Creation host, time localhost.localdomain, 2017-03-31 12:13:14 +0200
LV Status available
# open 1
LV Size 100.00 GiB
Current LE 3200
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 249:2

[root@oda-srv2046 ~]# lvdisplay /dev/mapper/VolGroupSys-LogVolU01

--- Logical volume ---

LV Path /dev/VolGroupSys/LogVolU01

LV Name LogVolU01

VG Name VolGroupSys

LV UUID lKPvcP-w0xv-Gezy-v6Pj-gtdU-TMMz-xtaj7s

LV Write Access read/write

LV Creation host, time localhost.localdomain, 2017-03-31 12:13:14 +0200

LV Status available

# open 1

LV Size 100.00 GiB

Current LE 3200

Segments 1

Allocation inherit

Read ahead sectors auto

- currently set to 256

Block device 249:2

Extend /u01

Now we extend online the LVM with the lvextend command from 100 GB to 150 GB.

[root@oda-srv2046 ~]# lvextend --size +50G /dev/VolGroupSys/LogVolU01
Size of logical volume VolGroupSys/LogVolU01 changed from 100.00 GiB (3200 extents) to 150.00 GiB (4800 extents).
Logical volume LogVolU01 successfully resized.

[root@oda-srv2046 ~]# lvextend --size +50G /dev/VolGroupSys/LogVolU01

Size of logical volume VolGroupSys/LogVolU01 changed from 100.00 GiB (3200 extents) to 150.00 GiB (4800 extents).

Logical volume LogVolU01 successfully resized.

The logical volume is now 150 GB.

[root@oda-srv2046 ~]# lvdisplay /dev/mapper/VolGroupSys-LogVolU01
--- Logical volume ---
LV Path /dev/VolGroupSys/LogVolU01
LV Name LogVolU01
VG Name VolGroupSys
LV UUID lKPvcP-w0xv-Gezy-v6Pj-gtdU-TMMz-xtaj7s
LV Write Access read/write
LV Creation host, time localhost.localdomain, 2017-03-31 12:13:14 +0200
LV Status available
# open 1
LV Size 150.00 GiB
Current LE 4800
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 249:2

[root@oda-srv2046 ~]# lvdisplay /dev/mapper/VolGroupSys-LogVolU01

--- Logical volume ---

LV Path /dev/VolGroupSys/LogVolU01

LV Name LogVolU01

VG Name VolGroupSys

LV UUID lKPvcP-w0xv-Gezy-v6Pj-gtdU-TMMz-xtaj7s

LV Write Access read/write

LV Creation host, time localhost.localdomain, 2017-03-31 12:13:14 +0200

LV Status available

# open 1

LV Size 150.00 GiB

Current LE 4800

Segments 2

Allocation inherit

Read ahead sectors auto

- currently set to 256

Block device 249:2

Finally the filesystem has to be resized.

[root@oda-srv2046 ~]# resize2fs /dev/VolGroupSys/LogVolU01
resize2fs 1.43-WIP (20-Jun-2013)
Filesystem at /dev/VolGroupSys/LogVolU01 is mounted on /u01; on-line resizing required
old_desc_blocks = 7, new_desc_blocks = 10
Performing an on-line resize of /dev/VolGroupSys/LogVolU01 to 39321600 (4k) blocks.
The filesystem on /dev/VolGroupSys/LogVolU01 is now 39321600 blocks long.

[root@oda-srv2046 ~]# resize2fs /dev/VolGroupSys/LogVolU01

resize2fs 1.43-WIP (20-Jun-2013)

Filesystem at /dev/VolGroupSys/LogVolU01 is mounted on /u01; on-line resizing required

old_desc_blocks = 7, new_desc_blocks = 10

Performing an on-line resize of /dev/VolGroupSys/LogVolU01 to 39321600 (4k) blocks.

The filesystem on /dev/VolGroupSys/LogVolU01 is now 39321600 blocks long.

Now we have enough space for patching.

[root@oda-srv2046 ~]# df -m /u01
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/VolGroupSys-LogVolU01
151062 85933 57451 60% /u01

[root@oda-srv2046 ~]# df -m /u01

Filesystem 1M-blocks Used Available Use% Mounted on

/dev/mapper/VolGroupSys-LogVolU01

151062 85933 57451 60% /u01

Summary

Resizing the /u01 on an Oracle Database Appliance is very easy and straightforward. There are no special actions required, it’s just a logical volume, no downtime required. And there is more space left on the physical volume for other resize actions.

Oracle Engineered Systems, Oracle RDBMS