Email: [email protected] 7 x 24 online support!
Bug 13331814 : Oracle ASM DISKS TURNED INTO FORMER WHILE DISKGROUP IS MOUNTED.
Abstract: ASM DISKS TURNED INTO FORMER WHILE DISKGROUP IS MOUNTED.
PROBLEM:
--------
While the 2 diskgroups were mounted on 3 of 4 ASM instances, 1 member disk
(of each diskgroup) turned into FORMER, the disks associated with both
diskgroups appear as MEMBER (3) & FORMER(2) (note the diskgroups were mounted
while 2 disks turned into FORMER):
======================================================
GROUP_NUMBER DISK_NUMBER HEADER_STATU MODE_ST OS_MB TOTAL_MB FREE_MB NAME
FAILGROUP PATH
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3 0 MEMBER ONLINE 20468 20468 19335 DG_SCUT_ARCH1_0000 DG_SCUT_ARCH1_0000
/dev/rdsk/c3t60A980006466654F476F64317648572Fd0s6
3 1 MEMBER ONLINE 20468 20468 19341 DG_SCUT_ARCH1_0001 DG_SCUT_ARCH1_0001
/dev/rdsk/c3t60A980006466654F476F64317656586Dd0s6
3 2 FORMER ONLINE 20468 20468 19342 DG_SCUT_ARCH1_0002 DG_SCUT_ARCH1_0002
/dev/rdsk/c3t60A980006466654F476F643176575A4Fd0s6 <(== HERE
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
14 0 MEMBER ONLINE 20468 20468 20378 DG_MDSUT_ARCH2_0000 DG_MDSUT_ARCH2_0000
/dev/rdsk/c3t60A980006466654C436F643955687064d0s6
14 1 FORMER ONLINE 20468 20468 20378 DG_MDSUT_ARCH2_0001 DG_MDSUT_ARCH2_0001
/dev/rdsk/c3t60A980006466654C436F6439556B4E37d0s6 <(== HERE
======================================================
DIAGNOSTIC ANALYSIS:
--------------------
Both disks show a FORMER status:
======================================================
+ASM:oracle> kfed read c3t60A980006466654F476F643176575A4Fd0s6.dump | egrep
'grpname|dskname|hdrsts'
kfdhdb.hdrsts: 4 ; 0x027: KFDHDR_FORMER
kfdhdb.dskname: DG_SCUT_ARCH1_0002 ; 0x028: length=18
kfdhdb.grpname: DG_SCUT_ARCH1 ; 0x048: length=13
======================================================
+ASM:oracle> kfed read c3t60A980006466654C436F6439556B4E37d0s6.dump | egrep
'grpname|dskname|hdrsts'
kfdhdb.hdrsts: 4 ; 0x027: KFDHDR_FORMER
kfdhdb.dskname: DG_MDSUT_ARCH2_0001 ; 0x028: length=19
kfdhdb.grpname: DG_MDSUT_ARCH2 ; 0x048: length=14
======================================================
1) This an 11.2.0.2.0 ASM configuration (4 RAC nodes).
2) DG_MDSUT_ARCH2 & DG_SCUT_ARCH1 diskgroups were mounted on 4 ASM instances:
======================================================
GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB OFFLINE_DISKS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3 DG_SCUT_ARCH1 MOUNTED EXTERN 40936 38676 0
14 DG_MDSUT_ARCH2 MOUNTED EXTERN 20468 20378 0
======================================================
3) While the 2 diskgroups were mounted on 3 of 4 ASM instances, 1 member disk
(of each diskgroup) turned into FORMER, the disks associated with both
diskgroups appear as MEMBER (3) & FORMER(2) (note the diskgroups were mounted
while 2 disks turned into FORMER):
======================================================
GROUP_NUMBER DISK_NUMBER HEADER_STATU MODE_ST OS_MB TOTAL_MB FREE_MB NAME
FAILGROUP PATH
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3 0 MEMBER ONLINE 20468 20468 19335 DG_SCUT_ARCH1_0000 DG_SCUT_ARCH1_0000
/dev/rdsk/c3t60A980006466654F476F64317648572Fd0s6
3 1 MEMBER ONLINE 20468 20468 19341 DG_SCUT_ARCH1_0001 DG_SCUT_ARCH1_0001
/dev/rdsk/c3t60A980006466654F476F64317656586Dd0s6
3 2 FORMER ONLINE 20468 20468 19342 DG_SCUT_ARCH1_0002 DG_SCUT_ARCH1_0002
/dev/rdsk/c3t60A980006466654F476F643176575A4Fd0s6 <(== HERE
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
14 0 MEMBER ONLINE 20468 20468 20378 DG_MDSUT_ARCH2_0000 DG_MDSUT_ARCH2_0000
/dev/rdsk/c3t60A980006466654C436F643955687064d0s6
14 1 FORMER ONLINE 20468 20468 20378 DG_MDSUT_ARCH2_0001 DG_MDSUT_ARCH2_0001
/dev/rdsk/c3t60A980006466654C436F6439556B4E37d0s6 <(== HERE
======================================================
4) Diskgroups were not mounted on the node #4 (+ASM4), since node #4 was
evicted. After node reboot 2 the diskgroups cannot be mounted (DG_MDSUT_ARCH2
& DG_SCUT_ARCH1) on this node. Then the 2 disk showed header_status =
'FORMER'.
5) Since this is ASM 11.2.0.2, I asked to fix the disks headers as follow
(without success):
======================================================
5.1) Please backup all the database contained in the DG_SCUT_ARCH1 &
DG_MDSUT_ARCH2 diskgroups and validate the backups.
5.2) Then shutdown all the database instances referencing the DG_SCUT_ARCH1 &
DG_MDSUT_ARCH2 diskgroups (from all the nodes).
5.3) Then dismount the DG_SCUT_ARCH1 & DG_MDSUT_ARCH2 diskgroups from all the
ASM instances and keep them dismounted on all the nodes:
======================================================
SQL> alter diskgroup DG_MDSUT_ARCH2 dismount;
SQL> alter diskgroup DG_SCUT_ARCH1 dismount;
======================================================
5.4) Then fix the disks as follow:
======================================================
$>kfed repair /dev/rdsk/c3t60A980006466654F476F643176575A4Fd0s6
======================================================
$> kfed repair /dev/rdsk/c3t60A980006466654C436F6439556B4E37d0s6
======================================================
5.5) Then mount the 2 diskgroup on all the ASM instances:
======================================================
SQL> alter diskgroup DG_MDSUT_ARCH2 mount;
SQL> alter diskgroup DG_SCUT_ARCH1 mount;
======================================================
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> alter diskgroup DG_MDSUT_ARCH2 mount;
alter diskgroup DG_MDSUT_ARCH2 mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "1" is missing from group number "4"
SQL> alter diskgroup DG_SCUT_ARCH1 mount;
alter diskgroup DG_SCUT_ARCH1 mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "2" is missing from group number "4"
======================================================
*** 10/31/11 11:55 am ***
6) Then asked for the 50 MB of the affected disks:
======================================================
$> dd if=/dev/rdsk/c3t60A980006466654F476F643176575A4Fd0s6
of=/tmp/c3t60A980006466654F476F643176575A4Fd0s6.dump bs=1048576 count=50
$> dd if=/dev/rdsk/c3t60A980006466654C436F6439556B4E37d0s6
of=/tmp/c3t60A980006466654C436F6439556B4E37d0s6.dump bs=1048576 count=50
======================================================
7) Then dump the disk header using kfed, but both disks showed a FORMER
status:
======================================================
+ASM:oracle> kfed read c3t60A980006466654F476F643176575A4Fd0s6.dump | egrep
'grpname|dskname|hdrsts'
kfdhdb.hdrsts: 4 ; 0x027: KFDHDR_FORMER
kfdhdb.dskname: DG_SCUT_ARCH1_0002 ; 0x028: length=18
kfdhdb.grpname: DG_SCUT_ARCH1 ; 0x048: length=13
======================================================
+ASM:oracle> kfed read c3t60A980006466654C436F6439556B4E37d0s6.dump | egrep
'grpname|dskname|hdrsts'
kfdhdb.hdrsts: 4 ; 0x027: KFDHDR_FORMER
kfdhdb.dskname: DG_MDSUT_ARCH2_0001 ; 0x028: length=19
kfdhdb.grpname: DG_MDSUT_ARCH2 ; 0x048: length=14
======================================================
8) So, I edited the disk headers to change the status from FORMER to MEMBER:
======================================================
From:
kfdhdb.hdrsts: 4 ; 0x027: KFDHDR_FORMER
======================================================
To:
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
======================================================
9) Then attempted the repair the disk header on both disks using the fixed
header as follow:
======================================================
kfed merge /dev/rdsk/c3t60A980006466654C436F6439556B4E37d0s6
text=c3t60A980006466654C436F6439556B4E37d0s6_affected_DG_MDSUT_ARCH2_0001_kfed
_fix.txt
======================================================
kfed merge /dev/rdsk/c3t60A980006466654F476F643176575A4Fd0s6
text=c3t60A980006466654F476F643176575A4Fd0s6_affected_DG_SCUT_ARCH1_0002_kfed_
fix.txt
======================================================
10) But diskgroup could not be mounted due to the next new ORA-15203 error (I
double checked they are using the correct 11.2.0.2 ASM release to mount the
diskgroups):
======================================================
SQL> alter diskgroup DG_MDSUT_ARCH2 mount;
alter diskgroup DG_MDSUT_ARCH2 mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15203: diskgroup DG_MDSUT_ARCH2 contains disks from an incompatible
version
of ASM
======================================================
SQL> SQL> alter diskgroup DG_SCUT_ARCH1 mount;
alter diskgroup DG_SCUT_ARCH1 mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15203: diskgroup DG_SCUT_ARCH1 contains disks from an incompatible
version
of ASM
======================================================
11) We need help please from the Development team for the 2 things:
11.1) Why the 2 disk turned into FORMER disks?
11.1) Is there any option to fix the ASM header since kfed repair is not
working and or manual fix header patch is also having issues?