[Urgent] ORA-15042: ASM disk “76” is missing

Posted by PDSERVICE on Feb 05, 2016 In

If you cannot recover the data by yourself, ask Parnassusdata, the professional ORACLE database recovery team for help.

Parnassusdata Software Database Recovery Team

Service Hotline: +86 13764045638 E-mail: service@parnassusdata.com

One customer has a ASM problem about ORA-15042.
O/S: Linux X86 64bit 2.6.18-194.el5
DB Version : 10.2.0.5

Although we can access the ASM header using kfed & dd, the asm instance cannot read these devices.
For example, the ASM instance can read the 75th disk, but cannot read the 76th disk.

Do you have this experience?

# Environment
LGEDGDMS01:/engn001/orasvc01/product/10.2.0] uname -a
Linux LGEDGDMS01 2.6.18-194.el5 #1 SMP Tue Mar 16 21:52:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

# Error
SQL> startup
ASM instance started

Total System Global Area 130023424 bytes
Fixed Size                  2094544 bytes
Variable Size             102763056 bytes
ASM Cache                  25165824 bytes
ORA-15042: ASM disk “23” is missing
ORA-15042: ASM disk “22” is missing
ORA-15040: diskgroup is incomplete
ORA-15066: offlining disk “” may result in a data loss
ORA-15042: ASM disk “88” is missing
…
ORA-15042: ASM disk “77” is missing
ORA-15042: ASM disk “76” is missing   ==> 76 th device
ORA-15040: diskgroup is incomplete
ORA-15066: offlining disk “” may result in a data loss
ORA-15042: ASM disk “88” is missing
ORA-15042: ASM disk “87” is missing
…
ORA-15042: ASM disk “81” is missing

SQL> show parameter asm_diskstring
NAME TYPE VALUE
———————————— ——————————— ——————————
asm_diskstring string /dev/mapper/mpath_asm*

# v$asm_disks results.
select name, group_number,disk_number, path, state, header_status from v$asm_disk order by disk_number
/
NAME       GROUP_NUMBER DISK_NUMBER PATH                                     STATE                    HEADER_STATUS
———- ———— ———– —————————————- ———————— ————————————
0          66 /dev/mapper/mpath_asm129p1               NORMAL                   MEMBER
0          67 /dev/mapper/mpath_asm130p1               NORMAL                   MEMBER
0          68 /dev/mapper/mpath_asm131p1               NORMAL                   MEMBER
0          69 /dev/mapper/mpath_asm132p1               NORMAL                   MEMBER
0          70 /dev/mapper/mpath_asm133p1               NORMAL                   MEMBER
0          71 /dev/mapper/mpath_asm134p1               NORMAL                   MEMBER
0          72 /dev/mapper/mpath_asm135p1               NORMAL                   MEMBER
0          73 /dev/mapper/mpath_asm136p1               NORMAL                   MEMBER
0          74 /dev/mapper/mpath_asm137p1               NORMAL                   MEMBER
0          75 /dev/mapper/mpath_asm138p1               NORMAL                   MEMBER
0          89 /dev/mapper/mpath_asm063p1               NORMAL                   MEMBER   ==> Cannot see the 76 th device
0          90 /dev/mapper/mpath_asm064p1               NORMAL                   MEMBER
0          91 /dev/mapper/mpath_asm065p1               NORMAL                   MEMBER
0          92 /dev/mapper/mpath_asm066p1               NORMAL                   MEMBER
# Permission – OK
* 75th asm file (Good Device)
LGEDGDMS01:/engn001/orasvc01/product/10.2.0] ls -al /dev/mapper/mpath_asm138p1
brw-rw—- 1 orasvc01 dba 253, 248 Jan 30 17:06 /dev/mapper/mpath_asm138p1

* 76th the asm file (Cannot read this device)
LGEDGDMS01:/engn001/orasvc01/product/10.2.0] ls -al /dev/mapper/mpath_asm175
brw-rw—- 1 orasvc01 dba 253, 197 Jan 30 17:06 /dev/mapper/mpath_asm175

# kfed result – OK
* 75th asm file (Good Device)
+ /engn001/orasvc01/product/10.2.0/bin/kfed read /dev/mapper/mpath_asm138p1
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj:              2147483723 ; 0x008: TYPE=0x8 NUMB=0x4b
kfbh.check:                  2774762225 ; 0x00c: 0xa56382f1
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:ORCLDISKASMDISK138 ; 0x000: length=18
kfdhdb.driver.reserved[0]:   1145918273 ; 0x008: 0x444d5341
kfdhdb.driver.reserved[1]:    827020105 ; 0x00c: 0x314b5349
kfdhdb.driver.reserved[2]:        14387 ; 0x010: 0x00003833
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000
kfdhdb.compat:                168820736 ; 0x020: 0x0a100000
kfdhdb.dsknum:                       75 ; 0x024: 0x004b                       ==> The 75th device
kfdhdb.grptyp:                        2 ; 0x026: KFDGTP_NORMAL
kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname:           DGDATA01_0075 ; 0x028: length=13
kfdhdb.grpname:                DGDATA01 ; 0x048: length=8
kfdhdb.fgname:            DGDATA01_0075 ; 0x068: length=13
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.crestmp.hi:             32973218 ; 0x0a8: HOUR=0x2 DAYS=0xd MNTH=0x8 YEAR=0x7dc
kfdhdb.crestmp.lo:           1898247168 ; 0x0ac: USEC=0x0 MSEC=0x13d SECS=0x12 MINS=0x1c
kfdhdb.mntstmp.hi:             32973219 ; 0x0b0: HOUR=0x3 DAYS=0xd MNTH=0x8 YEAR=0x7dc
kfdhdb.mntstmp.lo:           1163180032 ; 0x0b4: USEC=0x0 MSEC=0x12e SECS=0x15 MINS=0x11
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000
kfdhdb.mfact:                    113792 ; 0x0c0: 0x0001bc80
kfdhdb.dsksize:                   13892 ; 0x0c4: 0x00003644

* 76th the asm file (Cannot read this device)
+ /engn001/orasvc01/product/10.2.0/bin/kfed read /dev/mapper/mpath_asm175
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj:              2147483724 ; 0x008: TYPE=0x8 NUMB=0x4c
kfbh.check:                  2433973412 ; 0x00c: 0x91137ca4
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:ORCLDISKASMDISK175 ; 0x000: length=18
kfdhdb.driver.reserved[0]:   1145918273 ; 0x008: 0x444d5341
kfdhdb.driver.reserved[1]:    827020105 ; 0x00c: 0x314b5349
kfdhdb.driver.reserved[2]:        13623 ; 0x010: 0x00003537
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000
kfdhdb.compat:                168820736 ; 0x020: 0x0a100000
kfdhdb.dsknum:                       76 ; 0x024: 0x004c            ==> the 76th device, ASM instance cannot read this device.
kfdhdb.grptyp:                        2 ; 0x026: KFDGTP_NORMAL
kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname:           DGDATA01_0076 ; 0x028: length=13
kfdhdb.grpname:                DGDATA01 ; 0x048: length=8
kfdhdb.fgname:            DGDATA01_0076 ; 0x068: length=13
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.crestmp.hi:             32982981 ; 0x0a8: HOUR=0x5 DAYS=0x1e MNTH=0x1 YEAR=0x7dd
kfdhdb.crestmp.lo:            366295040 ; 0x0ac: USEC=0x0 MSEC=0x14e SECS=0x1d MINS=0x5
kfdhdb.mntstmp.hi:             32982981 ; 0x0b0: HOUR=0x5 DAYS=0x1e MNTH=0x1 YEAR=0x7dd
kfdhdb.mntstmp.lo:            366307328 ; 0x0b4: USEC=0x0 MSEC=0x15a SECS=0x1d MINS=0x5
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000
kfdhdb.mfact:                    113792 ; 0x0c0: 0x0001bc80
kfdhdb.dsksize:                   55572 ; 0x0c4: 0x0000d914

# Check the dd results – OK
* the 75th asm device (Good)
dd if=/dev/mapper/mpath_asm138p1 bs=4096|od -tx1z|more

0000000 01 82 01 01 00 00 00 00 4b 00 00 80 f1 82 63 a5 >……..K…..c.<
0000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >…………….<
0000040 4f 52 43 4c 44 49 53 4b 41 53 4d 44 49 53 4b 31 >ORCLDISKASMDISK1<
0000060 33 38 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >38…………..<
0000100 00 00 10 0a 4b 00 02 03 44 47 44 41 54 41 30 31 >….K…DGDATA01<
0000120 5f 30 30 37 35 00 00 00 00 00 00 00 00 00 00 00 >_0075………..<
0000140 00 00 00 00 00 00 00 00 44 47 44 41 54 41 30 31 >……..DGDATA01<
0000160 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >…………….<
0000200 00 00 00 00 00 00 00 00 44 47 44 41 54 41 30 31 >……..DGDATA01<
0000220 5f 30 30 37 35 00 00 00 00 00 00 00 00 00 00 00 >_0075………..<
0000240 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >…………….<
*
0000300 00 00 00 00 00 00 00 00 a2 21 f7 01 00 f4 24 71 >………!….$q<
0000320 a3 21 f7 01 00 b8 54 45 00 02 00 10 00 00 10 00 >.!….TE……..<
0000340 80 bc 01 00 44 36 00 00 02 00 00 00 01 00 00 00 >….D6……….<
0000360 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >…………….<
0000400 00 00 10 0a 14 cd f6 01 00 2c 95 00 00 00 00 00 >………,……<
0000420 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >…………….<
*
0010000 01 82 02 01 01 00 00 00 4b 00 00 80 de 63 17 81 >……..K….c..<
0010020 af e0 35 00 00 00 00 00 00 00 00 00 00 00 00 00 >..5………….<
0010040 00 00 00 00 fe 00 20 00 c0 01 00 01 c0 01 00 01 >…… ………<
0010060 c0 01 00 01 c0 01 00 01 c0 01 00 01 c0 01 01 01 >…………….<
0010100 c0 01 00 01 c0 01 00 01 c0 01 00 01 c0 01 00 01 >…………….<
0010120 c0 01 00 01 c0 01 01 01 c0 01 01 01 c0 01 01 01 >…………….<
0010140 c0 01 01 01 c0 01 01 01 c0 01 01 01 c0 01 01 01 >…………….<
*
0010240 c0 01 01 01 04 00 01 01 00 00 00 00 00 00 00 00 >…………….<
0010260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >…………….<
*
0020000 01 82 03 01 02 00 00 00 4b 00 00 80 ce 10 bd 80 >……..K…….<
0020020 df ad 1e 00 00 00 00 00 00 00 00 00 00 00 00 00 >…………….<
0020040 00 00 00 00 c0 01 00 00 08 00 08 00 00 00 c0 01 >…………….<
0020060 10 00 10 00 00 00 00 00 18 00 18 00 00 00 00 00 >…………….<
0020100 20 00 20 00 00 00 00 00 00 00 00 00 00 00 80 00 > . ………….<
0020120 00 00 00 00 00 00 80 00 d9 0b 00 00 18 01 80 00 >…………….<

* 76th device (Read Failure)
dd if=/dev/mapper/mpath_asm175 bs=4096|od -tx1z|more

0000000 01 82 01 01 00 00 00 00 4c 00 00 80 a4 7c 13 91 >……..L….|..<
0000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >…………….<
0000040 4f 52 43 4c 44 49 53 4b 41 53 4d 44 49 53 4b 31 >ORCLDISKASMDISK1<
0000060 37 35 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >75…………..<
0000100 00 00 10 0a 4c 00 02 03 44 47 44 41 54 41 30 31 >….L…DGDATA01<
0000120 5f 30 30 37 36 00 00 00 00 00 00 00 00 00 00 00 >_0076………..<
0000140 00 00 00 00 00 00 00 00 44 47 44 41 54 41 30 31 >……..DGDATA01<
0000160 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >…………….<
0000200 00 00 00 00 00 00 00 00 44 47 44 41 54 41 30 31 >……..DGDATA01<
0000220 5f 30 30 37 36 00 00 00 00 00 00 00 00 00 00 00 >_0076………..<
0000240 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >…………….<
*
0000300 00 00 00 00 00 00 00 00 c5 47 f7 01 00 38 d5 15 >………G…8..<
0000320 c5 47 f7 01 00 68 d5 15 00 02 00 10 00 00 10 00 >.G…h……….<
0000340 80 bc 01 00 14 d9 00 00 02 00 00 00 01 00 00 00 >…………….<
0000360 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >…………….<
0000400 00 00 10 0a 14 cd f6 01 00 2c 95 00 00 00 00 00 >………,……<
0000420 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >…………….<
*
0000660 00 00 00 00 00 00 00 00 02 ec 44 ff 00 00 00 00 >……….D…..<
0000700 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >…………….<
*
0010000 01 82 02 01 01 00 00 00 4c 00 00 80 c3 62 4b 80 >……..L….bK.<
0010020 65 e0 35 00 00 00 00 00 00 00 00 00 00 00 00 00 >e.5………….<
0010040 00 00 00 00 fe 00 7d 00 c0 01 00 01 c0 01 00 01 >……}………<
0010060 c0 01 00 01 c0 01 00 01 c0 01 00 01 c0 01 00 01 >…………….<
*
0010460 c0 01 01 01 c0 01 01 01 c0 01 01 01 c0 01 01 01 >…………….<
*
0011020 c0 01 01 01 c0 01 01 01 14 00 01 01 00 00 00 00 >…………….<
0011040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >…………….<
*
0020000 01 82 03 01 02 00 00 00 4c 00 00 80 f0 45 ff 80 >……..L….E..<
0020020 9a d8 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 >…………….<
0020040 00 00 00 00 c0 01 00 00 08 00 08 00 00 00 c0 01 >…………….<
0020060 10 00 10 00 00 00 00 00 18 00 18 00 00 00 00 00 >…………….<

kfod status=true asm_diskstring=’/dev/mapper/mpath_asm*’ disk=ALL
——————————————————————————–
Disk          Size Header    Path
================================================================================
1:      13893 Mb CANDIDATE /dev/mapper/mpath_asm001
2:      13892 Mb MEMBER    /dev/mapper/mpath_asm001p1
3:      13893 Mb CANDIDATE /dev/mapper/mpath_asm002
4:      13892 Mb MEMBER    /dev/mapper/mpath_asm002p1
5:      13893 Mb CANDIDATE /dev/mapper/mpath_asm003
6:      13892 Mb MEMBER    /dev/mapper/mpath_asm003p1
7:      13893 Mb CANDIDATE /dev/mapper/mpath_asm004
…
274:      13892 Mb MEMBER    /dev/mapper/mpath_asm137p1
275:      13893 Mb CANDIDATE /dev/mapper/mpath_asm138
276:      13892 Mb MEMBER    /dev/mapper/mpath_asm138p1   ==> MEMBER
277:      13893 Mb CANDIDATE /dev/mapper/mpath_asm139
278:      13892 Mb MEMBER    /dev/mapper/mpath_asm139p1
…
343:      62400 Mb CANDIDATE /dev/mapper/mpath_asm172
344:      62393 Mb MEMBER    /dev/mapper/mpath_asm172p1
345:      62400 Mb CANDIDATE /dev/mapper/mpath_asm173
346:      62393 Mb MEMBER    /dev/mapper/mpath_asm173p1
347:      62400 Mb CANDIDATE /dev/mapper/mpath_asm174
348:      62393 Mb MEMBER    /dev/mapper/mpath_asm174p1
349:      55572 Mb CANDIDATE /dev/mapper/mpath_asm175    ==> CANDIDATE~!
350:      55572 Mb CANDIDATE /dev/mapper/mpath_asm176
351:      55572 Mb CANDIDATE /dev/mapper/mpath_asm177
352:      55572 Mb CANDIDATE /dev/mapper/mpath_asm178
353:      55572 Mb CANDIDATE /dev/mapper/mpath_asm179

I could find out one. The added partition don’t have any partition tables, but the original asm disks do it.
Due to storage engineer fault, I suppose that the KFED results is “MEMBER” and the KFOD result is “CANDICATE’ status.
I’ll replace it to the additional disks with partition tables.
If it successful, I will reply it.

# Reference
(Doc ID 580153.1) How To Setup ASM on Linux Using ASMLIB Disks, Raw Devices or Block Devices?
In order to use a disk (e.g. SAN) in Automatic Storage Management, the disk must have a partition table.

You are here

[Urgent] ORA-15042: ASM disk “76” is missing