ASMがdisk headerをなくした。ORA-15032、ORA-15040、ORA-15042 でDiskgrouがmountできなくなった

プロのOracle Databaseの復旧サービスを提供
携帯番号： +86 13764045638 メール：service@parnassusdata.com

ASMがdisk headerをなくして、ORA-15032、ORA-15040、ORA-15042で Diskgroupがmountできないインスタンスも少なくない。この文で、どうやって解決できるかを紹介できる。

SQL> select * from v$version;

BANNER
——————————————————————————–
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production
PL/SQL Release 11.2.0.3.0 – Production
CORE 11.2.0.3.0 Production
TNS for Linux: Version 11.2.0.3.0 – Production
NLSRTL Version 11.2.0.3.0 – Production
SQL> alter diskgroup datadg mount;
alter diskgroup datadg mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk “5” is missing from group number “1”
ERROR: alter diskgroup datadg mount
Wed Mar 13 07:42:03 2013
SQL> alter diskgroup datadg mount
NOTE: cache registered group DATADG number=1 incarn=0xccb845cd
NOTE: cache began mount (first) of group DATADG number=1 incarn=0xccb845cd
NOTE: Assigning number (1,2) to disk (/dev/asm-diskg)
NOTE: Assigning number (1,1) to disk (/dev/asm-diskf)
NOTE: Assigning number (1,0) to disk (/dev/asm-diske)
Wed Mar 13 16:42:09 2013
NOTE: GMON heartbeating for grp 1
GMON querying group 1 at 20 for pid 27, osid 5439
NOTE: Assigning number (1,5) to disk ()
GMON querying group 1 at 21 for pid 27, osid 5439
NOTE: cache dismounting (clean) group 1/0xCCB845CD (DATADG)
NOTE: messaging CKPT to quiesce pins Unix process pid: 5439, image: oracle@vmac1 (TNS V1-V3)
NOTE: dbwr not being msg’d to dismount
NOTE: lgwr not being msg’d to dismount
NOTE: cache dismounted group 1/0xCCB845CD (DATADG)
NOTE: cache ending mount (fail) of group DATADG number=1 incarn=0xccb845cd
NOTE: cache deleting context for group DATADG 1/0xccb845cd
GMON dismounting group 1 at 22 for pid 27, osid 5439
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
ERROR: diskgroup DATADG was not mounted
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk “5” is missing from group number “1”
ERROR: alter diskgroup datadg mount
Wed Mar 13 16:42:10 2013
ASM Health Checker found 1 new failures

[grid@vmac1 ~]$ kfed read /dev/asm-diskh
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
7FA1DA233400 00000000 00000000 00000000 00000000 […………….]
Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
col path for a20
set linesize 200 pagesize 1400
select path,header_status,state from v$asm_disk;
PATH HEADER_STATUS STATE
——————– ———————————— ————————
/dev/asm-diskh CANDIDATE NORMAL
/dev/asm-diskg MEMBER NORMAL
/dev/asm-diskf MEMBER NORMAL
/dev/asm-diske MEMBER NORMAL
/dev/asm-diskc MEMBER NORMAL
/dev/asm-diskd MEMBER NORMAL
/dev/asm-diskb MEMBER NORMAL

7 rows selected.

[grid@vmac1 ~]$ kfed repair /dev/asm-diskh
KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]

[grid@vmac1 ~]$ kfed repair /dev/asm-diskh ausz=1048576
KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]
ASMインスタンスを閉める
トラブルが起こったASMのheaderを優先的にバックアップしてください

dd if=<bad disk> of=<file> bs=4096 count=1

3. 今のディスクにfile 1 block 1を持っているものを探し出す

[grid@vmac1 ~]$ kfed read /dev/asm-diske |grep f1b1
kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002

[grid@vmac1 ~]$
[grid@vmac1 ~]$ kfed read /dev/asm-diskf |grep f1b1
kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000

[grid@vmac1 ~]$ kfed read /dev/asm-diskg |grep f1b1
kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000

ここのasm-diskeにf1b1に０を超えた数値が現れた。file 1 block 1が持っていれば、二つ目のauタイプがKFBTYP_LISTHEADかを確認できる。

[grid@vmac1 ~]$ kfed read /dev/asm-diske aun=2|grep kfbh.type
kfbh.type: 5 ; 0x002: KFBTYP_LISTHEAD

なくしたディスクにfile 1 block 1 F1B1を含んでいれば、KFBTYP_LISTHEADを見つけ出すまで、そのディスクにあるすべてのAUをスキャンする。LISTHEADを探せない場合に、仕方なく diskgroupを再構造しかない。
バーション11.1.0.7から(10gは10.2.0.5から、なるべく10.2.0.5前のASMを使われないでください)各I/OがASM disk header(AU 0 blocknum 0)へ提出するたびに、 AU 1にコピする。最後の二つのブロックは違ったAU sizeによって、そのブロックの位置も異なる。

Allocation Unit Size Block Number on AU 1
1048576 254
4194304 1022
8388608 2046
16777216 4094

まずはkfedでその位置に正確なdisk headerがあるかを確認する。さもなければ、人工的にふさわしいheaderを見つけ出してください。 テーブル1は以下の通り：

[grid@vmac1 ~]$ kfed read /dev/asm-diske ausz=1048576 aun=1 blkn=254|less
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 254 ; 0x004: blk=254
kfbh.block.obj: 2147483648 ; 0x008: disk=0
kfbh.check: 2086475720 ; 0x00c: 0x7c5d17c8
kfbh.fcn.base: 31322 ; 0x010: 0x00007a5a
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
後でkfed repairコマンドでdisk headerをリカバリする

[grid@vmac1 ~]$ kfed repair /dev/asm-diskh ausz=1048576

 
もしdisk headerの自動的なバックアップもなくした場合に、以下のエラになる。

KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]

もし以上のkfed repairが無効であれば、人工的にdisk headerをリカバリしてください：

10.2.0.5バーションに対して、kred repairが使用可能な情况は以下の通り：

トラブルdisk が同じなdiskgroupにあるが、f1b1ファイルを含むdisk headerを探し出してください。例えばここのasm-diskf
[grid@vmac1 ~]$ kfed read /dev/asm-diskf |grep f1b1
kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000

kfed read <device name> > fix.txt コマンドでヘッダを格納してください
[grid@vmac1 ~]$ kfed read /dev/asm-diskf > fix.txt
fix.txtを編集し、kfdhdb.dsknum 、kfdhdb.dskname 、 kfdhdb.fgname 三つの情報を修正してください：

同時にalert.logの情報を参考してください：

[grid@vmac1 trace]$ grep “cache opening” alert_+ASM1.log
NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb
NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc
NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd
NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb
NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc
NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd
NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb
NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc
NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd
NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb
NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc
NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd
NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb
NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc
NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd
NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb
NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc
NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd
NOTE: cache opening disk 0 of grp 2: DATADG_0000 path:/dev/asm-diske
NOTE: cache opening disk 1 of grp 2: DATADG_0001 path:/dev/asm-diskf
NOTE: cache opening disk 2 of grp 2: DATADG_0002 path:/dev/asm-diskg
NOTE: cache opening disk 0 of grp 2: DATADG_0000 path:/dev/asm-diske
NOTE: cache opening disk 1 of grp 2: DATADG_0001 path:/dev/asm-diskf
NOTE: cache opening disk 2 of grp 2: DATADG_0002 path:/dev/asm-diskg
NOTE: cache opening disk 0 of grp 1: DATADG_0000 path:/dev/asm-diske
NOTE: cache opening disk 1 of grp 1: DATADG_0001 path:/dev/asm-diskf
NOTE: cache opening disk 2 of grp 1: DATADG_0002 path:/dev/asm-diskg
NOTE: cache opening disk 0 of grp 2: SYSTEDG_0000 path:/dev/asm-diskb
NOTE: cache opening disk 1 of grp 2: SYSTEDG_0001 path:/dev/asm-diskc
NOTE: cache opening disk 2 of grp 2: SYSTEDG_0002 path:/dev/asm-diskd
NOTE: cache opening disk 5 of grp 1: DATADG_0005 path:/dev/asm-diskh

 
原fix.txt中的内容：

[grid@vmac1 ~]$ egrep “dsknum|grptyp|hdrsts|dskname|grpname|fgname” fix.txt
kfdhdb.dsknum: 1 ; 0x024: 0x0001
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: DATADG_0001 ; 0x028: length=11
kfdhdb.grpname: DATADG ; 0x048: length=6
kfdhdb.fgname: DATADG_0001 ; 0x068: length=11

修正したあと：

[grid@vmac1 ~]$ egrep “dsknum|grptyp|hdrsts|dskname|grpname|fgname” fix.txt
kfdhdb.dsknum: 5 ; 0x024: 0x0005
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: DATADG_0005 ; 0x028: length=11
kfdhdb.grpname: DATADG ; 0x048: length=6
kfdhdb.fgname: DATADG_0005 ; 0x068: length=11

そしてkfbh.block.objを修正してください
[grid@vmac1 ~]$ grep kfbh.block.obj fix.txt
kfbh.block.obj: 2147483649 ; 0x008: disk=1

2147483649==》0x80000001
0x80000001 最後の一位はASM DISK NUMBERでkfdhdb.dsknum に等しい。ここでは 0x80000005 ==》 2147483653
[grid@vmac1 ~]$ grep kfbh.block.obj fix.txt
kfbh.block.obj: 2147483653 ; 0x008: disk=5
windowsプラットフォームのASMLIBの場合に、kfdhdb.driver.reserved[0]を使う必要があるから、WindowsのASMLIBを使う人があまりいない。

そしてdisk directory を探し出せるために、aunum=2 blknum=2を確認してください。kfed read は f1b1のdiskの aunum=2 blknum=位置を持っているから:

kfed read <device name> aunum=2 blknum=2 | more
[grid@vmac1 ~]$ kfed read /dev/asm-diske|grep f1b1
kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002

[grid@vmac1 ~]$ kfed read /dev/asm-diske aunum=2 blknum=2|more
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 4 ; 0x002: KFBTYP_FILEDIR
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 2 ; 0x004: blk=2
kfbh.block.obj: 1 ; 0x008: file=1
kfbh.check: 322527999 ; 0x00c: 0x133962ff
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfffdb.node.incarn: 1 ; 0x000: A=1 NUMM=0x0
kfffdb.node.frlist.number: 4294967295 ; 0x004: 0xffffffff

…………………….
kfffde[0].xptr.au: 3 ; 0x4a0: 0x00000003
kfffde[0].xptr.disk: 0 ; 0x4a4: 0x0000
kfffde[0].xptr.flags: 0 ; 0x4a6: L=0 E=0 D=0 S=0
kfffde[0].xptr.chk: 41 ; 0x4a7: 0x29
kfffde[1].xptr.au: 4294967295 ; 0x4a8: 0xffffffff
kfffde[1].xptr.disk: 65535 ; 0x4ac: 0xffff
kfffde[1].xptr.flags: 0 ; 0x4ae: L=0 E=0 D=0 S=0
kfffde[1].xptr.chk: 42 ; 0x4af: 0x2a
==》disk directory 位于 disk=0 的 aunum=3

NOTE: cache opening disk 0 of grp 2: DATADG_0000 path:/dev/asm-diske ==> あるいはasm-diske
[grid@vmac1 ~]$ kfed read /dev/asm-diske aunum=3 blknum=0|more
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 6 ; 0x002: KFBTYP_DISKDIR
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 2 ; 0x008: file=2
kfbh.check: 389127513 ; 0x00c: 0x17319d59
kfbh.fcn.base: 31299 ; 0x010: 0x00007a43
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kffdnd.bnode.incarn: 1 ; 0x000: A=1 NUMM=0x0
kffdnd.bnode.frlist.number: 4294967295 ; 0x004: 0xffffffff
kffdnd.bnode.frlist.incarn: 0 ; 0x008: A=0 NUMM=0x0
kffdnd.overfl.number: 4294967295 ; 0x00c: 0xffffffff
kffdnd.overfl.incarn: 0 ; 0x010: A=0 NUMM=0x0
kffdnd.parent.number: 0 ; 0x014: 0x00000000
kffdnd.parent.incarn: 1 ; 0x018: A=1 NUMM=0x0
kffdnd.fstblk.number: 0 ; 0x01c: 0x00000000
kffdnd.fstblk.incarn: 1 ; 0x020: A=1 NUMM=0x0
kfddde[0].entry.incarn: 1 ; 0x024: A=1 NUMM=0x0
kfddde[0].entry.hash: 0 ; 0x028: 0x00000000
kfddde[0].entry.refer.number:4294967295 ; 0x02c: 0xffffffff
kfddde[0].entry.refer.incarn: 0 ; 0x030: A=0 NUMM=0x0
………………………………………………….

kfddde 構造は disk directory構造で、kfddde[0].entry.incarn A=1だけを持っている記録は割り当てられた記録。A=0とはその記録が削除されたと意味している。

[grid@vmac1 ~]$ grep “kfddde\[5\]” disk.txt
kfddde[5].entry.incarn: 1 ; 0x8e4: A=1 NUMM=0x0
kfddde[5].entry.hash: 5 ; 0x8e8: 0x00000005
kfddde[5].entry.refer.number:4294967295 ; 0x8ec: 0xffffffff
kfddde[5].entry.refer.incarn: 0 ; 0x8f0: A=0 NUMM=0x0
kfddde[5].dsknum: 5 ; 0x8f4: 0x0005
kfddde[5].state: 2 ; 0x8f6: KFDSTA_NORMAL
kfddde[5].ddchgfl: 132 ; 0x8f7: 0x84
kfddde[5].dskname: DATADG_0005 ; 0x8f8: length=11
kfddde[5].fgname: DATADG_0005 ; 0x918: length=11
kfddde[5].crestmp.hi: 32984459 ; 0x938: HOUR=0xb DAYS=0xc MNTH=0x3 YEAR=0x7dd
kfddde[5].crestmp.lo: 2470649856 ; 0x93c: USEC=0x0 MSEC=0xc8 SECS=0x34 MINS=0x24
kfddde[5].failstmp.hi: 0 ; 0x940: HOUR=0x0 DAYS=0x0 MNTH=0x0 YEAR=0x0
kfddde[5].failstmp.lo: 0 ; 0x944: USEC=0x0 MSEC=0x0 SECS=0x0 MINS=0x0
kfddde[5].timer: 0 ; 0x948: 0x00000000
kfddde[5].size: 5120 ; 0x94c: 0x00001400
kfddde[5].srRloc.super.hiStart: 0 ; 0x950: 0x00000000
kfddde[5].srRloc.super.loStart: 0 ; 0x954: 0x00000000
kfddde[5].srRloc.super.length: 0 ; 0x958: 0x00000000
kfddde[5].srRloc.incarn: 0 ; 0x95c: 0x00000000
kfddde[5].dskrprtm: 0 ; 0x960: 0x00000000
kfddde[5].zones[0].start: 0 ; 0x964: 0x00000000
kfddde[5].zones[0].size: 5120 ; 0x968: 0x00001400
kfddde[5].zones[0].used: 2 ; 0x96c: 0x00000002

fix.txtを編集することに戻って、 crestmp.hi とcrestmp.lo が以上に示された情報とマッチしてください。

元々

[grid@vmac1 ~]$ egrep “hi|lo” fix.txt
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 2147483653 ; 0x008: disk=5
kfdhdb.crestmp.hi: 32983191 ; 0x0a8: HOUR=0x17 DAYS=0x4 MNTH=0x2 YEAR=0x7dd
kfdhdb.crestmp.lo: 2328519680 ; 0x0ac: USEC=0x0 MSEC=0x299 SECS=0x2c MINS=0x22
kfdhdb.mntstmp.hi: 32984468 ; 0x0b0: HOUR=0x14 DAYS=0xc MNTH=0x3 YEAR=0x7dd
kfdhdb.mntstmp.lo: 1231840256 ; 0x0b4: USEC=0x0 MSEC=0x319 SECS=0x16 MINS=0x12
kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001
kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000
kfdhdb.grpstmp.hi: 32983191 ; 0x0e4: HOUR=0x17 DAYS=0x4 MNTH=0x2 YEAR=0x7dd
kfdhdb.grpstmp.lo: 2328331264 ; 0x0e8: USEC=0x0 MSEC=0x1e1 SECS=0x2c MINS=0x22
リカバリしたあと

kfdhdb.crestmp.hi: 32984459 ; 0x938: HOUR=0xb DAYS=0xc MNTH=0x3 YEAR=0x7dd
kfdhdb.crestmp.lo: 2470649856 ; 0x93c: USEC=0x0 MSEC=0xc8 SECS=0x34 MINS=0x24
kfdhdb.mntstmp.hi: 32984468 ; 0x0b0: HOUR=0x14 DAYS=0xc MNTH=0x3 YEAR=0x7dd
kfdhdb.mntstmp.lo: 1231840256 ; 0x0b4: USEC=0x0 MSEC=0x319 SECS=0x16 MINS=0x12
そしてkfed mergeコマンドでdisk headerを合併する

kfed merge <device name> text=fix.txt

[grid@vmac1 ~]$ kfed merge /dev/asm-diskh text=fix.txt

ASMLIbを使っている場合に、以下のコマンドでheaderのasmli情報をリカバリしてください。

/etc/init.d/oracleasm force-renamedisk /dev/sdbg1 <ASMLIB Disk Name>
/etc/init.d/oracleasm scandisks
/etc/init.d/oracleasm listdisks

そしてASMインスタンスをnomountに起動する

SQL> startup nomount;
SQL> col path for a20
SQL> set linesize 200 pagesize 1400
SQL> select path,header_status,state from v$asm_disk;

PATH HEADER_STATUS STATE
——————– ———————————— ————————
/dev/asm-diskh MEMBER NORMAL
/dev/asm-diskg MEMBER NORMAL
/dev/asm-diskf MEMBER NORMAL
/dev/asm-diske MEMBER NORMAL
/dev/asm-diskc MEMBER NORMAL
/dev/asm-diskd MEMBER NORMAL
/dev/asm-diskb MEMBER NORMAL

7 rows selected.
ヘッダ情報はmemberか否かを確認する

[grid@vmac1 ~]$ kfed read /dev/asm-diskh
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 2147483653 ; 0x008: disk=5
kfbh.check: 3412972861 ; 0x00c: 0xcb6dd53d
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8
kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000
kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000

そして alter diskgroup トラブルDG mount;

以上のステップにトラブルが起こらない場合にmount dgが無事に進行できる;
SQL>
SQL> alter diskgroup datadg mount;

Diskgroup altered.
NOTE: cache registered group DATADG number=1 incarn=0x01c845f0
NOTE: cache began mount (first) of group DATADG number=1 incarn=0x01c845f0
NOTE: Assigning number (1,5) to disk (/dev/asm-diskh)
NOTE: Assigning number (1,2) to disk (/dev/asm-diskg)
NOTE: Assigning number (1,1) to disk (/dev/asm-diskf)
NOTE: Assigning number (1,0) to disk (/dev/asm-diske)
Wed Mar 13 19:39:49 2013
NOTE: GMON heartbeating for grp 1
GMON querying group 1 at 56 for pid 27, osid 8690
NOTE: cache opening disk 0 of grp 1: DATADG_0000 path:/dev/asm-diske
NOTE: F1X0 found on disk 0 au 2 fcn 0.31322
NOTE: cache opening disk 1 of grp 1: DATADG_0001 path:/dev/asm-diskf
NOTE: cache opening disk 2 of grp 1: DATADG_0002 path:/dev/asm-diskg
NOTE: cache opening disk 5 of grp 1: DATADG_0005 path:/dev/asm-diskh
NOTE: cache mounting (first) external redundancy group 1/0x01C845F0 (DATADG)
Wed Mar 13 19:39:49 2013
* allocate domain 1, invalid = TRUE
kjbdomatt send to inst 2
Wed Mar 13 19:39:49 2013
NOTE: attached to recovery domain 1
NOTE: starting recovery of thread=1 ckpt=11.2351 group=1 (DATADG)
NOTE: advancing ckpt for group 1 (DATADG) thread=1 ckpt=11.2351
NOTE: cache recovered group 1 to fcn 0.33763
NOTE: redo buffer size is 256 blocks (1053184 bytes)
Wed Mar 13 19:39:49 2013
NOTE: LGWR attempting to mount thread 1 for diskgroup 1 (DATADG)
NOTE: LGWR found thread 1 closed at ABA 11.2350
NOTE: LGWR mounted thread 1 for diskgroup 1 (DATADG)
NOTE: LGWR opening thread 1 at fcn 0.33763 ABA 12.2351
NOTE: cache mounting group 1/0x01C845F0 (DATADG) succeeded
NOTE: cache ending mount (success) of group DATADG number=1 incarn=0x01c845f0
GMON querying group 1 at 57 for pid 18, osid 2911
Wed Mar 13 19:39:49 2013
NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1
SUCCESS: diskgroup DATADG was mounted
SUCCESS: alter diskgroup datadg mount
Wed Mar 13 19:39:49 2013
NOTE: diskgroup resource ora.DATADG.dg is online
NOTE: diskgroup resource ora.DATADG.dg is updated
Wed Mar 13 19:39:59 2013
NOTE: client PROD1:PROD registered, osid 10169, mbr 0x1
Wed Mar 13 19:40:11 2013
NOTE: ASM client PROD1:PROD disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file /g01/orabase/diag/asm/+asm/+ASM1/trace/+ASM1_ora_10169.trc

 
注意 以上のリカバリは約20MBのasm headerがディスクヘッダデータをなくした場合だけに効ける。それより上回った場合にmount diskgroupできなくなるかもしれない。
もしfile numbe=4 Continuing Operations Directory (COD) – ASM file number 4もなくなったであれば、mountできなくなる。以下のインスタンスで、22MBをなくしたから。たとえkfed mergeであっても取り戻せない。
SELECT x.xnum_kffxp “Extent”,
x.au_kffxp “AU”,
x.disk_kffxp “Disk #”,
d.name “Disk name”
FROM x$kffxp x, v$asm_disk_stat d
WHERE x.group_kffxp=d.group_number
and x.disk_kffxp=d.disk_number
and x.group_kffxp=1
and x.number_kffxp=4
ORDER BY 1, 2;

Extent AU Disk # Disk name
———- ———- ———- ——————————————————————————————
0 21 5 DATADG_0005
1 16 1 DATADG_0001
2 33 2 DATADG_0002
3 34 0 DATADG_0000
4 22 5 DATADG_0005
5 34 2 DATADG_0002
6 35 0 DATADG_0000
7 33 1 DATADG_0001

SQL> alter diskgroup datadg dismount;

Diskgroup altered.

[grid@vmac1 ~]$ dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=20
20+0 records in
20+0 records out
20971520 bytes (21 MB) copied, 0.0165823 s, 1.3 GB/s
[grid@vmac1 ~]$ kfed merge /dev/asm-diskh text=fix.txt

SQL> alter diskgroup datadg mount;

Diskgroup altered.

SQL> alter diskgroup datadg mount;

Diskgroup altered.
SQL> alter diskgroup datadg dismount;

Diskgroup altered.

[grid@vmac1 ~]$ dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=21
21+0 records in
21+0 records out
22020096 bytes (22 MB) copied, 0.0182842 s, 1.2 GB/s
[grid@vmac1 ~]$ kfed merge /dev/asm-diskh text=fix.txt
SQL> alter diskgroup datadg mount;

Diskgroup altered.

[grid@vmac1 ~]$ dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=22
22+0 records in
22+0 records out
23068672 bytes (23 MB) copied, 0.0312157 s, 739 MB/s
[grid@vmac1 ~]$ kfed merge /dev/asm-diskh text=fix.txt
SQL> alter diskgroup datadg mount;
alter diskgroup datadg mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15130: diskgroup “DATADG” is being dismounted
ORA-15066: offlining disk “DATADG_0005” in group “DATADG” may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]

Errors in file /g01/orabase/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8690.trc:
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
ERROR: cache failed to read group=1(DATADG) fn=4 blk=0 from disk(s): 5(DATADG_0005)
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
NOTE: cache initiating offline of disk 5 group DATADG
NOTE: process _user8690_+asm1 (8690) initiating offline of disk 5.3915953639 (DATADG_0005) with mask 0x7e in group 1
WARNING: Disk 5 (DATADG_0005) in group 1 in mode 0x7f is now being taken offline on ASM inst 1
NOTE: initiating PST update: grp = 1, dsk = 5/0xe968b5e7, mask = 0x6a, op = clear
GMON updating disk modes for group 1 at 108 for pid 27, osid 8690
ERROR: Disk 5 cannot be offlined, since diskgroup has external redundancy.
ERROR: too many offline disks in PST (grp 1)
WARNING: Offline of disk 5 (DATADG_0005) in group 1 and mode 0x7f failed on ASM inst 1
Wed Mar 13 20:00:56 2013
NOTE: halting all I/Os to diskgroup 1 (DATADG)
System State dumped to trace file /g01/orabase/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8690.trc
NOTE: AMDU dump of disk group DATADG created at /g01/orabase/diag/asm/+asm/+ASM1/trace
ERROR: ORA-15130 signalled during mount of diskgroup DATADG
NOTE: cache dismounting (clean) group 1/0xEB784617 (DATADG)
NOTE: messaging CKPT to quiesce pins Unix process pid: 8690, image: oracle@vmac1 (TNS V1-V3)
NOTE: LGWR doing non-clean dismount of group 1 (DATADG)
NOTE: LGWR sync ABA=18.2360 last written ABA 18.2360
kjbdomdet send to inst 2
detach from dom 1, sending detach message to inst 2
Wed Mar 13 20:00:57 2013
List of instances:
1 2
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 12)
Global Resource Directory partially frozen for dirty detach
* dirty detach – domain 1 invalid = TRUE
0 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
freeing rdom 1
WARNING: dirty detached from domain 1
NOTE: cache dismounted group 1/0xEB784617 (DATADG)
NOTE: cache ending mount (fail) of group DATADG number=1 incarn=0xeb784617
NOTE: cache deleting context for group DATADG 1/0xeb784617
GMON dismounting group 1 at 109 for pid 27, osid 8690
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
ERROR: diskgroup DATADG was not mounted
ORA-15032: not all alterations performed
ORA-15130: diskgroup “DATADG” is being dismounted
ORA-15066: offlining disk “DATADG_0005” in group “DATADG” may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
ERROR: alter diskgroup datadg mount
SQL> alter diskgroup datadg mount;
alter diskgroup datadg mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15130: diskgroup “DATADG” is being dismounted
ORA-15066: offlining disk “DATADG_0005” in group “DATADG” may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 !=1]
ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 !=1]

[grid@vmac1 trace]$ dd if=/dev/asm-diske of=/dev/asm-diskh bs=4096 skip=3 seek=3 count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.000617397 s, 6.6 MB/s

[grid@vmac1 trace]$ dd if=/dev/asm-diske of=/dev/asm-diskh bs=4096 skip=4 seek=4 count=1
kfffde[0].xptr.au: 21 ; 0x4a0: 0x00000015
kfffde[0].xptr.disk: 5 ; 0x4a4: 0x0005
kfffde[0].xptr.flags: 0 ; 0x4a6: L=0 E=0 D=0 S=0
kfffde[0].xptr.chk: 58 ; 0x4a7: 0x3a
kfffde[1].xptr.au: 16 ; 0x4a8: 0x00000010
[grid@vmac1 trace]$ dd if=/dev/asm-diske of=/dev/asm-diskh bs=1048576 skip=21 seek=21 count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00296742 s, 353 MB/s

ORA-15066: offlining disk “DATADG_0005” in group “DATADG” may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:26077] [obj_kfbl] [4] [0] [3 != 4]
ORA-15196: invalid ASM block header [kfc.c:26077] [obj_kfbl] [4] [0] [3 != 4]
ERROR: alter diskgroup datadg mount force