Oracle 在”WARNING: Disk … is now being offlined”, “WARNING: Write Failed.”, “ORA-15081” 之后，添加磁盘到ASM 磁盘组导致ORA-15130

如果自己搞不定可以找诗檀软件专业ORACLE数据库修复团队成员帮您恢复!

诗檀软件专业数据库修复团队

服务热线 ： 13764045638 QQ号:47079569 邮箱：service@parnassusdata.com

ora-15081
oerr ora 15081
15081, 00000, "failed to submit an I/O operation to a disk"
// *Cause: A submission of an I/O operation to a disk has failed.
// *Action: Make sure that all the disks are operational.
//

ora-15130
 oerr ora 15130
15130, 00000, "diskgroup \"%s\" is being dismounted"
// *Cause: The diskgroup is being dismounted by request or because an I/O
// error was encountered that could not be handled by taking the
// disks offline. A disk cannot be offlined whenever doing so
// could result in all copies of a redundant extent being unavailable.
// *Action: Repair the hardware problem and re-mount the diskgroup.
// Refer to the alert log to determine which disks have failed.
//

适用于：

Oracle Database – Enterprise Edition – 版本 11.2.0.1 及以上
本文信息适用于任何平台。

症状

Cluster (2 节点) — 11204 — Oracle Solaris on SPARC (64位) 11     <—– 但这个问题也适用于支持ASM或UNIX平台的任何Oracle release
添加磁盘到ASM磁盘组导致磁盘组的强制dismount：

alert__ASM1.log :
~~~~~~~~~~~~~
…
Thu Oct 09 15:06:10 2014
SQL> alter diskgroup PRAFR01_DATA add                                                                                           
  failgroup FG_PATH1 disk ‘/dev/rdsk/c5t50060E8006D19F12d84s0’
  failgroup FG_PATH2 disk ‘/dev/rdsk/c5t50060E8006D19F12d85s0’ REBALANCE POWER 11
NOTE: GroupBlock outside rolling migration privileged region
NOTE: Assigning number (2,6) to disk (/dev/rdsk/c5t50060E8006D19F12d84s0)
NOTE: Assigning number (2,7) to disk (/dev/rdsk/c5t50060E8006D19F12d85s0)
NOTE: requesting all-instance membership refresh for group=2
NOTE: initializing header on grp 2 disk PRAFR01_DATA_0006
NOTE: initializing header on grp 2 disk PRAFR01_DATA_0007
NOTE: requesting all-instance disk validation for group=2
Thu Oct 09 15:06:13 2014
NOTE: skipping rediscovery for group 2/0xdb4fb0fa (PRAFR01_DATA) on local instance.
NOTE: requesting all-instance disk validation for group=2
NOTE: skipping rediscovery for group 2/0xdb4fb0fa (PRAFR01_DATA) on local instance.
Thu Oct 09 15:06:20 2014
GMON updating for reconfiguration, group 2 at 13 for pid 29, osid 17246
Thu Oct 09 15:06:20 2014
NOTE: group 2 PST updated.
NOTE: initiating PST update: grp = 2
GMON updating group 2 at 14 for pid 29, osid 17246
NOTE: group PRAFR01_DATA: updated PST location: disk 0000 (PST copy 0)
NOTE: group PRAFR01_DATA: updated PST location: disk 0003 (PST copy 1)
NOTE: PST update grp = 2 completed successfully
NOTE: membership refresh pending for group 2/0xdb4fb0fa (PRAFR01_DATA)
GMON querying group 2 at 15 for pid 18, osid 9218
NOTE: cache opening disk 6 of grp 2: PRAFR01_DATA_0006 path:/dev/rdsk/c5t50060E8006D19F12d84s0
NOTE: cache opening disk 7 of grp 2: PRAFR01_DATA_0007 path:/dev/rdsk/c5t50060E8006D19F12d85s0
Thu Oct 09 15:06:24 2014
NOTE: Attempting voting file refresh on diskgroup PRAFR01_DATA
NOTE: Refresh completed on diskgroup PRAFR01_DATA. No voting file found.
GMON querying group 2 at 16 for pid 18, osid 9218
SUCCESS: refreshed membership for 2/0xdb4fb0fa (PRAFR01_DATA)
SUCCESS: alter diskgroup PRAFR01_DATA add
failgroup FG_PATH1 disk ‘/dev/rdsk/c5t50060E8006D19F12d84s0’
failgroup FG_PATH2 disk ‘/dev/rdsk/c5t50060E8006D19F12d85s0’ REBALANCE POWER 11
NOTE: starting rebalance of group 2/0xdb4fb0fa (PRAFR01_DATA) at power 11
Starting background process ARB0
Thu Oct 09 15:06:25 2014
ARB0 started with pid=42, OS id=18217
NOTE: assigning ARB0 to group 2/0xdb4fb0fa (PRAFR01_DATA) with 11 parallel I/Os
NOTE: Attempting voting file refresh on diskgroup PRAFR01_DATA
NOTE: Refresh completed on diskgroup PRAFR01_DATA. No voting file found.
cellip.ora not found.
Thu Oct 09 15:07:20 2014
NOTE: process _user19665_+asm1 (19665) initiating offline of disk 7.4043259954 (PRAFR01_DATA_0007) with mask 0x7e in group 2
NOTE: checking PST: grp = 2
Thu Oct 09 15:07:20 2014
GMON checking disk modes for group 2 at 17 for pid 43, osid 19665
NOTE: checking PST for grp 2 done.
NOTE: sending set offline flag message 3512885393 to 1 disk(s) in group 2
WARNING: Disk PRAFR01_DATA_0007 in mode 0x7f is now being offlined                               <—————- disk been offlined
Thu Oct 09 15:07:20 2014
WARNING: Write Failed. group:2 disk:7 AU:3344 offset:0 size:1048576                              <—————- Write failed on disk#7 (PRAFR01_DATA_0007 or /dev/rdsk/c5t50060E8006D19F12d85s0)
ERROR: failed to copy file +PRAFR01_DATA.257, extent 9651
NOTE: initiating PST update: grp = 2, dsk = 7/0xf0ff4032, mask = 0x6a, op = clear
GMON updating disk modes for group 2 at 18 for pid 43, osid 19665
NOTE: group PRAFR01_DATA: updated PST location: disk 0000 (PST copy 0)
NOTE: group PRAFR01_DATA: updated PST location: disk 0003 (PST copy 1)
ERROR: ORA-15081 thrown in ARB0 for group number 2
Errors in file /app/oracle/base/11.2.0/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_18217.trc:
ORA-15081: failed to submit an I/O operation to a disk                                           <—————- I/O failed
Thu Oct 09 15:07:20 2014
NOTE: stopping process ARB0
Thu Oct 09 15:07:20 2014
WARNING: Write Failed. group:2 disk:7 AU:0 offset:32768 size:4096
WARNING: Write Failed. group:2 disk:7 AU:0 offset:4096 size:4096
NOTE: PST update grp = 2 completed successfully
NOTE: initiating PST update: grp = 2, dsk = 7/0xf0ff4032, mask = 0x7e, op = clear
GMON updating disk modes for group 2 at 19 for pid 43, osid 19665
NOTE: group PRAFR01_DATA: updated PST location: disk 0000 (PST copy 0)
NOTE: group PRAFR01_DATA: updated PST location: disk 0003 (PST copy 1)
NOTE: cache closing disk 7 of grp 2: PRAFR01_DATA_0007
NOTE: DBWR successfully wrote to at least one mirror side
NOTE: DBWR successfully wrote to at least one mirror side
NOTE: PST update grp = 2 completed successfully
NOTE: process _user19665_+asm1 (19665) initiating offline of disk 6.4043259953 (PRAFR01_DATA_0006) with mask 0x7e in group 2
NOTE: checking PST: grp = 2
GMON checking disk modes for group 2 at 20 for pid 43, osid 19665
NOTE: cache closing disk 7 of grp 2: (not open) PRAFR01_DATA_0007
ERROR: too many offline disks in PST (grp 2)
NOTE: checking PST for grp 2 done.
NOTE: initiating PST update: grp = 2, dsk = 6/0xf0ff4031, mask = 0x6a, op = clear
GMON updating disk modes for group 2 at 21 for pid 43, osid 19665
NOTE: cache closing disk 7 of grp 2: (not open) PRAFR01_DATA_0007
ERROR: Disk 6 cannot be offlined, since all the disks [6, 7] with mirrored data would be offline.
ERROR: too many offline disks in PST (grp 2)
Thu Oct 09 15:07:20 2014
NOTE: cache dismounting (not clean) group 2/0xDB4FB0FA (PRAFR01_DATA)
WARNING: Offline for disk PRAFR01_DATA_0006 in mode 0x7f failed.
NOTE: messaging CKPT to quiesce pins Unix process pid: 19696, image: oracle@23RNBAFPRA01 (B000)
NOTE: halting all I/Os to diskgroup 2 (PRAFR01_DATA)

原因

在磁盘上写入权限，而不是set Write permissions on the disks, not set

解决方案

在镜像之前（错误发生时）：

crw-r—–   1 grid     asmadmin 231, 1400 Oct  9 15:12 /dev/rdsk/c5t50060E8006D19F12d84s0
crw-r—–   1 grid     asmadmin 231, 1384 Oct  9 15:12 /dev/rdsk/c5t50060E8006D19F12d86s0
crw-r—–   1 grid     asmadmin 231, 1376 Oct  9 15:12 /dev/rdsk/c5t50060E8006D19F12d87s0
crw-r—–   1 grid     asmadmin 231, 1432 Oct  9 15:12 /dev/rdsk/c6t50060E8006D19F02d88s0
crw-r—–   1 grid     asmadmin 231, 1424 Oct  9 15:12 /dev/rdsk/c6t50060E8006D19F02d89s0
crw-r—–   1 grid     asmadmin 231, 1416 Oct  9 15:12 /dev/rdsk/c6t50060E8006D19F02d90s0
crw-r—–   1 grid     asmadmin 231, 1408 Oct  9 15:12 /dev/rdsk/c6t50060E8006D19F02d91s0

在镜像之后（这解决了问题）

crw-rw—-   1 grid     asmadmin 231, 1400 Oct  9 15:12 /dev/rdsk/c5t50060E8006D19F12d84s0
crw-rw—-   1 grid     asmadmin 231, 1384 Oct  9 15:12 /dev/rdsk/c5t50060E8006D19F12d86s0
crw-rw—-   1 grid     asmadmin 231, 1376 Oct  9 15:12 /dev/rdsk/c5t50060E8006D19F12d87s0
crw-rw—-   1 grid     asmadmin 231, 1432 Oct  9 15:12 /dev/rdsk/c6t50060E8006D19F02d88s0
crw-rw—-   1 grid     asmadmin 231, 1424 Oct  9 15:12 /dev/rdsk/c6t50060E8006D19F02d89s0
crw-rw—-   1 grid     asmadmin 231, 1416 Oct  9 15:12 /dev/rdsk/c6t50060E8006D19F02d90s0
crw-rw—-   1 grid     asmadmin 231, 1408 Oct  9 15:12 /dev/rdsk/c6t50060E8006D19F02d91s0

在磁盘权限被更正后，磁盘组能mount 并保持mounted。