Email: service@parnassusdata.com 7 x 24 online support!

Home

DOWNLOAD DBRECOVER

ORA-00600: internal error code, arguments:[2667] 一例

ORA-00600: internal error code, arguments:[2667] 一例

Posted by PDSERVICE on Jun 01, 2016 In

一套Power AIX上的9.2.0.1系统在数据库打开过程中遇到ORA-00600:[2667]内部错误，详细日志如下:

ORA-00600: internal error code, arguments:[2667] 

 

如果自己搞不定可以找诗檀软件专业ORACLE数据库修复团队成员帮您恢复!

诗檀软件专业数据库修复团队

服务热线 ： 13764045638    QQ号:47079569    邮箱：service@parnassusdata.com

 

Wed Mar 9 19:03:38 2011
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 117699122479
Resetting resetlogs activation ID 2197911857 (0x83017931)
Wed Mar 9 19:03:47 2011
LGWR: Primary database is in CLUSTER CONSISTENT mode
Assigning activation ID 2284878888 (0x88307c28)
Thread 1 opened at log sequence 1
Current log# 3 seq# 1 mem# 0: /s01/maclean/oradata/PROD/redo03.log
Successful open of redo thread 1.
Wed Mar 9 19:03:47 2011
SMON: enabling cache recovery
Wed Mar 9 19:03:47 2011
Errors in file /s01/maclean/admin/PROD/udump/PROD_ora_585914.trc:
ORA-07445: exception encountered: core dump [] [] [] [] [] []
Wed Mar 9 19:03:48 2011
Errors in file /s01/maclean/admin/PROD/bdump/PROD_pmon_602286.trc:
ORA-07445: exception encountered: core dump [] [] [] [] [] []
Wed Mar 9 19:03:50 2011
Errors in file /s01/maclean/admin/PROD/bdump/PROD_lgwr_548884.trc:
ORA-00600: internal error code, arguments: [2667], [1], [1], [3], [1739273391], [1739273391], [1739273391], [7267]
LGWR: terminating instance due to error 600

相关的初始化参数
fast_start_mttr_target = 300
_allow_resetlogs_corruption= TRUE
undo_management = AUTO
undo_tablespace = UNDOTBS1

以上可以看到lgwr关键进程在数据库open后几秒后遭遇了ORA-00600:[2667]内部错误后终止了实例。
该数据库在之前因为丢失当前日志文件进行了已经实施了一系列的非常规恢复操作，包括设置一系列的underscore参数:

Before I provide the steps to fix the ora-00600 error, I want to tell you that this database is opened with the unsupported parameter
"allow_resetlogs_corruption".

*************************************************************************
* By forcing open the database using this parameter, there is a strong *
* likelihood of logical corruption, possibly affecting the data *
* dictionary. Oracle does not guarantee that all of the data will be *
* accessible nor will it support a database that has been opened by *
* this method and that the database users will be allowed to continue *
* work. All this does is provide a way to get at the contents of the *
* database for extraction, usually by export. It is up to you to *
* determine the amount of lost data and to correct any logical *
* corruption issues. *
* *
*************************************************************************

2) The steps to get rid of the ora-00600 are as follows:

+ Change UNDO_MANAGEMENT=AUTO to

UNDO_MANAGEMENT=MANUAL

+ Remove or comment out UNDO_TABLESPACE and UNDO_RETENTION.

+ Add
_CORRUPTED_ROLLBACK_SEGMENTS =(comma separated list of Automatic Undo segments)

Example:

_CORRUPTED_ROLLBACK_SEGMENTS = (_SYSSMU1$, _SYSSMU2$, _SYSSMU3$, _SYSSMU4$,
_SYSSMU5$, _SYSSMU6$, _SYSSMU7$, _SYSSMU8$, _SYSSMU9$, _SYSSMU10$)

Note, sometimes the alert log will tell you what Automatic Undo segments are in use. 
Search the alert log for SYSS. If the alert log does not contain that information then use _SYSSMU1$ 
through _SYSSMU10$ as shown in the example above.

In UNIX you can issue this command to get the undo segment names:

$ strings system01.dbf | grep _SYSSMU | cut -d $ -f 1 | sort -u

From the output of the strings command above, add a $ to end of each _SYSSMU undo segment name.

++ Startup mount the database as follows:

SQL > startup mount
SQL > recover database;
SQl > alter database open;

*._corrupted_rollback_segments= (_SYSSMU730$, _SYSSMU731$, _SYSSMU732$, _SYSSMU733$, _SYSSMU734$, 
_SYSSMU735$, _SYSSMU736$, _SYSSMU737$, _SYSSMU738$, _SYSSMU739$, _SYSSMU744$, _SYSSMU740$, _SYSSMU741$, 
_SYSSMU742$, _SYSSMU743$, _SYSSMU744$, _SYSSMU745$, _SYSSMU746$, _SYSSMU747$, _SYSSMU748$, _SYSSMU749$, _SYSSMU74t$, 
_SYSSMU75$, _SYSSMU750$, _SYSSMU751$, _SYSSMU752$, _SYSSMU753$, _SYSSMU754$, _SYSSMU755$, _SYSSMU756$, _SYSSMU757$, 
_SYSSMU758$, _SYSSMU759$, _SYSSMU76$, _SYSSMU760$, _SYSSMU761$, _SYSSMU762$, _SYSSMU763$, _SYSSMU764$, _SYSSMU765$, 
_SYSSMU766$, _SYSSMU767$, _SYSSMU768$)

但在完成以上设置后仍不能避免ora-00600[2667]的发生，下位errpt日志中磁盘阵列损坏信息:

errpt -a

---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR6
IDENTIFIER: B9735AF4

Date/Time: Thu Mar 10 19:01:51 THAIST 2011
Sequence Number: 106800
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: PERM
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000

Description
SUBSYSTEM COMPONENT FAILURE

Probable Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT

Failure Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600
0000 0098 0000 0000 3FC6 0600 0000 0000 0000 0000 0000 D544 0000 0000 0000 0000
0008 5000 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020
2020 0660 2200 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5B8C 32C1 3033 3130 3131 2F30 3630 3533 3000 0000 0000 0000 0000 0000
0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR6
IDENTIFIER: B9735AF4

Date/Time: Thu Mar 10 19:01:51 THAIST 2011
Sequence Number: 106799
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: PERM
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000

Description
SUBSYSTEM COMPONENT FAILURE

Probable Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT

Failure Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600
0000 0098 0000 0000 3FC6 0600 0000 0000 0000 0000 0000 D524 0000 0000 0000 0000
0008 5000 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020
2020 0660 2200 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5B8C 32C0 3033 3130 3131 2F30 3630 3533 3000 0000 0000 0000 0000 0000
0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR10
IDENTIFIER: C86ACB7E

Date/Time: Thu Mar 10 19:01:51 THAIST 2011
Sequence Number: 106798
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: INFO
Resource Name: dac0
Resource Class: array
Resource Type: ibm-dac-V4
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21
VPD:
Manufacturer................IBM
Machine Type and Model......1814 FAStT
Part Number.................24288-00
ROS Level and ID............0916

Description
ARRAY CONFIGURATION CHANGED

Probable Causes
ARRAY CONTROLLER
CABLES AND CONNECTIONS

Failure Causes
ARRAY CONTROLLER
CABLES AND CONNECTIONS

Recommended Actions
NO ACTION NECESSARY

Detail Data
SENSE DATA
0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600
0000 0098 0000 0000 9502 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0008 1800 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020
2020 0660 2200 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5B8C 32BF 3033 3130 3131 2F30 3630 3533 3000 0000 0000 0000 0000 0000
0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR6
IDENTIFIER: B9735AF4

Date/Time: Thu Mar 10 19:01:47 THAIST 2011
Sequence Number: 106797
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: PERM
Resource Name: hdisk4
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L0

Description
SUBSYSTEM COMPONENT FAILURE

Probable Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT

Failure Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600
0000 0098 0000 0000 3FC6 0600 0000 0000 0000 0000 0000 D544 0000 0000 0000 0000
0008 5000 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020
2020 0660 2200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5B8C 3215 3033 3130 3131 2F30 3630 3532 3600 0000 0000 0000 0000 0000
0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR6
IDENTIFIER: B9735AF4

Date/Time: Thu Mar 10 19:01:47 THAIST 2011
Sequence Number: 106796
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: PERM
Resource Name: hdisk4
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L0

Description
SUBSYSTEM COMPONENT FAILURE

Probable Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT

Failure Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600
0000 0098 0000 0000 3FC6 0600 0000 0000 0000 0000 0000 D524 0000 0000 0000 0000
0008 5000 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020
2020 0660 2200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5B8C 3214 3033 3130 3131 2F30 3630 3532 3600 0000 0000 0000 0000 0000
0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR10
IDENTIFIER: C86ACB7E

Date/Time: Thu Mar 10 19:01:47 THAIST 2011
Sequence Number: 106795
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: INFO
Resource Name: dac0
Resource Class: array
Resource Type: ibm-dac-V4
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21
VPD:
Manufacturer................IBM
Machine Type and Model......1814 FAStT
Part Number.................24288-00
ROS Level and ID............0916

Description
ARRAY CONFIGURATION CHANGED

Probable Causes
ARRAY CONTROLLER
CABLES AND CONNECTIONS

Failure Causes
ARRAY CONTROLLER
CABLES AND CONNECTIONS

Recommended Actions
NO ACTION NECESSARY

Detail Data
SENSE DATA
0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600
0000 0098 0000 0000 9502 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0008 1800 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020
2020 0660 2200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5B8C 3213 3033 3130 3131 2F30 3630 3532 3600 0000 0000 0000 0000 0000
0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18

Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106794
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000

Description
ARRAY OPERATION ERROR

Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE

Failure Causes
DASD MEDIA
DISK DRIVE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0A00 2800 4644 2F00 0003 0004 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18

Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106793
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000

Description
ARRAY OPERATION ERROR

Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE

Failure Causes
DASD MEDIA
DISK DRIVE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0A00 2A00 132E C948 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18

Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106792
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000

Description
ARRAY OPERATION ERROR

Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE

Failure Causes
DASD MEDIA
DISK DRIVE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0A00 2A00 1335 55F0 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18

Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106791
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000

Description
ARRAY OPERATION ERROR

Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE

Failure Causes
DASD MEDIA
DISK DRIVE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0A00 2A00 136C 0F98 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18

Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106790
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000

Description
ARRAY OPERATION ERROR

Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE

Failure Causes
DASD MEDIA
DISK DRIVE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0A00 2A00 13D7 29B0 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18

Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106789
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000

Description
ARRAY OPERATION ERROR

Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE

Failure Causes
DASD MEDIA
DISK DRIVE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0A00 2A00 1337 FD88 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18

Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106788
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000

Description
ARRAY OPERATION ERROR

Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE

Failure Causes
DASD MEDIA
DISK DRIVE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0A00 2A00 136C 1F08 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18

Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106787
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000

Description
ARRAY OPERATION ERROR

Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE

Failure Causes
DASD MEDIA
DISK DRIVE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0A00 2A00 2578 E838 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18

Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106786
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000

Description
ARRAY OPERATION ERROR

Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE

Failure Causes
DASD MEDIA
DISK DRIVE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0A00 2800 4644 3300 0001 0004 0000 0000 0000 0000 0007 7497 0200 0400 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18

Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106785
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000

Description
ARRAY OPERATION ERROR

Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE

Failure Causes
DASD MEDIA
DISK DRIVE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0A00 2800 4643 BA00 0004 0004 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR8
IDENTIFIER: 483C9D10

Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106784
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: INFO
Resource Name: dac0
Resource Class: array
Resource Type: ibm-dac-V4
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21
VPD:
Manufacturer................IBM
Machine Type and Model......1814 FAStT
Part Number.................24288-00
ROS Level and ID............0916

Description
ARRAY ACTIVE CONTROLLER SWITCH

Probable Causes
ARRAY CONTROLLER
CABLES AND CONNECTIONS

Failure Causes
ARRAY CONTROLLER
CABLES AND CONNECTIONS

Recommended Actions
NO ACTION NECESSARY

Detail Data
SENSE DATA
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0400 00EE 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0002 0000 0001
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR3
IDENTIFIER: D9770360

Date/Time: Thu Mar 10 18:48:47 THAIST 2011
Sequence Number: 106783
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: PERM
Resource Name: dac0utm
Resource Class: NONE
Resource Type: NONE
Location:

Description
ARRAY OPERATION ERROR

Probable Causes
ARRAY DASD DEVICE
STORAGE DEVICE CABLE
EB4F

Failure Causes
DISK DRIVE
DISK DRIVE ELECTRONICS
STORAGE DEVICE CABLE
ARRAY CONTROLLER

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0A00 2800 0000 0200 0000 0104 0000 0000 0000 0000 0000 0013 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 1103 8000 F205 3703 0000 0200 0000 0000 0000 0000 0040 0002 0000 0000
0000 0000
---------------------------------------------------------------------------
LABEL: FSCSI_ERR4
IDENTIFIER: 3074FEB7

Date/Time: Thu Mar 10 18:48:47 THAIST 2011
Sequence Number: 106782
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: fscsi0
Resource Class: driver
Resource Type: efscsi
Location: U787B.001.DNWGK9Y-P1-C4-T1

Description
ADAPTER ERROR

Probable Causes
ADAPTER HARDWARE OR CABLE
ADAPTER MICROCODE
FIBRE CHANNEL SWITCH OR FC-AL HUB

Failure Causes
ADAPTER
CABLES AND CONNECTIONS
DEVICE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
CHECK CABLES AND THEIR CONNECTIONS
VERIFY DEVICE CONFIGURATION

Detail Data
SENSE DATA
0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0500 0000 0000
0001 0400 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 422F 0000 0812 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2005 00A0
B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
1105 C000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR9
IDENTIFIER: 8B79A4BD

Date/Time: Thu Mar 10 18:48:45 THAIST 2011
Sequence Number: 106781
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: PERM
Resource Name: dac1
Resource Class: array
Resource Type: ibm-dac-V4
Location: U787B.001.DNWGK9Y-P1-C1-T1-W200400A0B8484C21
VPD:
Manufacturer................IBM
Machine Type and Model......1814 FAStT
Part Number.................24288-00
ROS Level and ID............0916

Description
ARRAY CONTROLLER SWITCH FAILURE

Probable Causes
ARRAY CONTROLLER
CABLES AND CONNECTIONS

Failure Causes
ARRAY CONTROLLER
CABLES AND CONNECTIONS

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0A00 5A00 2C01 0000 0002 0004 0000 0000 0000 0000 0000 0000 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 526D 6000 F205 3703 0000 0000 0000 0010 0000 0000 0040 0002 0000 0000
0000 0000
---------------------------------------------------------------------------
LABEL: FSCSI_ERR4
IDENTIFIER: 3074FEB7

Date/Time: Thu Mar 10 18:48:45 THAIST 2011
Sequence Number: 106780
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: fscsi1
Resource Class: driver
Resource Type: efscsi
Location: U787B.001.DNWGK9Y-P1-C1-T1

Description
ADAPTER ERROR

Probable Causes
ADAPTER HARDWARE OR CABLE
ADAPTER MICROCODE
FIBRE CHANNEL SWITCH OR FC-AL HUB

Failure Causes
ADAPTER
CABLES AND CONNECTIONS
DEVICE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
CHECK CABLES AND THEIR CONNECTIONS
VERIFY DEVICE CONFIGURATION

Detail Data
SENSE DATA
0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0100 0000 0000
0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 422F 0000 1212 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2004 00A0
B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0FFF A000
---------------------------------------------------------------------------
LABEL: FSCSI_ERR4
IDENTIFIER: 3074FEB7

Date/Time: Thu Mar 10 18:48:43 THAIST 2011
Sequence Number: 106779
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: fscsi1
Resource Class: driver
Resource Type: efscsi
Location: U787B.001.DNWGK9Y-P1-C1-T1

Description
ADAPTER ERROR

Probable Causes
ADAPTER HARDWARE OR CABLE
ADAPTER MICROCODE
FIBRE CHANNEL SWITCH OR FC-AL HUB

Failure Causes
ADAPTER
CABLES AND CONNECTIONS
DEVICE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
CHECK CABLES AND THEIR CONNECTIONS
VERIFY DEVICE CONFIGURATION

Detail Data
SENSE DATA
0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0100 0000 0000
0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 422F 0000 1112 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2004 00A0
B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0FFF A000
---------------------------------------------------------------------------
LABEL: FSCSI_ERR4
IDENTIFIER: 3074FEB7

Date/Time: Thu Mar 10 18:48:42 THAIST 2011
Sequence Number: 106778
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: fscsi0
Resource Class: driver
Resource Type: efscsi
Location: U787B.001.DNWGK9Y-P1-C4-T1

Description
ADAPTER ERROR

Probable Causes
ADAPTER HARDWARE OR CABLE
ADAPTER MICROCODE
FIBRE CHANNEL SWITCH OR FC-AL HUB

Failure Causes
ADAPTER
CABLES AND CONNECTIONS
DEVICE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
CHECK CABLES AND THEIR CONNECTIONS
VERIFY DEVICE CONFIGURATION

Detail Data
SENSE DATA
0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0500 0000 0000
0001 0400 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 422F 0000 0712 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2005 00A0
B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
1105 C000
---------------------------------------------------------------------------
LABEL: FSCSI_ERR4
IDENTIFIER: 3074FEB7

Date/Time: Thu Mar 10 18:48:41 THAIST 2011
Sequence Number: 106777
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: fscsi1
Resource Class: driver
Resource Type: efscsi
Location: U787B.001.DNWGK9Y-P1-C1-T1

Description
ADAPTER ERROR

Probable Causes
ADAPTER HARDWARE OR CABLE
ADAPTER MICROCODE
FIBRE CHANNEL SWITCH OR FC-AL HUB

Failure Causes
ADAPTER
CABLES AND CONNECTIONS
DEVICE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
CHECK CABLES AND THEIR CONNECTIONS
VERIFY DEVICE CONFIGURATION

Detail Data
SENSE DATA
0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0100 0000 0000
0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 422F 0000 1012 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2004 00A0
B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0FFF A000
---------------------------------------------------------------------------
LABEL: FSCSI_ERR4
IDENTIFIER: 3074FEB7

Date/Time: Thu Mar 10 18:48:39 THAIST 2011
Sequence Number: 106776
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: fscsi1
Resource Class: driver
Resource Type: efscsi
Location: U787B.001.DNWGK9Y-P1-C1-T1

Description
ADAPTER ERROR

Probable Causes
ADAPTER HARDWARE OR CABLE
ADAPTER MICROCODE
FIBRE CHANNEL SWITCH OR FC-AL HUB

Failure Causes
ADAPTER
CABLES AND CONNECTIONS
DEVICE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
CHECK CABLES AND THEIR CONNECTIONS
VERIFY DEVICE CONFIGURATION

Detail Data
SENSE DATA
0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0100 0000 0000
0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 422F 0000 0F12 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2004 00A0
B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0FFF A000
---------------------------------------------------------------------------
LABEL: FSCSI_ERR4
IDENTIFIER: 3074FEB7

因为lgwr的crash是发生在database open后，所以实际上我们是可以在这段时间内操作数据库的，这个case最终通过新建undo tablespace代替老的问题回滚表空间解决了。

We have been repair the disk controller, but ora-600[2667] still occur when we try to open database.
After recreating the undo tablespace we were able to open the database.