Email: service@parnassusdata.com 7 x 24 online support!

Home

DOWNLOAD DBRECOVER

    You are here

    • You are here:
      • Home > Blogs > PDSERVICE's blog > Grid Infrastructure (GI) startup fails because crsd fails to start in a flex asm environment (Doc ID 2392762.1)

Grid Infrastructure (GI) startup fails because crsd fails to start in a flex asm environment (Doc ID 2392762.1)

Grid Infrastructure (GI) startup fails because crsd fails to start in a flex asm environment (Doc ID 2392762.1)

Posted by PDSERVICE on Apr 05, 2020 In
SYMPTOMS
In a Flex ASM environment, Grid Infrastructure (GI) startup fails when GI is running on one or more of other nodes,
and "crsctl stat res -t -init" output shows that all resources except ora.crsd is up.
The ora.crsd shows either offline or intermediate state.
 
The cluster alert.log shows the following error:
2018-04-05 15:16:53.918 [CRSD(2697)]CRS-8500: Oracle Clusterware CRSD process is starting with operating system process ID 2697
2018-04-05 15:17:00.608 [CRSD(2697)]CRS-1013: The OCR location in an ASM disk group is inaccessible. Details in /u01/app/grid/diag/crs/<name>/crs/trace/crsd.trc.
2018-04-05 15:17:00.615 [CRSD(2697)]CRS-0804: Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage Storage layer error [Insufficient quorum to open OCR devices] [0]]. Details at (:CRSD00111:) in /u01/app/grid/diag/crs/<name>/crs/trace/crsd.trc.
 
 
 
The crsd.trc shows the following error:
2018-04-05 15:17:01.732 : CLSCRED:2919112768: (:CLSCRED0101:)clsCredDomInitRootDom: Using user given storage context for repository access.
2018-04-05 15:17:01.757 : OCRRAW:2919112768: 8033 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS
 
2018-04-05 15:17:01.761 : OCRRAW:2919112768: 8033 Error 4 querying length of attr ASM_STATIC_DISCOVERY_ADDRESS
 
2018-04-05 15:17:01.798 : CLSCRED:2919112768: (:CLSCRED1079:)clsCredOcrKeyExists: Obj dom : SYSTEM.credentials.domains.root.ASM.Self.076fa97b2ac84f70ff7035254e98f38d.root not found
2018-04-05 15:17:01.798 : OCRRAW:2919112768: 7755 Error 4 opening dom root in 0x4d37e30
 
2018-04-05 15:17:01.816 : OCRRAW:2919112768: kgfnConnect2: kgfnGetBeqData failed
 
2018-04-05 15:17:01.816*:kgfn.c@4933: kgfnConnect2: kgfnGetBeqData failed
2018-04-05 15:17:01.816 : CSSCLNT:2919112768: clsssinit: initialized context: (0x4f23d50) flags 0x104
2018-04-05 15:17:01.821 : CSSCLNT:2919112768: clsssterm: terminating context (0x4f23d50)
2018-04-05 15:17:01.862 : OCRRAW:2919112768: kgfnConnect2Int: cstr=(DESCRIPTION=(TCP_USER_TIMEOUT=1)(TRANSPORT_CONNECT_TIMEOUT=60)(EXPIRE_TIME=1)(ADDRESS_LIST=(LOAD_BALANCE=ON)(ADDRESS=(PROTOCOL=tcp)(HOST=nn.nn.255.13))(PORT=1526)))(CONNECT_DATA=(SERVICE_NAME=+ASM)))
 
2018-04-05 15:17:01.862*:kgfn.c@6685: kgfnConnect2Int: cstr=(DESCRIPTION=(TCP_USER_TIMEOUT=1)(TRANSPORT_CONNECT_TIMEOUT=60)(EXPIRE_TIME=1)(ADDRESS_LIST=(LOAD_BALANCE=ON)(ADDRESS=(PROTOCOL=tcp)(HOST=nn.nn.255.13)(PORT=1526)))(CONNECT_DATA=(SERVICE_NAME=+ASM)))
2018-04-05 15:17:01.862*:kgfn.c@6853: kgfnConnect2Int: OCISessionBegin failed
2018-04-05 15:17:03.139 : OCRRAW:2919112768: kgfnRecordErr 1017 OCI error:
ORA-01017: invalid username/password; logon denied
 
2018-04-05 15:17:03.139*:kgfn.c@1707: kgfnRecordErrPriv: 1017 error=ORA-01017: invalid username/password; logon denied
 
2018-04-05 15:17:03.140 : default:2919112768: clsCredDomClose: Credctx deleted 0x4d45890
2018-04-05 15:17:03.140 : OCRRAW:2919112768: kgfnConnect2: failed to connect
 
2018-04-05 15:17:03.140*:kgfn.c@5253: kgfnConnect2: failed to connect
2018-04-05 15:17:03.140 : OCRRAW:2919112768: kgfnConnect2Retry: failed to connect connect after 1 attempts, 143s elapsed
 
2018-04-05 15:17:03.140 : OCRRAW:2919112768: kgfo_kge2slos error stack at kgfoAl06: ORA-01017: invalid username/password; logon denied
ORA-27300: OS system dependent operation:sslssunreghdlr failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: sskgpreset1
ORA-15077: could not locate ASM instance serving a required diskgroup
 
 
 
 
 
CAUSE
The common causes are:
 
1) The sqlnet.ora shows
SQLNET.AUTHENTICATION_SERVICES=none
 
The above setting invalidates any OS authentication that crsd needs to connect to a remote asm instance running on another node.
 
2) The asm password is not correct
 
3) ASMlistener subnet do not match with the configured interconnect for the private interconnect.
Issue "oifcfg getif" to get the subnet for the private interconnect (cluster interconnect).
 This problem is also reported when SQLNET.AUTHENTICATION_SERVICES=all
 
SOLUTION
1) if the sqlnet.ora shows SQLNET.AUTHENTICATION_SERVICES=none or SQLNET.AUTHENTICATION_SERVICES=all
 
1) Remove "SQLNET.AUTHENTICATION_SERVICES=none" or "SQLNET.AUTHENTICATION_SERVICES=all" from the Grid Home SQLNET.ORA file (location $ORACLE_HOME/network/admin)
 
2) Restart the CRS with force.
 
crsctl stop crs -f
crsctl start crs
 
Refer to the "Unable to startup CRS as ASM failed to startup with "ORA-01017: invalid username/password; logon denied Document 1681849.1"
 
 2) The asm password is incorrect (this is likely cause if the sqlnet.ora is set up correctly)
 
1) Recreate the asm password as instructed in the MOS note " How to recreate shared ASM password file in 12c GI cluster Document 1929673.1".
 
2) Restart the CRS with force.
crsctl stop crs -f
crsctl start crs
 
 3) ASMlistener subnet do not match with the configured interconnect
 
Recreate the ASMlistener as mentioned in Step 3 from the section "C. For 12c and 18c Oracle Clusterware with Flex ASM" in the Document 283684.1.
 
 
A quick workaround is to start the asm manually on the local node using sqlplus.
If the ora.crsd does not become online after a couple of minutes of manually starting asm, then issue "crsctl start res ora.crsd -init" as root.