Home » Server Options » RAC & Failsafe » crsctl start crs not responding after manually crsctl stop crs (2 node 11gr2 RAC, OEL 6.4 (3rd Node is configured but downed before ages))
crsctl start crs not responding after manually crsctl stop crs [message #603400] Sun, 15 December 2013 23:23 Go to next message
BlackEagle
Messages: 8
Registered: December 2013
Location: India
Junior Member
Hi!

First I have manually stopped CRS:
[root@orcl1 /]# /u01/app/11.2.0/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'orcl1'
CRS-2673: Attempting to stop 'ora.crsd' on 'orcl1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'orcl1'
CRS-2673: Attempting to stop 'ora.orcl3.vip' on 'orcl1'
CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'orcl1'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'orcl1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'orcl1'
CRS-2677: Stop of 'ora.orcl3.vip' on 'orcl1' succeeded
CRS-2672: Attempting to start 'ora.orcl3.vip' on 'orcl2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.orcl.db' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'orcl1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'orcl1'
CRS-2676: Start of 'ora.orcl3.vip' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.orcl1.vip' on 'orcl1'
CRS-2677: Stop of 'ora.orcl1.vip' on 'orcl1' succeeded
CRS-2672: Attempting to start 'ora.orcl1.vip' on 'orcl2'
CRS-2676: Start of 'ora.orcl1.vip' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.OCR_VOTE.dg' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'orcl1'
CRS-2677: Stop of 'ora.asm' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'orcl1'
CRS-2677: Stop of 'ora.ons' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'orcl1'
CRS-2677: Stop of 'ora.net1.network' on 'orcl1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'orcl1' has completed
CRS-2677: Stop of 'ora.crsd' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'orcl1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'orcl1'
CRS-2673: Attempting to stop 'ora.evmd' on 'orcl1'
CRS-2673: Attempting to stop 'ora.asm' on 'orcl1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'orcl1'
CRS-2677: Stop of 'ora.evmd' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.crf' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.asm' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'orcl1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'orcl1'
CRS-2677: Stop of 'ora.cssd' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'orcl1'
CRS-2677: Stop of 'ora.gipcd' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'orcl1'
CRS-2677: Stop of 'ora.gpnpd' on 'orcl1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'orcl1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

At second step I have checked status:
[root@orcl1 /]# /u01/app/11.2.0/grid/bin/crsctl check crs
CRS-4639: Could not contact Oracle High Availability Services

At third step I have manually started CRS:
[root@orcl1 /]# /u01/app/11.2.0/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

At fourth step I have checked CRS status:
[root@orcl1 /]# /u01/app/11.2.0/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager

At fifth step attempt to shut down once again:
[root@orcl1 /]# /u01/app/11.2.0/grid/bin/crsctl stop crs
CRS-2796: The command may not proceed when Cluster Ready Services is not running
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.

So I thought to do down CRS on all running nodes (Node1 & Node2 are online while Node3 was allready down since ages)
[root@orcl1 /]# /u01/app/11.2.0/grid/bin/crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'orcl1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'orcl1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'orcl1'
CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'orcl1'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'orcl1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.orcl1.vip' on 'orcl1'
CRS-2677: Stop of 'ora.orcl.db' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'orcl1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'orcl1'
CRS-2677: Stop of 'ora.orcl1.vip' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.crsd' on 'orcl2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'orcl2'
CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'orcl2'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'orcl2'
CRS-2673: Attempting to stop 'ora.cvu' on 'orcl2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'orcl2'
CRS-2673: Attempting to stop 'ora.orcl2.vip' on 'orcl2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'orcl2'
CRS-2673: Attempting to stop 'ora.oc4j' on 'orcl2'
CRS-2677: Stop of 'ora.orcl2.vip' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.OCR_VOTE.dg' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'orcl1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'orcl2'
CRS-2677: Stop of 'ora.asm' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'orcl1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.orcl3.vip' on 'orcl2'
CRS-2677: Stop of 'ora.cvu' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.ons' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'orcl1'
CRS-2677: Stop of 'ora.net1.network' on 'orcl1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'orcl1' has completed
CRS-2677: Stop of 'ora.scan1.vip' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.crsd' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'orcl1'
CRS-2673: Attempting to stop 'ora.evmd' on 'orcl1'
CRS-2673: Attempting to stop 'ora.asm' on 'orcl1'
CRS-2677: Stop of 'ora.orcl3.vip' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.orcl.db' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'orcl2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'orcl2'
CRS-2677: Stop of 'ora.ctssd' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.asm' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'orcl1'
CRS-2677: Stop of 'ora.FRA.dg' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'orcl1'
CRS-2677: Stop of 'ora.cssd' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.OCR_VOTE.dg' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'orcl2'
CRS-2677: Stop of 'ora.asm' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'orcl2'
CRS-2677: Stop of 'ora.ons' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'orcl2'
CRS-2677: Stop of 'ora.net1.network' on 'orcl2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'orcl2' has completed
CRS-2677: Stop of 'ora.crsd' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'orcl2'
CRS-2673: Attempting to stop 'ora.evmd' on 'orcl2'
CRS-2673: Attempting to stop 'ora.asm' on 'orcl2'
CRS-2677: Stop of 'ora.evmd' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.asm' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'orcl2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'orcl2'
CRS-2677: Stop of 'ora.cssd' on 'orcl2' succeeded

And then started cluster on all nodes from Node1
[root@orcl1 /]# /u01/app/11.2.0/grid/bin/crsctl start cluster -all
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'orcl1'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'orcl2'
CRS-2676: Start of 'ora.cssdmonitor' on 'orcl1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'orcl1'
CRS-2672: Attempting to start 'ora.diskmon' on 'orcl1'
CRS-2676: Start of 'ora.diskmon' on 'orcl1' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'orcl2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'orcl2'
CRS-2672: Attempting to start 'ora.diskmon' on 'orcl2'
CRS-2676: Start of 'ora.diskmon' on 'orcl2' succeeded

And then tried to check CRS status from Node1 and Node2 but still having error:
[root@orcl1 /]# /u01/app/11.2.0/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager


Re: crsctl start crs not responding after manually crsctl stop crs [message #603407 is a reply to message #603400] Mon, 16 December 2013 00:49 Go to previous messageGo to next message
John Watson
Messages: 8922
Registered: January 2010
Location: Global Village
Senior Member
When you copy/paste a terminal session into your post, please enclose it with a [/code] tag at the end and a [code] tag at the beginning.

For how many minutes did you wait between this
/u01/app/11.2.0/grid/bin/crsctl start crs
and
/u01/app/11.2.0/grid/bin/crsctl check crs



Re: crsctl start crs not responding after manually crsctl stop crs [message #603412 is a reply to message #603407] Mon, 16 December 2013 01:26 Go to previous messageGo to next message
BlackEagle
Messages: 8
Registered: December 2013
Location: India
Junior Member
I waited for around a minute and half in between

 /u01/app/11.2.0/grid/bin/crsctl start crs 

and

 /u01/app/11.2.0/grid/bin/crsctl check crs 

Re: crsctl start crs not responding after manually crsctl stop crs [message #603414 is a reply to message #603412] Mon, 16 December 2013 01:27 Go to previous messageGo to next message
John Watson
Messages: 8922
Registered: January 2010
Location: Global Village
Senior Member
You might want to wait a bit longer.
Re: crsctl start crs not responding after manually crsctl stop crs [message #603415 is a reply to message #603414] Mon, 16 December 2013 01:29 Go to previous messageGo to next message
John Watson
Messages: 8922
Registered: January 2010
Location: Global Village
Senior Member
If you get bored waiting, commands such as
crsctl stat res -init -t
may show you how it is going.

[Updated on: Mon, 16 December 2013 01:29]

Report message to a moderator

Re: crsctl start crs not responding after manually crsctl stop crs [message #603423 is a reply to message #603415] Mon, 16 December 2013 02:36 Go to previous message
BlackEagle
Messages: 8
Registered: December 2013
Location: India
Junior Member
Thanks John!

By waiting for a bit long it is fine.

Many thanks. Smile
Previous Topic: gsd unknown state
Next Topic: Difference between activeversion, releaseversion and softwareversion
Goto Forum:
  


Current Time: Thu Mar 28 16:49:24 CDT 2024