2013-10-25 16:52:53.545: [UiServer][14657] S(11b6262d0): set Properties ( root,11b625590) 2013-10-25 16:52:53.555: [UiServer][14400] processMessage called 2013-10-25 16:52:53.555: [UiServer][14400] Sending message to PE. ctx= 11b61b410 2013-10-25 16:52:53.556: [ CRSPE][13886] Processing PE command id=24151. Description: [Stat Resource : 11b61f2f0] 2013-10-25 16:52:53.557: [ CRSPE][13886] PE Command [ Stat Resource : 11b61f2f0 ] has completed 2013-10-25 16:52:53.557: [ CRSPE][13886] UI Command [Stat Resource : 11b61f2f0] is replying to sender. 2013-10-25 16:52:53.557: [UiServer][14400] Done for ctx=11b61b410 2013-10-25 16:57:53.551: [UiServer][14657] S(11b6262d0): set Properties ( root,11b625590) 2013-10-25 16:57:53.562: [UiServer][14400] processMessage called 2013-10-25 16:57:53.562: [UiServer][14400] Sending message to PE. ctx= 11b61bfd0 2013-10-25 16:57:53.563: [ CRSPE][13886] Processing PE command id=24152. Description: [Stat Resource : 11b61f2f0] 2013-10-25 16:57:53.563: [ CRSPE][13886] PE Command [ Stat Resource : 11b61f2f0 ] has completed 2013-10-25 16:57:53.564: [ CRSPE][13886] UI Command [Stat Resource : 11b61f2f0] is replying to sender. 2013-10-25 16:57:53.564: [UiServer][14400] Done for ctx=11b61bfd0 2013-10-25 17:02:53.604: [UiServer][14657] S(11b6262d0): set Properties ( root,11b625590) 2013-10-25 17:02:53.615: [UiServer][14400] processMessage called 2013-10-25 17:02:53.615: [UiServer][14400] Sending message to PE. ctx= 11b61c5d0 2013-10-25 17:02:53.616: [ CRSPE][13886] Processing PE command id=24153. Description: [Stat Resource : 11b61f2f0] 2013-10-25 17:02:53.616: [ CRSPE][13886] PE Command [ Stat Resource : 11b61f2f0 ] has completed 2013-10-25 17:02:53.616: [ CRSPE][13886] UI Command [Stat Resource : 11b61f2f0] is replying to sender. 2013-10-25 17:02:53.617: [UiServer][14400] Done for ctx=11b61c5d0 2013-10-25 17:04:57.967: [ AGFW][12601] Agfw Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:2368102 2013-10-25 17:04:57.967: [ AGFW][12601] Received state change for ora.net1.network db2 1 [old state = ONLINE, new state = OFFLINE] 2013-10-25 17:04:57.968: [ AGFW][12601] Agfw Proxy Server sending message to PE, Contents = [MIDTo:2|OpID:3|FromA:{Invalid|Node:0|Process:0|Type:0}|ToA:{Invalid|Node:-1|Process:-1|Type:-1}|MIDFrom:0|Type:4|Pri2|Id:3535762] 2013-10-25 17:04:57.968: [ AGFW][12601] Agfw Proxy Server replying to the message: RESOURCE_STATUS[Proxy] ID 20481:2368102 2013-10-25 17:04:57.968: [ CRSPE][13886] State change received from db2 for ora.net1.network db2 1 2013-10-25 17:04:57.968: [ CRSPE][13886] Processing PE command id=24154. Description: [Resource State Change (ora.net1.network db2 1) : 11b61b7f0] 2013-10-25 17:04:57.968: [ CRSPE][13886] RI [ora.net1.network db2 1] new external state [OFFLINE] old value: [ONLINE] on db2 label = [] 2013-10-25 17:04:57.969: [ CRSRPT][14143] Publishing event: Cluster Resource State Change Event for ora.net1.network:db2 : 11b61f330 2013-10-25 17:04:57.969: [ CRSPE][13886] Resource Resource Instance ID[ora.net1.network db2 1]. Values: STATE=OFFLINE TARGET=ONLINE LAST_SERVER=db2 CURRENT_RCOUNT=0 LAST_RESTART=1382678769 FAILURE_COUNT=0 FAILURE_HISTORY= STATE_DETAILS= INCARNATION=56 STATE_CHANGE_VERS=154 LAST_FAULT=1382674398 DEGREE_ID=1 ID=ora.net1.network db2 1 Lock Info: Write Locks:none ReadLocks:|STATE INITED| has failed! 2013-10-25 17:04:57.969: [ CRSRPT][14143] Publish to eons buffered event : 11b61f330 2013-10-25 17:04:57.970: [ CRSPE][13886] Processing unplanned state change for [ora.net1.network db2 1] 2013-10-25 17:04:57.971: [ CRSPE][13886] Scheduled local recovery for [ora.net1.network db2 1] 2013-10-25 17:04:57.973: [ AGFW][12601] Agfw Proxy Server received the message: RESOURCE_PROBE[ora.db2.vip 1 1] ID 4097:3535766 2013-10-25 17:04:57.973: [ AGFW][12601] Agfw Proxy Server forwarding the message: RESOURCE_PROBE[ora.db2.vip 1 1] ID 4097:3535766 to the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:57.973: [ AGFW][12601] Agfw Proxy Server received the message: RESOURCE_PROBE[ora.LISTENER.lsnr db2 1] ID 4097:3535768 2013-10-25 17:04:57.974: [ AGFW][12601] Agfw Proxy Server forwarding the message: RESOURCE_PROBE[ora.LISTENER.lsnr db2 1] ID 4097:3535768 to the agent /oracle/product/11.2.0/crs/bin/oraagent_grid 2013-10-25 17:04:57.974: [ AGFW][12601] Agfw Proxy Server received the message: RESOURCE_PROBE[ora.ons db2 1] ID 4097:3535771 2013-10-25 17:04:57.975: [ AGFW][12601] Agfw Proxy Server forwarding the message: RESOURCE_PROBE[ora.ons db2 1] ID 4097:3535771 to the agent /oracle/product/11.2.0/crs/bin/oraagent_grid 2013-10-25 17:04:57.978: [ CRSPE][13886] Sending message to agfw: id = 3535775 2013-10-25 17:04:57.978: [ AGFW][12601] Agfw Proxy Server received the message: RESOURCE_START[ora.net1.network db2 1] ID 4098:3535775 2013-10-25 17:04:57.979: [ CRSPE][13886] CRS-2672: Attempting to start 'ora.net1.network' on 'db2' 2013-10-25 17:04:57.980: [ AGFW][12601] Agfw Proxy Server forwarding the message: RESOURCE_START[ora.net1.network db2 1] ID 4098:3535775 to the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:57.981: [ AGFW][12601] Received the reply to the message: RESOURCE_PROBE[ora.db2.vip 1 1] ID 4097:3535769 from the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:57.982: [ AGFW][12601] ora.db2.vip 1 1 received state from probe request. Old state = ONLINE, New state = ONLINE 2013-10-25 17:04:57.982: [ AGFW][12601] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_PROBE[ora.db2.vip 1 1] ID 4097:3535766 2013-10-25 17:04:57.984: [ CRSCOMM][11830][FFAIL] Ipc: Couldnt clscreceive message, no message: 11 2013-10-25 17:04:57.984: [ CRSCOMM][11830] Ipc: Client disconnected. 2013-10-25 17:04:57.985: [ CRSCOMM][11830][FFAIL] IpcL: Listener got clsc error 11 for memNum. 128 2013-10-25 17:04:57.985: [ CRSCOMM][11830] IpcL: connection to member 128 has been removed 2013-10-25 17:04:57.985: [CLSFRAME][11830] Removing IPC Member:{Relative|Node:0|Process:128|Type:3} 2013-10-25 17:04:57.985: [CLSFRAME][11830] Disconnected from AGENT process: {Relative|Node:0|Process:128|Type:3} 2013-10-25 17:04:57.985: [ AGFW][12601] Received the reply to the message: RESOURCE_START[ora.net1.network db2 1] ID 4098:3535778 from the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:57.985: [ CRSPE][13886] Disconnected from server: 2013-10-25 17:04:57.985: [ AGFW][12601] Agfw received DESTINATION_DOWN error for msg: 4098for resource: ora.net1.network db2 1 2013-10-25 17:04:57.987: [ AGFW][12601] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_START[ora.net1.network db2 1] ID 4098:3535775 2013-10-25 17:04:57.987: [ CRSPE][13886] Received reply to action [Start] message ID: 3535775 2013-10-25 17:04:57.987: [ CRSPE][13886] Fatal Error from AGFW Proxy: Communication error with agent process 2013-10-25 17:04:57.987: [ AGFW][12601] Agfw Proxy Server received process disconnected notification, count=1 2013-10-25 17:04:57.987: [ AGFW][12601] /oracle/product/11.2.0/crs/bin/orarootagent_root disconnected. 2013-10-25 17:04:57.987: [ AGFW][12601] Restarting the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:57.988: [ AGFW][12601] Agent /oracle/product/11.2.0/crs/bin/orarootagent_root[11927852] stopped! 2013-10-25 17:04:57.988: [ CRSPE][13886] CRS-2674: Start of 'ora.net1.network' on 'db2' failed 2013-10-25 17:04:57.988: [ CRSCOMM][12601] IpcL: removeConnection: Member 128 does not exist. 2013-10-25 17:04:57.988: [ AGFW][12601] Starting the agent: /oracle/product/11.2.0/crs/bin/orarootagent with user id: root and incarnation:46 2013-10-25 17:04:58.043: [ AGFW][12601] Starting the HB [Interval = 30000, misscount = 6kill allowed=1] for agent: /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.189: [ CRSPE][13886] Sending message to agfw: id = 3535792 2013-10-25 17:04:58.189: [ AGFW][12601] Agfw Proxy Server received the message: RESOURCE_CLEAN[ora.net1.network db2 1] ID 4100:3535792 2013-10-25 17:04:58.189: [ CRSPE][13886] CRS-2679: Attempting to clean 'ora.net1.network' on 'db2' 2013-10-25 17:04:58.190: [ AGFW][12601] Starting of the agent: /oracle/product/11.2.0/crs/bin/orarootagent with user id root is already in progress. 2013-10-25 17:04:58.190: [ AGFW][12601] Could not forward message [RESOURCE_CLEAN[ora.net1.network db2 1] ID 4100:3535792] to agent. /oracle/product/11.2.0/crs/bin/orarootagent_root is not running 2013-10-25 17:04:58.190: [ AGFW][12601] Starting of the agent: /oracle/product/11.2.0/crs/bin/orarootagent with user id root is already in progress. 2013-10-25 17:04:58.230: [ AGFW][12601] Received the reply to the message: RESOURCE_PROBE[ora.LISTENER.lsnr db2 1] ID 4097:3535772 from the agent /oracle/product/11.2.0/crs/bin/oraagent_grid 2013-10-25 17:04:58.230: [ AGFW][12601] ora.LISTENER.lsnr db2 1 received state from probe request. Old state = ONLINE, New state = ONLINE 2013-10-25 17:04:58.230: [ AGFW][12601] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_PROBE[ora.LISTENER.lsnr db2 1] ID 4097:3535768 2013-10-25 17:04:58.289: [CLSFRAME][11830] New IPC Member:{Relative|Node:0|Process:129|Type:3}:AGENT 2013-10-25 17:04:58.289: [CLSFRAME][11830] New process connected to us ID:{Relative|Node:0|Process:129|Type:3} Info:AGENT 2013-10-25 17:04:58.292: [ AGFW][12601] Agfw Proxy Server received the message: AGENT_HANDSHAKE[Proxy] ID 20484:14 2013-10-25 17:04:58.292: [ AGFW][12601] Agent /oracle/product/11.2.0/crs/bin/orarootagent_root with pid:12582946 connected to server. 2013-10-25 17:04:58.292: [ AGFW][12601] Could not find the resource type [ app.appvip.type ] 2013-10-25 17:04:58.293: [ AGFW][12601] Could not find the resource type [ ora.acfs.type ] 2013-10-25 17:04:58.293: [ AGFW][12601] Agfw Proxy Server sending message: RESTYPE_ADD[ora.cluster_vip.type] ID 8196:3535812 to the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.294: [ AGFW][12601] Agfw Proxy Server sending message: RESTYPE_ADD[ora.cluster_vip_net1.type] ID 8196:3535814 to the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.294: [ AGFW][12601] Could not find the resource type [ ora.crs.type ] 2013-10-25 17:04:58.295: [ AGFW][12601] Could not find the resource type [ ora.ctss.type ] 2013-10-25 17:04:58.295: [ AGFW][12601] Could not find the resource type [ ora.diskmon.type ] 2013-10-25 17:04:58.295: [ AGFW][12601] Could not find the resource type [ ora.drivers.acfs.type ] 2013-10-25 17:04:58.295: [ AGFW][12601] Could not find the resource type [ ora.gns.type ] 2013-10-25 17:04:58.296: [ AGFW][12601] Agfw Proxy Server sending message: RESTYPE_ADD[ora.network.type] ID 8196:3535816 to the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.296: [ AGFW][12601] Could not find the resource type [ ora.registry.acfs.type ] 2013-10-25 17:04:58.297: [ AGFW][12601] Agfw Proxy Server sending message: RESTYPE_ADD[ora.scan_vip.type] ID 8196:3535818 to the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.298: [ AGFW][12601] Agfw Proxy Server sending message: RESOURCE_ADD[ora.db1.vip 1 1] ID 4356:3535820 to the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.298: [ AGFW][12601] Agfw Proxy Server sending message: RESOURCE_ADD[ora.db2.vip 1 1] ID 4356:3535822 to the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.299: [ AGFW][12601] Agfw Proxy Server sending message: RESOURCE_ADD[ora.net1.network db2 1] ID 4356:3535824 to the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.300: [ AGFW][12601] Agfw Proxy Server forwarding the message: RESOURCE_CLEAN[ora.net1.network db2 1] ID 4100:3535792 to the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.301: [ AGFW][12601] Agfw Proxy Server sending message: RESOURCE_ADD[ora.scan1.vip 1 1] ID 4356:3535827 to the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.301: [ AGFW][12601] Agfw Proxy Server replying to the message: AGENT_HANDSHAKE[Proxy] ID 20484:14 2013-10-25 17:04:58.307: [ AGFW][12601] Received the reply to the message: RESTYPE_ADD[ora.cluster_vip.type] ID 8196:3535812 from the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.312: [ AGFW][12601] Received the reply to the message: RESTYPE_ADD[ora.cluster_vip_net1.type] ID 8196:3535814 from the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.317: [ AGFW][12601] Received the reply to the message: RESTYPE_ADD[ora.network.type] ID 8196:3535816 from the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.319: [ AGFW][12601] Received the reply to the message: RESTYPE_ADD[ora.scan_vip.type] ID 8196:3535818 from the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.320: [ AGFW][12601] Received the reply to the message: RESOURCE_ADD[ora.db1.vip 1 1] ID 4356:3535820 from the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.351: [ AGFW][12601] Received the reply to the message: RESOURCE_PROBE[ora.ons db2 1] ID 4097:3535773 from the agent /oracle/product/11.2.0/crs/bin/oraagent_grid 2013-10-25 17:04:58.351: [ AGFW][12601] ora.ons db2 1 received state from probe request. Old state = ONLINE, New state = ONLINE 2013-10-25 17:04:58.352: [ AGFW][12601] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_PROBE[ora.ons db2 1] ID 4097:3535771 2013-10-25 17:04:58.353: [ AGFW][12601] Received the reply to the message: RESOURCE_ADD[ora.db2.vip 1 1] ID 4356:3535822 from the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.354: [ AGFW][12601] Received the reply to the message: RESOURCE_ADD[ora.net1.network db2 1] ID 4356:3535824 from the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.360: [UiServer][14657] S(11b63c930): set Properties ( root,11b638010) 2013-10-25 17:04:58.370: [UiServer][14400] processMessage called 2013-10-25 17:04:58.371: [UiServer][14400] Sending message to PE. ctx= 11b620b30 2013-10-25 17:04:58.371: [ CRSPE][13886] Processing PE command id=24155. Description: [Stat Resource : 11b632170] 2013-10-25 17:04:58.376: [ CRSPE][13886] PE Command [ Stat Resource : 11b632170 ] has completed 2013-10-25 17:04:58.376: [ CRSPE][13886] UI Command [Stat Resource : 11b632170] is replying to sender. 2013-10-25 17:04:58.381: [UiServer][14400] Done for ctx=11b620b30 2013-10-25 17:04:58.424: [UiServer][14657] S(11b632590): set Properties ( root,11b6321b0) 2013-10-25 17:04:58.434: [UiServer][14400] processMessage called 2013-10-25 17:04:58.435: [UiServer][14400] Sending message to PE. ctx= 11b620b30 2013-10-25 17:04:58.435: [ CRSPE][13886] Processing PE command id=24156. Description: [Stat Resource : 11b632bd0] 2013-10-25 17:04:58.439: [ CRSPE][13886] PE Command [ Stat Resource : 11b632bd0 ] has completed 2013-10-25 17:04:58.440: [ CRSPE][13886] UI Command [Stat Resource : 11b632bd0] is replying to sender. 2013-10-25 17:04:58.444: [UiServer][14400] Done for ctx=11b620b30 2013-10-25 17:04:58.484: [UiServer][14657] S(11b632590): set Properties ( root,11b6321b0) 2013-10-25 17:04:58.494: [UiServer][14400] processMessage called 2013-10-25 17:04:58.494: [UiServer][14400] Sending message to PE. ctx= 11b620d10 2013-10-25 17:04:58.495: [ CRSPE][13886] Processing PE command id=24157. Description: [Stat Resource : 11b632bd0] 2013-10-25 17:04:58.499: [ CRSPE][13886] PE Command [ Stat Resource : 11b632bd0 ] has completed 2013-10-25 17:04:58.499: [ CRSPE][13886] UI Command [Stat Resource : 11b632bd0] is replying to sender. 2013-10-25 17:04:58.504: [UiServer][14400] Done for ctx=11b620d10 2013-10-25 17:04:58.543: [UiServer][14657] S(11b632590): set Properties ( root,11b6321b0) 2013-10-25 17:04:58.554: [UiServer][14400] processMessage called 2013-10-25 17:04:58.554: [UiServer][14400] Sending message to PE. ctx= 11b620b30 2013-10-25 17:04:58.555: [ CRSPE][13886] Processing PE command id=24158. Description: [Stat Resource : 11b632bd0] 2013-10-25 17:04:58.558: [ CRSPE][13886] PE Command [ Stat Resource : 11b632bd0 ] has completed 2013-10-25 17:04:58.559: [ CRSPE][13886] UI Command [Stat Resource : 11b632bd0] is replying to sender. 2013-10-25 17:04:58.560: [ AGFW][12601] Received the reply to the message: RESOURCE_ADD[ora.scan1.vip 1 1] ID 4356:3535827 from the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:58.564: [UiServer][14400] Done for ctx=11b620b30 2013-10-25 17:04:59.063: [ AGFW][12601] Received the reply to the message: RESOURCE_CLEAN[ora.net1.network db2 1] ID 4100:3535825 from the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:59.064: [ AGFW][12601] Agfw Proxy Server sending the reply to PE for message:RESOURCE_CLEAN[ora.net1.network db2 1] ID 4100:3535792 2013-10-25 17:04:59.064: [ CRSPE][13886] Received reply to action [Clean] message ID: 3535792 2013-10-25 17:04:59.565: [ AGFW][12601] Received the reply to the message: RESOURCE_CLEAN[ora.net1.network db2 1] ID 4100:3535825 from the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:04:59.566: [ AGFW][12601] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_CLEAN[ora.net1.network db2 1] ID 4100:3535792 2013-10-25 17:04:59.566: [ CRSPE][13886] Received reply to action [Clean] message ID: 3535792 2013-10-25 17:04:59.566: [ CRSPE][13886] CRS-2681: Clean of 'ora.net1.network' on 'db2' succeeded 2013-10-25 17:04:59.566: [ CRSPE][13886] Sequencer for [ora.net1.network db2 1] has completed with error: CRS-5804: Communication error with agent process 2013-10-25 17:04:59.567: [ CRSPE][13886] RIs of this resource are not relocatable:ora.net1.network: failover impossible 2013-10-25 17:04:59.568: [ CRSPE][13886] Failover cannot be completed for [ora.net1.network db2 1]. Stopping it and the resource tree 2013-10-25 17:04:59.577: [ CRSPE][13886] Sending message to agfw: id = 3535867 2013-10-25 17:04:59.577: [ AGFW][12601] Agfw Proxy Server received the message: RESOURCE_STOP[ora.ons db2 1] ID 4099:3535867 2013-10-25 17:04:59.577: [ CRSPE][13886] CRS-2673: Attempting to stop 'ora.ons' on 'db2' 2013-10-25 17:04:59.578: [ AGFW][12601] Agfw Proxy Server forwarding the message: RESOURCE_STOP[ora.ons db2 1] ID 4099:3535867 to the agent /oracle/product/11.2.0/crs/bin/oraagent_grid 2013-10-25 17:04:59.580: [ CRSPE][13886] Sending message to agfw: id = 3535870 2013-10-25 17:04:59.580: [ AGFW][12601] Agfw Proxy Server received the message: RESOURCE_STOP[ora.LISTENER.lsnr db2 1] ID 4099:3535870 2013-10-25 17:04:59.580: [ CRSPE][13886] CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'db2' 2013-10-25 17:04:59.581: [ AGFW][12601] Agfw Proxy Server forwarding the message: RESOURCE_STOP[ora.LISTENER.lsnr db2 1] ID 4099:3535870 to the agent /oracle/product/11.2.0/crs/bin/oraagent_grid 2013-10-25 17:05:00.237: [ AGFW][12601] Received the reply to the message: RESOURCE_STOP[ora.ons db2 1] ID 4099:3535868 from the agent /oracle/product/11.2.0/crs/bin/oraagent_grid 2013-10-25 17:05:00.238: [ AGFW][12601] Agfw Proxy Server sending the reply to PE for message:RESOURCE_STOP[ora.ons db2 1] ID 4099:3535867 2013-10-25 17:05:00.238: [ CRSPE][13886] Received reply to action [Stop] message ID: 3535867 2013-10-25 17:05:00.743: [ AGFW][12601] Received the reply to the message: RESOURCE_STOP[ora.ons db2 1] ID 4099:3535868 from the agent /oracle/product/11.2.0/crs/bin/oraagent_grid 2013-10-25 17:05:00.744: [ AGFW][12601] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_STOP[ora.ons db2 1] ID 4099:3535867 2013-10-25 17:05:00.744: [ CRSPE][13886] Received reply to action [Stop] message ID: 3535867 2013-10-25 17:05:00.744: [ CRSPE][13886] RI [ora.ons db2 1] new external state [OFFLINE] old value: [ONLINE] label = [] 2013-10-25 17:05:00.744: [ CRSPE][13886] CRS-2677: Stop of 'ora.ons' on 'db2' succeeded 2013-10-25 17:05:00.745: [ CRSRPT][14143] Publishing event: Cluster Resource State Change Event for ora.ons:db2 : 11b62e8b0 2013-10-25 17:05:00.745: [ CRSRPT][14143] Publish to eons buffered event : 11b62e8b0 2013-10-25 17:05:01.171: [UiServer][14657] S(11b63df10): set Properties ( grid,11b6322d0) 2013-10-25 17:05:01.181: [UiServer][14400] processMessage called 2013-10-25 17:05:01.182: [UiServer][14400] Container [ Name: UI_STOP CLIENT: TextMessage[] CLIENT_PRIMARY_GROUP: TextMessage[oinstall] EVENT_TAG: TextMessage[1] FILTER: TextMessage[((NAME==ora.LISTENER.lsnr)&&(LAST_SERVER==db2))USR_ORA_OPI=true] FILTER_TAG: TextMessage[1] LOCALE: TextMessage[AMERICAN_AMERICA.WE8ISO8859P1] NO_WAIT_TAG: TextMessage[1] ] 2013-10-25 17:05:01.182: [UiServer][14400] Sending message to PE. ctx= 11b6326f0 2013-10-25 17:05:01.182: [ CRSPE][13886] Cmd : 11b63e330 : flags: EVENT_TAG 2013-10-25 17:05:01.183: [ CRSPE][13886] Processing PE command id=24159. Description: [Stop Resource : 11b63e330] 2013-10-25 17:05:01.183: [ CRSPE][13886] Expression Filter : ((NAME == ora.LISTENER.lsnr) AND (LAST_SERVER == db2)) 2013-10-25 17:05:01.184: [ CRSPE][13886] Attribute overrides for the command: USR_ORA_OPI = true; 2013-10-25 17:05:01.187: [ CRSPE][13886] PE Command [ Stop Resource : 11b63e330 ] has completed 2013-10-25 17:05:01.187: [UiServer][14400] Container [ Name: ORDER MESSAGE: TextMessage[CRS-2545: Cannot operate on 'instance of ora.LISTENER.lsnr assigned to db2'. It is locked by 'SYSTEM' for command 'Unplanned Resource State Change : ora.net1.network'] MSGTYPE: TextMessage[1] OBJID: TextMessage[ora.LISTENER.lsnr db2 1] WAIT: TextMessage[0] ] 2013-10-25 17:05:01.187: [ CRSPE][13886] UI Command [Stop Resource : 11b63e330] is replying to sender. 2013-10-25 17:05:01.188: [UiServer][14400] Container [ Name: UI_DATA ora.LISTENER.lsnr db2 1: TextMessage[233] ] 2013-10-25 17:05:01.188: [UiServer][14400] Done for ctx=11b6326f0 2013-10-25 17:05:01.253: [ AGFW][12601] Received the reply to the message: RESOURCE_STOP[ora.LISTENER.lsnr db2 1] ID 4099:3535871 from the agent /oracle/product/11.2.0/crs/bin/oraagent_grid 2013-10-25 17:05:01.254: [ AGFW][12601] Agfw Proxy Server sending the reply to PE for message:RESOURCE_STOP[ora.LISTENER.lsnr db2 1] ID 4099:3535870 2013-10-25 17:05:01.254: [ CRSPE][13886] Received reply to action [Stop] message ID: 3535870 2013-10-25 17:05:01.763: [ AGFW][12601] Received the reply to the message: RESOURCE_STOP[ora.LISTENER.lsnr db2 1] ID 4099:3535871 from the agent /oracle/product/11.2.0/crs/bin/oraagent_grid 2013-10-25 17:05:01.763: [ AGFW][12601] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_STOP[ora.LISTENER.lsnr db2 1] ID 4099:3535870 2013-10-25 17:05:01.763: [ CRSPE][13886] Received reply to action [Stop] message ID: 3535870 2013-10-25 17:05:01.764: [ CRSPE][13886] RI [ora.LISTENER.lsnr db2 1] new external state [OFFLINE] old value: [ONLINE] label = [] 2013-10-25 17:05:01.764: [ CRSPE][13886] CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'db2' succeeded 2013-10-25 17:05:01.765: [ CRSRPT][14143] Publishing event: Cluster Resource State Change Event for ora.LISTENER.lsnr:db2 : 11b624690 2013-10-25 17:05:01.765: [ CRSRPT][14143] Publish to eons buffered event : 11b624690 2013-10-25 17:05:01.767: [ CRSPE][13886] Sending message to agfw: id = 3535889 2013-10-25 17:05:01.767: [ AGFW][12601] Agfw Proxy Server received the message: RESOURCE_STOP[ora.db2.vip 1 1] ID 4099:3535889 2013-10-25 17:05:01.767: [ CRSPE][13886] CRS-2673: Attempting to stop 'ora.db2.vip' on 'db2' 2013-10-25 17:05:01.768: [ AGFW][12601] Agfw Proxy Server forwarding the message: RESOURCE_STOP[ora.db2.vip 1 1] ID 4099:3535889 to the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:05:01.857: [ AGFW][12601] Received the reply to the message: RESOURCE_STOP[ora.db2.vip 1 1] ID 4099:3535890 from the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:05:01.858: [ AGFW][12601] Agfw Proxy Server sending the reply to PE for message:RESOURCE_STOP[ora.db2.vip 1 1] ID 4099:3535889 2013-10-25 17:05:01.858: [ CRSPE][13886] Received reply to action [Stop] message ID: 3535889 2013-10-25 17:05:02.363: [ AGFW][12601] Received the reply to the message: RESOURCE_STOP[ora.db2.vip 1 1] ID 4099:3535890 from the agent /oracle/product/11.2.0/crs/bin/orarootagent_root 2013-10-25 17:05:02.363: [ AGFW][12601] Agfw Proxy Server sending the last reply to PE for message:RESOURCE_STOP[ora.db2.vip 1 1] ID 4099:3535889 2013-10-25 17:05:02.364: [ CRSPE][13886] Received reply to action [Stop] message ID: 3535889 2013-10-25 17:05:02.364: [ CRSPE][13886] RI [ora.db2.vip 1 1] new external state [OFFLINE] old value: [ONLINE] label = [] 2013-10-25 17:05:02.364: [ CRSPE][13886] CRS-2677: Stop of 'ora.db2.vip' on 'db2' succeeded 2013-10-25 17:05:02.365: [ CRSRPT][14143] Publishing event: Cluster Resource State Change Event for ora.db2.vip:db2 : 11b635610 2013-10-25 17:05:02.365: [ CRSRPT][14143] Publish to eons buffered event : 11b635610 2013-10-25 17:05:02.370: [ CRSPE][13886] Sending message to agfw: id = 3535898 2013-10-25 17:05:02.370: [ CRSPE][13886] CRS-2672: Attempting to start 'ora.db2.vip' on 'db1' 2013-10-25 17:05:02.372: [ CRSCCL][11059]clscsend completed:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 1500 msglen = 4657 clschdr.size_clscmsgh= 4745 src= (2, 1360954526) dest= (1, 7514) 2013-10-25 17:05:04.993: [ CRSCCL][10802]clscreceive:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 963 msglen = 1582 clschdr.size_clscmsgh= 1670 src= (1, 7514) dest= (2, 1360954526) 2013-10-25 17:05:04.994: [ CRSPE][13886] Received reply to action [Start] message ID: 3535898 2013-10-25 17:05:05.495: [ CRSCCL][10802]clscreceive:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 964 msglen = 1653 clschdr.size_clscmsgh= 1741 src= (1, 7514) dest= (2, 1360954526) 2013-10-25 17:05:05.495: [ CRSPE][13886] Received reply to action [Start] message ID: 3535898 2013-10-25 17:05:05.496: [ CRSPE][13886] RI [ora.db2.vip 1 1] new external state [INTERMEDIATE] old value: [OFFLINE] on db1 label = [FAILED OVER] 2013-10-25 17:05:05.496: [ CRSPE][13886] Set State Details to [FAILED OVER] from [ ] for [ora.db2.vip 1 1] 2013-10-25 17:05:05.496: [ CRSPE][13886] CRS-2676: Start of 'ora.db2.vip' on 'db1' succeeded 2013-10-25 17:05:05.497: [ CRSRPT][14143] Publishing event: Cluster Resource State Change Event for ora.db2.vip:db1 : 11b63fa90 2013-10-25 17:05:05.497: [ CRSRPT][14143] Publish to eons buffered event : 11b63fa90 2013-10-25 17:05:05.498: [ CRSOCR][12858] Multi Write Batch processing... 2013-10-25 17:05:05.504: [ CRSOCR][12858] Multi Write Batch done. 2013-10-25 17:05:05.511: [ CRSPE][13886] PE Command [ Resource State Change (ora.net1.network db2 1) : 11b61b7f0 ] has completed 2013-10-25 17:05:05.512: [ AGFW][12601] Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:3535910 2013-10-25 17:05:05.512: [ CRSCCL][11059]clscsend completed:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 1501 msglen = 738 clschdr.size_clscmsgh= 826 src= (2, 1360954526) dest= (1, 7514) 2013-10-25 17:05:05.512: [ AGFW][12601] Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:3535910 2013-10-25 17:05:05.512: [ AGFW][12601] Agfw received reply from PE for resource state change for ora.net1.network db2 1 2013-10-25 17:05:05.513: [ CRSCCL][10802]clscreceive:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 965 msglen = 1104 clschdr.size_clscmsgh= 1192 src= (1, 7514) dest= (2, 1360954526) 2013-10-25 17:05:28.631: [UiServer][14657] S(11b630d10): set Properties ( root,11b633b70) 2013-10-25 17:05:28.642: [UiServer][14400] processMessage called 2013-10-25 17:05:28.642: [UiServer][14400] Sending message to PE. ctx= 11b61d7b0 2013-10-25 17:05:28.643: [ CRSPE][13886] Processing PE command id=24160. Description: [Stat Resource : 11b626710] 2013-10-25 17:05:28.665: [ CRSPE][13886] PE Command [ Stat Resource : 11b626710 ] has completed 2013-10-25 17:05:28.667: [ CRSPE][13886] UI Command [Stat Resource : 11b626710] is replying to sender. 2013-10-25 17:05:28.678: [UiServer][14400] Done for ctx=11b61d7b0