Node 1 ================================================================================================================== CSSD]2013-08-07 06:08:28.852 [14] >WARNING: clssnmPollingThread: node nr-del-racdb2-mts (2) at 50 2.123763e-314artbeat fatal, eviction in 14.715 seconds [ CSSD]2013-08-07 06:08:28.852 [14] >TRACE: clssnmPollingThread: node nr-del-racdb2-mts (2) is impending reconfig, flag 1037, misstime 15285 [ CSSD]2013-08-07 06:08:28.852 [14] >TRACE: clssnmPollingThread: diskTimeout set to (27000)ms impending reconfig status(1) [ CSSD]2013-08-07 06:08:29.852 [14] >WARNING: clssnmPollingThread: node nr-del-racdb2-mts (2) at 50 2.123763e-314artbeat fatal, eviction in 13.715 seconds [ CSSD]2013-08-07 06:08:36.850 [14] >WARNING: clssnmPollingThread: node nr-del-racdb2-mts (2) at 75 2.123763e-314artbeat fatal, eviction in 6.715 seconds [ CSSD]2013-08-07 06:08:40.849 [14] >WARNING: clssnmPollingThread: node nr-del-racdb2-mts (2) at 90 2.123763e-314artbeat fatal, eviction in 2.715 seconds [ CSSD]2013-08-07 06:08:41.849 [14] >WARNING: clssnmPollingThread: node nr-del-racdb2-mts (2) at 90 2.123763e-314artbeat fatal, eviction in 1.715 seconds [ CSSD]2013-08-07 06:08:42.848 [14] >WARNING: clssnmPollingThread: node nr-del-racdb2-mts (2) at 90 2.123763e-314artbeat fatal, eviction in 0.715 seconds [ CSSD]2013-08-07 06:08:43.568 [14] >TRACE: clssnmPollingThread: Eviction started for node nr-del-racdb2-mts (2), flags 0x040d, state 3, wt4c 0 [ CSSD]2013-08-07 06:08:43.568 [16] >TRACE: clssnmDoSyncUpdate: Initiating sync 21 [ CSSD]2013-08-07 06:08:43.568 [16] >TRACE: clssnmDoSyncUpdate: diskTimeout set to (27000)ms [ CSSD]2013-08-07 06:08:43.568 [16] >TRACE: clssnmSetupAckWait: Ack message type (11) [ CSSD]2013-08-07 06:08:43.568 [16] >TRACE: clssnmSetupAckWait: node(1) is ALIVE [ CSSD]2013-08-07 06:08:43.568 [16] >TRACE: clssnmSendSync: syncSeqNo(21) [ CSSD]2013-08-07 06:08:43.569 [16] >TRACE: clssnmWaitForAcks: Ack message type(11), ackCount(1) [ CSSD]2013-08-07 06:08:43.569 [9] >TRACE: clssnmHandleSync: diskTimeout set to (27000)ms [ CSSD]2013-08-07 06:08:43.569 [9] >TRACE: clssnmHandleSync: Acknowledging sync: src[1] srcName[nr-del-racdb1-mts] seq[17] sync[21] [ CSSD]2013-08-07 06:08:43.569 [1] >USER: NMEVENT_SUSPEND [00][00][00][06] [ CSSD]2013-08-07 06:08:43.569 [16] >TRACE: clssnmWaitForAcks: done, msg type(11) [ CSSD]2013-08-07 06:08:43.569 [16] >TRACE: clssnmDoSyncUpdate: Terminating node 2, nr-del-racdb2-mts, misstime(30005) state(5) [ CSSD]2013-08-07 06:08:43.569 [16] >TRACE: clssnmSetupAckWait: Ack message type (13) [ CSSD]2013-08-07 06:08:43.569 [16] >TRACE: clssnmSetupAckWait: node(1) is ACTIVE [ CSSD]2013-08-07 06:08:43.569 [16] >TRACE: clssnmWaitForAcks: Ack message type(13), ackCount(1) [ CSSD]2013-08-07 06:08:43.569 [9] >TRACE: clssnmSendVoteInfo: node(1) syncSeqNo(21) [ CSSD]2013-08-07 06:08:43.570 [16] >TRACE: clssnmWaitForAcks: done, msg type(13) [ CSSD]2013-08-07 06:08:43.570 [16] >TRACE: clssnmCheckDskInfo: Checking disk info... [ CSSD]2013-08-07 06:08:43.570 [16] >TRACE: clssnmCheckDskInfo: node 2, nr-del-racdb2-mts, state 5 with leader 2 has smaller cluster size 1; my cluster size 1 wit h leader 1 [ CSSD]2013-08-07 06:08:43.570 [16] >TRACE: clssnmEvict: Start [ CSSD]2013-08-07 06:08:43.570 [16] >TRACE: clssnmEvict: Evicting node 2, nr-del-racdb2-mts, birth 20, death 21, impendingrcfg 1, stateflags 0x40d [ CSSD]2013-08-07 06:08:43.570 [16] >TRACE: clssnmSendShutdown: req to node 2, kill time 2770365095 Node 2 ===================================================================================================================== CSSD]2013-08-07 06:08:29.571 [14] >WARNING: clssnmPollingThread: node nr-del-racdb1-mts (1) at 50 2.123763e-314artbeat fatal, eviction in 14.285 seconds [ CSSD]2013-08-07 06:08:29.571 [14] >TRACE: clssnmPollingThread: node nr-del-racdb1-mts (1) is impending reconfig, flag 1039, misstime 15715 [ CSSD]2013-08-07 06:08:29.571 [14] >TRACE: clssnmPollingThread: diskTimeout set to (27000)ms impending reconfig status(1) [ CSSD]2013-08-07 06:11:55.960 >USER: Copyright 2013, Oracle version 10.2.0.4.0 [ clsdmt]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=nr-del-racdb2-mtsDBG_CSSD)) [ CSSD]2013-08-07 06:11:55.960 >USER: CSS daemon log for node nr-del-racdb2-mts, number 2, in cluster crs [ CSSD]2013-08-07 06:11:56.027 [1] >TRACE: clssscmain: local-only set to false [ CSSD]2013-08-07 06:11:56.063 [1] >TRACE: clssnmReadNodeInfo: added node 1 (nr-del-racdb1-mts) to cluster [ CSSD]2013-08-07 06:11:56.077 [1] >TRACE: clssnmReadNodeInfo: added node 2 (nr-del-racdb2-mts) to cluster [ CSSD]2013-08-07 06:11:56.098 [5] >TRACE: clssnm_skgxninit: Compatible vendor clusterware not in use [ CSSD]2013-08-07 06:11:56.098 [5] >TRACE: clssnm_skgxnmon: skgxn init failed [ CSSD]2013-08-07 06:11:56.102 [1] >TRACE: clssnmNMInitialize: misscount set to (30) [ CSSD]2013-08-07 06:11:56.104 [1] >TRACE: clssnmNMInitialize: Network heartbeat thresholds are: impending reconfig 15000 ms, reconfig start (misscount) 30000 ms [ CSSD]2013-08-07 06:11:56.114 [1] >TRACE: clssnmDiskStateChange: state from 1 to 2 disk (0//dev/rdsk/c4t600A0B800067E686000003B74C2843F3d0s0) [ CSSD]2013-08-07 06:11:56.114 [6] >TRACE: clssnmvDPT: spawned for disk 0 (/dev/rdsk/c4t600A0B800067E686000003B74C2843F3d0s0) [ CSSD]2013-08-07 06:11:58.141 [6] >TRACE: clssnmDiskStateChange: state from 2 to 4 disk (0//dev/rdsk/c4t600A0B800067E686000003B74C2843F3d0s0) [ CSSD]2013-08-07 06:11:58.142 [1] >TRACE: clssnmFatalInit: fatal mode enabled [ CSSD]2013-08-07 06:11:58.141 [7] >TRACE: clssnmvKillBlockThread: spawned for disk 0 (/dev/rdsk/c4t600A0B800067E686000003B74C2843F3d0s0) initial sleep interval ( 1000)ms [ CSSD]2013-08-07 06:11:58.143 [6] >TRACE: clssnmReadDskHeartbeat: node(1) is down. rcfg(22) wrtcnt(6994935) LATS(180393) Disk lastSeqNo(6994935) [ CSSD]2013-08-07 06:11:58.172 [9] >TRACE: clssnmClusterListener: Listening on (ADDRESS=(PROTOCOL=tcp)(HOST=nr-del-racdb2-mts-priv)(PORT=49895)) [ CSSD]2013-08-07 06:11:58.174 [9] >TRACE: clssnmconnect: connecting to node(1), con(100684fd0), flags 0x0003 [ CSSD]2013-08-07 06:11:58.188 [10] >TRACE: clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLstnr_crs_2)) [ CSSD]2013-08-07 06:11:58.188 [10] >TRACE: clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_nr-del-racdb2-mts_crs))