[Logo] Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)
  [Search] Search   [Recent Topics] Recent Topics   [Members]  Member Listing   [Groups] Back to home page 
[Register] Register / 
[Login] Login 
[Expert]
Need help on Deadlock...  XML
Forum Index -> General
Author Message
Shake

master

Joined: 01/05/2008 07:27:49
Messages: 84
Offline

Hi,

I suddently got that dead lock (after several weeks without any problems).
But I don't really have an idea where the problem might be. So does anyone has any suggestions?

I killed all clients but one, but the dead lock still existed...

Code:
"Database Server" prio=10 tid=0x090c4c00 nid=0x2c9 in Object.wait() [0x511ad000..0x511adf40]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at java.lang.Object.wait(Object.java:485)
 	at com.tc.object.lockmanager.impl.ClientLockManagerImpl.waitUntilRunning(ClientLockManagerImpl.java:567)
 	at com.tc.object.lockmanager.impl.ClientLockManagerImpl.lock(ClientLockManagerImpl.java:294)
 	- locked <0x575a4690> (a com.tc.object.lockmanager.impl.ClientLockManagerImpl)
 	at com.tc.object.lockmanager.impl.ThreadLockManagerImpl.lock(ThreadLockManagerImpl.java:46)
 	at com.tc.object.tx.ClientTransactionManagerImpl.begin(ClientTransactionManagerImpl.java:187)
 	at com.tc.object.bytecode.ManagerImpl.begin(ManagerImpl.java:337)
 	at com.tc.object.bytecode.ManagerImpl.monitorEnter(ManagerImpl.java:507)
 	at com.tc.object.bytecode.ManagerImpl.monitorEnter(ManagerImpl.java:495)
 	at com.tc.object.bytecode.ManagerUtil.monitorEnter(ManagerUtil.java:485)
 	at java.util.concurrent.locks.ReentrantReadWriteLock$DsoLock.lock(ReentrantReadWriteLock/java:35)
 	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock/java)
 	at eu.cedarsoft.history.ElementsCollection.getElements(ElementsCollection.java:110)


Full stack traces:

Code:
"ConnectionEstablisher" daemon prio=10 tid=0x09133c00 nid=0x500f in Object.wait() [0x50d5a000..0x50d5b040]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at com.tc.util.concurrent.TCFuture.get(TCFuture.java:88)
 	- locked <0x576161f8> (a com.tc.net.protocol.transport.MessageTransportStatus)
 	at com.tc.util.concurrent.TCFuture.get(TCFuture.java:64)
 	at com.tc.net.protocol.transport.ClientMessageTransport.waitForSynAck(ClientMessageTransport.java:219)
 	at com.tc.net.protocol.transport.ClientMessageTransport.handShake(ClientMessageTransport.java:213)
 	at com.tc.net.protocol.transport.ClientMessageTransport.reconnect(ClientMessageTransport.java:262)
 	at com.tc.net.protocol.transport.ClientConnectionEstablisher.reconnect(ClientConnectionEstablisher.java:153)
 	at com.tc.net.protocol.transport.ClientConnectionEstablisher.access$100(ClientConnectionEstablisher.java:29)
 	at com.tc.net.protocol.transport.ClientConnectionEstablisher$AsyncReconnect.run(ClientConnectionEstablisher.ja
 va:274)
 	at java.lang.Thread.run(Thread.java:619)
 
 "AsyncWorkerThread AsynchroniousDao-CallbackCaller for HibernateDao for: de.avr.id.LastGivenExternalId. 19618574 25149
 781" prio=10 tid=0x08c39400 nid=0x2d2 in Object.wait() [0x500ad000..0x500adec0]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	- waiting on <0x57dad348> (a java.util.LinkedList)
 	at java.lang.Object.wait(Object.java:485)
 	at com.tc.object.lockmanager.impl.ClientLock.waitForLock(ClientLock.java:619)
 	- locked <0x57dad348> (a java.util.LinkedList)
 	at com.tc.object.lockmanager.impl.ClientLock.wait(ClientLock.java:353)
 	at com.tc.object.lockmanager.impl.ClientLockManagerImpl.wait(ClientLockManagerImpl.java:361)
 	at com.tc.object.lockmanager.impl.ThreadLockManagerImpl.wait(ThreadLockManagerImpl.java:54)
 	at com.tc.object.tx.ClientTransactionManagerImpl.wait(ClientTransactionManagerImpl.java:227)
 	at com.tc.object.bytecode.ManagerImpl.objectWait0(ManagerImpl.java:416)
 	at com.tc.object.bytecode.ManagerUtil.objectWait0(ManagerUtil.java:454)
 	at eu.cedarsoft.async.AsyncCallSupport.waitForNextCallback(AsyncCallSupport.java:158)
 	- locked <0x57dad348> (a java.util.LinkedList)
 	at eu.cedarsoft.async.AsyncCallSupport$AsyncWorker.run(AsyncCallSupport.java:255)
 	at java.lang.Thread.run(Thread.java:619)
 
 "AsyncWorkerThread AsynchroniousDao-CallbackCaller for HibernateDao for: de.avr.contact.StaffGroup. 1109242 24081726" 
 prio=10 tid=0x09786800 nid=0x2d1 in Object.wait() [0x500fe000..0x500fef40]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	- waiting on <0x57db6ce0> (a java.util.LinkedList)
 	at java.lang.Object.wait(Object.java:485)
 	at com.tc.object.lockmanager.impl.ClientLock.waitForLock(ClientLock.java:619)
 	- locked <0x57db6ce0> (a java.util.LinkedList)
 	at com.tc.object.lockmanager.impl.ClientLock.wait(ClientLock.java:353)
 	at com.tc.object.lockmanager.impl.ClientLockManagerImpl.wait(ClientLockManagerImpl.java:361)
 	at com.tc.object.lockmanager.impl.ThreadLockManagerImpl.wait(ThreadLockManagerImpl.java:54)
 	at com.tc.object.tx.ClientTransactionManagerImpl.wait(ClientTransactionManagerImpl.java:227)
 	at com.tc.object.bytecode.ManagerImpl.objectWait0(ManagerImpl.java:416)
 	at com.tc.object.bytecode.ManagerUtil.objectWait0(ManagerUtil.java:454)
 	at eu.cedarsoft.async.AsyncCallSupport.waitForNextCallback(AsyncCallSupport.java:158)
 	- locked <0x57db6ce0> (a java.util.LinkedList)
 	at eu.cedarsoft.async.AsyncCallSupport$AsyncWorker.run(AsyncCallSupport.java:255)
 	at java.lang.Thread.run(Thread.java:619)
 
 "AsyncWorkerThread AsynchroniousDao-CallbackCaller for HibernateDao for: de.avr.appointment.AbstractCalendar. 2458662 
 22103325" prio=10 tid=0x09786400 nid=0x2d0 in Object.wait() [0x5075c000..0x5075cdc0]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at java.lang.Object.wait(Object.java:485)
 	at com.tc.object.lockmanager.impl.ClientLock.waitForLock(ClientLock.java:619)
 	- locked <0x57db7eb0> (a java.util.LinkedList)
 	at com.tc.object.lockmanager.impl.ClientLock.wait(ClientLock.java:353)
 	at com.tc.object.lockmanager.impl.ClientLockManagerImpl.wait(ClientLockManagerImpl.java:361)
 	at com.tc.object.lockmanager.impl.ThreadLockManagerImpl.wait(ThreadLockManagerImpl.java:54)
 	at com.tc.object.tx.ClientTransactionManagerImpl.wait(ClientTransactionManagerImpl.java:227)
 	at com.tc.object.bytecode.ManagerImpl.objectWait0(ManagerImpl.java:416)
 	at com.tc.object.bytecode.ManagerUtil.objectWait0(ManagerUtil.java:454)
 	at eu.cedarsoft.async.AsyncCallSupport.waitForNextCallback(AsyncCallSupport.java:158)
 	- locked <0x57db7eb0> (a java.util.LinkedList)
 	at eu.cedarsoft.async.AsyncCallSupport$AsyncWorker.run(AsyncCallSupport.java:255)
 	at java.lang.Thread.run(Thread.java:619)
 
 "AsyncWorkerThread AsynchroniousDao-CallbackCaller for HibernateDao for: eu.cedarsoft.business.contact.City. 22670902 
 14444682" prio=10 tid=0x08d81400 nid=0x2cf in Object.wait() [0x507ad000..0x507ade40]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at java.lang.Object.wait(Object.java:485)
 	at com.tc.object.lockmanager.impl.ClientLock.waitForLock(ClientLock.java:619)
 	- locked <0x57db8e20> (a java.util.LinkedList)
 	at com.tc.object.lockmanager.impl.ClientLock.wait(ClientLock.java:353)
 	at com.tc.object.lockmanager.impl.ClientLockManagerImpl.wait(ClientLockManagerImpl.java:361)
 	at com.tc.object.lockmanager.impl.ThreadLockManagerImpl.wait(ThreadLockManagerImpl.java:54)
 	at com.tc.object.tx.ClientTransactionManagerImpl.wait(ClientTransactionManagerImpl.java:227)
 	at com.tc.object.bytecode.ManagerImpl.objectWait0(ManagerImpl.java:416)
 	at com.tc.object.bytecode.ManagerUtil.objectWait0(ManagerUtil.java:454)
 	at eu.cedarsoft.async.AsyncCallSupport.waitForNextCallback(AsyncCallSupport.java:158)
 	- locked <0x57db8e20> (a java.util.LinkedList)
 	at eu.cedarsoft.async.AsyncCallSupport$AsyncWorker.run(AsyncCallSupport.java:255)
 	at java.lang.Thread.run(Thread.java:619)
 
 "AsyncWorkerThread AsynchroniousDao-CallbackCaller for HibernateDao for: eu.cedarsoft.business.Bank. 20725235 21230798
 " prio=10 tid=0x08d81000 nid=0x2ce in Object.wait() [0x507fe000..0x507ff0c0]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at java.lang.Object.wait(Object.java:485)
 	at com.tc.object.lockmanager.impl.ClientLock.waitForLock(ClientLock.java:619)
 	- locked <0x57dba248> (a java.util.LinkedList)
 	at com.tc.object.lockmanager.impl.ClientLock.wait(ClientLock.java:353)
 	at com.tc.object.lockmanager.impl.ClientLockManagerImpl.wait(ClientLockManagerImpl.java:361)
 	at com.tc.object.lockmanager.impl.ThreadLockManagerImpl.wait(ThreadLockManagerImpl.java:54)
 	at com.tc.object.tx.ClientTransactionManagerImpl.wait(ClientTransactionManagerImpl.java:227)
 	at com.tc.object.bytecode.ManagerImpl.objectWait0(ManagerImpl.java:416)
 	at com.tc.object.bytecode.ManagerUtil.objectWait0(ManagerUtil.java:454)
 	at eu.cedarsoft.async.AsyncCallSupport.waitForNextCallback(AsyncCallSupport.java:158)
 	- locked <0x57dba248> (a java.util.LinkedList)
 	at eu.cedarsoft.async.AsyncCallSupport$AsyncWorker.run(AsyncCallSupport.java:255)
 	at java.lang.Thread.run(Thread.java:619)
 
 "AsyncWorkerThread AsynchroniousDao-CallbackCaller for HibernateDao for: de.avr.contact.AvrContact. 26811705 15864550"
  prio=10 tid=0x09018400 nid=0x2cd in Object.wait() [0x50dad000..0x50dae140]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at java.lang.Object.wait(Object.java:485)
 	at com.tc.object.lockmanager.impl.ClientLock.waitForLock(ClientLock.java:619)
 	- locked <0x57dbb1c8> (a java.util.LinkedList)
 	at com.tc.object.lockmanager.impl.ClientLock.wait(ClientLock.java:353)
 	at com.tc.object.lockmanager.impl.ClientLockManagerImpl.wait(ClientLockManagerImpl.java:361)
 	at com.tc.object.lockmanager.impl.ThreadLockManagerImpl.wait(ThreadLockManagerImpl.java:54)
 	at com.tc.object.tx.ClientTransactionManagerImpl.wait(ClientTransactionManagerImpl.java:227)
 	at com.tc.object.bytecode.ManagerImpl.objectWait0(ManagerImpl.java:416)
 	at com.tc.object.bytecode.ManagerUtil.objectWait0(ManagerUtil.java:454)
 	at eu.cedarsoft.async.AsyncCallSupport.waitForNextCallback(AsyncCallSupport.java:158)
 	- locked <0x57dbb1c8> (a java.util.LinkedList)
 	at eu.cedarsoft.async.AsyncCallSupport$AsyncWorker.run(AsyncCallSupport.java:255)
 	at java.lang.Thread.run(Thread.java:619)
 
 "AsyncWorkerThread AsynchroniousDao-CallbackCaller for HibernateDao for: de.avr.contracts.Contract. 7045653 24132223" 
 prio=10 tid=0x09017c00 nid=0x2cc in Object.wait() [0x50dfe000..0x50dfefc0]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at java.lang.Object.wait(Object.java:485)
 	at com.tc.object.lockmanager.impl.ClientLock.waitForLock(ClientLock.java:619)
 	- locked <0x57dbc5e0> (a java.util.LinkedList)
 	at com.tc.object.lockmanager.impl.ClientLock.wait(ClientLock.java:353)
 	at com.tc.object.lockmanager.impl.ClientLockManagerImpl.wait(ClientLockManagerImpl.java:361)
 	at com.tc.object.lockmanager.impl.ThreadLockManagerImpl.wait(ThreadLockManagerImpl.java:54)
 	at com.tc.object.tx.ClientTransactionManagerImpl.wait(ClientTransactionManagerImpl.java:227)
 	at com.tc.object.bytecode.ManagerImpl.objectWait0(ManagerImpl.java:416)
 	at com.tc.object.bytecode.ManagerUtil.objectWait0(ManagerUtil.java:454)
 	at eu.cedarsoft.async.AsyncCallSupport.waitForNextCallback(AsyncCallSupport.java:158)
 	- locked <0x57dbc5e0> (a java.util.LinkedList)
 	at eu.cedarsoft.async.AsyncCallSupport$AsyncWorker.run(AsyncCallSupport.java:255)
 	at java.lang.Thread.run(Thread.java:619)
 
 "AsyncWorkerThread AsynchroniousDao-CallbackCaller for HibernateDao for: de.avr.contracts.ContractBasket. 8117065 7013
 877" prio=10 tid=0x092cf800 nid=0x2cb in Object.wait() [0x5115c000..0x5115d040]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at java.lang.Object.wait(Object.java:485)
 	at com.tc.object.lockmanager.impl.ClientLock.waitForLock(ClientLock.java:619)
 	- locked <0x57dbd8e8> (a java.util.LinkedList)
 	at com.tc.object.lockmanager.impl.ClientLock.wait(ClientLock.java:353)
 	at com.tc.object.lockmanager.impl.ClientLockManagerImpl.wait(ClientLockManagerImpl.java:361)
 	at com.tc.object.lockmanager.impl.ThreadLockManagerImpl.wait(ThreadLockManagerImpl.java:54)
 	at com.tc.object.tx.ClientTransactionManagerImpl.wait(ClientTransactionManagerImpl.java:227)
 	at com.tc.object.bytecode.ManagerImpl.objectWait0(ManagerImpl.java:416)
 	at com.tc.object.bytecode.ManagerUtil.objectWait0(ManagerUtil.java:454)
 	at eu.cedarsoft.async.AsyncCallSupport.waitForNextCallback(AsyncCallSupport.java:158)
 	- locked <0x57dbd8e8> (a java.util.LinkedList)
 	at eu.cedarsoft.async.AsyncCallSupport$AsyncWorker.run(AsyncCallSupport.java:255)
 	at java.lang.Thread.run(Thread.java:619)
 
 "MySQL Statement Cancellation Timer" daemon prio=10 tid=0x09da7800 nid=0x2ca in Object.wait() [0x52050000..0x52050ec0]
 
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	- waiting on <0x578b4440> (a java.util.TaskQueue)
 	at java.lang.Object.wait(Object.java:485)
 	at java.util.TimerThread.mainLoop(Timer.java:483)
 	- locked <0x578b4440> (a java.util.TaskQueue)
 	at java.util.TimerThread.run(Timer.java:462)
 
 "Database Server" prio=10 tid=0x090c4c00 nid=0x2c9 in Object.wait() [0x511ad000..0x511adf40]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at java.lang.Object.wait(Object.java:485)
 	at com.tc.object.lockmanager.impl.ClientLockManagerImpl.waitUntilRunning(ClientLockManagerImpl.java:567)
 	at com.tc.object.lockmanager.impl.ClientLockManagerImpl.lock(ClientLockManagerImpl.java:294)
 	- locked <0x575a4690> (a com.tc.object.lockmanager.impl.ClientLockManagerImpl)
 	at com.tc.object.lockmanager.impl.ThreadLockManagerImpl.lock(ThreadLockManagerImpl.java:46)
 	at com.tc.object.tx.ClientTransactionManagerImpl.begin(ClientTransactionManagerImpl.java:187)
 	at com.tc.object.bytecode.ManagerImpl.begin(ManagerImpl.java:337)
 	at com.tc.object.bytecode.ManagerImpl.monitorEnter(ManagerImpl.java:507)
 	at com.tc.object.bytecode.ManagerImpl.monitorEnter(ManagerImpl.java:495)
 	at com.tc.object.bytecode.ManagerUtil.monitorEnter(ManagerUtil.java:485)
 	at java.util.concurrent.locks.ReentrantReadWriteLock$DsoLock.lock(ReentrantReadWriteLock/java:35)
 	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock/java)
 	at eu.cedarsoft.history.ElementsCollection.getElements(ElementsCollection.java:110)
 	at eu.cedarsoft.gdao.caching.CachingDao.findAll(CachingDao.java:120)
 	at eu.cedarsoft.gdao.AbstractGenericDao.getElements(AbstractGenericDao.java:25)
 	at eu.cedarsoft.history.DelegatingObservableObjectAccess.getElements(DelegatingObservableObjectAccess.java:25)
 
 	at de.avr.AbstractObjectManager.getContracts(AbstractObjectManager.java:920)
 	at de.avr.CommitAllAction.execute(CommitAllAction.java:38)
 	at de.avr.ConsoleServer.run(ConsoleServer.java:66)
 	at de.avr.DatabaseServer$3.run(DatabaseServer.java:84)
 	at java.lang.Thread.run(Thread.java:619)
 
 "Thread-11" daemon prio=10 tid=0x08bd1c00 nid=0x2bf in Object.wait() [0x51566000..0x51567040]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at java.lang.Object.wait(Object.java:485)
 	at com.tc.management.remote.protocol.terracotta.TunnelingEventHandler.accept(TunnelingEventHandler.java:84)
 	- locked <0x575a6868> (a com.tc.management.remote.protocol.terracotta.TunnelingEventHandler)
 	at com.tc.management.remote.protocol.terracotta.TunnelingMessageConnectionServer.accept(TunnelingMessageConnec
 tionServer.java:31)
 	at com.sun.jmx.remote.generic.SynchroMessageConnectionServerImpl.accept(SynchroMessageConnectionServerImpl.jav
 a:47)
 	at javax.management.remote.generic.GenericConnectorServer$Receiver.run(GenericConnectorServer.java:340)
 
 "Timer-3" daemon prio=10 tid=0x08bd4400 nid=0x2be in Object.wait() [0x515b7000..0x515b7ec0]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	- waiting on <0x57613590> (a java.util.TaskQueue)
 	at java.lang.Object.wait(Object.java:485)
 	at java.util.TimerThread.mainLoop(Timer.java:483)
 	- locked <0x57613590> (a java.util.TaskQueue)
 	at java.util.TimerThread.run(Timer.java:462)
 
 "Low Memory Detector" daemon prio=10 tid=0x51e42000 nid=0x2bd runnable [0x00000000..0x00000000]
    java.lang.Thread.State: RUNNABLE
 
 "CompilerThread1" daemon prio=10 tid=0x51e37000 nid=0x2bc waiting on condition [0x00000000..0x51689418]
    java.lang.Thread.State: RUNNABLE
 
 "CompilerThread0" daemon prio=10 tid=0x51e37c00 nid=0x2bb waiting on condition [0x00000000..0x5170a498]
    java.lang.Thread.State: RUNNABLE
 
 "Signal Dispatcher" daemon prio=10 tid=0x51e5b800 nid=0x2ba waiting on condition [0x00000000..0x00000000]
    java.lang.Thread.State: RUNNABLE
 
 "WorkerThread(client_coordination_stage,0)" daemon prio=10 tid=0x51e5a400 nid=0x2b9 in Object.wait() [0x517cb000..0x51
 7cc140]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(Unknown Source)
 	- locked <0x575a6780> (a java.lang.Object)
 	at com.tc.util.concurrent.TCBoundedLinkedQueue.poll(TCBoundedLinkedQueue.java:29)
 	at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
 	at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
 
 "WorkerThread(lock_response_stage,0)" daemon prio=10 tid=0x51e59400 nid=0x2b8 in Object.wait() [0x5181c000..0x5181cfc0
 ]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(Unknown Source)
 	- locked <0x575a74f8> (a java.lang.Object)
 	at com.tc.util.concurrent.TCBoundedLinkedQueue.poll(TCBoundedLinkedQueue.java:29)
 	at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
 	at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
 
 "WorkerThread(hydrate_message_stage,0)" daemon prio=10 tid=0x51e58000 nid=0x2b7 in Object.wait() [0x5186d000..0x5186e0
 40]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(Unknown Source)
 	- locked <0x575a7318> (a java.lang.Object)
 	at com.tc.util.concurrent.TCBoundedLinkedQueue.poll(TCBoundedLinkedQueue.java:29)
 	at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
 	at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
 
 "WorkerThread(receive_transaction_complete_stage,0)" daemon prio=10 tid=0x51e57000 nid=0x2b6 in Object.wait() [0x518be
 000..0x518beec0]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(Unknown Source)
 	- locked <0x575a7e68> (a java.lang.Object)
 	at com.tc.util.concurrent.TCBoundedLinkedQueue.poll(TCBoundedLinkedQueue.java:29)
 	at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
 	at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
 
 "WorkerThread(dmi_stage,0)" daemon prio=10 tid=0x51e67400 nid=0x2b5 in Object.wait() [0x5190f000..0x5190ff40]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(Unknown Source)
 	- locked <0x575a8ef8> (a java.lang.Object)
 	at com.tc.util.concurrent.TCBoundedLinkedQueue.poll(TCBoundedLinkedQueue.java:29)
 	at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
 	at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
 
 "WorkerThread(batch_txn_ack_stage,0)" daemon prio=10 tid=0x51e66400 nid=0x2b4 in Object.wait() [0x51960000..0x51960dc0
 ]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(Unknown Source)
 	- locked <0x575a7a70> (a java.lang.Object)
 	at com.tc.util.concurrent.TCBoundedLinkedQueue.poll(TCBoundedLinkedQueue.java:29)
 	at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
 	at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
 
 "WorkerThread(lock_statistics_response_stage,0)" daemon prio=10 tid=0x51e65000 nid=0x2b3 in Object.wait() [0x519b1000.
 .0x519b1e40]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(Unknown Source)
 	- locked <0x575a8618> (a java.lang.Object)
 	at com.tc.util.concurrent.TCBoundedLinkedQueue.poll(TCBoundedLinkedQueue.java:29)
 	at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
 	at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
 
 "WorkerThread(jmxremote_tunnel_stage,0)" daemon prio=10 tid=0x51e62800 nid=0x2b2 in Object.wait() [0x51a02000..0x51a03
 0c0]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(Unknown Source)
 	- locked <0x575a7638> (a java.lang.Object)
 	at com.tc.util.concurrent.TCBoundedLinkedQueue.poll(TCBoundedLinkedQueue.java:29)
 	at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
 	at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
 
 "WorkerThread(receive_object_stage,0)" daemon prio=10 tid=0x51e61800 nid=0x2b1 in Object.wait() [0x51a53000..0x51a5414
 0]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(Unknown Source)
 	- locked <0x575a78e0> (a java.lang.Object)
 	at com.tc.util.concurrent.TCBoundedLinkedQueue.poll(TCBoundedLinkedQueue.java:29)
 	at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
 	at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
 
 "WorkerThread(receive_root_id_stage,0)" daemon prio=10 tid=0x51e10800 nid=0x2b0 in Object.wait() [0x51aa4000..0x51aa4f
 c0]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(Unknown Source)
 	- locked <0x575a7778> (a java.lang.Object)
 	at com.tc.util.concurrent.TCBoundedLinkedQueue.poll(TCBoundedLinkedQueue.java:29)
 	at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
 	at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
 
 "WorkerThread(object_id_request_response_stage,0)" daemon prio=10 tid=0x51e0fc00 nid=0x2af in Object.wait() [0x51af500
 0..0x51af6040]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(Unknown Source)
 	- locked <0x575a7d28> (a java.lang.Object)
 	at com.tc.util.concurrent.TCBoundedLinkedQueue.poll(TCBoundedLinkedQueue.java:29)
 	at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
 	at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
 
 "WorkerThread(lock_statistics_enable_disable_stage,0)" daemon prio=10 tid=0x51e2b400 nid=0x2ae in Object.wait() [0x51b
 46000..0x51b46ec0]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(Unknown Source)
 	- locked <0x575a7c00> (a java.lang.Object)
 	at com.tc.util.concurrent.TCBoundedLinkedQueue.poll(TCBoundedLinkedQueue.java:29)
 	at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
 	at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
 
 "WorkerThread(receive_transaction_stage,0)" daemon prio=10 tid=0x51e23000 nid=0x2ad in Object.wait() [0x51b97000..0x51
 b97f40]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(Unknown Source)
 	- locked <0x575a7200> (a java.lang.Object)
 	at com.tc.util.concurrent.TCBoundedLinkedQueue.poll(TCBoundedLinkedQueue.java:29)
 	at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
 	at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
 
 "Reaper" daemon prio=10 tid=0x51e28800 nid=0x2ab in Object.wait() [0x51c39000..0x51c39e40]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
 	- locked <0x575a52b8> (a java.lang.ref.ReferenceQueue$Lock)
 	at com.tc.object.ClientObjectManagerImpl$2.run(ClientObjectManagerImpl.java:1200)
 
 "H2 Log Writer STATISTICS-BUFFER" daemon prio=10 tid=0x51e1f000 nid=0x2aa sleeping[0x51cba000..0x51cbb0c0]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
 	at java.lang.Thread.sleep(Native Method)
 	at org.h2.store.WriterThread.run(WriterThread.java:147)
 
 "H2 File Lock Watchdog /home/avr/avr/live/databaseServer/statistics-127.0.1.1/statistics-buffer.lock.db" daemon prio=1
 0 tid=0x51e0d400 nid=0x2a9 sleeping[0x51d0c000..0x51d0c140]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
 	at java.lang.Thread.sleep(Native Method)
 	at org.h2.store.FileLock$1.run(FileLock.java:198)
 	at java.lang.Thread.run(Thread.java:619)
 
 "com.tc.util.ToggleableReferenceManager$QueueProcessor" daemon prio=10 tid=0x51e08400 nid=0x2a7 in Object.wait() [0x51
 d5c000..0x51d5d040]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
 	- locked <0x575a63f0> (a java.lang.ref.ReferenceQueue$Lock)
 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
 	at com.tc.util.ToggleableReferenceManager$QueueProcessor.run(ToggleableReferenceManager.java:129)
 
 "DSO Lock Object.wait() timer" daemon prio=10 tid=0x51e03c00 nid=0x2a6 in Object.wait() [0x51dad000..0x51dadec0]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at java.lang.Object.wait(Object.java:485)
 	at com.tc.object.lockmanager.impl.ClientLockManagerImpl.waitUntilRunning(ClientLockManagerImpl.java:567)
 	at com.tc.object.lockmanager.impl.ClientLockManagerImpl.runGC(ClientLockManagerImpl.java:121)
 	- locked <0x575a4690> (a com.tc.object.lockmanager.impl.ClientLockManagerImpl)
 	at com.tc.object.lockmanager.impl.ClientLockManagerImpl$LockGCTask.run(ClientLockManagerImpl.java:631)
 	at java.util.TimerThread.mainLoop(Timer.java:512)
 	at java.util.TimerThread.run(Timer.java:462)
 
 "RemoteTransactionManager Flusher" daemon prio=10 tid=0x51e00800 nid=0x2a5 in Object.wait() [0x51dfe000..0x51dfef40]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at java.lang.Object.wait(Object.java:485)
 	at com.tc.object.tx.RemoteTransactionManagerImpl.waitUntilRunning(RemoteTransactionManagerImpl.java:440)
 	at com.tc.object.tx.RemoteTransactionManagerImpl.getCompletedTransactionIDLowWaterMark(RemoteTransactionManage
 rImpl.java:401)
 	- locked <0x575a5380> (a java.lang.Object)
 	at com.tc.object.tx.RemoteTransactionManagerImpl.access$100(RemoteTransactionManagerImpl.java:43)
 	at com.tc.object.tx.RemoteTransactionManagerImpl$RemoteTransactionManagerTimerTask.run(RemoteTransactionManage
 rImpl.java:461)
 	at java.util.TimerThread.mainLoop(Timer.java:512)
 	at java.util.TimerThread.run(Timer.java:462)
 
 "HealthChecker" daemon prio=10 tid=0x08379800 nid=0x2a4 sleeping[0x51f5d000..0x51f5ddc0]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
 	at java.lang.Thread.sleep(Native Method)
 	at java.lang.Thread.sleep(Thread.java:298)
 	at com.tc.util.concurrent.ThreadUtil.reallySleep(ThreadUtil.java:24)
 	at com.tc.util.concurrent.ThreadUtil.reallySleep(ThreadUtil.java:16)
 	at com.tc.net.protocol.transport.ConnectionHealthCheckerImpl$HealthCheckerMonitorThreadEngine.run(ConnectionHe
 althCheckerImpl.java:183)
 	at java.lang.Thread.run(Thread.java:619)
 
 "TCComm Main Selector Thread" daemon prio=10 tid=0x08373800 nid=0x2a3 runnable [0x51fae000..0x51faee40]
    java.lang.Thread.State: RUNNABLE
 	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
 	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:184)
 	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
 	- locked <0x57615e10> (a sun.nio.ch.Util$1)
 	- locked <0x57615e00> (a java.util.Collections$UnmodifiableSet)
 	- locked <0x57614690> (a sun.nio.ch.EPollSelectorImpl)
 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
 	at com.tc.net.core.CoreNIOServices.selectLoop(CoreNIOServices.java:338)
 	at com.tc.net.core.CoreNIOServices.run(CoreNIOServices.java:61)
 
 "SampledCounterManager Timer" daemon prio=10 tid=0x08516400 nid=0x2a2 in Object.wait() [0x51fff000..0x520000c0]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at java.util.TimerThread.mainLoop(Timer.java:509)
 	- locked <0x57614098> (a java.util.TaskQueue)
 	at java.util.TimerThread.run(Timer.java:462)
 
 "startlevel job thread" daemon prio=10 tid=0x08bbe400 nid=0x2a0 in Object.wait() [0x520b5000..0x520b5fc0]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at org.knopflerfish.framework.Queue.removeWait(Queue.java:108)
 	- locked <0x574bdc98> (a org.knopflerfish.framework.Queue)
 	at org.knopflerfish.framework.StartLevelImpl.run(StartLevelImpl.java:170)
 	at java.lang.Thread.run(Thread.java:619)
 
 "Finalizer" daemon prio=10 tid=0x080f2400 nid=0x29f in Object.wait() [0x52720000..0x52721040]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
 	- locked <0x570b38c0> (a java.lang.ref.ReferenceQueue$Lock)
 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
 
 "Reference Handler" daemon prio=10 tid=0x080f1c00 nid=0x29e in Object.wait() [0x52771000..0x52771ec0]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at java.lang.Object.wait(Object.java:485)
 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
 	- locked <0x570b38e0> (a java.lang.ref.Reference$Lock)
 
 "main" prio=10 tid=0x08058c00 nid=0x29a in Object.wait() [0xb7e2a000..0xb7e2b218]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	- waiting on <0x576f7408> (a java.lang.Object)
 	at java.lang.Object.wait(Object.java:485)
 	at com.tc.object.bytecode.ManagerImpl.objectWait0(ManagerImpl.java:423)
 	at com.tc.object.bytecode.ManagerUtil.objectWait0(ManagerUtil.java:454)
 	at de.avr.DatabaseServer.main(DatabaseServer.java:110)
 	- locked <0x576f7408> (a java.lang.Object)
 
amiller

ophanim

Joined: 08/29/2007 09:05:48
Messages: 722
Location: St. Louis, MO
Offline

What version of Terracotta?

What's in your tc-config?

What is happening on the Terracotta server? Is it doing anything? Do you have a server thread dump?

Alex Miller (Terracotta Engineer) - Want to post to the forums? Sign up here
[WWW]
teck

seraphim
[Avatar]
Joined: 05/24/2006 15:03:25
Messages: 1128
Offline

I second amiller's questions...

From those stack traces it looks like your process has been disconnected from the terracotta server. So some possibilities are that the server is no longer running, or there is a partition in the network, or the server is actively refusing reconnection.

Info from the server (thread dump and terracotta log file) should provide more information.

THe terracotta log file on client side might have some information as well. In particular I'm wondering if our health check decided that the TC server was down


Tim Eck (terracotta engineer)
Shake

master

Joined: 01/05/2008 07:27:49
Messages: 84
Offline

I am using terracotta-2.6.2.
I think it shouldn't be a network problem since the server and client run on the same host...

Sometimes I get some "Unable to find communications stack" messages. Don't know why. But I didn't care until now.
It shouldn't be caused by an old client because the Terracotta server has been up for several weeks now...

The terracotta server was still running (could quit it with Ctrl-C).


tc-config.xml:
Code:
<?xml version="1.0" encoding="UTF-8"?>
 <tc:tc-config xmlns:tc="http://www.terracotta.org/config"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="http://www.terracotta.org/config http://www.terracotta.org/schema/terracotta-4.xsd">
 
   <servers>
     <server host="192.168.123.100">
     <data>/home/avr/avr/live/data</data>
     <logs>/home/avr/avr/live/logs</logs>
     <dso>
     <persistence>
     <mode>temporary-swap-only</mode>
     </persistence>
     </dso>
     </server>
     <!--server host="192.168.0.158">
       <data>/tmp/avr/data</data>
       <logs>/tmp/avr/logs</logs>
       <dso>
         <persistence>
           <mode>temporary-swap-only</mode>
         </persistence>
       </dso>
     </server-->
   </servers>
 
   <clients>
     <modules>
       <module name="tim-annotations" version="1.0.3" />
       <module name="tim-hibernate-3.2.5" version="1.1.2" />
     </modules>
 
     <dso>
       <debugging>
         <instrumentation-logging>
           <!--<class>true</class>-->
           <!--<hierarchy>true</hierarchy>-->
           <!--<locks>true</locks>-->
           <transient-root>true</transient-root>
           <roots>true</roots>
           <distributed-methods>true</distributed-methods>
         </instrumentation-logging>
 
         <runtime-logging>
           <lock-debug>false</lock-debug>
           <!--<field-change-debug>true</field-change-debug>-->
           <!--<wait-notify-debug>true</wait-notify-debug>-->
           <!--<distributed-method-debug>true</distributed-method-debug>-->
           <!--<new-object-debug>true</new-object-debug>-->
           <non-portable-dump>false</non-portable-dump>
         </runtime-logging>
 
         <runtime-output-options>
           <auto-lock-details>true</auto-lock-details>
           <caller>true</caller>
           <full-stack>false</full-stack>
         </runtime-output-options>
       </debugging>
     </dso>
 
     <logs>logs/terracotta/client/%h/%D</logs>
   </clients>
 
   <application>
     <dso>
       <locks>
         <!--<autolock>-->
         <!--<method-expression>* eu.cedarsoft..*.*(..)</method-expression>-->
         <!--<lock-level>write</lock-level>-->
         <!--</autolock>-->
 
         <autolock>
           <method-expression>* eu.cedarsoft.async.AsyncCallSupport.invokeNullable(..)</method-expression>
           <lock-level>write</lock-level>
         </autolock>
         <autolock>
           <method-expression>* eu.cedarsoft.async.AsyncCallSupport.waitForNextCallback()</method-expression>
           <lock-level>write</lock-level>
         </autolock>
         <autolock>
           <method-expression>* eu.cedarsoft.async.AsyncCallSupport.acknowledge(..)</method-expression>
           <lock-level>write</lock-level>
         </autolock>
         <autolock>
           <method-expression>* eu.cedarsoft.async.AsyncCallSupport$AsyncWorker.run()</method-expression>
           <lock-level>write</lock-level>
         </autolock>
 
 
         <!-- Useage of synchronized statements -->
         <autolock>
           <method-expression>* de.avr.imports.access.AccessImporter.addContactInformations(..)</method-expression>
           <lock-level>write</lock-level>
         </autolock>
 
         <autolock>
           <method-expression>* de.avr.ui.dialogs.AbstractPropertiesDialog.onFinish(..)</method-expression>
           <lock-level>write</lock-level>
         </autolock>
 
         <autolock>
           <method-expression>* de.avr.imports.HsbcImporter.importInsuredPersons(..)</method-expression>
           <lock-level>write</lock-level>
         </autolock>
 
         <autolock>
           <method-expression>* de.avr.ui.FormPage.commit()</method-expression>
           <lock-level>write</lock-level>
         </autolock>
       </locks>
 
       <distributed-methods>
         <method-expression>void eu.cedarsoft.history.CollectionSupport.entryDeleted(Object)</method-expression>
         <method-expression>void eu.cedarsoft.history.CollectionSupport.entryAdded(Object)</method-expression>
         <method-expression>void eu.cedarsoft.history.CollectionSupport.entryChanged(Object)</method-expression>
 
         <method-expression>void eu.cedarsoft.history.HistoryListenerSupport.notifyEntryAdded(Object)</method-expression>
         <method-expression>void eu.cedarsoft.history.HistoryListenerSupport.notifyEntryRemoved(Object)
         </method-expression>
         <method-expression>void eu.cedarsoft.history.HistoryListenerSupport.notifyEntryChanged(Object)
         </method-expression>
 
         <method-expression>void eu.cedarsoft.history.ContinuousEntriesInformation.notifyEntryRemoved(Object)
         </method-expression>
         <method-expression>void eu.cedarsoft.history.ContinuousEntriesInformation.notifyEntryAdded(Object)
         </method-expression>
         <method-expression>void eu.cedarsoft.history.ContinuousEntriesInformation.notifyEntryChanged(Object)
         </method-expression>
 
         <method-expression>void eu.cedarsoft.utils.ChangeListenerSupport.changed(..)</method-expression>
 
         <!-- property change support -->
         <method-expression>void java.beans.PropertyChangeSupport.fire*(..)</method-expression>
       </distributed-methods>
 
       <instrumented-classes>
         <include>
           <class-expression>eu.cedarsoft..*..*</class-expression>
           <honor-transient>true</honor-transient>
         </include>
         <include>
           <class-expression>de.avr..*..*</class-expression>
           <honor-transient>true</honor-transient>
         </include>
 
         <!-- Anonymous inner classes -->
         <include>
           <class-expression>de.avr.AbstractObjectManager$*</class-expression>
           <honor-transient>true</honor-transient>
         </include>
 
         <include>
           <class-expression>de.avr.contracts.offer.PurchaseOffer$*</class-expression>
           <honor-transient>true</honor-transient>
         </include>
 
         <include>
           <class-expression>de.avr.contracts.offer.PurchaseOfferProvision$*</class-expression>
           <honor-transient>true</honor-transient>
         </include>
 
 
         <!-- Hibernate -->
         <include>
           <class-expression>org.hibernate.collection.Persistent*</class-expression>
         </include>
         <include>
           <class-expression>org.hibernate.collection.AbstractPersistentCollection</class-expression>
         </include>
 
         <!-- Exceptions -->
         <include>
           <class-expression>org.springframework..*Exception</class-expression>
         </include>
         <include>
           <class-expression>org.hibernate.*Exception</class-expression>
         </include>
         <include>
           <class-expression>org.hibernate.exception.*</class-expression>
         </include>
         <include>
           <class-expression>java.sql.*Exception</class-expression>
         </include>
         <include>
           <class-expression>java.lang.*Exception</class-expression>
         </include>
 
         <!-- Joda time -->
         <include>
           <class-expression>org.joda.time..*</class-expression>
           <honor-transient>false</honor-transient>
         </include>
 
         <!-- Property Change support -->
         <include>
           <class-expression>java.beans.PropertyChangeSupport</class-expression>
           <honor-transient>true</honor-transient>
         </include>
         <include>
           <class-expression>java.beans.PropertyChangeEvent</class-expression>
           <honor-transient>true</honor-transient>
         </include>
 
       </instrumented-classes>
 
 
       <additional-boot-jar-classes>
         <include>java.beans.PropertyChangeSupport</include>
         <include>java.beans.PropertyChangeEvent</include>
         <include>java.sql.SQLException</include>
         <include>java.sql.BatchUpdateException</include>
         <include>java.lang.NullPointerException</include>
       </additional-boot-jar-classes>
 
     </dso>
   </application>
 </tc:tc-config>
 
 Filename terracotta-client.log [Disk] Download
 Description Client log
 Filesize 7582 Kbytes
 Downloaded:  399 time(s)

 Filename terracotta-server.log.1 [Disk] Download
 Description Server log
 Filesize 590 Kbytes
 Downloaded:  254 time(s)

teck

seraphim
[Avatar]
Joined: 05/24/2006 15:03:25
Messages: 1128
Offline

So the client does think the server is dead:
2008-08-15 11:35:34,473 [HealthChecker] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Client - avrserver.local:9510 is DEAD
 
Of course the server wasn't dead (I can see that from the log, and you have confirned that fact as well).

So the question is why was it unresponsive beyond the timeout period for the health check? Usually the problem is a long GC in the terracotta server. Are you running the terracotta server with the stock start-tc-server.sh script, or have you changed it to increase the heap size? Any chance the host where all this is running had to do any swapping?

I recommend monitoring the heap for the terracota server process with -verbose:gc or jstat to make more data in case this happens again


Tim Eck (terracotta engineer)
teck

seraphim
[Avatar]
Joined: 05/24/2006 15:03:25
Messages: 1128
Offline

I forget to mention that you can disable the health check function if you don't ever want clients to disconnect themselves automatically.

-Dcom.tc.l1.healthcheck.l2.ping.enabled=false

Tim Eck (terracotta engineer)
steve

ophanim

Joined: 05/24/2006 14:22:53
Messages: 619
Offline

One small correction... Long GC in the server should not trip up the Health check. We can detect that. It's long GC's in the client that can lead to confusion for the health checker (we have some plans to look at that over
the next few months).

Want to post to this forum? Join the Terracotta Community
Shake

master

Joined: 01/05/2008 07:27:49
Messages: 84
Offline

Thanks for your help.

Yes - I have increased the memory for the TC server. And I will try the command line parameter "-Dcom.tc.l1.healthcheck.l2.ping.enabled=false".

Hope it helps....
 
Forum Index -> General
Go to:   
Powered by JForum 2.1.7 © JForum Team