Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

We are having issues related to the instability of terracotta client in terracotta cluster environment. Here is our system setup:

1. Terracotta server cluster using opensource version 3.6.0 (one master and one hot standby).

2. Two tomcat service connect to TC server cluster to maintain the cache through two different process (one is JMS listener and one is triggered by schedule - every 1 mins). Those two process will update one cache all day lone.

3. In all the cases, all the updates are occurred on node2, however what we found is that the node 2 terracotta client is dead when the update loads are high.

Here are some log messages we found from the system:

1. From terracotta server:

2012-11-19 17:09:05,685 [WorkerThread(managed_object_fault_stage, 0, 0)] WARN com.tc.objectserver.api.ObjectManager - Request for non-existent object : ObjectID=[53078738] context = ObjectManagerLookupContext@23005557 : [ processed count = 1, responseContext = Lookup Context@6685409[ clientID = ClientID[109] , requestID = ObjectRequestID=[608984] , ids = BitSetObjectIDSet [ ObjectID=[53078738] ] , lookedup objects.size() = 0 , missingObjects = null , maxRequestDepth = 500 , requestingThreadName = WorkerThread(receive_transaction_stage, 0) , serverInitiated = CLIENT , respondObjectRequestSink = StageQueue(respond_to_request_stage) ] , missing = BitSetObjectIDSet [ ] ]
2012-11-19 17:09:05,685 [WorkerThread(respond_to_request_stage, 0, 0)] WARN com.tc.objectserver.impl.ObjectRequestManagerImpl - Sending missing ids: 1 , to client: ClientID[109]
2012-11-19 17:09:21,685 [L2_L1:TCWorkerComm # 3_R] INFO com.tc.net.protocol.transport.ConnectionPolicyImpl - Removed connection [109] from jvm 60d5c37c-015f-439f-92ff-3d33fe2c121d-13ae7ddcd09
2012-11-19 17:09:22,060 [L2_L1:TCWorkerComm # 3_R] INFO com.tc.net.protocol.transport.ConnectionPolicyImpl - De-allocated connection license for jvm 60d5c37c-015f-439f-92ff-3d33fe2c121d-13ae7ddcd09; ConnectionPolicy[maxConnections=2147483647, connectedJvmCount=5]
2012-11-19 17:09:22,060 [L2_L1:TCWorkerComm # 3_R] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl: DSO Server - Connection to [prodwarch2.wam.westernasset.local:3912] CLOSED. Health Monitoring for this node is now disabled.
2012-11-19 17:09:22,060 [L2_L1:TCWorkerComm # 3_R] WARN com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(109.c4c6a255b36a4fe3970d80d7c0beece2.60d5c37c-015f-439f-92ff-3d33fe2c121d-13ae7ddcd09): CLOSE EVENT : com.tc.net.core.TCConnectionImpl@1427083: connected: false, closed: true local=10.199.0.171:9511 remote=10.199.1.64:3912 connect=[Fri Nov 09 17:11:32 PST 2012] idle=375ms [1299769265 read, 1907766491 write]. STATUS : CLOSED
2012-11-19 17:09:22,060 [L2_L1:TCWorkerComm # 3_R] WARN com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(109.c4c6a255b36a4fe3970d80d7c0beece2.60d5c37c-015f-439f-92ff-3d33fe2c121d-13ae7ddcd09): closing down connection - com.tc.net.core.TCConnectionImpl@1427083: connected: false, closed: true local=10.199.0.171:9511 remote=10.199.1.64:3912 connect=[Fri Nov 09 17:11:32 PST 2012] idle=375ms [1299769265 read, 1907766491 write]
2012-11-19 17:09:22,060 [L2_L1:TCWorkerComm # 3_R] WARN com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(109.c4c6a255b36a4fe3970d80d7c0beece2.60d5c37c-015f-439f-92ff-3d33fe2c121d-13ae7ddcd09): Can only close an open connection
2012-11-19 17:09:22,435 [L2_L1:TCWorkerComm # 3_R] INFO com.tc.objectserver.persistence.db.DBPersistorImpl - Deleted client state for ChannelID=[109]
2012-11-19 17:09:25,873 [Job_Executor382] INFO com.tc.management.remote.connect.ClientBeanBag - unregisterBean: org.terracotta:feature=org.terracotta.modules.tim-ehcache-2.x-ui-1.8.1,name=Ehcache,type=Loader,clients=Clients,node=prodwarch2.wam.westernasset.local/3912
2012-11-19 17:09:25,873 [Job_Executor382] INFO com.tc.management.remote.connect.ClientBeanBag - Unregistered Tunneled MBean 'org.terracotta:clients=Clients,feature=org.terracotta.modules.tim-ehcache-2.x-ui-1.8.1,name=Ehcache,type=Loader,node=prodwarch2.wam.westernasset.local/3912'
2012-11-19 17:09:25,873 [Job_Executor382] INFO com.tc.management.remote.connect.ClientBeanBag - unregisterBean: net.sf.ehcache:SampledCacheManager=__DEFAULT__,name=com.westernasset.srm.InstrumentAssetIdIndex,type=SampledCache,clients=Clients,node=prodwarch2.wam.westernasset.local/3912
2012-11-19 17:09:25,873 [Job_Executor382] INFO com.tc.management.remote.connect.ClientBeanBag - Unregistered Tunneled MBean 'net.sf.ehcache:SampledCacheManager=__DEFAULT__,clients=Clients,name=com.westernasset.srm.InstrumentAssetIdIndex,type=SampledCache,node=prodwarch2.wam.westernasset.local/3912'

2. From terracotta client:

2012-11-19 17:01:18,180 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[109]: Lock GC collected 1 garbage locks
2012-11-19 17:09:06,785 [WorkerThread(receive_object_stage, 0)] WARN com.tc.object.RemoteObjectManager - ClientID[109]: Received Missing Object ID from server : ObjectID=[53078738] ObjectLookup State : com.tc.object.RemoteObjectManagerImpl$ObjectLookupState[Mon Nov 19 17:09:05 PST 2012, requestID =ObjectRequestID=[608984], lookupID =ObjectID=[53078738], parent = ObjectID=[-1], depth = 500, state = LOOKUP_REQUEST]
2012-11-19 17:09:08,348 [WorkerThread(receive_transaction_stage, 0)] WARN com.tc.object.ClientObjectManager - ClientID[109]: Exception retrieving object ObjectID=[53078738]
com.tc.exception.TCObjectNotFoundException:
*************************************************
Requested Object is missing : ObjectID=[53078738]
*************************************************

at com.tc.object.RemoteObjectManagerImpl.basicRetrieve(RemoteObjectManagerImpl.java:239)
at com.tc.object.RemoteObjectManagerImpl.retrieve(RemoteObjectManagerImpl.java:207)
at com.tc.object.ClientObjectManagerImpl.lookup(ClientObjectManagerImpl.java:558)
at com.tc.object.ClientObjectManagerImpl.lookup(ClientObjectManagerImpl.java:502)
at com.tc.object.tx.ClientTransactionManagerImpl.basicApply(ClientTransactionManagerImpl.java:333)
at com.tc.object.tx.ClientTransactionManagerImpl.apply(ClientTransactionManagerImpl.java:378)
at com.tc.object.handler.ReceiveTransactionHandler.handleEvent(ReceiveTransactionHandler.java:100)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:145)
2012-11-19 17:09:10,520 [WorkerThread(receive_transaction_stage, 0)] ERROR com.tc.object.DistributedObjectClient - Thread:Thread[WorkerThread(receive_transaction_stage, 0),5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
com.tc.exception.TCObjectNotFoundException:
*************************************************
Requested Object is missing : ObjectID=[53078738]
*************************************************

at com.tc.object.RemoteObjectManagerImpl.basicRetrieve(RemoteObjectManagerImpl.java:239)
at com.tc.object.RemoteObjectManagerImpl.retrieve(RemoteObjectManagerImpl.java:207)
at com.tc.object.ClientObjectManagerImpl.lookup(ClientObjectManagerImpl.java:558)
at com.tc.object.ClientObjectManagerImpl.lookup(ClientObjectManagerImpl.java:502)
at com.tc.object.tx.ClientTransactionManagerImpl.basicApply(ClientTransactionManagerImpl.java:333)
at com.tc.object.tx.ClientTransactionManagerImpl.apply(ClientTransactionManagerImpl.java:378)
at com.tc.object.handler.ReceiveTransactionHandler.handleEvent(ReceiveTransactionHandler.java:100)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:145)
2012-11-19 17:09:12,551 [WorkerThread(receive_transaction_stage, 0)] ERROR com.tc.logging.ThreadDumpHandler - Mon Nov 19 17:09:10 PST 2012
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.6-b01 mixed mode):

"Dump On Timeout Timer" Id=193 RUNNABLE

"JarManager idle thread" Id=192 RUNNABLE

"pool-8-thread-7" Id=191 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.westernasset.cache.srm.service.InstrumentRatingUpdateService.runInstrumentRatingUpdate(InstrumentRatingUpdateService.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
....

Please help to point us where could be the problem and how to avoid it.

Thanks

Louie

I would at least upgrade to 3.6.5. Lots of useful bug fixes on the 3.6 line in there.