Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

Hi,
We are using ehcache version 2.5.1 in our application. Recently we have seen a issue where terracotta server went down due to long run of GC, all the JVM threads were in waiting state and the heap space went out of memory.

Every time when server data is full, we are deleting the objectdb folder and restarting the server. We had deleted the objectdb a day before the issue was happened. Is this fine to delete objectdb when server data is full or should we follow any other approach.

The terracotta server logs are as follows:

""

2014-07-16 09:46:31,888 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:5720)] WARN com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Server - 172.16.1.121:39132 might be in Long GC. Ping-probe cycles completed since last reply : 1
2014-07-16 09:48:52,514 [WorkerThread(managed_object_flush_stage, 6, 0)] ERROR com.tc.server.TCServerMain - Thread:Thread[WorkerThread(managed_object_flush_stage, 6, 0),5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
com.tc.objectserver.persistence.db.DBException: com.tc.objectserver.persistence.db.TCDatabaseException: (JE 4.1.17) JAVA_ERROR: Java Error occurred, recovery may not be possible.
at com.tc.objectserver.persistence.db.ManagedObjectPersistorImpl.saveAllObjects(ManagedObjectPersistorImpl.java:417)
at com.tc.objectserver.impl.PersistentManagedObjectStore.commitAllObjects(PersistentManagedObjectStore.java:90)
at com.tc.objectserver.impl.ObjectManagerImpl.flushAllAndCommit(ObjectManagerImpl.java:922)
at com.tc.objectserver.impl.ObjectManagerImpl.flushAndEvict(ObjectManagerImpl.java:1081)
at com.tc.objectserver.handler.ManagedObjectFlushHandler.handleEvent(ManagedObjectFlushHandler.java:30)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:145)
Caused by: com.tc.objectserver.persistence.db.TCDatabaseException: (JE 4.1.17) JAVA_ERROR: Java Error occurred, recovery may not be possible.
at com.tc.objectserver.persistence.db.ManagedObjectPersistorImpl.basicSaveObject(ManagedObjectPersistorImpl.java:360)
at com.tc.objectserver.persistence.db.ManagedObjectPersistorImpl.saveAllObjects(ManagedObjectPersistorImpl.java:404)
... 5 more
2014-07-16 09:48:52,960 [WorkerThread(managed_object_flush_stage, 6, 0)] ERROR com.tc.logging.ThreadDumpHandler - Wed Jul 16 09:48:52 IST 2014
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.6-b01 mixed mode):

"Dump On Timeout Timer" Id=385 TIMED_WAITING on java.util.TaskQueue@2c80b6a6
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:509)
at java.util.TimerThread.run(Timer.java:462)

"Thread-130" Id=374 WAITING on [I@3657631d
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.sendWithReturn(ClientSynchroMessageConnectionImpl.java:234)
at javax.management.remote.generic.ClientIntermediary$GenericClientNotifForwarder.fetchNotifs(ClientIntermediary.java:922)
at com.sun.jmx.remote.opt.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:473)
at com.sun.jmx.remote.opt.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:371)
at java.lang.Thread.run(Thread.java:662)

"Thread-129" Id=373 WAITING on [I@232c20ca
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.sendWithReturn(ClientSynchroMessageConnectionImpl.java:234)
at javax.management.remote.generic.ClientIntermediary$GenericClientNotifForwarder.fetchNotifs(ClientIntermediary.java:922)
at com.sun.jmx.remote.opt.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:473)
at com.sun.jmx.remote.opt.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:371)
at java.lang.Thread.run(Thread.java:662)

"Thread-128" Id=372 WAITING on [I@261cbfd0
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.sendWithReturn(ClientSynchroMessageConnectionImpl.java:234)
at javax.management.remote.generic.ClientIntermediary$GenericClientNotifForwarder.fetchNotifs(ClientIntermediary.java:922)
at com.sun.jmx.remote.opt.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:473)
at com.sun.jmx.remote.opt.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:371)
at java.lang.Thread.run(Thread.java:662)

"Job_Executor18" Id=363 WAITING on [I@2975043c
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at com.sun.jmx.remote.opt.util.ThreadService$ThreadServiceJob.run(ThreadService.java:223)
at com.sun.jmx.remote.opt.util.JobExecutor.run(JobExecutor.java:99)

"btpool0-13" Id=360 TIMED_WAITING on org.mortbay.thread.BoundedThreadPool$PoolThread@4be26f56
.
.
.
.

2014-07-16 09:48:54,110 [WorkerThread(managed_object_flush_stage, 6, 0)] ERROR com.tc.objectserver.api.ObjectManager - exception printing roots in ObjectManagerImpl
com.tc.objectserver.persistence.db.DBException: com.sleepycat.je.EnvironmentFailureException: (JE 4.1.17) JAVA_ERROR: Java Error occurred, recovery may not be possible.
at com.tc.objectserver.storage.berkeleydb.BerkeleyDBTCRootDatabase.getRootNames(BerkeleyDBTCRootDatabase.java:69)
at com.tc.objectserver.persistence.db.ManagedObjectPersistorImpl.loadRootNames(ManagedObjectPersistorImpl.java:257)
at com.tc.objectserver.impl.PersistentManagedObjectStore.getRootNames(PersistentManagedObjectStore.java:61)
at com.tc.objectserver.impl.ObjectManagerImpl.getRootNames(ObjectManagerImpl.java:239)
at com.tc.objectserver.impl.ObjectManagerImpl.prettyPrint(ObjectManagerImpl.java:200)
at com.tc.text.PrettyPrinterImpl$PrettyPrintablePrintPolicy.visit(PrettyPrinterImpl.java:145)
at com.tc.text.PrettyPrinterImpl.basicVisit(PrettyPrinterImpl.java:112)
at com.tc.text.PrettyPrinterImpl.visit(PrettyPrinterImpl.java:106)
at com.tc.handler.CallbackDumpAdapter.callbackOnExit(CallbackDumpAdapter.java:29)
at com.tc.handler.CallbackDumpHandler.dump(CallbackDumpHandler.java:21)
at com.tc.handler.CallbackDumpHandler.callbackOnExit(CallbackDumpHandler.java:26)
at com.tc.lang.ThrowableHandler.handleDefaultException(ThrowableHandler.java:139)
at com.tc.lang.ThrowableHandler.handleThrowable(ThrowableHandler.java:108)
at com.tc.lang.TCThreadGroup.uncaughtException(TCThreadGroup.java:29)
at java.lang.Thread.dispatchUncaughtException(Thread.java:1874)
Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 4.1.17) JAVA_ERROR: Java Error occurred, recovery may not be possible.
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1452)
at com.sleepycat.je.Database.checkEnv(Database.java:1778)
at com.sleepycat.je.Database.openCursor(Database.java:625)
at com.tc.objectserver.storage.berkeleydb.BerkeleyDBTCRootDatabase.getRootNames(BerkeleyDBTCRootDatabase.java:62)
... 14 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at com.sleepycat.je.log.LogManager.marshallIntoBuffer(LogManager.java:700)
at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:385)
at com.sleepycat.je.log.LogManager.log(LogManager.java:334)
at com.sleepycat.je.tree.LN.log(LN.java:560)
at com.sleepycat.je.tree.LN.optionalLog(LN.java:430)
at com.sleepycat.je.tree.Tree.insertNewSlot(Tree.java:2534)
at com.sleepycat.je.tree.Tree.insert(Tree.java:2467)
at com.sleepycat.je.dbi.CursorImpl.put(CursorImpl.java:1209)
at com.sleepycat.je.Cursor.putAllowPhantoms(Cursor.java:1799)
at com.sleepycat.je.Cursor.putNoNotify(Cursor.java:1756)
at com.sleepycat.je.Cursor.putNotify(Cursor.java:1689)
at com.sleepycat.je.Cursor.putInternal(Cursor.java:1626)
at com.sleepycat.je.Database.putInternal(Database.java:1186)
at com.sleepycat.je.Database.put(Database.java:1058)
at com.tc.objectserver.storage.berkeleydb.BerkeleyDBTCBytesBytesDatabase.put(BerkeleyDBTCBytesBytesDatabase.java:50)
at com.tc.objectserver.storage.berkeleydb.BerkeleyDBTCLongToBytesDatabase.put(BerkeleyDBTCLongToBytesDatabase.java:45)
at com.tc.objectserver.storage.berkeleydb.BerkeleyDBTCLongToBytesDatabase.insert(BerkeleyDBTCLongToBytesDatabase.java:82)
at com.tc.objectserver.persistence.db.ManagedObjectPersistorImpl.basicSaveObject(ManagedObjectPersistorImpl.java:344)
at com.tc.objectserver.persistence.db.ManagedObjectPersistorImpl.saveAllObjects(ManagedObjectPersistorImpl.java:404)
at com.tc.objectserver.impl.PersistentManagedObjectStore.commitAllObjects(PersistentManagedObjectStore.java:90)
at com.tc.objectserver.impl.ObjectManagerImpl.flushAllAndCommit(ObjectManagerImpl.java:922)
at com.tc.objectserver.impl.ObjectManagerImpl.flushAndEvict(ObjectManagerImpl.java:1081)
at com.tc.objectserver.handler.ManagedObjectFlushHandler.handleEvent(ManagedObjectFlushHandler.java:30)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:145)

Could you please help us on preventing this kind of issues in future.

Your help is appreciated.
--
...
Best Regards,
Anil Kumar Konam,
Hyderabad,
mobile# +91-9676861475