[Logo] Terracotta Discussion Forums
  [Search] Search   [Recent Topics] Recent Topics   [Members]  Member Listing   [Groups] Back to home page 
[Register] Register /  [Login] Login 
[Expert]
Terracotta Server runs out of memory  XML
Forum Index -> Terracotta Platform
Author Message
[Post New]07/02/2009 09:02:43
    Subject: Terracotta Server runs out of memory
[Up]
troy216

journeyman

Joined: 03/05/2008 20:39:27
Messages: 34
Offline
I am running a load test that simply stuffs objects into Terracotta. After a few hours, the Terracotta server runs out of memory and dies. Shouldn’t I be able to continually add new objects until the disk fills? Why is the Server running out of memory?

The server-data files add up to 25,770,000,000 bytes.

tc-config.xml:
Code:
<?xml version="1.0" encoding="UTF-8"?>
 <!--
 
   All content copyright (c) 2003-2008 Terracotta, Inc.,
   except as may otherwise be noted in a separate copyright notice.
   All rights reserved.
 
 -->
 <!--
  This is a Terracotta configuration file that has been pre-configured
  for use with DSO.  All classes are included for instrumentation,
  and all instrumented methods are write locked.
 
  For more information, please see the product documentation.
 -->
 <tc:tc-config xmlns:tc="http://www.terracotta.org/config"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd">
 
     <system>
       <configuration-model>production</configuration-model>
     </system>
     
     <servers>
         <!-- Tell DSO where the Terracotta server can be found. -->
         <server host="%i" name="Server1">
             <data>../server-data</data>
             <logs>../server-logs</logs>
             <statistics>../server-stats</statistics>
             <dso>
                 <client-reconnect-window>120</client-reconnect-window>
 
                 <!-- Information about how DSO should persist data to disk. -->
                 <persistence>
                     <mode>permanent-store</mode>
                     <!--<mode>temporary-swap-only</mode>-->
                 </persistence>
             </dso>
         </server>
     </servers>
 
     <!-- Tell DSO where to put the generated client logs -->
     <clients>
         <logs>../client-logs/%(tc.client)</logs>
         <statistics>../client-stats/%(tc.client)</statistics>
         <modules>
             <module name="tim-annotations"
                     version="1.3.2"
                     group-id="org.terracotta.modules"/>
             <module name="tim-concurrent-collections"
                     version="1.1.2"
                     group-id="org.terracotta.modules"/>
             <module name="tim-map-evictor"
                     version="1.1.2"
                     group-id="org.terracotta.modules"/>
         </modules>
         <dso>
 
             <!-- This section allows you to declare object fault count, which is
                the max number of reachable objects that are prefetched from DSO server
                to DSO client when an object is faulted from DSO server to DSO client.
                It has a default value of 500.
             -->
             <fault-count>500</fault-count>
 
             <!-- This section controls output from the DSO client that can help you
          debug your program. -->
             <debugging>
 
                 <!-- This section controls output from the DSO client related to
               class instrumentation; each individual element controls a different kind
               of output. -->
 
                 <instrumentation-logging>
                     <class>false</class>
                     <locks>false</locks>
                     <transient-root>false</transient-root>
                     <roots>false</roots>
                     <distributed-methods>false</distributed-methods>
                 </instrumentation-logging>
 
                 <!-- This section controls output from the DSO client related to
             operations occurring at runtime; each individual element controls a
             different kind of output. -->
 
                 <runtime-logging>
                     <non-portable-dump>false</non-portable-dump>
                     <lock-debug>false</lock-debug>
                     <wait-notify-debug>false</wait-notify-debug>
                     <distributed-method-debug>false</distributed-method-debug>
                     <new-object-debug>false</new-object-debug>
                 </runtime-logging>
 
                 <!-- This section controls the exact format of the output from
     any options enabled under 'runtime-logging', above.
     Each individual element controls a different kind of output.
 
     Setting the value of 'caller' element to true is now deprecated.  -->
 
                 <runtime-output-options>
                     <auto-lock-details>false</auto-lock-details>
                     <caller>false</caller>
                     <full-stack>false</full-stack>
                 </runtime-output-options>
             </debugging>
         </dso>
     </clients>
 
     <application>
         <dso>
             <!-- Start by including all classes for instrumentation.
                It's more efficient to instrument only those classes that
                hold shared roots or are part of a shared root's graph.
             -->
             <!--<instrumented-classes>-->
             <!--<include>-->
             <!--<class-expression>*..*</class-expression>-->
             <!--</include>-->
             <!--</instrumented-classes>-->
 
             <!-- Apply write level autolocks for all instrumented methods.
                It's more efficient to create finer-grain locks as dictated
                by your application needs.
             -->
             <!--<locks>-->
             <!--<autolock>-->
             <!--<lock-level>write</lock-level>-->
             <!--<method-expression>* *..*.*(..)</method-expression>-->
             <!--</autolock>-->
             <!--</locks>-->
         </dso>
     </application>
 
 
 </tc:tc-config>


server log tail:
Code:
 2009-07-01 23:23:04,877 [TC Memory Monitor] INFO com.tc.object.cache.LFUEvictionPolicy - Time taken to compute removal candidates : 2450 ms
 2009-07-01 23:23:04,877 [WorkerThread(managed_object_request_stage,0)] WARN com.tc.objectserver.handler.ManagedObjectRequestHandler - Time to Remove 10000 is 4660 ms
 2009-07-01 23:23:09,385 [WorkerThread(managed_object_request_stage,0)] WARN com.tc.objectserver.handler.ManagedObjectRequestHandler - Time to Remove 10000 is 4508 ms
 2009-07-01 23:23:28,417 [WorkerThread(managed_object_request_stage,0)] WARN com.tc.objectserver.handler.ManagedObjectRequestHandler - Time to Remove 10000 is 19032 ms
 2009-07-01 23:23:30,906 [TCWorkerComm # 3] WARN com.tc.net.core.CoreNIOServices - Exception trying to shutdown socket output: Bad file descriptor
 2009-07-01 23:23:30,909 [WorkerThread(managed_object_request_stage,0)] WARN com.tc.objectserver.handler.ManagedObjectRequestHandler - Time to Remove 10000 is 2491 ms
 2009-07-01 23:23:33,384 [TC Memory Monitor] INFO com.tc.object.cache.LFUEvictionPolicy - Time taken to compute removal candidates : 2474 ms
 2009-07-01 23:23:37,814 [WorkerThread(managed_object_request_stage,0)] WARN com.tc.objectserver.handler.ManagedObjectRequestHandler - Time to Remove 10000 is 6904 ms
 2009-07-01 23:23:56,719 [WorkerThread(managed_object_request_stage,0)] WARN com.tc.objectserver.handler.ManagedObjectRequestHandler - Time to Remove 10000 is 18904 ms
 2009-07-01 23:24:10,824 [TCWorkerComm # 1] ERROR com.tc.bytes.TCByteBufferFactory - OOME trying to allocate non-direct buffer of size 4096 (index 0 of count 1)
 2009-07-01 23:24:10,824 [TCWorkerComm # 1] ERROR com.tc.net.core.CoreNIOServices - Unhandled exception from selectLoop
 java.lang.OutOfMemoryError: GC overhead limit exceeded
 2009-07-01 23:24:13,060 [WorkerThread(managed_object_fault_stage,2)] ERROR com.tc.server.TCServerMain - Thread:Thread[WorkerThread(managed_object_fault_stage,2),5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
 java.lang.OutOfMemoryError: GC overhead limit exceeded
         at com.tc.objectserver.impl.ObjectManagerImpl.basicLookupObjectsFor(ObjectManagerImpl.java:498)
         at com.tc.objectserver.impl.ObjectManagerImpl.processPendingLookups(ObjectManagerImpl.java:828)
         at com.tc.objectserver.impl.ObjectManagerImpl.postRelease(ObjectManagerImpl.java:653)
         at com.tc.objectserver.impl.ObjectManagerImpl.addFaultedObject(ObjectManagerImpl.java:364)
         at com.tc.objectserver.handler.ManagedObjectFaultHandler.handleEvent(ManagedObjectFaultHandler.java:59)
         at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:142)
 2009-07-01 23:24:13,061 [ObjectID-Checkpoint] ERROR com.tc.objectserver.persistence.sleepycat.FastObjectIDManagerImpl - Error ojectID checkpoint: Environment invalid because of previous exception: com.sleepycat.je.RunRecoveryException
 2009-07-01 23:24:13,061 [ObjectID-Checkpoint] ERROR com.tc.objectserver.persistence.sleepycat.SleepycatPersistor$SleepycatPersistorBase - Error on abortOnError
 Environment invalid because of previous exception: com.sleepycat.je.RunRecoveryException
         at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1296)
         at com.sleepycat.je.Transaction.checkEnv(Transaction.java:468)
         at com.sleepycat.je.Transaction.abort(Transaction.java:88)
         at com.tc.objectserver.persistence.sleepycat.SleepycatPersistor$SleepycatPersistorBase.abortOnError(SleepycatPersistor.java:236)
         at com.tc.objectserver.persistence.sleepycat.SleepycatPersistor$SleepycatPersistorBase.abortOnError(SleepycatPersistor.java:231)
         at com.tc.objectserver.persistence.sleepycat.FastObjectIDManagerImpl.oidFlushLogToBitsArray(FastObjectIDManagerImpl.java:236)
         at com.tc.objectserver.persistence.sleepycat.FastObjectIDManagerImpl.processCheckpoint(FastObjectIDManagerImpl.java:250)
         at com.tc.objectserver.persistence.sleepycat.FastObjectIDManagerImpl.access$300(FastObjectIDManagerImpl.java:37)
         at com.tc.objectserver.persistence.sleepycat.FastObjectIDManagerImpl$CheckpointRunner.run(FastObjectIDManagerImpl.java:294)
 Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
 2009-07-01 23:24:15,547 [WorkerThread(managed_object_request_stage,0)] ERROR com.tc.server.TCServerMain - Thread:Thread[WorkerThread(managed_object_request_stage,0),5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
 java.lang.OutOfMemoryError: GC overhead limit exceeded
 2009-07-01 23:24:15,548 [WorkerThread(commit_changes_stage,1)] ERROR com.tc.server.TCServerMain - Thread:Thread[WorkerThread(commit_changes_stage,1),5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
 java.lang.OutOfMemoryError: GC overhead limit exceeded
 2009-07-01 23:24:15,548 [WorkerThread(commit_changes_stage,0)] ERROR com.tc.server.TCServerMain - Thread:Thread[WorkerThread(commit_changes_stage,0),5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
 com.tc.objectserver.persistence.sleepycat.DBException: Environment invalid because of previous exception: com.sleepycat.je.RunRecoveryException
         at com.tc.objectserver.persistence.sleepycat.SleepycatPersistenceTransactionProvider.newTransaction(SleepycatPersistenceTransactionProvider.java:25)
         at com.tc.objectserver.tx.ServerTransactionManagerImpl.commit(ServerTransactionManagerImpl.java:374)
         at com.tc.objectserver.handler.CommitTransactionChangeHandler.handleEvent(CommitTransactionChangeHandler.java:30)
         at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:142)
 Caused by: Environment invalid because of previous exception: com.sleepycat.je.RunRecoveryException
         at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1296)
         at com.sleepycat.je.Environment.checkEnv(Environment.java:1790)
         at com.sleepycat.je.Environment.beginTransactionInternal(Environment.java:1042)
         at com.sleepycat.je.Environment.beginTransaction(Environment.java:1028)
         at com.tc.objectserver.persistence.sleepycat.SleepycatPersistenceTransactionProvider.newNativeTransaction(SleepycatPersistenceTransactionProvider.java:34)
         at com.tc.objectserver.persistence.sleepycat.SleepycatPersistenceTransactionProvider.newTransaction(SleepycatPersistenceTransactionProvider.java:23)
         ... 3 more
 Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
 2009-07-01 23:24:22,766 [Cleaner-1] ERROR com.tc.server.TCServerMain - Thread:Thread[Cleaner-1,5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
 java.lang.OutOfMemoryError: GC overhead limit exceeded
         at java.lang.Long.valueOf(Long.java:550)
         at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:487)
         at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:233)
         at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:138)
         at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:141)
         at java.lang.Thread.run(Thread.java:619)
 2009-07-01 23:24:22,766 [WorkerThread(jmx_events_stage,0)] ERROR com.tc.server.TCServerMain - Thread:Thread[WorkerThread(jmx_events_stage,0),5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
 2009-07-01 23:24:22,766 [TCWorkerComm # 1] ERROR com.tc.server.TCServerMain - Thread:Thread[TCWorkerComm # 1,5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
 java.lang.RuntimeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
         at com.tc.net.core.CoreNIOServices.run(CoreNIOServices.java:64)
 Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
 2009-07-01 23:24:22,766 [WorkerThread(managed_object_fault_stage,0)] ERROR com.tc.server.TCServerMain - Thread:Thread[WorkerThread(managed_object_fault_stage,0),5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
 java.lang.OutOfMemoryError: GC overhead limit exceeded
         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1739)
         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1954)
         at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
         at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:30)
         at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
         at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
 2009-07-01 23:24:34,389 [WorkerThread(jmx_events_stage,0)] ERROR com.tc.server.TCServerMain - Error while handling uncaught expectionnull
 java.lang.OutOfMemoryError: GC overhead limit exceeded
 2009-07-01 23:24:34,384 [WorkerThread(hydrate_message_stage,0)] ERROR com.tc.net.protocol.tcm.HydrateHandler - Error hydrating message of type REQUEST_MANAGED_OBJECT_MESSAGE (12)
 java.lang.OutOfMemoryError: Java heap space
 2009-07-01 23:24:25,236 [WorkerThread(cluster_metadata_stage,0)] ERROR com.tc.server.TCServerMain - Thread:Thread[WorkerThread(cluster_metadata_stage,0),5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
 java.lang.OutOfMemoryError: GC overhead limit exceeded
         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1739)
         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1954)
         at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
         at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:30)
         at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
         at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
 2009-07-01 23:24:37,985 [TC Memory Monitor] WARN com.terracottatech.dso - Detected Long GC > 8000 ms. Event Type : LongGCEventType.LONG_GC GC Collection Count: 10 GC Collection Time: 11517 ms
 2009-07-01 23:24:39,353 [TCComm Main Selector Thread (listen 0:0:0:0:0:0:0:0:9510)] WARN com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Server - 10.10.223.32:40546 might be in Long GC. GC count since last ping reply : 1
 2009-07-01 23:24:44,004 [TCComm Main Selector Thread (listen 0:0:0:0:0:0:0:0:9510)] WARN com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Server - 10.10.223.32:40551 might be in Long GC. GC count since last ping reply : 1
 2009-07-01 23:24:44,004 [TCComm Main Selector Thread (listen 0:0:0:0:0:0:0:0:9510)] WARN com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Server - 10.10.223.32:39447 might be in Long GC. GC count since last ping reply : 1
 2009-07-01 23:24:37,984 [WorkerThread(hydrate_message_stage,0)] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl: DSO Server - Connection to [10.10.223.32:40546] CLOSED. Health Monitoring for this node is now disabled.
 2009-07-01 23:24:44,005 [ObjectID-Checkpoint] ERROR com.tc.server.TCServerMain - Thread:Thread[ObjectID-Checkpoint,5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
 com.tc.objectserver.persistence.sleepycat.DBException: Environment invalid because of previous exception: com.sleepycat.je.RunRecoveryException
         at com.tc.objectserver.persistence.sleepycat.SleepycatPersistenceTransactionProvider.newTransaction(SleepycatPersistenceTransactionProvider.java:25)
         at com.tc.objectserver.persistence.sleepycat.FastObjectIDManagerImpl.oidFlushLogToBitsArray(FastObjectIDManagerImpl.java:179)
         at com.tc.objectserver.persistence.sleepycat.FastObjectIDManagerImpl.processCheckpoint(FastObjectIDManagerImpl.java:250)
         at com.tc.objectserver.persistence.sleepycat.FastObjectIDManagerImpl.access$300(FastObjectIDManagerImpl.java:37)
         at com.tc.objectserver.persistence.sleepycat.FastObjectIDManagerImpl$CheckpointRunner.run(FastObjectIDManagerImpl.java:294)
 Caused by: Environment invalid because of previous exception: com.sleepycat.je.RunRecoveryException
         at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1296)
         at com.sleepycat.je.Environment.checkEnv(Environment.java:1790)
         at com.sleepycat.je.Environment.beginTransactionInternal(Environment.java:1042)
         at com.sleepycat.je.Environment.beginTransaction(Environment.java:1028)
         at com.tc.objectserver.persistence.sleepycat.SleepycatPersistenceTransactionProvider.newNativeTransaction(SleepycatPersistenceTransactionProvider.java:34)
         at com.tc.objectserver.persistence.sleepycat.SleepycatPersistenceTransactionProvider.newTransaction(SleepycatPersistenceTransactionProvider.java:23)
         ... 4 more
 Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
 2009-07-01 23:24:58,228 [TCWorkerComm # 2] INFO com.terracottatech.console - Client Cannot Reconnect ConnectionID(9.23e71d1e622b424eb4e6e1bd43e34313) not found. Connection attempts from the Terracotta client at 10.10.223.32:36643 are being rejected by the Terracotta server array. Restart the client to allow it to rejoin the cluster. Many client reconnection failures can be avoided by configuring the Terracotta server array for "permanent-store" and tuning reconnection parameters. For more information, see http://www.terracotta.org/ha
 2009-07-01 23:25:00,449 [CommonShutDownHook] INFO com.terracottatech.dso - L2 Exiting...
 2009-07-01 23:25:02,931 [TCWorkerComm # 2] INFO com.tc.net.protocol.transport.ServerStackProvider - Client Cannot Reconnect ConnectionID(9.23e71d1e622b424eb4e6e1bd43e34313) not found. Connection attempts from the Terracotta client at 10.10.223.32:36643 are being rejected by the Terracotta server array. Restart the client to allow it to rejoin the cluster. Many client reconnection failures can be avoided by configuring the Terracotta server array for "permanent-store" and tuning reconnection parameters. For more information, see http://www.terracotta.org/ha
 2009-07-01 23:24:58,228 [TCComm Main Selector Thread (listen 0:0:0:0:0:0:0:0:9510)] WARN com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Server - 10.10.223.32:39447 might be in Long GC. GC count since last ping reply : 2
 2009-07-01 23:25:02,931 [TC Memory Monitor] ERROR com.tc.runtime.TCMemoryManagerImpl - Exception thrown
 java.lang.OutOfMemoryError: GC overhead limit exceeded
 2009-07-01 23:25:06,530 [TCComm Main Selector Thread (listen 0:0:0:0:0:0:0:0:9510)] WARN com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Server - 10.10.223.32:40551 might be in Long GC. GC count since last ping reply : 2
 2009-07-01 23:25:06,536 [TCWorkerComm # 3] ERROR com.tc.net.core.CoreNIOServices - Unhandled exception from selectLoop
 java.lang.OutOfMemoryError: GC overhead limit exceeded
 2009-07-01 23:25:06,537 [WorkerThread(apply_changes_stage,0)] ERROR com.tc.server.TCServerMain - Thread:Thread[WorkerThread(apply_changes_stage,0),5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
 java.lang.OutOfMemoryError: GC overhead limit exceeded
 2009-07-01 23:25:06,537 [WorkerThread(hydrate_message_stage,0)] ERROR com.tc.server.TCServerMain - Thread:Thread[WorkerThread(hydrate_message_stage,0),5,TC Thread Group] got an uncaught exception. calling CallbackOnExitDefaultHandlers.
 com.tc.objectserver.persistence.sleepycat.DBException: Environment invalid because of previous exception: com.sleepycat.je.RunRecoveryException
         at com.tc.objectserver.persistence.sleepycat.SleepycatPersistenceTransactionProvider.newTransaction(SleepycatPersistenceTransactionProvider.java:25)
         at com.tc.objectserver.persistence.sleepycat.ClientStatePersistorImpl.deleteClientState(ClientStatePersistorImpl.java:113)
         at com.tc.objectserver.persistence.sleepycat.ConnectionIDFactoryImpl.channelRemoved(ConnectionIDFactoryImpl.java:114)
         at com.tc.object.net.DSOChannelManagerImpl.fireChannelRemovedEvent(DSOChannelManagerImpl.java:168)
         at com.tc.object.net.DSOChannelManagerImpl.access$200(DSOChannelManagerImpl.java:36)
         at com.tc.object.net.DSOChannelManagerImpl$GenericChannelEventListener.channelRemoved(DSOChannelManagerImpl.java:180)
         at com.tc.net.protocol.tcm.ChannelManagerImpl.fireChannelRemovedEvent(ChannelManagerImpl.java:59)
         at com.tc.net.protocol.tcm.ChannelManagerImpl.removeChannel(ChannelManagerImpl.java:119)
         at com.tc.net.protocol.tcm.ChannelManagerImpl.notifyChannelEvent(ChannelManagerImpl.java:100)
         at com.tc.net.protocol.tcm.AbstractMessageChannel.fireEvent(AbstractMessageChannel.java:243)
         at com.tc.net.protocol.tcm.AbstractMessageChannel.fireChannelClosedEvent(AbstractMessageChannel.java:134)
         at com.tc.net.protocol.tcm.AbstractMessageChannel.close(AbstractMessageChannel.java:146)
         at com.tc.net.protocol.tcm.HydrateHandler.handleEvent(HydrateHandler.java:26)
         at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:142)
 Caused by: Environment invalid because of previous exception: com.sleepycat.je.RunRecoveryException
         at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1296)
         at com.sleepycat.je.Environment.checkEnv(Environment.java:1790)
         at com.sleepycat.je.Environment.beginTransactionInternal(Environment.java:1042)
         at com.sleepycat.je.Environment.beginTransaction(Environment.java:1028)
         at com.tc.objectserver.persistence.sleepycat.SleepycatPersistenceTransactionProvider.newNativeTransaction(SleepycatPersistenceTransactionProvider.java:34)
         at com.tc.objectserver.persistence.sleepycat.SleepycatPersistenceTransactionProvider.newTransaction(SleepycatPersistenceTransactionProvider.java:23)
         ... 13 more
 Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
[Post New]07/02/2009 10:23:56
    Subject: Re:Terracotta Server runs out of memory
[Up]
amiller

ophanim

Joined: 08/29/2007 09:05:48
Messages: 722
Location: St. Louis, MO
Offline
As with any Java process, it's possible to overwhelm the heap and OOME. We do a bunch of stuff to protect against that but there are limits to what you can do. There are a whole bunch of ways to tune GC and distributed GC and with a data set of this size, I suspect some of them would be helpful.

This is a pretty good starting point for those efforts:
http://www.terracotta.org/confluence/display/wiki/TUNING+TERRACOTTA#TUNINGTERRACOTTA-MEMORYMANAGEMENT%3A

Are you just running this test to stress the max size or are you actually testing a real app? It might help to know what you're trying to do so we can direct your efforts appropriately.

How many shared objects do you have in your database? Depending how many, it might be useful to instead use a striped server array with Terracotta
FX, which is not free but you can trial it from here: http://www.terracotta.org/web/display/enterprise/Trial+Terracotta+FX
Alex Miller (Terracotta Engineer) - Want to post to the forums? Sign up here
[WWW]
[Post New]07/02/2009 10:35:23
    Subject: Re:Terracotta Server runs out of memory
[Up]
tgautier

seraphim

Joined: 06/05/2006 12:19:26
Messages: 1781
Offline
Simple question related to Alex's response - how much memory are you giving the server?

It defaults to 512mb which is reasonable for development environments and low stress environments.

If you are stress testing it, and/or expecting to deploy to a high load environment, I recommend you give it 2GB.

[WWW]
[Post New]07/02/2009 11:05:43
    Subject: Re:Terracotta Server runs out of memory
[Up]
troy216

journeyman

Joined: 03/05/2008 20:39:27
Messages: 34
Offline
I am just doing a load test to verify that Terracotta can manage the amount of data we expect to have. That test simply continues to load random data without rest. I did run with the default of 512M heap. The test is probably unrealistic, but I wanted see what would happend. I will increase the heap to 2GB and try again.

Is there a recommended GC configurations so that GC can keep up with this scenario of high volume object creation? I am using a significat transaction size... so the transaction rate should not be too high.

Thanks.

-Troy
[Post New]07/02/2009 11:45:49
    Subject: Re:Terracotta Server runs out of memory
[Up]
tgautier

seraphim

Joined: 06/05/2006 12:19:26
Messages: 1781
Offline
Sizing is tricky - the best is to observe your expected usage in a staging environment and tune as appropriate.

The primary thing to consider here is managing spikes, if you are throwing enough volume, where volume is data size * txns/s whereby the Server cannot handle the spike you could get an OOME.

Generally speaking, there are a lot of algorithms in play that prevent this from happening, the server pages memory out to disk, it can push back on (throttle) clients and so forth.

512MB is on the lower end of providing the server enough room/time to manage memory, so it's likely that 1GB, 1.5GB or 2GB will solve the problem completely.

[WWW]
[Post New]09/24/2009 12:16:06
    Subject: Re:Terracotta Server runs out of memory
[Up]
javiervegas

neo

Joined: 09/24/2009 12:12:52
Messages: 1
Offline
Is it possible to go over 2GB?
[Post New]09/24/2009 13:15:32
    Subject: Re:Terracotta Server runs out of memory
[Up]
gkeim

ophanim

Joined: 12/05/2006 10:22:37
Messages: 555
Location: Terracotta, Inc.
Offline
Yes, with a 64bit OS.
Gary Keim (terracotta developer) Want to post to this forum? Join the Terracotta Community
[Post New]01/08/2010 09:26:11
    Subject: Re:Terracotta Server runs out of memory
[Up]
msnuser168

jedi

Joined: 03/09/2009 06:38:39
Messages: 142
Offline
Did Terracotta support CentOS 64bit and Glassfish V2.0?

If in CentOS 32bit then Terracotta only can use up to 2GB, right?

Thanks!!
[Post New]01/09/2010 19:01:52
    Subject: Re:Terracotta Server runs out of memory
[Up]
ari

seraphim

Joined: 05/24/2006 14:23:21
Messages: 1356
Location: San Francisco, CA
Offline
We test RHEL 64 bit in our automation labs. CentOS is known to work too.

And, you could go above 2GB on 32 bit CentOS with changes to Linux kernel.

Cheers,

--Ari
[WWW]
 
Forum Index -> Terracotta Platform
Go to:   
Powered by JForum 2.1.7 © JForum Team