| Author |
Message |
![[Post New]](/forums/templates/default/images/icon_minipost_new.gif) 05/08/2008 07:14:29
|
Shake
master
Joined: 01/05/2008 07:27:49
Messages: 84
Offline
|
Some seconds ago I run into an OutOfMemoryError. Everything I see is
Code:
[INFO] [databaseServer] java.lang.OutOfMemoryError: GC overhead limit exceeded
[INFO] [databaseServer] java.lang.OutOfMemoryError: GC overhead limit exceeded
[INFO] [databaseServer]
[INFO] [databaseServer] ********************************** ERROR ***********************************
[INFO] [databaseServer] * Terracotta client shutting down due to error java.lang.OutOfMemoryError:
[INFO] [databaseServer] * GC overhead limit exceeded
[INFO] [databaseServer] ****************************************************************************
[INFO] [databaseServer]
[INFO] [databaseServer] java.lang.OutOfMemoryError: GC overhead limit exceeded
The same application runs fine as standalone version - with the same database...
Any ideas how I could solve that? Didn't find anything within the log files.
I try to load several thound objects from the database using Hibernate and add them to a root.
The standalone just needs a few seconds (<10) to start up - so the problem seems to be related to Terracotta.
Uuh - now it failed fast (java.lang.OutOfMemoryError: Java heap space) after setting jvmargs="-Xmx2024m".... Trying one more time:
The log:
Code:
2008-05-08 16:07:17,278 [main] INFO com.tc.properties.TCProperties - Loading default properties from tc.properties
2008-05-08 16:07:17,279 [main] INFO com.tc.properties.TCProperties - Loaded TCProperties : TCProperties = { console.showObjectID = true, cvt.buffer.randomsuffix.enabled = false, cvt.retriever.notification.interval = 60, cvt.store.randomsuffix.enabled = false, ehcache.concurrency = 1, ehcache.evictor.logging.enabled = false, ehcache.evictor.pool.size = 1, ehcache.global.eviction.enable = true, ehcache.global.eviction.frequency = 10, ehcache.global.eviction.rest.timeMillis = 10, ehcache.global.eviction.segments = 2, ehcache.lock.readLevel = READ, ehcache.lock.writeLevel = WRITE, ehcache.logging.enabled = false, http.defaultservlet.attribute.aliases = false;, http.defaultservlet.attribute.dirallowed = false;, http.defaultservlet.enabled = false;, l1.cachemanager.criticalObjectThreshold = -1, l1.cachemanager.criticalThreshold = 90, l1.cachemanager.enabled = true, l1.cachemanager.leastCount = 2, l1.cachemanager.logging.enabled = false, l1.cachemanager.monitorOldGenOnly = true, l1.cachemanager.percentageToEvict = 10, l1.cachemanager.sleepInterval = 3000, l1.cachemanager.threshold = 70, l1.connect.versionMatchCheck.enabled = true, l1.healthcheck.l2.ping.enabled = true, l1.healthcheck.l2.ping.idletime = 5000, l1.healthcheck.l2.ping.interval = 1000, l1.healthcheck.l2.ping.probes = 3, l1.healthcheck.l2.socketConnect = true, l1.healthcheck.l2.socketConnectCount = 10, l1.healthcheck.l2.socketConnectTimeout = 2, l1.jvm.check.compatibility = true, l1.lock.statistics.gatherInterval = 1, l1.lock.statistics.traceDepth = 1, l1.lockmanager.timeout.interval = 60000, l1.max.connect.retries = -1, l1.modules.additional = , l1.modules.default = org.terracotta.modules.excludes-config;bundle-version:=2.6.0.SNAPSHOT,org.terracotta.modules.guimodels-config;bundle-version:=2.6.0.SNAPSHOT,org.terracotta.modules.jdk15-preinst-config;bundle-version:=2.6.0.SNAPSHOT,org.terracotta.modules.spring-config;bundle-version:=2.6.0.SNAPSHOT,org.terracotta.modules.standard-config;bundle-version:=2.6.0.SNAPSHOT, l1.modules.repositories = /home/johannes/.m2/repository, l1.modules.tc-version-check = , l1.objectmanager.objectid.request.size = 50000, l1.objectmanager.remote.logging.enabled = false, l1.objectmanager.remote.maxDNALRUSize = 60, l1.socket.connect.timeout = 10000, l1.socket.reconnect.waitInterval = 1000, l1.transactionmanager.completedAckFlushTimeout = 5000, l1.transactionmanager.folding.debug = false, l1.transactionmanager.folding.enabled = true, l1.transactionmanager.folding.lock.limit = 0, l1.transactionmanager.folding.object.limit = 0, l1.transactionmanager.logging.enabled = false, l1.transactionmanager.maxBatchSizeInKiloBytes = 128, l1.transactionmanager.maxOutstandingBatchSize = 4, l1.transactionmanager.maxPendingBatches = 88, l1.transactionmanager.maxSleepTimeBeforeHalt = 1024, l1.transactionmanager.strings.compress.enabled = true, l1.transactionmanager.strings.compress.logging.enabled = false, l1.transactionmanager.strings.compress.minSize = 512, l2.beanshell.enabled = false, l2.beanshell.port = 9929, l2.berkeleydb.je.checkpointer.bytesInterval = 100000000, l2.berkeleydb.je.cleaner.bytesInterval = 20000000, l2.berkeleydb.je.cleaner.detailMaxMemoryPercentage = 5, l2.berkeleydb.je.cleaner.lookAheadCacheSize = 32768, l2.berkeleydb.je.cleaner.minAge = 5, l2.berkeleydb.je.lock.nLockTables = 7, l2.berkeleydb.je.lock.timeout = 180000000, l2.berkeleydb.je.maxMemoryPercent = 25, l2.cachemanager.criticalObjectThreshold = -1, l2.cachemanager.criticalThreshold = 90, l2.cachemanager.enabled = true, l2.cachemanager.leastCount = 2, l2.cachemanager.logging.enabled = false, l2.cachemanager.monitorOldGenOnly = true, l2.cachemanager.percentageToEvict = 10, l2.cachemanager.sleepInterval = 3000, l2.cachemanager.threshold = 70, l2.healthcheck.l1.ping.enabled = true, l2.healthcheck.l1.ping.idletime = 30000, l2.healthcheck.l1.ping.interval = 10000, l2.healthcheck.l1.ping.probes = 6, l2.healthcheck.l1.socketConnect = false, l2.healthcheck.l1.socketConnectCount = 2, l2.healthcheck.l1.socketConnectTimeout = 2, l2.healthcheck.l2.ping.enabled = true, l2.healthcheck.l2.ping.idletime = 5000, l2.healthcheck.l2.ping.interval = 1000, l2.healthcheck.l2.ping.probes = 3, l2.healthcheck.l2.socketConnect = true, l2.healthcheck.l2.socketConnectCount = 10, l2.healthcheck.l2.socketConnectTimeout = 2, l2.l1reconnect.enabled = false, l2.l1reconnect.timeout.millis = 5000, l2.lfu.agingFactor = 1, l2.lfu.debug.enabled = false, l2.lfu.recentlyAccessedIgnorePercentage = 20, l2.lockmanager.greedy.lease.enabled = true, l2.lockmanager.greedy.lease.leaseTimeInMillis = 50, l2.nha.groupcomm.type = tc-group-comm, l2.nha.mcast.enabled = false, l2.nha.ooo.maxDelayedAcks = 16, l2.nha.ooo.sendWindow = 32, l2.nha.send.timeout.millis = 16000, l2.nha.tcgroupcomm.discovery.interval = 1000, l2.nha.tcgroupcomm.handshake.timeout = 5000, l2.nha.tcgroupcomm.reconnect.enabled = false, l2.nha.tcgroupcomm.reconnect.timeout = 5000, l2.nha.tribes.failuredetector.millis = 5000, l2.nha.tribes.mcast.mcastAddress = 228.0.0.4, l2.nha.tribes.mcast.mcastFrequency = 500, l2.nha.tribes.mcast.mcastPort = 7777, l2.nha.tribes.mcast.memberDropTime = 3000, l2.nha.tribes.mcast.tcpListenHost = 127.0.0.1, l2.nha.tribes.mcast.tcpListenPort = 4000, l2.nha.tribes.orderinterceptor.enabled = false, l2.objectmanager.cachePolicy = lfu, l2.objectmanager.deleteBatchSize = 5000, l2.objectmanager.fault.logging.enabled = false, l2.objectmanager.loadObjectID.checkpoint.changes = 2000, l2.objectmanager.loadObjectID.checkpoint.maxlimit = 1000, l2.objectmanager.loadObjectID.checkpoint.timeperiod = 10000, l2.objectmanager.loadObjectID.fastLoad = true, l2.objectmanager.loadObjectID.longsPerDiskEntry = 8, l2.objectmanager.maxObjectsInTxnObjGrouping = 5000, l2.objectmanager.maxObjectsToCommit = 5000, l2.objectmanager.maxTxnsInTxnObjectGrouping = 500, l2.objectmanager.passive.sync.batch.size = 500, l2.objectmanager.passive.sync.throttle.timeInMillis = 0, l2.objectmanager.persistor.logging.enabled = false, l2.seda.commitstage.threads = 4, l2.seda.faultstage.threads = 4, l2.seda.flushstage.threads = 4, l2.startuplock.retries.enabled = false, l2.transactionmanager.logging.enabled = false, l2.transactionmanager.logging.printCommits = false, l2.transactionmanager.logging.printStats = true, l2.transactionmanager.logging.verbose = false, l2.transactionmanager.passive.throttle.enabled = true, l2.transactionmanager.passive.throttle.maxSleepSeconds = 5, l2.transactionmanager.passive.throttle.threshold = 20000, lock.statistics.enabled = false, logging.maxBackups = 20, logging.maxLogFileSize = 512, memory.monitor.forcebasic = false, net.core.keepalive = false, net.core.recv.buffer = 65536, net.core.send.buffer = 65536, net.core.tcpnodelay = true, session.debug.hops = false, session.debug.hops.interval = 100, session.debug.invalidate = false, session.debug.sessions = false, session.invalidator.bench.enabled = true, session.invalidator.sleep = 300, session.request.bench.enabled = false, session.request.tracking = false, session.request.tracking.dump = false, session.request.tracking.interval = 2500, session.request.tracking.threshold = 15000, session.vhosts.excluded = , tc.bytebuffer.common.pool.maxcount = 3000, tc.bytebuffer.pooling.enabled = true, tc.bytebuffer.threadlocal.pool.maxcount = 2000, tc.management.mbeans.enabled = true, tc.management.test.mbeans.enabled = false, tc.stage.monitor.delay = 5000, tc.stage.monitor.enabled = false, tcm.monitor.delay = 5, tcm.monitor.enabled = false }
2008-05-08 16:07:17,290 [main] INFO com.terracottatech.general - New logging session started.
2008-05-08 16:07:17,308 [main] INFO com.terracottatech.console - Terracotta 2.6-nightly-rev8465, as of 20080507-180511 (Revision 8465 by cruise@rh4mo0 from 2.6)
2008-05-08 16:07:17,309 [main] INFO com.tc.logging.TCLogging - Terracotta 2.6-nightly-rev8465, as of 20080507-180511 (Revision 8465 by cruise@rh4mo0 from 2.6)
2008-05-08 16:07:17,487 [main] INFO com.tc.config.schema.setup.StandardXMLFileConfigurationCreator - Attempting to load configuration from the file at '/home/johannes/projects/de/avr/execution/databaseServer/tc-config.xml'...
2008-05-08 16:07:17,489 [main] INFO com.tc.config.schema.setup.StandardXMLFileConfigurationCreator - Successfully loaded configuration from the file at '/home/johannes/projects/de/avr/execution/databaseServer/tc-config.xml'. Config is:
<?xml version="1.0" encoding="UTF-8"?>
<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.terracotta.org/config http://www.terracotta.org/schema/terracotta-4.xsd">
<servers>
<!--<server host="192.168.123.100">-->
<server host="192.168.0.36">
<data>/home/johannes/projects/de/avr/data</data>
<logs>/home/johannes/projects/de/avr/execution/databaseServer/logs</logs>
</server>
</servers>
<clients>
<modules>
<module name="tim-annotations" version="1.0.1" />
<module name="clustered-hibernate-3.2.5" version="2.6.0-SNAPSHOT" />
</modules>
<dso>
<debugging>
<instrumentation-logging>
<!--<class>true</class>-->
<!--<hierarchy>true</hierarchy>-->
<!--<locks>true</locks>-->
<transient-root>true</transient-root>
<roots>true</roots>
<distributed-methods>true</distributed-methods>
</instrumentation-logging>
<runtime-logging>
<!--<lock-debug>false</lock-debug>-->
<!--<field-change-debug>true</field-change-debug>-->
<!--<wait-notify-debug>true</wait-notify-debug>-->
<distributed-method-debug>true</distributed-method-debug>
<!--<new-object-debug>true</new-object-debug>-->
<non-portable-dump>false</non-portable-dump>
</runtime-logging>
<runtime-output-options>
<auto-lock-details>false</auto-lock-details>
<caller>true</caller>
<full-stack>false</full-stack>
</runtime-output-options>
</debugging>
</dso>
<logs>logs/terracotta/client/%h/%D</logs>
</clients>
<application>
<dso>
<locks>
<autolock>
<method-expression>* eu.cedarsoft..*.*(..)</method-expression>
<lock-level>write</lock-level>
</autolock>
<autolock>
<method-expression>* de.avr..*.*(..)</method-expression>
<lock-level>write</lock-level>
</autolock>
</locks>
<distributed-methods>
<method-expression>void eu.cedarsoft.history.CollectionSupport.entryDeleted(Object)</method-expression>
<method-expression>void eu.cedarsoft.history.CollectionSupport.entryAdded(Object)</method-expression>
<method-expression>void eu.cedarsoft.history.HistoryListenerSupport.notifyEntryAdded(Object)</method-expression>
<method-expression>void eu.cedarsoft.history.HistoryListenerSupport.notifyEntryRemoved(Object)
</method-expression>
<method-expression>void eu.cedarsoft.history.ContinuousEntriesInformation.notifyEntryRemoved(Object)
</method-expression>
<method-expression>void eu.cedarsoft.history.ContinuousEntriesInformation.notifyEntryAdded(Object)
</method-expression>
<method-expression>void eu.cedarsoft.utils.ChangeListenerSupport.changed(..)</method-expression>
<!-- property change support -->
<method-expression>void java.beans.PropertyChangeSupport.fire*(..)</method-expression>
</distributed-methods>
<instrumented-classes>
<include>
<class-expression>eu.cedarsoft..*..*</class-expression>
<honor-transient>true</honor-transient>
</include>
<!-- Hibernate -->
<include>
<class-expression>org.hibernate.collection.Persistent*</class-expression>
</include>
<include>
<class-expression>org.hibernate.collection.AbstractPersistentCollection</class-expression>
</include>
<!-- Exceptions -->
<include>
<class-expression>org.springframework..*Exception</class-expression>
</include>
<include>
<class-expression>org.hibernate.*Exception</class-expression>
</include>
<include>
<class-expression>org.hibernate.exception.*</class-expression>
</include>
<include>
<class-expression>java.sql.*Exception</class-expression>
</include>
<include>
<class-expression>java.lang.*Exception</class-expression>
</include>
<!-- Joda time -->
<include>
<class-expression>org.joda.time..*</class-expression>
<honor-transient>false</honor-transient>
</include>
<!-- Property Change support -->
<include>
<class-expression>java.beans.PropertyChangeSupport</class-expression>
<honor-transient>true</honor-transient>
</include>
<include>
<class-expression>java.beans.PropertyChangeEvent</class-expression>
<honor-transient>true</honor-transient>
</include>
</instrumented-classes>
<additional-boot-jar-classes>
<include>java.beans.PropertyChangeSupport</include>
<include>java.beans.PropertyChangeEvent</include>
<include>java.sql.SQLException</include>
<include>java.sql.BatchUpdateException</include>
<include>java.lang.NullPointerException</include>
</additional-boot-jar-classes>
</dso>
</application>
</tc:tc-config>
2008-05-08 16:07:17,630 [main] INFO com.terracottatech.console - Configuration loaded from the file at '/home/johannes/projects/de/avr/execution/databaseServer/tc-config.xml'.
2008-05-08 16:07:17,727 [main] INFO com.terracottatech.console - Log file: '/home/johannes/projects/de/avr/execution/databaseServer/logs/terracotta-server.log'.
2008-05-08 16:07:17,729 [main] INFO com.tc.logging.TCLogging - ========================================================================
All Java System Properties for this Terracotta instance:
com.tc.l1.modules.repositories: /home/johannes/.m2/repository
file.encoding : UTF-8
file.encoding.pkg : sun.io
file.separator : /
java.awt.graphicsenv : sun.awt.X11GraphicsEnvironment
java.awt.printerjob : sun.print.PSPrinterJob
java.class.path : /home/johannes/.m2/repository/commons-lang/commons-lang/2.2/commons-lang-2.2.jar:/home/johannes/.m2/repository/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.jar:/home/johannes/.m2/repository/commons-io/commons-io/1.2/commons-io-1.2.jar:/home/johannes/.m2/repository/commons-collections/commons-collections/3.1/commons-collections-3.1.jar:/home/johannes/.m2/repository/org/codehaus/plexus/plexus-utils/1.1/plexus-utils-1.1.jar:/home/johannes/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar:/home/johannes/.m2/repository/com/h2database/h2/1.0.69/h2-1.0.69.jar:/home/johannes/.m2/repository/org/codehaus/plexus/plexus-archiver/1.0-alpha-7/plexus-archiver-1.0-alpha-7.jar:/home/johannes/.m2/repository/org/terracotta/terracotta/2.6.0-SNAPSHOT/terracotta-2.6.0-SNAPSHOT.jar:/home/johannes/.m2/repository/org/terracotta/dependencies/2.6.0-SNAPSHOT/dependencies-2.6.0-SNAPSHOT.jar:/home/johannes/.m2/repository/log4j/log4j/1.2.9/log4j-1.2.9.jar:/home/johannes/.m2/repository/org/slf4j/slf4j-nop/1.3.0/slf4j-nop-1.3.0.jar:/home/johannes/.m2/repository/org/slf4j/slf4j-api/1.3.0/slf4j-api-1.3.0.jar:/home/johannes/.m2/repository/commons-httpclient/commons-httpclient/3.0.1/commons-httpclient-3.0.1.jar:/home/johannes/.m2/repository/commons-codec/commons-codec/1.2/commons-codec-1.2.jar:/home/johannes/.m2/repository/javax/management/jmxri/1.2.1/jmxri-1.2.1.jar:/home/johannes/.m2/repository/javax/management/jmxremote/1.0.1_04/jmxremote-1.0.1_04.jar:/home/johannes/.m2/repository/javax/management/jmxremote_optional/1.0.1_04/jmxremote_optional-1.0.1_04.jar:/home/johannes/.m2/repository/org/beanshell/bsh/2.0b4/bsh-2.0b4.jar:/home/johannes/.m2/repository/xmlbeans/xbean/2.1.0/xbean-2.1.0.jar:/home/johannes/.m2/repository/xmlbeans/xbean_xpath/2.1.0/xbean_xpath-2.1.0.jar:/home/johannes/.m2/repository/xmlbeans/xmlpublic/2.1.0/xmlpublic-2.1.0.jar:/home/johannes/.m2/repository/knopflerfish-tc/knopflerfish-tc/2.0.1/knopflerfish-tc-2.0.1.jar:/home/johannes/.m2/repository/javax/servlet/servlet-api/2.4/servlet-api-2.4.jar:/home/johannes/.m2/repository/org/mortbay/jetty/jetty-util/6.1.8/jetty-util-6.1.8.jar:/home/johannes/.m2/repository/org/mortbay/jetty/jetty/6.1.8/jetty-6.1.8.jar:/home/johannes/.m2/repository/org/mortbay/jetty/servlet-api-2.5/6.1.8/servlet-api-2.5-6.1.8.jar:/home/johannes/.m2/repository/concurrent/concurrent/1.3.4/concurrent-1.3.4.jar:/home/johannes/.m2/repository/trove/trove/1.1-beta-5/trove-1.1-beta-5.jar:/home/johannes/.m2/repository/berkeleydb/je/3.2.74/je-3.2.74.jar:/home/johannes/.m2/repository/jline/jline/0.9.9/jline-0.9.9.jar:/home/johannes/.m2/repository/jfree/jfreechart/1.0.0/jfreechart-1.0.0.jar:/home/johannes/.m2/repository/jfree/jcommon/1.0.0/jcommon-1.0.0.jar:/home/johannes/.m2/repository/org/dijon/appframe/1.0/appframe-1.0.jar:/home/johannes/.m2/repository/treemap/treemap/2.5.1/treemap-2.5.1.jar:/home/johannes/.m2/repository/tomcat/catalina/5.0.28/catalina-5.0.28.jar:/home/johannes/.m2/repository/org/apache/tomcat/tribes/6.0.11-20071126/tribes-6.0.11-20071126.jar:/home/johannes/.m2/repository/org/apache/tomcat/juli/6.0.9/juli-6.0.9.jar:/home/johannes/.m2/repository/stax/stax/1.2.0/stax-1.2.0.jar:/home/johannes/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/home/johannes/.m2/repository/dom4j/dom4j/1.6.1/dom4j-1.6.1.jar:/home/johannes/.m2/repository/org/terracotta/tcconfig1/1.0/tcconfig1-1.0.jar:/home/johannes/.m2/repository/org/terracotta/tcconfig2/2.0/tcconfig2-2.0.jar:/home/johannes/.m2/repository/org/terracotta/tcconfig/2.6.0-SNAPSHOT/tcconfig-2.6.0-SNAPSHOT.jar:/home/johannes/.m2/repository/org/terracotta/tc-l1-prop-from-l2/2.6.0-SNAPSHOT/tc-l1-prop-from-l2-2.6.0-SNAPSHOT.jar:/home/johannes/.m2/repository/ant/ant/1.5.4/ant-1.5.4.jar:/home/johannes/.m2/repository/jdom/jdom/1.0/jdom-1.0.jar:/home/johannes/.m2/repository/jaxen/jaxen/1.0-FCS/jaxen-1.0-FCS.jar:/home/johannes/.m2/repository/org/apache/geronimo/specs/geronimo-j2ee-deployment_1.1_spec/1.0/geronimo-j2ee-deployment_1.1_spec-1.0.jar:/home/johannes/.m2/repository/commons-cli/commons-cli/1.0/commons-cli-1.0.jar:
java.class.version : 50.0
java.endorsed.dirs : /usr/lib/jvm/java-6-sun-1.6.0.06/jre/lib/endorsed
java.ext.dirs : /usr/lib/jvm/java-6-sun-1.6.0.06/jre/lib/ext:/usr/java/packages/lib/ext
java.home : /usr/lib/jvm/java-6-sun-1.6.0.06/jre
java.io.tmpdir : /tmp
java.library.path : /usr/lib/jvm/java-6-sun-1.6.0.06/jre/lib/amd64/server:/usr/lib/jvm/java-6-sun-1.6.0.06/jre/lib/amd64:/usr/lib/jvm/java-6-sun-1.6.0.06/jre/../lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib
java.runtime.name : Java(TM) SE Runtime Environment
java.runtime.version : 1.6.0_06-b02
java.specification.name : Java Platform API Specification
java.specification.vendor : Sun Microsystems Inc.
java.specification.version : 1.6
java.vendor : Sun Microsystems Inc.
java.vendor.url : http://java.sun.com/
java.vendor.url.bug : http://java.sun.com/cgi-bin/bugreport.cgi
java.version : 1.6.0_06
java.vm.info : mixed mode
java.vm.name : Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name : Java Virtual Machine Specification
java.vm.specification.vendor : Sun Microsystems Inc.
java.vm.specification.version : 1.0
java.vm.vendor : Sun Microsystems Inc.
java.vm.version : 10.0-b22
line.separator :
os.arch : amd64
os.name : Linux
os.version : 2.6.24-17-generic
path.separator : :
sun.arch.data.model : 64
sun.boot.class.path : /usr/lib/jvm/java-6-sun-1.6.0.06/jre/lib/resources.jar:/usr/lib/jvm/java-6-sun-1.6.0.06/jre/lib/rt.jar:/usr/lib/jvm/java-6-sun-1.6.0.06/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-6-sun-1.6.0.06/jre/lib/jsse.jar:/usr/lib/jvm/java-6-sun-1.6.0.06/jre/lib/jce.jar:/usr/lib/jvm/java-6-sun-1.6.0.06/jre/lib/charsets.jar:/usr/lib/jvm/java-6-sun-1.6.0.06/jre/classes
sun.boot.library.path : /usr/lib/jvm/java-6-sun-1.6.0.06/jre/lib/amd64
sun.cpu.endian : little
sun.cpu.isalist :
sun.desktop : gnome
sun.io.unicode.encoding : UnicodeLittle
sun.java.launcher : SUN_STANDARD
sun.jnu.encoding : UTF-8
sun.management.compiler : HotSpot 64-Bit Server Compiler
sun.os.patch.level : unknown
tc.classpath : file:/tmp/tc-classpath50370.tmp
user.country : US
user.dir : /home/johannes/projects/de/avr/execution/databaseServer
user.home : /home/johannes
user.language : en
user.name : johannes
user.timezone : Europe/Berlin
========================================================================
2008-05-08 16:07:20,081 [main] INFO com.terracottatech.console - Statistics store: '/home/johannes/projects/de/avr/execution/databaseServer/statistics'.
2008-05-08 16:07:20,081 [main] INFO com.terracottatech.dso - Statistics store: '/home/johannes/projects/de/avr/execution/databaseServer/statistics'.
2008-05-08 16:07:22,348 [main] INFO com.terracottatech.console - Statistics buffer: '/home/johannes/projects/de/avr/execution/databaseServer/statistics'.
2008-05-08 16:07:22,348 [main] INFO com.terracottatech.dso - Statistics buffer: '/home/johannes/projects/de/avr/execution/databaseServer/statistics'.
2008-05-08 16:07:22,407 [main] INFO com.terracottatech.console - JMX Server started. Available at URL[service:jmx:jmxmp://0.0.0.0:9520]
2008-05-08 16:07:22,435 [main] INFO com.tc.objectserver.persistence.sleepycat.DBEnvironment - Env config = allowCreate=true
{je.cleaner.minAge=5, je.cleaner.bytesInterval=20000000, je.env.isTransactional=true, je.cleaner.lookAheadCacheSize=32768, je.env.isReadOnly=false, je.maxMemoryPercent=25, je.lock.nLockTables=7, je.checkpointer.bytesInterval=100000000, je.cleaner.detailMaxMemoryPercentage=5, je.lock.timeout=180000000} DB Config = com.sleepycat.je.DatabaseConfig@4cc72243 JE Properties = {je.cleaner.detailMaxMemoryPercentage=5, je.cleaner.bytesInterval=20000000, je.maxMemoryPercent=25, je.checkpointer.bytesInterval=100000000, je.cleaner.lookAheadCacheSize=32768, je.lock.nLockTables=7, je.cleaner.minAge=5, je.lock.timeout=180000000}
2008-05-08 16:07:22,698 [main] INFO com.tc.net.core.TCWorkerCommManager - Creating 4 worker comm threads.
2008-05-08 16:07:22,708 [main] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl: DSO Server - HealthChecker Started
2008-05-08 16:07:22,906 [main] WARN com.tc.statistics.retrieval.impl.StatisticsRetrievalRegistryImpl - Statistic retrieval action com.tc.statistics.retrieval.actions.SRACpu wasn't activated since its definition couldn't be found.
2008-05-08 16:07:22,907 [main] WARN com.tc.statistics.retrieval.impl.StatisticsRetrievalRegistryImpl - Statistic retrieval action com.tc.statistics.retrieval.actions.SRANetworkActivity wasn't activated since its definition couldn't be found.
2008-05-08 16:07:22,907 [main] WARN com.tc.statistics.retrieval.impl.StatisticsRetrievalRegistryImpl - Statistic retrieval action com.tc.statistics.retrieval.actions.SRADiskActivity wasn't activated since its definition couldn't be found.
2008-05-08 16:07:22,911 [main] INFO com.tc.statistics.StatisticRetrievalAction - "message monitor" statistic is not enabled. Please enable the property "tcm.monitor.delay" to collect this statistics.
2008-05-08 16:07:22,912 [main] INFO com.tc.statistics.retrieval.actions.SRAL2FaultsFromDisk - "l2 faults from disk" statistic is not enabled. Please enable the property "l2.objectmanager.fault.logging.enabled" to collect this statistic.
2008-05-08 16:07:22,913 [main] INFO com.tc.objectserver.tx.ServerTransactionManager - Waiting for txns to complete
2008-05-08 16:07:22,913 [main] INFO com.tc.objectserver.tx.ServerTransactionManager - No more txns in the system.
2008-05-08 16:07:22,913 [main] INFO com.tc.objectserver.handshakemanager.ServerClientHandshakeManager - Starting DSO services...
2008-05-08 16:07:22,914 [main] INFO com.tc.objectserver.persistence.impl.TransactionStoreImpl - shutdownAllClientsExcept() : Removing txns from DB : 0
2008-05-08 16:07:22,914 [main] INFO com.tc.objectserver.impl.ObjectRequestManagerImpl - Processing Pending Lookups = 0
2008-05-08 16:07:22,915 [main] INFO com.tc.objectserver.tx.ResentTransactionSequencer - Unregistering ResentTransactionSequencer since no more resent Transactions : 0
2008-05-08 16:07:22,915 [main] INFO com.tc.objectserver.gtx.GlobalTransactionIDLowWaterMarkProvider - Switching GlobalTransactionID Low Water mark provider since all resent transactions are applied
2008-05-08 16:07:22,924 [main] INFO com.terracottatech.console - Terracotta Server has started up as ACTIVE node on 0:0:0:0:0:0:0:0:9510 successfully, and is now ready for work.
2008-05-08 16:07:27,874 [TCWorkerComm # 1] INFO com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(0.5657b1222df9441fa29960a8e27a3f8e): Attaching new connection: com.tc.net.core.TCConnectionJDK14@2073810741: connected: true, closed: false local=192.168.0.36:9510 remote=192.168.0.36:41854 connect=[Thu May 08 16:07:27 CEST 2008] idle=13ms [28 read, 0 write]
2008-05-08 16:07:27,898 [TCWorkerComm # 1] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Server - Health monitoring agent started for morquad64.local:41854
2008-05-08 16:07:27,920 [WorkerThread(client_handshake_stage,0)] INFO com.tc.objectserver.handshakemanager.ServerClientHandshakeManager - Client connected ClientID[0]
2008-05-08 16:07:28,014 [WorkerThread(jmxremote_tunnel_stage,0)] INFO com.tc.management.remote.protocol.terracotta.ClientTunnelingEventHandler - L1[ChannelID=[0]] notified us that their JMX server is now available
2008-05-08 16:09:10,942 [TCWorkerComm # 1] WARN com.tc.bytes.TCByteBufferFactory - Asking for a large amount of memory: 60989661 bytes
2008-05-08 16:09:17,018 [TC Memory Monitor] INFO com.tc.runtime.TCMemoryManagerImpl - Sleep time changed to : 600
2008-05-08 16:09:18,311 [TC Memory Monitor] INFO com.tc.runtime.TCMemoryManagerImpl - Sleep time changed to : 109
2008-05-08 16:09:20,504 [TC Memory Monitor] INFO com.tc.runtime.TCMemoryManagerImpl - Sleep time changed to : 15
2008-05-08 16:09:22,761 [TC Memory Monitor] INFO com.tc.runtime.TCMemoryManagerImpl - Sleep time changed to : 1
2008-05-08 16:09:35,048 [WorkerThread(process_transaction_stage,0)] ERROR com.tc.objectserver.tx.TransactionalObjectManagerImpl - Exception thrown
java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.resize(HashMap.java:462)
at java.util.HashMap.addEntry(HashMap.java:755)
at java.util.HashMap.put(HashMap.java:385)
at com.tc.objectserver.impl.ObjectManagerImpl.addNewReference(ObjectManagerImpl.java:380)
at com.tc.objectserver.impl.ObjectManagerImpl.addNewReference(ObjectManagerImpl.java:376)
at com.tc.objectserver.impl.ObjectManagerImpl.createObject(ObjectManagerImpl.java:753)
at com.tc.objectserver.impl.ObjectManagerImpl.createNewObjects(ObjectManagerImpl.java:486)
at com.tc.objectserver.impl.ObjectManagerImpl.preFetchObjectsAndCreate(ObjectManagerImpl.java:354)
at com.tc.objectserver.tx.TransactionalObjectManagerImpl.createAndPreFetchObjectsFor(TransactionalObjectManagerImpl.java:116)
at com.tc.objectserver.tx.TransactionalObjectManagerImpl.addTransactions(TransactionalObjectManagerImpl.java:84)
at com.tc.objectserver.tx.ResentTransactionSequencer.addTransactions(ResentTransactionSequencer.java:56)
at com.tc.objectserver.tx.ServerTransactionManagerImpl.incomingTransactions(ServerTransactionManagerImpl.java:411)
at com.tc.objectserver.handler.ProcessTransactionHandler.handleEvent(ProcessTransactionHandler.java:76)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:142)
2008-05-08 16:09:37,093 [TC Memory Monitor] INFO com.tc.object.cache.LFUEvictionPolicy - Time taken to compute removal candidates : 2041 ms
2008-05-08 16:09:41,267 [TC Memory Monitor] INFO com.tc.object.cache.LFUEvictionPolicy - Time taken to compute removal candidates : 4172 ms
2008-05-08 16:09:43,169 [TC Memory Monitor] INFO com.tc.object.cache.LFUEvictionPolicy - Time taken to compute removal candidates : 1901 ms
2008-05-08 16:09:47,011 [TC Memory Monitor] INFO com.tc.object.cache.LFUEvictionPolicy - Time taken to compute removal candidates : 3840 ms
2008-05-08 16:09:48,934 [TC Memory Monitor] INFO com.tc.object.cache.LFUEvictionPolicy - Time taken to compute removal candidates : 1922 ms
2008-05-08 16:09:52,702 [TC Memory Monitor] INFO com.tc.object.cache.LFUEvictionPolicy - Time taken to compute removal candidates : 3767 ms
2008-05-08 16:09:56,535 [TC Memory Monitor] INFO com.tc.object.cache.LFUEvictionPolicy - Time taken to compute removal candidates : 3832 ms
2008-05-08 16:10:00,306 [WorkerThread(process_transaction_stage,0)] ERROR com.tc.server.TCServerMain - Thread:Thread[WorkerThread(process_transaction_stage,0),5,TC Thread Group] got an uncaught exception. calling CallbackOnExitHandlers.
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at java.lang.StringBuilder.append(StringBuilder.java:115)
at java.util.AbstractCollection.toString(AbstractCollection.java:422)
at java.lang.String.valueOf(String.java:2827)
at java.lang.StringBuilder.append(StringBuilder.java:115)
at com.tc.objectserver.tx.ServerTransactionImpl.toString(ServerTransactionImpl.java:139)
at java.lang.String.valueOf(String.java:2827)
at java.lang.StringBuilder.append(StringBuilder.java:115)
at com.tc.objectserver.tx.TransactionalObjectManagerImpl.addTransactions(TransactionalObjectManagerImpl.java:92)
at com.tc.objectserver.tx.ResentTransactionSequencer.addTransactions(ResentTransactionSequencer.java:56)
at com.tc.objectserver.tx.ServerTransactionManagerImpl.incomingTransactions(ServerTransactionManagerImpl.java:411)
at com.tc.objectserver.handler.ProcessTransactionHandler.handleEvent(ProcessTransactionHandler.java:76)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:142)
2008-05-08 16:10:00,350 [WorkerThread(process_transaction_stage,0)] ERROR com.tc.logging.ThreadDumpHandler - Thu May 08 16:10:00 CEST 2008
Full thread dump Java HotSpot(TM) 64-Bit Server VM (10.0-b22 mixed mode):
"Thread-19" Id=75 WAITING on [I@1769c398
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.sendWithReturn(ClientSynchroMessageConnectionImpl.java:194)
at javax.management.remote.generic.ClientIntermediary$GenericClientNotifForwarder.fetchNotifs(ClientIntermediary.java:864)
at com.sun.jmx.remote.opt.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:433)
at com.sun.jmx.remote.opt.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:331)
at java.lang.Thread.run(Thread.java:619)
"DestroyJavaVM" Id=74 RUNNABLE
"GC" Id=33 TIMED_WAITING on java.lang.Object@5b184326
at java.lang.Object.wait(Native Method)
at com.tc.objectserver.impl.ObjectManagerImpl$1.run(ObjectManagerImpl.java:802)
"WorkerThread(request_batch_global_transaction_id_sequence_stage,0)" Id=72 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4ecd51ad
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(process_transaction_stage,0)" Id=71 RUNNABLE
at sun.management.ThreadImpl.getThreadInfo0(Native Method)
at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:123)
at com.tc.util.runtime.ThreadDumpUtilJdk15.getThreadDump(ThreadDumpUtilJdk15.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.tc.util.runtime.ThreadDumpUtil.getThreadDump(ThreadDumpUtil.java:60)
at com.tc.logging.ThreadDumpHandler.callbackOnExit(ThreadDumpHandler.java:15)
at com.tc.lang.ThrowableHandler.handleDefaultException(ThrowableHandler.java:104)
at com.tc.lang.ThrowableHandler.handleThrowable(ThrowableHandler.java:79)
at com.tc.lang.TCThreadGroup.uncaughtException(TCThreadGroup.java:29)
at java.lang.Thread.dispatchUncaughtException(Thread.java:1821)
"WorkerThread(hydrate_message_stage,0)" Id=70 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d2088db
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(managed_object_fault_stage,3)" Id=69 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@9bed3d1
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(managed_object_fault_stage,2)" Id=68 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@9bed3d1
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(managed_object_fault_stage,1)" Id=67 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@9bed3d1
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(managed_object_fault_stage,0)" Id=66 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@9bed3d1
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(commit_changes_stage,0)" Id=65 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@708420ae
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(respond_to_lock_request_stage,0)" Id=64 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1c0b049a
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(transaction_lookup_stage,0)" Id=63 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61d688e2
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(channel_life_cycle_stage,0)" Id=62 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4fc4a198
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(transaction_lowwatermark_stage,0)" Id=61 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4446a5c4
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(request_lock_stage,0)" Id=60 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22e38fca
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(recall_objects_stage,0)" Id=59 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e40f7a8
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(respond_to_request_stage,3)" Id=58 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61e58565
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(respond_to_request_stage,2)" Id=57 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61e58565
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(respond_to_request_stage,1)" Id=56 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61e58565
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(respond_to_request_stage,0)" Id=55 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@61e58565
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(managed_object_flush_stage,3)" Id=54 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5965e55e
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(managed_object_flush_stage,2)" Id=53 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5965e55e
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(managed_object_flush_stage,1)" Id=52 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5965e55e
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(managed_object_flush_stage,0)" Id=51 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5965e55e
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(jmxremote_tunnel_stage,0)" Id=50 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4b5386c2
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(dso-http-bridge,0)" Id=49 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@263e53cf
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(client_handshake_stage,0)" Id=48 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23bb9829
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(broadcast_changes_stage,0)" Id=47 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b0cc9b4
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(send_managed_object_stage,0)" Id=46 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@17496e57
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(client_lock_statistics_respond_stage,0)" Id=45 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3eeabd2f
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(jmxremote_disconnect_stage,0)" Id=44 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7696452
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(apply_changes_stage,0)" Id=43 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58779959
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(jmx_events_stage,0)" Id=42 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d5e8aa5
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(jmxremote_connect_stage,0)" Id=41 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d4fa79d
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(managed_object_request_stage,0)" Id=40 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d7a7de4
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(object_id_batch_request_stage,0)" Id=39 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@49b9ef36
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(transaction_acknowledgement_stage,0)" Id=38 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6913108b
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"WorkerThread(apply_complete_stage,0)" Id=37 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4eb64f2e
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"Reconnect timer" Id=36 WAITING on java.util.TaskQueue@25f7d3f
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
at java.util.TimerThread.run(Timer.java:462)
"DSO Lock Object.wait() timer" Id=35 WAITING on java.util.TaskQueue@13f16b4
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
at java.util.TimerThread.run(Timer.java:462)
"TC Memory Monitor" Id=34 RUNNABLE
at com.tc.object.cache.LFUEvictionPolicy.getRemovalCandidatesInternal(LFUEvictionPolicy.java:171)
at com.tc.object.cache.LFUEvictionPolicy.getRemovalCandidates(LFUEvictionPolicy.java:104)
at com.tc.objectserver.impl.ObjectManagerImpl.evictCache(ObjectManagerImpl.java:877)
at com.tc.object.cache.CacheManager.memoryUsed(CacheManager.java:64)
at com.tc.runtime.TCMemoryManagerImpl.fireMemoryEvent(TCMemoryManagerImpl.java:101)
at com.tc.runtime.TCMemoryManagerImpl.access$500(TCMemoryManagerImpl.java:16)
at com.tc.runtime.TCMemoryManagerImpl$MemoryMonitor.fire(TCMemoryManagerImpl.java:192)
at com.tc.runtime.TCMemoryManagerImpl$MemoryMonitor.reportUsage(TCMemoryManagerImpl.java:176)
at com.tc.runtime.TCMemoryManagerImpl$MemoryMonitor.run(TCMemoryManagerImpl.java:133)
at java.lang.Thread.run(Thread.java:619)
"SampledCounterManager Timer" Id=32 TIMED_WAITING on java.util.TaskQueue@2d58497c
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:509)
at java.util.TimerThread.run(Timer.java:462)
"HealthChecker" Id=31 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at java.lang.Thread.sleep(Thread.java:298)
at com.tc.util.concurrent.ThreadUtil.reallySleep(ThreadUtil.java:24)
at com.tc.util.concurrent.ThreadUtil.reallySleep(ThreadUtil.java:16)
at com.tc.net.protocol.transport.ConnectionHealthCheckerImpl$HealthCheckerMonitorThreadEngine.run(ConnectionHealthCheckerImpl.java:183)
at java.lang.Thread.run(Thread.java:619)
"TCComm Main Selector Thread (listen 0:0:0:0:0:0:0:0:9510)" Id=30 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at com.tc.net.core.CoreNIOServices.selectLoop(CoreNIOServices.java:338)
at com.tc.net.core.CoreNIOServices.run(CoreNIOServices.java:61)
"TCWorkerComm # 3" Id=29 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at com.tc.net.core.CoreNIOServices.selectLoop(CoreNIOServices.java:338)
at com.tc.net.core.CoreNIOServices.run(CoreNIOServices.java:61)
"TCWorkerComm # 2" Id=28 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at com.tc.net.core.CoreNIOServices.selectLoop(CoreNIOServices.java:338)
at com.tc.net.core.CoreNIOServices.run(CoreNIOServices.java:61)
"TCWorkerComm # 1" Id=27 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at com.tc.net.core.CoreNIOServices.selectLoop(CoreNIOServices.java:338)
at com.tc.net.core.CoreNIOServices.run(CoreNIOServices.java:61)
"TCWorkerComm # 0" Id=26 RUNNABLE (in native)
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at com.tc.net.core.CoreNIOServices.selectLoop(CoreNIOServices.java:338)
at com.tc.net.core.CoreNIOServices.run(CoreNIOServices.java:61)
"Checkpointer" Id=24 WAITING on java.lang.Object@1ae2b9e5
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163)
at java.lang.Thread.run(Thread.java:619)
"Cleaner-1" Id=23 WAITING on java.lang.Object@66863941
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163)
at java.lang.Thread.run(Thread.java:619)
"INCompressor" Id=22 TIMED_WAITING on java.lang.Object@6726fef6
at java.lang.Object.wait(Native Method)
at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:165)
at java.lang.Thread.run(Thread.java:619)
"Job_Executor2" Id=21 WAITING on com.tc.management.remote.protocol.terracotta.TunnelingMessageConnection@af4627
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at com.tc.management.remote.protocol.terracotta.TunnelingMessageConnection.readMessage(TunnelingMessageConnection.java:64)
at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl$MessageReader.run(ClientSynchroMessageConnectionImpl.java:401)
at com.sun.jmx.remote.opt.util.ThreadService$ThreadServiceJob.run(ThreadService.java:208)
at com.sun.jmx.remote.opt.util.JobExecutor.run(JobExecutor.java:59)
"Job_Executor1" Id=20 WAITING on [I@6f96eed5
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at com.sun.jmx.remote.opt.util.ThreadService$ThreadServiceJob.run(ThreadService.java:183)
at com.sun.jmx.remote.opt.util.JobExecutor.run(JobExecutor.java:59)
"Job_Executor0" Id=19 TIMED_WAITING on [I@43eeba19
at java.lang.Object.wait(Native Method)
at com.sun.jmx.remote.opt.util.JobExecutor.run(JobExecutor.java:87)
"Thread-9" Id=18 RUNNABLE (in native)
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at com.sun.jmx.remote.socket.SocketConnectionServer.accept(SocketConnectionServer.java:173)
at com.sun.jmx.remote.generic.SynchroMessageConnectionServerImpl.accept(SynchroMessageConnectionServerImpl.java:47)
at javax.management.remote.generic.GenericConnectorServer$Receiver.run(GenericConnectorServer.java:340)
"Timer-0" Id=17 WAITING on java.util.TaskQueue@159811ce
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
at java.util.TimerThread.run(Timer.java:462)
"H2 Log Writer STATISTICS-BUFFER" Id=15 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.h2.store.WriterThread.run(WriterThread.java:149)
"H2 File Lock Watchdog /home/johannes/projects/de/avr/execution/databaseServer/statistics/statistics-buffer.lock.db" Id=14 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.h2.store.FileLock$1.run(FileLock.java:200)
at java.lang.Thread.run(Thread.java:619)
"WorkerThread(dso-http-bridge,0)" Id=13 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@263e53cf
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at com.tc.util.concurrent.TCLinkedBlockingQueue.poll(TCLinkedBlockingQueue.java:26)
at com.tc.async.impl.StageQueueImpl.poll(StageQueueImpl.java:103)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:129)
"btpool0-0" Id=12 TIMED_WAITING on org.mortbay.thread.BoundedThreadPool$PoolThread@6cf1f051
at java.lang.Object.wait(Native Method)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:490)
"H2 Log Writer STATISTICS-STORE" Id=10 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.h2.store.WriterThread.run(WriterThread.java:149)
"H2 File Lock Watchdog /home/johannes/projects/de/avr/execution/databaseServer/statistics/statistics-store.lock.db" Id=9 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.h2.store.FileLock$1.run(FileLock.java:200)
at java.lang.Thread.run(Thread.java:619)
"Signal Dispatcher" Id=4 RUNNABLE
"Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@3b46ad8b
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@1ad8e902
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
2008-05-08 16:10:00,395 [TC Memory Monitor] INFO com.tc.object.cache.LFUEvictionPolicy - Time taken to compute removal candidates : 3859 ms
2008-05-08 16:10:00,400 [WorkerThread(process_transaction_stage,0)] INFO com.tc.objectserver.lockmanager.impl.LockManagerImpl - com.tc.objectserver.lockmanager.impl.LockManagerImpl
--> locks: java.util.HashMap.size()=1401
2008-05-08 16:10:00,401 [WorkerThread(process_transaction_stage,0)] INFO com.tc.objectserver.tx.TransactionalObjectManagerImpl - com.tc.objectserver.tx.TransactionalObjectManagerImpl
--> checkedOutObjects: java.util.HashMap.size()=0
--> applyPendingTxns: java.util.HashMap.size()=0
--> commitPendingTxns: java.util.LinkedHashMap.size()=0
--> pendingTxnList: PendingList : pending Txns = {}
--> pendingObjectRequest: java.util.HashSet.size()=0
2008-05-08 16:10:00,404 [WorkerThread(process_transaction_stage,0)] INFO com.tc.objectserver.api.ObjectManager - com.tc.objectserver.impl.ObjectManagerImpl
--> collector: com.tc.objectserver.core.impl.MarkAndSweepGarbageCollector[State[ GC_SLEEP ]]
--> references: java.util.HashMap.size()=393185
--> checkedOutCount: 0
--> pending: PendingList { pending lookups = 0, blocked count = 0, blocked oids = [] }
--> objectStore: com.tc.objectserver.impl.PersistentManagedObjectStore
| +--> extantObjectIDs: com.tc.util.SyncObjectIdSetImpl.size()=71
--> stateManager: com.tc.objectserver.l1.impl.ClientStateManagerImpl
| +--> client states:
| | +--> ClientID[0]=com.tc.objectserver.l1.impl.ClientStateManagerImpl$ClientStateImpl
| | | +--> managed: com.tc.util.ObjectIDSet2.size()=71
--> roots: de.avr.spring.ObjectManagerAccess.INSTANCE,de.avr.spring.ObjectManagerAccess.INSTANCE_MONITOR
2008-05-08 16:10:00,404 [WorkerThread(process_transaction_stage,0)] INFO com.tc.objectserver.tx.ServerTransactionManager - com.tc.objectserver.tx.ServerTransactionManagerImpl
--> transactionAccounts: java.util.Collections$SynchronizedMap.size()=1
2008-05-08 16:10:00,404 [WorkerThread(process_transaction_stage,0)] ERROR com.tc.server.TCServerMain - Thread:Thread[WorkerThread(process_transaction_stage,0),5,TC Thread Group] got an uncaught exception. About to sleep then exit.
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at java.lang.StringBuilder.append(StringBuilder.java:115)
at java.util.AbstractCollection.toString(AbstractCollection.java:422)
at java.lang.String.valueOf(String.java:2827)
at java.lang.StringBuilder.append(StringBuilder.java:115)
at com.tc.objectserver.tx.ServerTransactionImpl.toString(ServerTransactionImpl.java:139)
at java.lang.String.valueOf(String.java:2827)
at java.lang.StringBuilder.append(StringBuilder.java:115)
at com.tc.objectserver.tx.TransactionalObjectManagerImpl.addTransactions(TransactionalObjectManagerImpl.java:92)
at com.tc.objectserver.tx.ResentTransactionSequencer.addTransactions(ResentTransactionSequencer.java:56)
at com.tc.objectserver.tx.ServerTransactionManagerImpl.incomingTransactions(ServerTransactionManagerImpl.java:411)
at com.tc.objectserver.handler.ProcessTransactionHandler.handleEvent(ProcessTransactionHandler.java:76)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:142)
2008-05-08 16:10:03,416 [CommonShutDownHook] INFO com.terracottatech.dso - L2 Exiting...
|
|
|
 |
05/08/2008 07:44:47
|
Shake
master
Joined: 01/05/2008 07:27:49
Messages: 84
Offline
|
I have another stack trace here:
I have read the database objects successfully. And I have added them to the root, too.
Now I try to exit a call to getHibernateTemplate().execute().
But there seems to be an issue anywhere...:
"main" prio=10 tid=0x000000004011ac00 nid=0x4893 in Object.wait() [0x000000004076c000..0x000000004076dee0]
[INFO] [databaseServer] java.lang.Thread.State: WAITING (on object monitor)
[INFO] [databaseServer] at java.lang.Object.wait(Native Method)
[INFO] [databaseServer] - waiting on <0x00007f9e65002b30> (a java.lang.Object)
[INFO] [databaseServer] at java.lang.Object.wait(Object.java:485)
[INFO] [databaseServer] at com.tc.object.lockmanager.impl.ClientLock.waitForLock(ClientLock.java:619)
[INFO] [databaseServer] - locked <0x00007f9e65002b30> (a java.lang.Object)
[INFO] [databaseServer] at com.tc.object.lockmanager.impl.ClientLock.basicLock(ClientLock.java:204)
[INFO] [databaseServer] at com.tc.object.lockmanager.impl.ClientLock.lock(ClientLock.java:116)
[INFO] [databaseServer] at com.tc.object.lockmanager.impl.ClientLock.lock(ClientLock.java:106)
[INFO] [databaseServer] at com.tc.object.lockmanager.impl.ClientLockManagerImpl.lock(ClientLockManagerImpl.java:305)
[INFO] [databaseServer] at com.tc.object.lockmanager.impl.ThreadLockManagerImpl.lock(ThreadLockManagerImpl.java:46)
[INFO] [databaseServer] at com.tc.object.tx.ClientTransactionManagerImpl.begin(ClientTransactionManagerImpl.java:187)
[INFO] [databaseServer] at com.tc.object.bytecode.ManagerImpl.begin(ManagerImpl.java:337)
[INFO] [databaseServer] at com.tc.object.bytecode.ManagerImpl.monitorEnter(ManagerImpl.java:507)
[INFO] [databaseServer] at com.tc.object.bytecode.ManagerUtil.monitorEnterWithContextInfo(ManagerUtil.java:496)
[INFO] [databaseServer] at org.hibernate.collection.AbstractPersistentCollection.wasInitialized(AbstractPersistentCollection.java)
[INFO] [databaseServer] at org.hibernate.engine.CascadingAction.collectionIsInitialized(CascadingAction.java:435)
[INFO] [databaseServer] at org.hibernate.engine.CascadingAction.getLoadedElementsIterator(CascadingAction.java:423)
[INFO] [databaseServer] at org.hibernate.engine.CascadingAction$5.getCascadableChildrenIterator(CascadingAction.java:222)
[INFO] [databaseServer] at org.hibernate.engine.Cascade.cascadeCollectionElements(Cascade.java:294)
[INFO] [databaseServer] at org.hibernate.engine.Cascade.cascadeCollection(Cascade.java:242)
[INFO] [databaseServer] at org.hibernate.engine.Cascade.cascadeAssociation(Cascade.java:219)
[INFO] [databaseServer] at org.hibernate.engine.Cascade.cascadeProperty(Cascade.java:169)
[INFO] [databaseServer] at org.hibernate.engine.Cascade.cascade(Cascade.java:130)
[INFO] [databaseServer] at org.hibernate.event.def.AbstractFlushingEventListener.cascadeOnFlush(AbstractFlushingEventListener.java:131)
[INFO] [databaseServer] at org.hibernate.event.def.AbstractFlushingEventListener.prepareEntityFlushes(AbstractFlushingEventListener.java:122)
[INFO] [databaseServer] at org.hibernate.event.def.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:65)
[INFO] [databaseServer] at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:26)
[INFO] [databaseServer] at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
[INFO] [databaseServer] at org.springframework.orm.hibernate3.HibernateAccessor.flushIfNecessary(HibernateAccessor.java:390)
[INFO] [databaseServer] at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:420)
[INFO] [databaseServer] at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:339)
[INFO] [databaseServer] at de.avr.CachedDaoInitializer.initialize(CachedDaoInitializer.java:60)
[INFO] [databaseServer] at de.avr.spring.ObjectManagerAccess.initializeObjectManager(ObjectManagerAccess.java:68)
[INFO] [databaseServer] - locked <0x00007f9e114172e8> (a java.lang.Object)
[INFO] [databaseServer] at de.avr.spring.ServerObjectManagerFactory.getObject(ServerObjectManagerFactory.java:20)
[INFO] [databaseServer] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport$1.run(FactoryBeanRegistrySupport.java:121)
[INFO] [databaseServer] at java.security.AccessController.doPrivileged(Native Method)
[INFO] [databaseServer] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:116)
[INFO] [databaseServer] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:91)
[INFO] [databaseServer] - locked <0x00007f9e0f38ce98> (a java.util.concurrent.ConcurrentHashMap)
[INFO] [databaseServer] at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1285)
[INFO] [databaseServer] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:217)
[INFO] [databaseServer] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
[INFO] [databaseServer] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
[INFO] [databaseServer] at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:881)
[INFO] [databaseServer] at de.avr.DatabaseServer.main(DatabaseServer.java:25)
|
|
|
|
05/08/2008 07:55:23
|
Shake
master
Joined: 01/05/2008 07:27:49
Messages: 84
Offline
|
Okay, now I refactored the hibernate stuff (removed the addition of the objects to the root from the hibernate callback).
But now terracotta "hangs" at another lock:
Code:
"main" prio=10 tid=0x000000004011ac00 nid=0x49f0 in Object.wait() [0x000000004135c000..0x000000004135dee0]
[INFO] [databaseServer] java.lang.Thread.State: WAITING (on object monitor)
[INFO] [databaseServer] at java.lang.Object.wait(Native Method)
[INFO] [databaseServer] - waiting on <0x00007f8d17af44c0> (a com.tc.object.RemoteObjectManagerImpl)
[INFO] [databaseServer] at java.lang.Object.wait(Object.java:485)
[INFO] [databaseServer] at com.tc.object.RemoteObjectManagerImpl.waitUntilRunning(RemoteObjectManagerImpl.java:120)
[INFO] [databaseServer] at com.tc.object.RemoteObjectManagerImpl.basicRetrieve(RemoteObjectManagerImpl.java:173)
[INFO] [databaseServer] - locked <0x00007f8d17af44c0> (a com.tc.object.RemoteObjectManagerImpl)
[INFO] [databaseServer] at com.tc.object.RemoteObjectManagerImpl.retrieve(RemoteObjectManagerImpl.java:154)
[INFO] [databaseServer] at com.tc.object.ClientObjectManagerImpl.lookup(ClientObjectManagerImpl.java:572)
[INFO] [databaseServer] at com.tc.object.ClientObjectManagerImpl.lookupObject(ClientObjectManagerImpl.java:483)
[INFO] [databaseServer] at com.tc.object.ClientObjectManagerImpl.lookupObject(ClientObjectManagerImpl.java:472)
[INFO] [databaseServer] at com.tc.object.TCObjectPhysical.resolveReference(TCObjectPhysical.java:133)
[INFO] [databaseServer] - locked <0x00007f8d1de4c5f0> (a com.tc.object.ObjectID)
[INFO] [databaseServer] at java.util.concurrent.locks.ReentrantReadWriteLock.__tc_getreaderLock(Unknown Source)
[INFO] [databaseServer] - locked <0x00007f8d1de4c5f0> (a com.tc.object.ObjectID)
[INFO] [databaseServer] at java.util.concurrent.locks.ReentrantReadWriteLock.readLock(Unknown Source)
[INFO] [databaseServer] at de.avr.contact.Company.getName(Company.java:44)
I really don't understand what could be wrong here...
|
|
|
|
05/09/2008 00:39:03
|
Shake
master
Joined: 01/05/2008 07:27:49
Messages: 84
Offline
|
After many hours of searching and trying I found the problem.
I added many objects to the root using addAll within one transaction.
After I splitted up the addition, everything seems to work ok now....
|
|
|
|
07/12/2010 08:46:20
|
apaliwal
praetor
Joined: 01/05/2010 20:52:24
Messages: 223
Offline
|
Thanks for sharing !
|
cheers
ashish
|
|
|
|
04/20/2012 14:33:58
|
valtih78
journeyman
Joined: 03/17/2012 07:56:22
Messages: 16
Offline
|
what is the point of limiting eviction to committed objects only? Why not to fragment transactions and swap any dirty data to disk automatically?
You want POJO style because it means that instead of trying to combine computational resources, I just describe my algorithm in Java. In Java, you just process you data structure. You do not count how many objects you accessed on the way and, because the graph you are working on may be arbitrary large, you never know how to partition it on your machine in advance and where it is better to place a lock. In the simplest case, you start with a single thread. But, Teraccotta's global lock = OutOfMem forces you to lock every bit. The pervasive 'synchronize' sections bloat your code and nullify your chances to have efficient computation.
Clusters are very good. But enabling huge data structures (+persistence) would be the simplest use of Terracotta. And, it seems feasible because it should count the amount of changes during the transaction anyway.
|
|
|
|
|
|
|
|
![[Logo]](/forums/templates/default/images/logo.png){kind=logo}
![[Expert]](/img/training.png)
![[Search]](/forums/templates/default/images/icon_mini_search.gif){kind=icon}
![[Recent Topics]](/forums/templates/default/images/icon_mini_recentTopics.gif){kind=icon}
![[Members]](/forums/templates/default/images/icon_mini_members.gif){kind=icon}
![[Groups]](/forums/templates/default/images/icon_mini_groups.gif){kind=icon}
![[Register]](/forums/templates/default/images/icon_mini_register.gif){kind=icon}
Register![[Login]](/forums/templates/default/images/icon_mini_login.gif){kind=icon}
Login
![[Up]](/forums/templates/default/images/icon_up.gif){kind=icon}