Terracotta Discussion Forums

Terracotta version: 2.7.1
The cluster was happy for quite a while, and then has switched in this mode where a thread requesting the W lock of a shared RRWL gets stuck for hours, but eventually gets the W lock and can progress. It just happened that some workers need a lock (not shared) that this stuck thread acquires prior to requesting the cluster-wide W lock, so it became very apparent. I saw CDV-940 as fixed in 2.7.3, so I am trying to figure out if this bug is the one causing this behavior for us, and if we should upgrade (we are very close to release and wanted to avoid moving to a newer Terracotta version this late in the cycle).

The L1 and L2 log files reveal nothing, so I am pasting bellow the stack trace for the stuck thread (the workers are blocked on locked java.util.HashMap@36f2ecb1, showing just one of them). Thanks for any clarifications.

Stuck thread:

"301100868@qtp0-262" - Thread t@323
java.lang.Thread.State: WAITING on java.lang.Object@230e66d8
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at com.tc.object.lockmanager.impl.ClientLock.waitForTryLock(ClientLock.java:583)
at com.tc.object.lockmanager.impl.ClientLock.basicLock(ClientLock.java:204)
at com.tc.object.lockmanager.impl.ClientLock.lock(ClientLock.java:118)
at com.tc.object.lockmanager.impl.ClientLock.tryLock(ClientLock.java:103)
at com.tc.object.lockmanager.impl.ClientLockManagerImpl.tryLock(ClientLockManagerImpl.java:337)
at com.tc.object.lockmanager.impl.StripedClientLockManagerImpl.tryLock(StripedClientLockManagerImpl.java:159)
at com.tc.object.lockmanager.impl.ThreadLockManagerImpl.tryLock(ThreadLockManagerImpl.java:59)
at com.tc.object.tx.ClientTransactionManagerImpl.tryBegin(ClientTransactionManagerImpl.java:154)
at com.tc.object.bytecode.ManagerImpl.tryBegin(ManagerImpl.java:341)
at com.tc.object.bytecode.ManagerImpl.tryMonitorEnter(ManagerImpl.java:553)
at com.tc.object.bytecode.ManagerUtil.tryMonitorEnter(ManagerUtil.java:503)
at java.util.concurrent.locks.ReentrantReadWriteLock$DsoLock.tryLock(ReentrantReadWriteLock/java:58)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(Unknown Source)
at com.hp.dharma.lockmanager.RWLockManagerImpl._aquireLock(RWLockManagerImpl.java:162)
at com.hp.dharma.lockmanager.RWLockManagerImpl.lock(RWLockManagerImpl.java:74)
at com.iconclude.dharma.commons.repo.LockManager.lockWrite(LockManager.java:30)
at com.iconclude.dharma.commons.repo.LocalConnection.executeWrite(LocalConnection.java:1131)
at com.iconclude.dharma.commons.repo.LocalConnection.start(LocalConnection.java:86)
at com.iconclude.dharma.commons.repo.RepoService.openConnection(RepoService.java:175)
at com.iconclude.dharma.commons.repo.RepoService.openWorkspaceConnection(RepoService.java:258)
at com.iconclude.dharma.commons.repo.RepoService.getConnectionForRemoteClient(RepoService.java:219)
- locked java.util.HashMap@36f2ecb1
at com.iconclude.dharma.commons.repo.servlet.HttpRepoService.getLocalConnection(HttpRepoService.java:342)
at com.iconclude.dharma.commons.repo.servlet.HttpRepoService.doOpen(HttpRepoService.java:1149)
at com.iconclude.dharma.commons.repo.servlet.HttpRepoService.doService(HttpRepoService.java:97)
at com.iconclude.dharma.commons.http.AbstractChainedHttpService.service(AbstractChainedHttpService.java:56)
at com.iconclude.dharma.commons.http.AbstractChainedHttpService.service(AbstractChainedHttpService.java:63)
at com.iconclude.dharma.http.HttpDispatchService.service(HttpDispatchService.java:57)
at com.iconclude.dharma.http.DharmaServlet.doGet(DharmaServlet.java:34)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1124)
at com.iconclude.dharma.commons.util.http.DharmaFilterToBeanProxy.doFilter(DharmaFilterToBeanProxy.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
at com.iconclude.dharma.commons.util.http.DharmaFilterToBeanProxy.doFilter(DharmaFilterToBeanProxy.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
at com.iconclude.dharma.commons.util.http.DharmaFilterToBeanProxy.doFilter(DharmaFilterToBeanProxy.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
at com.iconclude.dharma.commons.util.http.DharmaFilterToBeanProxy.doFilter(DharmaFilterToBeanProxy.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:361)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)

Locked ownable synchronizers:
- None

Worker waiting for the non-DSO regular lock:

"1301700068@qtp0-243" - Thread t@304
java.lang.Thread.State: BLOCKED on java.util.HashMap@36f2ecb1 owned by: 301100868@qtp0-262
at com.iconclude.dharma.commons.repo.RepoService.getConnectionForRemoteClient(RepoService.java:212)
at com.iconclude.dharma.commons.repo.servlet.HttpRepoService.getLocalConnection(HttpRepoService.java:342)
at com.iconclude.dharma.commons.repo.servlet.HttpRepoService.doPing(HttpRepoService.java:300)
at com.iconclude.dharma.commons.repo.servlet.HttpRepoService.doService(HttpRepoService.java:99)
at com.iconclude.dharma.commons.http.AbstractChainedHttpService.service(AbstractChainedHttpService.java:56)
at com.iconclude.dharma.commons.http.AbstractChainedHttpService.service(AbstractChainedHttpService.java:63)
at com.iconclude.dharma.http.HttpDispatchService.service(HttpDispatchService.java:57)
at com.iconclude.dharma.http.DharmaServlet.doGet(DharmaServlet.java:34)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1124)
at com.iconclude.dharma.commons.util.http.DharmaFilterToBeanProxy.doFilter(DharmaFilterToBeanProxy.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
at com.iconclude.dharma.commons.util.http.DharmaFilterToBeanProxy.doFilter(DharmaFilterToBeanProxy.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
at com.iconclude.dharma.commons.util.http.DharmaFilterToBeanProxy.doFilter(DharmaFilterToBeanProxy.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
at com.iconclude.dharma.commons.util.http.DharmaFilterToBeanProxy.doFilter(DharmaFilterToBeanProxy.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:361)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)

Locked ownable synchronizers:
- None

Your stack trace and description did remind me of CDV-940 and I think it's quite likely that is the problem you are seeing. I would recommend upgrading to 2.7.3 to see if it fixes your problem.