Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

We have a relatively small clustered cache that is used to coordinate exclusive access to resources across a number of distributed processes. Entries in the cache represent resources reserved by one of these processes for its temporary exclusive use. Operations that mutate the cache are guarded by the acquisition (using tryWriteLockOnKey) of a write lock on a distinguished cache entry that exists for this purpose.

Occasionally the following situation occurs: The tryWriteLockOnKey fails with a CacheException whose cause is an InterruptedException (see below). Once this happens, all subsequent attempts to acquire the write lock by that client process time out (the timeout is 10 seconds), requiring us to restart the client process. We're running Terracotta 3.7.3 on Java 1.6.0_37-b06.

There is nothing suspicious in the Terracotta or application logs around the event, except for these entries around the time of the event (which happened at 11.17.57):

2013-07-22 10:40:49,537 [TC Memory Monitor] INFO com.tc.runtime.TCMemoryManagerImpl - Sleep time changed to : 600
2013-07-22 10:42:40,353 [Statistics Logger] INFO com.terracottatech.dso - memory free : 318.051025 MB
2013-07-22 10:42:40,354 [Statistics Logger] INFO com.terracottatech.dso - memory used : 137.073975 MB
2013-07-22 10:42:40,354 [Statistics Logger] INFO com.terracottatech.dso - memory max : 455.125000 MB
2013-07-22 10:45:51,047 [TC Memory Monitor] INFO com.tc.runtime.TCMemoryManagerImpl - Sleep time changed to : 133
2013-07-22 10:57:40,353 [Statistics Logger] INFO com.terracottatech.dso - memory free : 265.152359 MB
2013-07-22 10:57:40,356 [Statistics Logger] INFO com.terracottatech.dso - memory used : 189.972641 MB
2013-07-22 10:57:40,362 [Statistics Logger] INFO com.terracottatech.dso - memory max : 455.125000 MB
2013-07-22 11:10:47,296 [TC Memory Monitor] INFO com.tc.runtime.TCMemoryManagerImpl - Sleep time changed to : 26
2013-07-22 11:10:47,375 [TC Memory Monitor] INFO com.tc.runtime.TCMemoryManagerImpl - Sleep time changed to : 5
2013-07-22 11:12:40,353 [Statistics Logger] INFO com.terracottatech.dso - memory free : 296.172562 MB
2013-07-22 11:12:40,354 [Statistics Logger] INFO com.terracottatech.dso - memory used : 158.952438 MB
2013-07-22 11:12:40,355 [Statistics Logger] INFO com.terracottatech.dso - memory max : 455.125000 MB
2013-07-22 11:15:49,755 [TC Memory Monitor] INFO com.tc.runtime.TCMemoryManagerImpl - Sleep time changed to : 1
2013-07-22 11:27:40,353 [Statistics Logger] INFO com.terracottatech.dso - memory free : 265.387367 MB
2013-07-22 11:27:40,353 [Statistics Logger] INFO com.terracottatech.dso - memory used : 189.737633 MB
2013-07-22 11:27:40,354 [Statistics Logger] INFO com.terracottatech.dso - memory max : 455.125000 MB

But the actual memory consumption never seems to get particularly high. Still, we're investigating heap utilization in our application. The error occurs sporadically; sometimes soon after the application comes up, sometimes after it has been up for days.

Two obvious questions: Any idea where to look for the cause of this? And, any suggestions about how to recover without having to restart the application?

The exception we're seeing:

2013-07-22 11.17.57.177:ERROR:[dispatcher-10:WYMACE@AWE] net.sf.ehcache.CacheException: java.lang.InterruptedException
Method ------ executeWithExecutor
Stack trace--
net.sf.ehcache.constructs.nonstop.store.ExecutorServiceStore.executeWithExecutor(ExecutorServiceStore.java:163)
net.sf.ehcache.constructs.nonstop.store.ExecutorServiceStore.executeWithExecutor(ExecutorServiceStore.java:129)
net.sf.ehcache.constructs.nonstop.store.ExecutorServiceStore.executeClusterOperation(ExecutorServiceStore.java:1185)
net.sf.ehcache.constructs.nonstop.store.NonstopStoreImpl.executeClusterOperation(NonstopStoreImpl.java:704)
net.sf.ehcache.constructs.nonstop.concurrency.NonstopSync.tryLock(NonstopSync.java:106)
net.sf.ehcache.Cache.tryWriteLockOnKey(Cache.java:4116)
vue.core.resource.ResourceCoordinatorImpl$CommittedResourceStore.acquireStoreLock(ResourceCoordinatorImpl.java:315)
vue.core.resource.ResourceCoordinatorImpl$CommittedResourceStore.commit(ResourceCoordinatorImpl.java:419)
vue.core.resource.ResourceCoordinatorImpl.requestResources(ResourceCoordinatorImpl.java:208)
vue.awe.WorkEngine$WorkerHostImpl.onWorkReceived(WorkEngine.java:1038)
vue.awe.Worker.execute(Worker.java:158)
vue.awe.Dispatcher$WorkRunner.run(Dispatcher.java:1174)
java.lang.Thread.run(Thread.java:662)

I'm in the process of moving a dso mode application running on Terracotta 3.3 to express mode on 3.7.3. To get started, I installed (the open source) terracotta-3.7.3-installer.jar, ran it, and created a minimal server configuration. When I try to start this configuration, the server fails with a NoClassDefFoundError for HttpServletRequest. Taking a look at the Class-Path: attribute of the tc.jar manifest, I see some questionable entries, including the following:

thirdparty-3.7.3.jar
servlet-api-2.5-6.1.8.jar

Neither of these JARs is present in TC_HOME/lib, and the name of the second (which should contain HttpServletRequest) looks corrupt to me.

If I edit the class path in the start-tc-server script to include a known good copy of servlet-api.jar, the server comes up without error.

Attached is the log produced when I attempt to bring up this configuration.