Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

Hello everyone.

We use in our grails web application the spring cache plugin to realise fragment caching. Few hours after deploying the application on live system we get a lot if net.sf.ehcache.constructs.nonstop.NonStopCacheException: lock() timed out. The underlying ehcache is configured to use terracotta for distributed caching between application nodes. I think that the problem is not the spring cache plugin but the ehcache non stop feature. Does anybody did already the same and has resolved this issue in your own web application or has an idea what could be going wrong here?

Also in my opinion the nonstop timeouts does not work as described on
http://ehcache.org/documentation/configuration/non-stop-cache#nonstop-timeouts-and-behaviors

If terracotta is offline or the timeout comes from an other reason, the noop oder localReads options does not work as described. Instead of returning null or read from local cache, a LockTimedOut or in this case NonStopException is thrown, but I whouldn't expect it. I whould expect that instead of throwing any exception local ehcache in the heap is used. Thus do we really need to catch such exceptions in our own application and if then why is it not handled by the non stop cache feature?

In the following more details about versions, the configuration and exceptions messages.

used software:

grails 1.3.7
ehcache-core 2.4.6
ehcache-terracotta 2.4.6
terracotta-toolkit-runtime 3.3.0
grails springcache plugin 1.3.1

We have configured the springCacheManager to use a seperate EhCacheManager instance with:

Code:

 springcacheCacheManager(org.springframework.cache.ehcache.EhCacheManagerFactoryBean){
         shared = false
         configLocation = new ClassPathResource("custom_ehcache.xml")
     }

Where the custom_ehcache_cache.xml looks like:

Code:

 <?xml version="1.0" encoding="UTF-8"?>
 <ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:noNamespaceSchemaLocation="ehcache.xsd"
          name="SPRING_CACHE"
         >
 
     <terracottaConfig url="tc-server1:9510,tc-server2:9510" rejoin="true"/>
 
     <defaultCache
             maxElementsInMemory="10000"
             maxElementsOnDisk="1000000"
             eternal="false"
             overflowToDisk="false"
             timeToIdleSeconds="300"
             timeToLiveSeconds="600"
             diskPersistent="false"
             memoryStoreEvictionPolicy="LRU"
             >
         <terracotta consistency="eventual" synchronousWrites="false">
             <nonstop
                     enabled="true"
                     immediateTimeout="false"
                     timeoutMillis="5000"
                     >
                 <timeoutBehavior type="localReads"/>
             </nonstop>
         </terracotta>
     </defaultCache>
 
     <cache name="cache1"
            maxElementsInMemory="10000"
            maxElementsOnDisk="1000000"
            eternal="false"
            overflowToDisk="false"
            timeToIdleSeconds="300"
            timeToLiveSeconds="600"
            diskPersistent="false"
            memoryStoreEvictionPolicy="LRU"
             >
         <terracotta consistency="eventual" synchronousWrites="false">
             <nonstop
                     enabled="true"
                     immediateTimeout="false"
                     timeoutMillis="5000"
                     >
                 <timeoutBehavior type="localReads"/>
             </nonstop>
         </terracotta>
     </cache>
 
     <cache name="cache2"
            maxElementsInMemory="10000"
            maxElementsOnDisk="1000000"
            eternal="false"
            overflowToDisk="false"
            timeToIdleSeconds="300"
            timeToLiveSeconds="600"
            diskPersistent="false"
            memoryStoreEvictionPolicy="LRU"
             >
         <terracotta consistency="eventual" synchronousWrites="false">
             <nonstop
                     enabled="true"
                     immediateTimeout="false"
                     timeoutMillis="5000"
                     >
                 <timeoutBehavior type="localReads"/>
             </nonstop>
         </terracotta>
     </cache>
 
 </ehcache>

the more detailed stack trace message:

Code:

 org.codehaus.groovy.grails.web.pages.exceptions.GroovyPagesException: Error processing GroovyPageView: Error executing tag <g:include>: Unable to execute include: net.sf.ehcache.constructs.nonstop.NonStopCacheException: lock() timed out at /WEB-INF/grails-app/views/myView.gsp:11
         at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)
         at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
         ...
         at java.lang.Thread.run(Thread.java:662)
 Caused by: org.codehaus.groovy.grails.web.taglib.exceptions.GrailsTagException: Error executing tag <g:include>: Unable to execute include: net.sf.ehcache.constructs.nonstop.NonStopCacheException: lock() timed out at /WEB-INF/grails-app/views/myView.gsp:11
         ... 4 more
 Caused by: org.codehaus.groovy.grails.web.servlet.mvc.exceptions.ControllerExecutionException: Unable to execute include: net.sf.ehcache.constructs.nonstop.NonStopCacheException: lock() timed out
         ... 7 more
 Caused by: javax.servlet.ServletException: net.sf.ehcache.constructs.nonstop.NonStopCacheException: lock() timed out
         at net.sf.ehcache.constructs.web.filter.Filter.logThrowable(Filter.java:143)
         at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:91)
         ... 7 more
 Caused by: net.sf.ehcache.constructs.nonstop.NonStopCacheException: lock() timed out
         at net.sf.ehcache.constructs.nonstop.concurrency.NonstopSync$2.performClusterOperationTimedOut(NonstopSync.java:104)
         at net.sf.ehcache.constructs.nonstop.concurrency.NonstopSync$2.performClusterOperationTimedOut(NonstopSync.java:94)
         at net.sf.ehcache.constructs.nonstop.store.ExecutorServiceStore.executeClusterOperation(ExecutorServiceStore.java:1037)
         at net.sf.ehcache.constructs.nonstop.store.NonstopStoreImpl.executeClusterOperation(NonstopStoreImpl.java:463)
         at net.sf.ehcache.constructs.nonstop.concurrency.NonstopSync.lock(NonstopSync.java:94)
         at net.sf.ehcache.constructs.blocking.BlockingCache.acquiredLockForKey(BlockingCache.java:186)
         at net.sf.ehcache.constructs.blocking.BlockingCache.get(BlockingCache.java:151)
         at net.sf.ehcache.constructs.blocking.BlockingCache.get(BlockingCache.java:243)
         at net.sf.ehcache.Ehcache$get.call(Unknown Source)
         at grails.plugin.springcache.web.GrailsFragmentCachingFilter.buildPageInfo(GrailsFragmentCachingFilter.groovy:88)
         at grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:62)
         at net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)
         ... 7 more

BR, Oleg Mayevskiy.

Hi,
Are you using any type of explicit locking or any of these caches in your code ? i.e: BlockingCache, SelfPopulatingCache, UpdatingSelfPopulatingCache

Ridhav

Hi. We don't, but the springcache Plugin deals with it.
E.g. grails.plugin.springcache.SpringcacheService
We decorate just a controller action and setup an ehcache for it as described in the ehcache.xml above

BR, Oleg.

Hello. BlockingCache is used, if using NonStop feature.

Also there some strange code in NonstopSync class. For example in the lock and tryLock methods the performClusterOperationTimedOut action just always throws the NonStopCacheException. It was not the case in version 2.4.0, thus it was changed for some reason. But the result is if using NonStop, then exceptions are always thrown instead of doing noop or localReads!