Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

i think we have a lot of possibilites in our code where we do a get and then on a cache miss we dont make a put call.

our app is on high concurrently use and we have no locking or jta on a cache.get and cache.put. i wondering for your question because ehcache have no putifabsent-method like jdk-concurrenthashmap.

is this needed ?

anyway i will try (very strange) to make a testcase.

I am very confused right now.
You do use BlockingCache right ?
Code:

net.sf.ehcache.constructs.blocking.BlockingCache.put(BlockingCache.java:204)

Now that's how BlockingCache behaves: on miss, the segment will be locked until the same thread does the put. Maybe we are totally misunderstanding each other. More info here:
http://ehcache.org/documentation/constructs.html

i think we missunderstand us. I will send you tommorow our code fragment.

Also, on a side note, coherent methods (like putIfAbsent) were introduced in ehcache 2.1.0...

I'm sorry I haven't responded to some of the questions here. We're in the process of upgrading from Terracotta 3.2.1 to 3.4 over the next week. While we still see some hangs, I don't think it's worth the effort to pursue until we're on the latest Terracotta, Ehcache, and Java versions in the environment. I'll certainly post back here if issues continue.

Interesting note on that blocking cache bit... I don't recall seeing those in our stack traces, however.

We were having the same problem on our application and could not figure out the problem for the longest time. Finally we found this thread and the above mentioned JVM bug. After upgrading our JVM (Hotspot 1.6.0_23) the issue was resolved. We were not using a Blocking Cache however, we were using a LinkedBlockingQueue at first and then the tim-async module.

Same issue, (see CPU skyrocketing thread: http://forums.terracotta.org/forums/posts/list/4995.page)

setup is slightly different though

TC: 3.2.0
express EH cache setup ( ehcache-core-1.7.2, ehcache-terracotta-1.8.0 )
jdk 1.6.0_14

we are just trying jdk 1.6.0_22...

We were using Jrockit 1.6.0_11 (old!) and we had that issue. I believe based on the bug report, this was fixed in 1.6.0_18, so with 1.6.0_22 you might have better luck.

I may have spoken too soon, we still see this issue but it just happens after a longer period of time.

Hard to tell because a couple of people have posted on this thread. Are you on the latest versions of ehcache? Do you have a thread dump of the issue?

Issue seems to be resolved.Please let us know if more information is required.

We've got similar problems with read-write ehcache regions used from hibernate. Threads get stuck at:

at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
at net.sf.ehcache.store.chm.SelectableConcurrentHashMap$Segment.get(SelectableConcurrentHashMap.java:690)
at net.sf.ehcache.store.chm.SelectableConcurrentHashMap.get(SelectableConcurrentHashMap.java:308)
at net.sf.ehcache.store.MemoryStore.get(MemoryStore.java:326)
at net.sf.ehcache.Cache.get(Cache.java:1634)
at org.hibernate.cache.ehcache.internal.regions.EhcacheTransactionalDataRegion.get(EhcacheTransactionalDataRegion.java:115)
at org.hibernate.cache.ehcache.internal.strategy.AbstractReadWriteEhcacheAccessStrategy.get(AbstractReadWriteEhcacheAccessStrategy.java:77)
at org.hibernate.cache.ehcache.internal.nonstop.NonstopAwareEntityRegionAccessStrategy.get(NonstopAwareEntityRegionAccessStrategy.java:133)
at org.hibernate.event.internal.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:546)
...

at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
at net.sf.ehcache.concurrent.ReadWriteLockSync.lock(ReadWriteLockSync.java:50)
at org.hibernate.cache.ehcache.internal.regions.EhcacheTransactionalDataRegion.writeLock(EhcacheTransactionalDataRegion.java:210)
at org.hibernate.cache.ehcache.internal.strategy.AbstractReadWriteEhcacheAccessStrategy.putFromLoad(AbstractReadWriteEhcacheAccessStrategy.java:102)
at org.hibernate.cache.ehcache.internal.nonstop.NonstopAwareEntityRegionAccessStrategy.putFromLoad(NonstopAwareEntityRegionAccessStrategy.java:195)
at org.hibernate.engine.internal.TwoPhaseLoad.doInitializeEntity(TwoPhaseLoad.java:224)
...

Hibernate 4.1.9, Ehcache 2.7.0, java version "1.7.0_17" (WebSphere Application Server 8.5 at Solaris 10 x86_64)

EDIT: We've found this likely related stuff:
http://forums.terracotta.org/forums/posts/list/6091.page
http://stackoverflow.com/questions/7809768/debugging-hibernate-ehcache-deadlock
Could it be that? We'll try to check it, but it probably isn't the same case we in our case we go via ehcache (in each thread) only once, not twice like there.

I really don't know why you think you face a similar issue.
Correct me if I'm wrong, but you don't use BlockingCache there ?! Do you ? This is all using Ehcache's Hibernate 2LC, isn't it ?
All I can see from the thread dump is contention ? I must be missing something. I'd start a new thread and share much more details. Thanks.

Basically I didn't want to start a new thread for that, however it's similar in its nature - threads hang in very similar manner under heavy load. Of course I can create a new thread, however I don't have much more information available at this moment. The caches affected all seem to be configured as read-write and they're also replicated (RMI) - and yes I'm aware of this: http://lists.terracotta.org/pipermail/ehcache-list/2011-April/000354.html which shouldn't be related to this case.

Still can't help with this... Also, you see threads waiting to acquire lock, but do you see threads holding locks ? Or do these threads hang forever ?