Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

We encountered deadlock with 2 threads.
Here is an extract of the thread dump:
[list]
"gdteam.tmo.common_sv.sfs.MWOExtensionController-20" Id=96 in WAITING on lock=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@57ade323
owned by pool-1-thread-4 Id=109
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
at net.sf.ehcache.store.chm.SelectableConcurrentHashMap$Segment.remove(SelectableConcurrentHashMap.java:563)
at net.sf.ehcache.store.chm.SelectableConcurrentHashMap.remove(SelectableConcurrentHashMap.java:393)
at net.sf.ehcache.store.MemoryStore.remove(MemoryStore.java:359)
at net.sf.ehcache.store.MemoryStore.evict(MemoryStore.java:864)
at net.sf.ehcache.store.MemoryStore.removeElementChosenByEvictionPolicy(MemoryStore.java:608)
at net.sf.ehcache.store.MemoryStore.access$600(MemoryStore.java:96)
at net.sf.ehcache.store.MemoryStore$Participant.evict(MemoryStore.java:1245)
at net.sf.ehcache.pool.impl.BalancedAccessEvictor.freeSpace(BalancedAccessEvictor.java:93)
at net.sf.ehcache.pool.impl.AtomicPoolAccessor.add(AtomicPoolAccessor.java:71)
at net.sf.ehcache.pool.impl.AbstractPoolAccessor.add(AbstractPoolAccessor.java:67)
at net.sf.ehcache.store.MemoryStore.put(MemoryStore.java:266)
at net.sf.ehcache.Cache.putInternal(Cache.java:1501)
at net.sf.ehcache.Cache.put(Cache.java:1427)
at net.sf.ehcache.Cache.put(Cache.java:1392)
at org.hibernate.cache.ehcache.internal.regions.EhcacheTransactionalDataRegion.put(EhcacheTransactionalDataRegion.java:141)
at org.hibernate.cache.ehcache.internal.strategy.ReadWriteEhcacheEntityRegionAccessStrategy.afterUpdate(ReadWriteEhcacheEntityRegionAccessStrategy.java:115)
at org.hibernate.cache.ehcache.internal.nonstop.NonstopAwareEntityRegionAccessStrategy.afterUpdate(NonstopAwareEntityRegionAccessStrategy.java:90)
at org.hibernate.action.internal.EntityUpdateAction.doAfterTransactionCompletion(EntityUpdateAction.java:295)
at org.hibernate.engine.spi.ActionQueue$AfterTransactionCompletionProcessQueue.afterTransactionCompletion(ActionQueue.java:705)
at org.hibernate.engine.spi.ActionQueue.afterTransactionCompletion(ActionQueue.java:300)
at org.hibernate.internal.SessionImpl.afterTransactionCompletion(SessionImpl.java:623)
at org.hibernate.engine.transaction.internal.TransactionCoordinatorImpl.afterTransaction(TransactionCoordinatorImpl.java:146)
at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.afterTransactionCompletion(JdbcTransaction.java:138)
at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:180)
at gdteam.tmo.sv.database.entity.transactions.TransactionalTask.commit(TransactionalTask.java:84)
...
Locked synchronizers: count = 3
java.util.concurrent.locks.ReentrantLock$NonfairSync@4814cbb5
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@591b3bcd
java.util.concurrent.locks.ReentrantLock$NonfairSync@43abe051
[/list]
[list]
"pool-1-thread-4" Id=109 in WAITING on lock=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@591b3bcd
owned by gdteam.tmo.common_sv.sfs.MWOExtensionController-20 Id=96
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
at net.sf.ehcache.store.chm.SelectableConcurrentHashMap$Segment.remove(SelectableConcurrentHashMap.java:563)
at net.sf.ehcache.store.chm.SelectableConcurrentHashMap.remove(SelectableConcurrentHashMap.java:393)
at net.sf.ehcache.store.MemoryStore.remove(MemoryStore.java:359)
at net.sf.ehcache.store.MemoryStore.evict(MemoryStore.java:864)
at net.sf.ehcache.store.MemoryStore.removeElementChosenByEvictionPolicy(MemoryStore.java:608)
at net.sf.ehcache.store.MemoryStore.access$600(MemoryStore.java:96)
at net.sf.ehcache.store.MemoryStore$Participant.evict(MemoryStore.java:1245)
at net.sf.ehcache.pool.impl.BalancedAccessEvictor.freeSpace(BalancedAccessEvictor.java:93)
at net.sf.ehcache.pool.impl.AtomicPoolAccessor.add(AtomicPoolAccessor.java:71)
at net.sf.ehcache.pool.impl.AbstractPoolAccessor.add(AbstractPoolAccessor.java:67)
at net.sf.ehcache.store.MemoryStore.put(MemoryStore.java:266)
at net.sf.ehcache.Cache.putInternal(Cache.java:1501)
at net.sf.ehcache.Cache.put(Cache.java:1427)
at net.sf.ehcache.Cache.put(Cache.java:1392)
at org.hibernate.cache.ehcache.internal.regions.EhcacheTransactionalDataRegion.put(EhcacheTransactionalDataRegion.java:141)
at org.hibernate.cache.ehcache.internal.strategy.AbstractReadWriteEhcacheAccessStrategy.putFromLoad(AbstractReadWriteEhcacheAccessStrategy.java:107)
at org.hibernate.cache.ehcache.internal.nonstop.NonstopAwareEntityRegionAccessStrategy.putFromLoad(NonstopAwareEntityRegionAccessStrategy.java:195)
at org.hibernate.engine.internal.TwoPhaseLoad.doInitializeEntity(TwoPhaseLoad.java:224)
at org.hibernate.engine.internal.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:134)
at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:1103)
at org.hibernate.loader.Loader.processResultSet(Loader.java:960)
at org.hibernate.loader.Loader.doQuery(Loader.java:910)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:341)
at org.hibernate.loader.Loader.doList(Loader.java:2516)
at org.hibernate.loader.Loader.listUsingQueryCache(Loader.java:2361)
at org.hibernate.loader.Loader.list(Loader.java:2324)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:124)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1621)
at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:374)
at gdteam.tmo.sv.database.impl.CarDAO.getByUserId(CarDAO.java:24)
...
Locked synchronizers: count = 3
java.util.concurrent.locks.ReentrantLock$NonfairSync@1f1bc339
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@57ade323
java.util.concurrent.locks.ReentrantLock$NonfairSync@72123b72
[/list]

The reason of the deadlock is unordered acquiring of different locks. More precisely, every thread (MWOExtensionController-20 and pool-1-thread-4) acquired locks (ReentrantReadWriteLock$NonfairSync@57ade323 and ReentrantReadWriteLock$NonfairSync@591b3bcd) in org.hibernate.cache.ehcache.internal.strategy.ReadWriteEhcacheEntityRegionAccessStrategy. And then they thy acquire this locks in reverse order in net.sf.ehcache.store.chm.SelectableConcurrentHashMap$Segment.remove. After that both threads enter in deadlock

Could you please file an issue at https://jira.terracotta.org/jira/browse/EHC ?

Thanks

I created the same issue in hibernate jira https://hibernate.atlassian.net/browse/HHH-9070
Probably this is bug of hibernate, not ehcache.
Please read and comment if necessary.

We had a look and the issue is an Ehcache one.

We created EHC-1079 and will be working on a fix.

Thanks for the report.

Hi,

I have a different scenario where I have problems in the call to getWithLoader. It looks like a problem with SelectableConcurrentHashMap because when I enable overflow to disk the problem does not happen (so instead of a MemoryStore with a SelectableConcurrentHashMap it uses a CacheStore with a AuthoritativeTier).

Here is more or less what my code looks like:

userCache.acquireWriteLockOnKey(key);
try {
userCache.getWithLoader(key, null, null);
} finally {
userCache.releaseWriteLockOnKey(key);
}

If I set cacheLoaderTimeoutMillis to 0 it works, but if I set it to any other value it always fail after the specified timeout. It stays waiting in the method containsKey.

For my surprise if I configure the cache with:

<persistence strategy="localTempSwap"/>

it works regardless of the timeout (0 or any other value).

I am using version 2.8.2.

Regards,
Constantino