Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

I am using ehcache in a cluster, and I want both cluster nodes to always have the same cache entries.

I am using RMICacheManagerPeerProviderFactory and multicast to find other peers. Peers are found and the logs show that things are working. On the cache, ReplicateAsynchronously is true.

This does what I want most the time. But we sometimes (once a day?) see consistency issues. I tried to read the documentation about replication consistency but I did not understand the sentences. The doc is here:
* http://ehcache.org/documentation/user-guide/cache-topologies#potential-issues-with-replicated-caching

The document says this:

For those times when it is important, Ehcache provides for synchronous delivery of puts and updates via invalidation.

Does this mean that synchronously delivery guarantees consistency? What happens when delivery fails? Does the original cache.put(e: Element) fail as well?

The document also says this:

If the replicateUpdatesViaCopy property is set to false in the replication configuration, updates are made by removing the element in any other cache peers. This forces the applications using the cache peers to return to a canonical source for the data.

This is not the behavior I am seeing. I have two caches that are out of sync (cache A contains key 1234 and cache B does not). When cache B is asked for key 1234 a null is returned. Please explain the meaning of this sentence in more depth.

By setting the replicateUpdatesViaCopy to false, you are forcing all the peers to remove the value from its cache. So you would get a null which is the behavior you are expecting. Then in your application logic you need to handle this scenario to go to the source of the data (DB or ??).

If data consistency is critical to your application you should look at distributed caching provided by Terracotta Server Array. Try BigMemory Max http://terracotta.org/products/bigmemorymax

- Shiva

Hi Friends,

I want to implement ehcache on cluster environment, But getting errors..
I am using ehcache-core-2.3.1.jar and RMI Replicating caching methodology to implement clustering

EHCACHE configuration is below (ehcache.xml)

<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd" updateCheck="false">

<diskStore path="java.io.tmpdir/REPCache" />

<cacheManagerEventListenerFactory class="" properties=""/>

<cacheManagerPeerProviderFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
properties="peerDiscovery=automatic, multicastGroupAddress=230.0.0.1,
multicastGroupPort=4446, timeToLive=32"/>
<cacheManagerPeerListenerFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
properties="port=40001,
socketTimeoutMillis=20000"/>

<defaultCache
maxElementsInMemory="10"
eternal="false"
timeToIdleSeconds="120"
timeToLiveSeconds="120"
overflowToDisk="true"
maxElementsOnDisk="100000"
diskPersistent="false"
diskExpiryThreadIntervalSeconds="120"
memoryStoreEvictionPolicy="LRU">
<cacheEventListenerFactory
class="net.sf.ehcache.distribution.RMICacheReplicatorFactory"
properties="replicatePuts = false,
replicateUpdates = true,
replicateRemovals = true,
replicateUpdatesViaCopy = false,
replicateAsynchronously = false"/>
</defaultCache>

<cache name="descriptorsCache" eternal="false"
maxElementsInMemory="500" maxElementsOnDisk="10000" overflowToDisk="true" diskPersistent="true"
timeToIdleSeconds="0" timeToLiveSeconds="300"
memoryStoreEvictionPolicy="LRU" />

<cache name="drlRulesCache" eternal="true"
maxElementsInMemory="500" maxElementsOnDisk="10000" overflowToDisk="true" diskPersistent="true"
timeToIdleSeconds="0" timeToLiveSeconds="300"
memoryStoreEvictionPolicy="LRU" />

<cache name="rqaflowcache" eternal="true"
maxElementsInMemory="500" maxElementsOnDisk="10000" overflowToDisk="false" diskPersistent="false"
timeToIdleSeconds="0" timeToLiveSeconds="300"
memoryStoreEvictionPolicy="LRU" >
<cacheEventListenerFactory
class="net.sf.ehcache.distribution.RMICacheReplicatorFactory"
properties="replicatePuts = false,
replicateUpdates = true,
replicateRemovals = true,
replicateUpdatesViaCopy = false,
replicateAsynchronously = false"/>
<bootstrapCacheLoaderFactory
class="net.sf.ehcache.distribution.RMIBootstrapCacheLoaderFactory"
properties="bootstrapAsynchronously=true, maximumChunkSizeBytes=5000000"/>
</cache>

I am getting below errors:

java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:

java.lang.ClassNotFoundException: net.sf.ehcache.distribution.RMICachePeer_Stub from bundle 235 (AppEntryRoute) (no security manager: RMI class loader disabled)
net.sf.ehcache.CacheException: Problem starting listener for RMICachePeer //20.198.236.222:50137/rqaflowcache. Initial cause was RemoteException occurred in server thread; nested exception is:

java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:

java.lang.ClassNotFoundException: net.sf.ehcache.distribution.RMICachePeer_Stub from bundle 235 (AppEntryRoute) (no security manager: RMI class loader disabled)

at net.sf.ehcache.distribution.RMICacheManagerPeerListener.notifyCacheAdded(RMICacheManagerPeerListener.java:529)

at net.sf.ehcache.event.CacheManagerEventListenerRegistry.notifyCacheAdded(CacheManagerEventListenerRegistry.java:159)

at net.sf.ehcache.CacheManager.addCacheNoCheck(CacheManager.java:1031)

at net.sf.ehcache.CacheManager.addConfiguredCaches(CacheManager.java:609)

at net.sf.ehcache.CacheManager.init(CacheManager.java:340)

at net.sf.ehcache.CacheManager.<init>(CacheManager.java:281)

at net.sf.ehcache.CacheManager.create(CacheManager.java:651)

at net.sf.ehcache.CacheManager.getInstance(CacheManager.java:671)

at com.csc.fs.BreezWay.AppEntry.impl.RQAStartFlownImpl.startRQAFlow(RQAStartFlownImpl.java:31)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.aries.proxy.impl.ProxyHandler$1.invoke(ProxyHandler.java:50)

at org.apache.aries.proxy.impl.DefaultWrapper.invoke(DefaultWrapper.java:31)

at org.apache.aries.proxy.impl.ProxyHandler.invoke(ProxyHandler.java:78)

at $Proxy89.startRQAFlow(Unknown Source)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:340)

at org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:237)

at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:166)

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)

at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:73)

at org.apache.camel.impl.ProcessorEndpoint.onExchange(ProcessorEndpoint.java:101)

at org.apache.camel.impl.ProcessorEndpoint$1.process(ProcessorEndpoint.java:71)

at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)

at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:119)

at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:285)

at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:114)

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)

at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)

at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)

at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)

at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)

at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)

at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)

at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)

at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)

at org.apache.camel.fabric.FabricTraceProcessor.process(FabricTraceProcessor.java:81)

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)

at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:322)

at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:213)

at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)

at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)

at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303)

at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)

at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)

at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150)

at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117)

at org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:50)

at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)

at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)

at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)

at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:71)

at org.apache.camel.component.jetty.CamelContinuationServlet.service(CamelContinuationServlet.java:113)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:547)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1359)

at org.eclipse.jetty.servlets.MultiPartFilter.doFilter(MultiPartFilter.java:97)

at org.apache.camel.component.jetty.CamelFilterWrapper.doFilter(CamelFilterWrapper.java:44)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1330)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:478)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:941)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:409)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:875)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)

at org.eclipse.jetty.server.Server.handle(Server.java:345)

at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:441)

at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:936)

at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:801)

at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:224)

at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:51)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:586)

at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:44)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:598)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:533)

at java.lang.Thread.run(Thread.java:619)
Caused by: java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:

java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:

java.lang.ClassNotFoundException: net.sf.ehcache.distribution.RMICachePeer_Stub from bundle 235 (AppEntryRoute) (no security manager: RMI class loader disabled)

at sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:396)

at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:250)

at sun.rmi.transport.Transport$1.run(Transport.java:159)

at java.security.AccessController.doPrivileged(Native Method)

at sun.rmi.transport.Transport.serviceCall(Transport.java:155)

at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:619)

at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)

at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)

at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:359)

at sun.rmi.registry.RegistryImpl_Stub.rebind(Unknown Source)

at java.rmi.Naming.rebind(Naming.java:160)

at net.sf.ehcache.distribution.RMICacheManagerPeerListener.bind(RMICacheManagerPeerListener.java:234)

at net.sf.ehcache.distribution.RMICacheManagerPeerListener.notifyCacheAdded(RMICacheManagerPeerListener.java:527)

... 84 more
Caused by: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:

java.lang.ClassNotFoundException: net.sf.ehcache.distribution.RMICachePeer_Stub from bundle 235 (AppEntryRoute) (no security manager: RMI class loader disabled)

at sun.rmi.registry.RegistryImpl_Skel.dispatch(Unknown Source)

at sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:386)

at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:250)

at sun.rmi.transport.Transport$1.run(Transport.java:159)

at java.security.AccessController.doPrivileged(Native Method)

at sun.rmi.transport.Transport.serviceCall(Transport.java:155)

at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

... 1 more
Caused by: java.lang.ClassNotFoundException: net.sf.ehcache.distribution.RMICachePeer_Stub from bundle 235 (AppEntryRoute) (no security manager: RMI class loader disabled)

at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:375)

at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:165)

at java.rmi.server.RMIClassLoader$2.loadClass(RMIClassLoader.java:620)

at java.rmi.server.RMIClassLoader.loadClass(RMIClassLoader.java:247)

at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:197)

at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)

at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)

... 12 more
Caused by: java.lang.ClassNotFoundException: net.sf.ehcache.distribution.RMICachePeer_Stub not found by AppEntryRoute [235]

at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1467)

at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:73)

at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1850)

at java.lang.ClassLoader.loadClass(ClassLoader.java:248)

at org.apache.felix.framework.Felix.loadBundleClass(Felix.java:1745)

at org.apache.felix.framework.BundleImpl.loadClass(BundleImpl.java:929)

at org.apache.camel.core.osgi.utils.BundleDelegatingClassLoader.findClass(BundleDelegatingClassLoader.java:47)

at org.apache.camel.core.osgi.utils.BundleDelegatingClassLoader.loadClass(BundleDelegatingClassLoader.java:69)

at java.lang.ClassLoader.loadClass(ClassLoader.java:248)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:247)

at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:359)

... 21 more

Can any help me to resolve this issue, i am deploying this bundle on fuse ESB.

Thanks in advance

Hi All,

I am facing same issue when i am deploying this bundle on fuse ESB. Exact same error. Have you able to solve this problem ?

2014-10-16 10:57:29,593 | WARN | plication Thread | RMIAsynchronousCacheReplicator | n.RMIAsynchronousCacheReplicator 325 | 200 - net.sf.ehcache - 2.7.4 | Unable to send message to remote peer. Message was: RemoteException occurred in server thread; nested exception is:
java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.lang.ClassNotFoundException: com.sample.dataaccess.translation.entity.Translation (no security manager: RMI class loader disabled)
java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.lang.ClassNotFoundException: com.sample.dataaccess.translation.entity.Translation (no security manager: RMI class loader disabled)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:353)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:275)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:252)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
at net.sf.ehcache.distribution.RMICachePeer_Stub.send(Unknown Source)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.writeReplicationQueue(RMIAsynchronousCacheReplicator.java:314)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.replicationThreadMain(RMIAsynchronousCacheReplicator.java:127)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.access$000(RMIAsynchronousCacheReplicator.java:58)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator$ReplicationThread.run(RMIAsynchronousCacheReplicator.java:389)