Terracotta Discussion Forums

Code:

 net.sf.ehcache.CacheException: java.io.StreamCorruptedException: invalid stream header
 ------
                 at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
                 at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
                 at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
                 at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
                 at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)
                 at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
                 at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3244)
                 at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
                 at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
                 at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2010)
                 at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:1916)
                 at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366)
                 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
                 at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
 Caused by: net.sf.ehcache.CacheException: java.io.StreamCorruptedException: invalid stream header
                 at net.sf.ehcache.store.compound.factories.DiskPersistentStorageFactory.retrieve(DiskPersistentStorageFactory.java:211)
                 at net.sf.ehcache.store.compound.factories.DiskPersistentStorageFactory.retrieve(DiskPersistentStorageFactory.java:59)
                 at net.sf.ehcache.store.compound.Segment.decode(Segment.java:177)
                 at net.sf.ehcache.store.compound.Segment.get(Segment.java:229)
                 at net.sf.ehcache.store.compound.CompoundStore.get(CompoundStore.java:169)
                 at net.sf.ehcache.Cache.searchInStoreWithStats(Cache.java:1899)
                 at net.sf.ehcache.Cache.get(Cache.java:1549)
                 ... 20 more
 Caused by: java.io.StreamCorruptedException: invalid stream header
                 at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:764)
                 at java.io.ObjectInputStream.<init>(ObjectInputStream.java:277)
                 at net.sf.ehcache.store.compound.factories.DiskStorageFactory$2.<init>(DiskStorageFactory.java:272)
                 at net.sf.ehcache.store.compound.factories.DiskStorageFactory.read(DiskStorageFactory.java:272)
                 at net.sf.ehcache.store.compound.factories.DiskPersistentStorageFactory.retrieve(DiskPersistentStorageFactory.java:203)
                 ... 34 more

Note I've removed stack trace entries specific to our organization. This is with ehcache 2.4.5. Got one report of this happening semi-regularly, and not totally sure what's going on with it. I've not seen it locally yet, but it is of concern right now.

This just took a more important level:

net.sf.ehcache.CacheException: java.io.StreamCorruptedException: invalid stream header
at net.sf.ehcache.store.compound.factories.DiskPersistentStorageFactory.retrieve(DiskPersistentStorageFactory.java:211)
at net.sf.ehcache.store.compound.factories.DiskPersistentStorageFactory.retrieve(DiskPersistentStorageFactory.java:59)
at net.sf.ehcache.store.compound.Segment.decode(Segment.java:177)
at net.sf.ehcache.store.compound.Segment.get(Segment.java:229)
at net.sf.ehcache.store.compound.CompoundStore.get(CompoundStore.java:169)
Truncated. see log file for complete stacktrace
java.io.StreamCorruptedException: invalid stream header
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:764)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:277)
at net.sf.ehcache.store.compound.factories.DiskStorageFactory$2.<init>(DiskStorageFactory.java:272)
at net.sf.ehcache.store.compound.factories.DiskStorageFactory.read(DiskStorageFactory.java:272)
at net.sf.ehcache.store.compound.factories.DiskPersistentStorageFactory.retrieve(DiskPersistentStorageFactory.java:203)
Truncated. see log file for complete stacktrace
>

After new deployment we're consistently getting the above results. We added a simple servlet to allow us to purge the caches, and that fixes the problem, but the next time we deploy the worry is we'll be seeing this again.

Hi ,

Can you please share your, use case , configuration and deployment scheme for better understanding.

Use case:
Caching about all of our oracle calls. This includes at this point 6 daos. However, JDBC calls are only about 2-5% of the overall system time per request.

Configuration:
Code:

       <cache name="urlInfo"
                 maxElementsInMemory="20000"
                 maxElementsOnDisk="500000"
                 overflowToDisk="false"
                 timeToLiveSeconds="86400"
                 diskPersistent="false"
                 timeToIdleSeconds="0"
         />
         <cache name="postalInfo"
                 maxElementsInMemory="10000"
                 maxElementsOnDisk="1000000"
                 overflowToDisk="false"
                 timeToLiveSeconds="2592000"
                 diskPersistent="false"
                 timeToIdleSeconds="0"
         />
         <cache name="inventory"
                 maxElementsInMemory="5000"
                 overflowToDisk="false"
                 timeToLiveSeconds="86400"
                 timeToIdleSeconds="0"
         />
         <cache name="mixedData"
                 maxElementsInMemory="6000"
                 maxElementsOnDisk="1000000"
                 overflowToDisk="false"
                 timeToLiveSeconds="2592000"
                 diskPersistent="false"
                 timeToIdleSeconds="0"
         />
         <cache name="oemInfo"
                 maxElementsInMemory="1000"
                 maxElementsOnDisk="1000"
                 overflowToDisk="false"
                 timeToLiveSeconds="86400"
                 diskPersistent="false"
                 timeToIdleSeconds="0"
         />
                 
         <cache name="warrantyInfo"
                 maxElementsInMemory="1000"
                 maxElementsOnDisk="100000"
                 overflowToDisk="false"
                 timeToLiveSeconds="2592000"
                 diskPersistent="false"
                 timeToIdleSeconds="0"
         />
 
         <cache name="plantCode"
                 maxElementsInMemory="1000"
                 maxElementsOnDisk="100000"
                 overflowToDisk="false"
                 timeToLiveSeconds="2592000"
                 diskPersistent="false"
             timeToIdleSeconds="0"
         />

This is used in WebLogic with side-by-side deploys. We have the shutdown listener in place. We're looking at 8 total boxes in a weblogic cluster, local cache location specific to each configuration (set through programatic configuration in a static method which loads the cache manager to a host-specific drive location). We've seen this though even on local workstations. Overall, at the end where the problem occurred, we've seen local cache files over 100MB in size. At this point, we've removed EHCache entirely, but we're looking to try and replicate the issue with load or other factors. Last, we'll probably try disabling the disk persistent system and persist to disk parameters, and just use in memory caching if we decide to go with EhCache at all.

Adding to the above, using this code:
Code:

 public abstract class AbstractCache<K, V> {
         private final EhCacheWrapper <K,V> cache;
         public AbstractCache(String cacheName) {
                 this.cache = new EhCacheWrapper <K,V>(cacheName);
         }
 
         public V get(K key) {
                 V resultsFromCache = cache.get(key);
                 if (resultsFromCache == null) {
                         V resultsFromOracle = getFromOracle(key);
                         cache.put(key, resultsFromOracle);
                         resultsFromCache = resultsFromOracle;
                 }
                 return resultsFromCache;
         }
         protected abstract V getFromOracle(K key);
         public void clear() {
               cache.getEhCache().removeAll();
         }
 }

along with the standard released cache wrapper code modified to the below (adds a few utils to disable all references to the cache):
Code:

 public class EhCacheWrapper<K, V> {
         private final String cacheName;
         private static boolean cachingEnabled = true;
       static final CacheManager cacheManager = init();
       static {
               MBeanServer mBeanServer = ManagementFactory.getPlatformMBeanServer();
               ManagementService.registerMBeans(cacheManager, mBeanServer, false, true, true, true);
       }
 
       protected static CacheManager init() {
               InputStream cacheInputStream = EhCacheWrapper.class.getResourceAsStream("/ehcache.xml");
               Configuration cacheConfig = ConfigurationFactory.parseConfiguration(cacheInputStream);
               cacheConfig.diskStore(new DiskStoreConfiguration().path(getLocationForCache()));
               CacheManager cacheManager = CacheManager.create(cacheConfig);
               return cacheManager;
       }
 
       protected static String getLocationForCache() {
               String diskStoreLocation = System.getProperty("java.io.tmpdir");
               if (NetworkTools.isLocalMachine()) {
                       diskStoreLocation = "C:/ehCache";
               }
               if (NetworkTools.isProductionlMachine()) {
                       diskStoreLocation = "D:/ehCache";
               }
               return diskStoreLocation;
       }
 
         public EhCacheWrapper(final String cacheName) {
                 this.cacheName = cacheName;
               getEhCache().setStatisticsEnabled(true);
         }
 
         public void put(final K key, final V value) {
               if (cachingEnabled ) {
                       getEhCache().put(new Element(key, value));
               }
         }
 
         @SuppressWarnings("unchecked")
         public V get(final K key) {
               if (cachingEnabled ) {
                       Element element = getEhCache().get(key);
                       if (element != null) {
                               return (V) element.getValue();
                       }
               }
                 return null;
         }
 
       public Ehcache getEhCache() {
               return cacheManager.getEhcache(cacheName);
       }
 
       public static CacheManager getCachemanager() {
               return cacheManager;
       }
 
         public static void setCachingEnabled(boolean cachingEnabled) {
                 EhCacheWrapper.cachingEnabled = cachingEnabled;
         }
 
         public static boolean isCachingEnabled() {
                 return cachingEnabled;
         }
 
         public static void setStatisticsEnabled(boolean enabled) {
               for (String cacheName : CacheManager.getInstance().getCacheNames()) {
                       getCachemanager().getCache(cacheName).setStatisticsEnabled(enabled);
               }
         }
 
 }

The exceptions you're seeing are because the in-heap structure within Ehcache is claiming a valid serialized value is present at a specific file offset and yet none is found there. From the information I have so far it's hard to say what might be going wrong. If you could supply me with answers to the following questions that would be helpful.

1. Are the disk store files being written to on a local or a remote (network) disk?
2. Does this only happen when using a previously persisted cache (i.e. not on first run)?
3. What JVM are you running?
4. How do you terminate the JVM?
5. Do you move the Ehcache disk store files around at all?

Hopefully this information will help to narrow down the circumstances that trigger the issue, and allow us to work towards getting a reproducible test case.

Thanks,

Chris

The files are written to a local disk. The IO didn't seem to be very high at all when monitoring disk IO. Higher obviously than without caching.

Initially the thought was that it was after a new deploy, but it looks like we've also seen it without a new deploy.

JVM Is JRockit with weblogic 9.2 (JRockit 1.5.0_12)

It varies on how we terminate the JVM's. Usually it's done through the various control panels, unless there are stuck threads or the box is not responding.

We don't move the disk store files at all.

As stated previously, we do have the listener in the web.xml to "shutdown" the cache manager.

Going to do some load testing over the weekend to see if we can replicate the issue.

Is there any fix or resolution for this issue?

I see this exception in my application. Any inputs are highly appreciated