[Logo] Terracotta Discussion Forums
  [Search] Search   [Recent Topics] Recent Topics   [Members]  Member Listing   [Groups] Back to home page 
[Register] Register / 
[Login] Login 
[Expert]
Memory usage in Ehcache  XML
Forum Index -> Ehcache
Author Message
pshukla

neo

Joined: 02/24/2011 04:58:29
Messages: 5
Offline

Hi I want one clarifications :

Is Ehcache Stores in memory index of all the elements that has been persist in disk , If yes can we restrict it . Actually , i have 5 million key/value pairs to be cached and when ever i am loading the cache from the cache manager it fills up lots of memory. So as the no of records increases the memory consumption increases . looking at profiler i found that following are the Objects Holding major part of RAM.
[[
One instance of "net.sf.ehcache.store.compound.impl.DiskPersistentStore" loaded by "sun.misc.Launcher$AppClassLoader @ 0x1a5ceef0" occupies 172,405,256 (58.91%) bytes. The memory is accumulated in one instance of "net.sf.ehcache.store.compound.Segment[]" loaded by "sun.misc.Launcher$AppClassLoader @ 0x1a5ceef0".

Keywords
sun.misc.Launcher$AppClassLoader @ 0x1a5ceef0
net.sf.ehcache.store.compound.impl.DiskPersistentStore
net.sf.ehcache.store.compound.Segment[]




]]
Ehcache implementation has brought a huge increase in the performance but due to memory usage the JVm is becoming unstable(crashing all the time with OOME).

Could anyone please help me on this.
Thanks
cdennis

master

Joined: 01/19/2009 10:03:26
Messages: 89
Offline

As you have noticed the open-source disk stores do not store the entire cache on disk. Instead the values are stored on the disk, and the keys are stored in memory mapped to value objects which reference where on disk the associated value object can be found. This means each on-disk mapping has an associated in-memory overhead equal to it's key-size plus a fixed overhead per mapping.

Assuming that you cannot increase the heap size and/or cache less of your data set there are really only a couple of other avenues open to you. Firstly if your keys are large objects you can reduce their size (although this isn't normally that common). Secondly you may be able to reduce the number of mappings that you store by restructuring your 5 million key/value pairs in to a smaller number of mappings that have much larger value objects. The value objects in this case may even be maps themselves, which hold the original value objects.

Hope this helps,

Chris

Chris Dennis (Terracotta Engineer)
pshukla

neo

Joined: 02/24/2011 04:58:29
Messages: 5
Offline

Hi Chris ,

Thank you very much for you reply this made me clear that with open source ehcache it always maintains the in memory index ,which consumes lots of memory and there is no out of the box way to get rid of it.

Could you please confirm me that if we can achieve the same with enterprise ehcache.
Is BigMemory concept will help for this case .

We have to cache key/value pair for 5 to 10 million records in a box with 4 GB RAM.
[In our case key size is little bigger i.e. 60 chars]

Could you pleas advise me on the same.

Thank you.
kbhasin

consul

Joined: 12/04/2006 13:08:21
Messages: 340
Offline

Enterprise Ehcache with BigMemory will definitely help you in this instance. My guess is that with BigMemory you will probably end up needing a lot less than even 4GB as all keys and values can be stored on BigMemory instead of the Java Heap and hence do not need to go through the Java GC leading to much smaller GC pauses and most likely no OOMEs.

You can download the latest version of BigMemory here: http://www.terracotta.org/dl/download-bigmemory and the documentation for BigMemory is available here: http://ehcache.org/documentation/offheap_store.html

Regards,

Kunal Bhasin,
Terracotta, Inc.

Be a part of the Terracotta community: Join now!
pshukla

neo

Joined: 02/24/2011 04:58:29
Messages: 5
Offline

Thank you , i will give it a try .

One more thing i wanted to know , what are the scenarios where cache get corrupt. I get Invalid Stream Header sometime .
[[
net.sf.ehcache.CacheException: java.io.StreamCorruptedException: invalid stream header: 624D5F05

]]
Is this issue as well addressed in this Enterprise Ehcache.

Please advise me on the same.

Thanks a lot.
kbhasin

consul

Joined: 12/04/2006 13:08:21
Messages: 340
Offline

Can you share the entire stack trace please?

Regards,

Kunal Bhasin,
Terracotta, Inc.

Be a part of the Terracotta community: Join now!
pshukla

neo

Joined: 02/24/2011 04:58:29
Messages: 5
Offline

Yes , please find it here

[[
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : Unexpected error :
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : net.sf.ehcache.CacheException: java.io.StreamCorruptedException: invalid stream header: 624D5F05
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at net.sf.ehcache.store.compound.factories.DiskPersistentStorageFactory.retrieve(DiskPersistentStorageFactory.java:209)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at net.sf.ehcache.store.compound.factories.DiskPersistentStorageFactory.retrieve(DiskPersistentStorageFactory.java:59)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at net.sf.ehcache.store.compound.Segment.decode(Segment.java:168)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at net.sf.ehcache.store.compound.Segment.get(Segment.java:206)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at net.sf.ehcache.store.compound.CompoundStore.get(CompoundStore.java:156)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at net.sf.ehcache.store.compound.CompoundStore.getQuiet(CompoundStore.java:163)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at net.sf.ehcache.Cache.searchInStoreWithoutStats(Cache.java:1718)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at net.sf.ehcache.Cache.get(Cache.java:1375)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at net.sf.ehcache.Cache.get(Cache.java:1335)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at com.bmc.atrium.ngie.CMDBLookUp.cache.AICacheManager.get(AICacheManager.java:204)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at com.bmc.atrium.ngie.CMDBLookUp.CMDBLookUp.lookupValues(CMDBLookUp.java:235)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at com.bmc.atrium.ngie.CMDBLookUp.CMDBLookUp.processRow(CMDBLookUp.java:735)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at org.pentaho.di.trans.step.BaseStep.runStepThread(BaseStep.java:2889)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at com.bmc.atrium.ngie.CMDBLookUp.CMDBLookUp.run(CMDBLookUp.java:971)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : Caused by: java.io.StreamCorruptedException: invalid stream header: 624D5F05
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at java.io.ObjectInputStream.readStreamHeader(Unknown Source)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at java.io.ObjectInputStream.<init>(Unknown Source)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at net.sf.ehcache.store.compound.factories.DiskStorageFactory$2.<init>(DiskStorageFactory.java:247)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at net.sf.ehcache.store.compound.factories.DiskStorageFactory.read(DiskStorageFactory.java:247)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : at net.sf.ehcache.store.compound.factories.DiskPersistentStorageFactory.retrieve(DiskPersistentStorageFactory.java:201)
2011/02/25 00:18:18 - CMDBLookUp.0 - ERROR (version 3.2.4-stable, build 11672 from 2010-01-28 11.09.00 by buildguy) : ... 13 more

]]

Here CMDBLookUp is our code where we have created a class AICacheManager where in we are accessing all the cache related stuff such as creation and accessing.

Thanks
kbhasin

consul

Joined: 12/04/2006 13:08:21
Messages: 340
Offline

This can possibly happen with the open source disk store if the java process is not shut down cleanly.

You will not face this issue with Enterprise Ehcache with Terracotta as the disk persistence is transactional.

Regards,

Kunal Bhasin,
Terracotta, Inc.

Be a part of the Terracotta community: Join now!
pshukla

neo

Joined: 02/24/2011 04:58:29
Messages: 5
Offline

Hi,

We are in process to get the enterprise version of ehcache. meanwhile to do another round of testing i need a trial version.

The problem is i have already used trial version once on my machine which expired on 25-Mar-11. Now when i download the new jars and license key and using that . It is giving me a message that you license expired on 25-mar-11.

Is there any way i can do one more round of testing.

Please reply its little urgent.

Thanks,
Pramod
rajoshi

seraphim

Joined: 07/04/2011 04:36:10
Messages: 1491
Offline

Issue seems to be resolved.Please let us know if more information is required.

Rakesh Joshi
Senior Consultant
Terracotta.
 
Forum Index -> Ehcache
Go to:   
Powered by JForum 2.1.7 © JForum Team