Terracotta Discussion Forums

I have the following configuration:
- a cluster of 2 terracotta nodes version : Terracotta 3.5.3, as of 20111005-171028 (Revision 18868)
- the servers are started with -Xms12g -Xmx12g
- I use also the local EhCache.
- I have to put in the cache a structure like:
CacheKey {
private final long projectID;
private final Long subprojectID;
private final String fileID;
private final Integer segmentID;
}

CacheValue {
private final Map<CacheKey, Status> members;
private Status overallStatus;
}

these objects form a hierachy, and for many objects in hierarchy, the CacheValue can have up to 50000 elements in members.

The problem is that after I insert these elements in memory (50000 CacheKeys+CacheValues with just one member + 1 CacheKey with one CacheValue with 50000 members) communication with terracotta becomes very slow, taking up to few hundreds of milliseconds for a put, or even seconds when GC is running on terracotta servers, and after some hours one of the server crashes with OutOfMemory.

Also, when I restarted the crashed server, it took it around 3 hours to recovery. Which is not acceptable for 50000 objects in cache.

Can you give me some advice how to solve this. Please note that Enterprise Terracotta and BigMemory are not a fesable solution for me.
And also, can you explain me why when I just insert/remove an object from a distributed cache, the number of live objects that I see in the Developer Console constantly rises and it's around hundreds of thousands or even million, until the Distributed Garbagge Collecor steps in.

Can I avoid using DGC, and let the local GC to cleanup all the objects?
Are there any methods to use distributed search on community version of Terracotta?

Thank you,
Stefan.

A few things

a) When you say, cluster of 2 - you perhaps mean one mirror group with active and mirror. 12GB of heap is a lot. It possibly means lots of GC's on the Terracotta side which might need additional tuning. Be wary of that.

b) You need to move to the latest version of TC - 3.5.3 is an older version.

c) What is the size of one object? And the total data set size. You need to size your ehcache.xml as per these findings.

d) With 3.6.2, you can disable DGC - its performed inline. So you should be good there.

Aside from the above generic points, its hard to comment more without actively looking at your Terracotta logs, tc_config. Your usecase might be on the borderline b/w choosing Enterprise and going with open source. A call with a Terracotta sales rep might perhaps help you with the options you might have. Good Luck.

Thank you for your prompt response.

a) yes, it's a master-slave. Indeed, the GC is a bit intrusive, and affects the performance.
b) I will.
c) the big object ( containing 50K references) has around 1.5 Mb. This value is taken from profiler ( deep size) and also confirmed by manually calculating the Java object sizes. What properties from ehcache.xml are you reffering to?

- another observation is that updating a member from this CacheValue takes a lot of time (900 milliseconds), also putting it into cache takes 200 around millis. Is this expected for big objects? Can a lot of updates on this objects cause an OutOfMemory Exception on the distributed server?

d) this means that the cache will rely only on JVM GC by default? That sounds great.

Stefan.