Terracotta Discussion Forums

I was trying to track down why loading from a populated EHCache instance takes so long.

I've attached a file (20080314-app.log) that has a snippet from my client (Tomcat), the Terracotta client log and the server log around the same time. As you can see the initial load from the Tomcat section took 53 seconds and the second request took 1.6 sec. When I looked at the Terracotta client log, I saw the cache hit/miss entries around the same time for the first one and nothing for the second one. The cache entry requested is fairly small (~600 objects). I'm a little concerned that it took almost a minute to move the cache element from L2 to L1 over a GB network. But that's another issue.

If you continue down the log, you'll see at about the same time from the server log, the fault counts start going through the roof. Since it appeared to be running away, I decided to shutdown the L2 and restart it without restarting the L1s.

Once I restarted L2, within a couple of minutes I received a divide by zero error (20080314-terracotta-server.log).

I'm not real sure what's happening, but hopefully there is enough information to point me in the right direction. I'm hoping the right direction is not scrapping EHCache...

Unfortunately, there is not a sample app that I can give to try to duplicate this issue. At this point, we are in a load testing environment with multiple boxes and instances running. However, I cannot make this happen when Terracotta is not being used or when there is a single L1 and L2 running on the same box.

Thanks,

Darin

Windows 2003 server
JDK 1.5.0_10
Tomcat 5.5.25
Spring 2.0.4
EHCache 1.2.4
Terracotta trunk-nightly-rev7223

tc.properties (but best way is to extract from the terracotta-server.log):

ehcache.concurrency=64
session.id.length=16
l2.cachemanager.percentageToEvict=20
l2.cachemanager.criticalThreshold=70
l2.cachemanager.threshold=70
l2.nha.tribes.mcast.mcastAddress=228.0.0.6
l2.nha.tribes.mcast.mcastPort=7779
l2.nha.tribes.mcast.tcpListenPort=4002
ehcache.logging.enabled=true
ehcache.evictor.logging.enabled=true
l1.cachemanager.logging.enabled=true
l1.objectmanager.remote.logging.enabled=true
l1.transactionmanager.logging.enabled=true
l1.transactionmanager.strings.compress.logging.enabled=true
l2.cachemanager.logging.enabled=true
l2.lfu.debug.enabled=true
l2.objectmanager.fault.logging.enabled=true
l2.transactionmanager.logging.enabled=true
l2.transactionmanager.logging.verbose=true

Hi,

Can you also share your ehcache.config file.
Thanks,

It's in the server log, but I have now attached it.

Darin

I'm not sure what the problem is, but it seems worth noting that you're on the bleeding edge using a non-stable trunk build.

The ehcache TIM is no longer being maintained in the main source tree (as of 2.5.2) and will no longer be in any future kits.

It is now being maintained as a module on the Forge (http://forge.terracotta.org). More specifically the tim-ehcache is at http://forge.terracotta.org/projects/tim-ehcache/. The latest released version is 1.0.2 and the trunk is 1.1.0-SNAPSHOT.

I had to go to a trunk version because of a fix that I needed that came in after 2.5.2.

I'm working on swapping out from EHCache to ConcurrentHashMap to see if I have the same issues or not. I'll post my findings.

Darin

From the logs it looks like you are using the local disk as the permanent store for the TC Server (C:/terra-server-data).

To zero in on the issue, we can start with a couple of things:

1. Turn off the Distributed Garbage Collection:

<garbage-collection>

<enabled>false</enabled>

BTW, how much is the total data size here and are you monitoring the the TC Server and Application heaps to see if any GCs are happening?

2. Increase the <fault-count> of the TC clients:

<clients>
<dso>
<fault-count>2000</fault-count>
</dso>
</clients>

Default is 500 and 2000 is just a wild guess on my part, you will have to play around with this number some.

3. As far as the division by zero arithmetic error, I would follow Alex's suggestion and move to a stable build and see if it still happens.