Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

I'm using tomcat 6.0.13 and terracotta-2.4-nightly-rev4149. I added an HttpSessionListener to my web app to observe session creation and destruction with Terracotta, and I'm using JMeter to create sessions with an HTTP request sampler. I am also using a load balancer to distribute the requests across a cluster of 5 tomcat servers.

I can see the sessions getting created in a distributed fashion. However, when the session expires on each of the servers, only one of the servers actually shows a "session destroyed" event, but for all the sessions created. For example, if I issue 10 requests to the load balancer, then two sessions are created for each server. But, when the sessions time out, all of the sessions are expired on a single server, so it looks like 10 sessions are destroyed on that server instead of two for each server.

At first I suspected some routing issue. However, I'm able to reproduce this behavior with the session-configurator.sh app on my desktop without a load balancer. I use JMeter to point to localhost:9081 and localhost:9082 and create ten sessions on each instance, and only on one of them twenty sessions are invalidated.

Is this expected behavior? If so, why isn't the session invalidation distributed? If not.. then is there a bug filed for this somewhere? It could be harmful to a webapp that relies on this event handler to be fired more than expected.

That sounds likes the expected behavior (coming from someone who knows the Terracotta code at least). The session invalidator (ie. the thing/thread that reaps expired sessions) only executes on a single node within your cluster for any one particular sweep. It isn't really possible to determine which node the session will be destroyed on (but it will be one and only one node). There is no guarantee that the session will be destroyed on the node where it was first created (as you have observed)

You should be able to see the invalidation occurring in more distributed fashion if you watch over a longer time period. The invalidation thread in each node wakes up at the configured interval (5 min default) -- if that thread finds that another node is currently running invalidation, it just goes back to sleep again.

I don't want to dismiss the idea that there might still be a bug in here, please let me know what you think

After some poking around, I eventually came to the same conclusion about the single node reaping. Though it's correct, it seems odd that a single node faults-in session objects from the other nodes in order to reclaim memory. Wouldn't there be a lot of I/O between the TC server and the reaper node, particularly in large clusters? Presumably the other nodes are flushing their session objects to the TC server, then the TC server faults in the session state to the reaper node. It also would seem that there's the potential that the TC server would end up writing a lot of state to disk if it becomes memory constrained during this process. Has this been a performance issue?

We haven't experienced a performance problem of that nature in our test lab. Although it is not visible in the TC admin console, there is another root which holds the last access time for sessions. When a reaper node is running an invalidation, it is at most pulling a single long value (ie. not the actual session object). There is also a trick to how those last access times are updated to optimize things a little further.

Given that nodes might come and go over time, any scheme that solely depends on the creator node (or even last accessor node) to do it's own session invalidation leaves open the possibility that sessions may never be expired.

hope this helps