Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

I apologize, I did not look at all of the files you had attached. Your stack trace does not appear to involve the HttpSessionListener.sessionDestroyed() event callback.

I recommend attaching a debugger and breaking on the method com.terracotta.session.util.StandardSession.setInvalid() to observe when the session is being marked invalid. Another approach would be to download the terracotta source code and insert some log statements yourself. Some more information on building terracotta code can be found here:
http://www.terracotta.org/confluence/display/orgsite/Building+Terracotta

I also created a new feature request around this too
http://jira.terracotta.org/jira/browse/CDV-158

I apologize, I did not look at all of the files you had attached. Your stack trace does not appear to involve the HttpSessionListener.sessionDestroyed() event callback.

I recommend attaching a debugger and breaking on the method com.terracotta.session.util.StandardSession.setInvalid() to observe when the session is being marked invalid. Another approach would be to download the terracotta source code and insert some log statements yourself. Some more information on building terracotta code can be found here:
http://www.terracotta.org/confluence/display/orgsite/Building+Terracotta

I also created a new feature request around this too
http://jira.terracotta.org/jira/browse/CDV-158

Hi TIM,
Thanks for that. Will do the debugging. In the mean time I guess it would be practical to do a workaround for the creating a session immediately after invalidating one for the same user. Would definitely be very interested in any suggestions you may have. I still haven't got the sessions clustered though due to the other problems. Once i can get the session invalidation problems out of the way, would like to see the fail over due to clustering. Hope to get it up today. Will post on any errors.

Thanks,

Regards,
Anoop Gopalakrishnan
Project Manager,
Device Driven
www.devicedriven.com

Hi Tim/Sreeni/Gordon,
I got all the illegal state exceptions out by removing the invalidate statements and instead checking for tokens inserted during authentication. However i still am not able to get a clustered session. The problem being that on every request that goes to the server, it is in the form of a new session, i.e there is no memory for the earlier session. The session timeout has been set at 30 mins , so it could not have been due to the session timeout. I tried working out of the clustered environment and it works fine. Only on the clustered environment, does it ask for the login for every click (i.e the session is getting invalidated on any request). We are using the getSession(false) method to get the existing session, but it returns a new session everytime. Any help is very welcome. We are using Struts as our MVC framework.

Thanks,

Regards,
Anoop Gopalakrishnan
Project Manager,
Device Driven
www.devicedriven.com

It sounds like the session cookie is not getting sent from one request to the next.

One tool I find very helpful for session cookies is the "web developer" add-on for firefox. It shows a nice view of the cookies that apply to a given page.
https://addons.mozilla.org/firefox/60/

It might be interesting to inspect the differences in the session cookie issued with and without clustered session. Of particular interest are the domain and path attributes of the cookie.

In the first post you mentioned you are using a load balancer, which one is that? I'm just trying to figure out if it is interfering somehow.

When you are testing requests between servers, are you going through the load balancer, or just changing the host in your browser manually?

Hi Tim,
I am using Zeus as the load balancer. Yes, I am using the LB while testing. I first check from the load balancer active connections , the currently serving web server and then kill that process to check if the session is forwarded as the same to the second web server. I am not sure myself if it is the LB thats causing the trouble. Will check with the firefox add-on and get back to you. In the mean time, thanks for the responsiveness shown by all at Terracotta. I really hope you guys make it big! Hope to be using it also for the hibernate second level cache.

Thanks,

Regards,
Anoop Gopalakrishnan
Project Manager,
Device Driven
www.devicedriven.com

By the way could you please tell me by what protocol is the data transferred between the client and the terracotta servers? Does it use multicast for multiple client machines? If it does, is it possible to use it in a multicast disabled cluster environment?

Thanks,

Regards,
Anoop Gopalakrishnan
Project Manager,
Device Driven
www.devicedriven.com

Zeus wrote:

By the way could you please tell me by what protocol is the data transferred between the client and the terracotta servers? Does it use multicast for multiple client machines? If it does, is it possible to use it in a multicast disabled cluster environment?
www.devicedriven.com

It's TCP, actually, no multicast.

Zeus wrote:

Hi Tim,
I am using Zeus as the load balancer. Yes, I am using the LB while testing. I first check from the load balancer active connections , the currently serving web server and then kill that process to check if the session is forwarded as the same to the second web server. I am not sure myself if it is the LB thats causing the trouble. Will check with the firefox add-on and get back to you. In the mean time, thanks for the responsiveness shown by all at Terracotta. I really hope you guys make it big! Hope to be using it also for the hibernate second level cache.
www.devicedriven.com

I'm not sure I follow this procedure - are you saying that you point your web browser to the LB and then find out what active server is running and then point your browser to the server? That wouldn't work because of cookie domains...but I imagine this isn't the case because you say you have it working without TC.

The first thing I always do when debugging Terracotta issues is to use the Admin Console. It is extremely useful to find out if what you think is going on is actually going on. In your particular case, once you connect the Admin Console to the TC server, you should see a "+" next to the Roots section. If you open the roots, you should see a root for each Web App that is configured, usually it's something like tc:tomcat_sessionXXX where XXX is the name of the Session. You can open up the session and see your Session data. If you aren't able to get this far with a single node, you definitely won't be seeing clustered data on two nodes.

If you have made it past the step above, then you have to debug your connection to the web servers.

The way I usually test failover is to use a very very simple Load Balancer. I found one some time back called balance : http://rpmfind.net/linux/rpm2html/search.php?query=balance

Then what I do is start the LB and point it only at one server. I make a few requests, then shut the LB down, and point it at the next server. This forces the "sticky" and "failover" scenarios.

Hope that helps!

H Taylor,
The procedure I followed is as follows:
1. Two linux boxes were set up as web servers
2. One box set up as Load Balancer.
3. The LB is configured to have sticky sessions , so all the requests from a client would be served from the same server through the LB unless that server come down.
4. The browser always points only to the LB ip.
5. Once a session has been setup and one of the servers start serving the sticky browser session (and after it has gone some distance so that there is enough data in the HttpSession), I do a kill -9 on the server which is serving.
6. Now the loss of one node would be picked up by the load balancer and all the subsequent requests would be routed to the next web server which was idle till now.
7. Ideally, if the session objects were replicated by TC, the user should be able to continue normally without even having noticed that one server had come down (w.r.t speed).

I hope I have been lucid. I am still doing the code review to check for possibilities where a session invalidation could be causing such an error. Got caught up in some other matter of urgency. Will keep you posted on the issue.

Thanks,

Regards,
Anoop Gopalakrishnan
Project Manager,
Device Driven
www.devicedriven.com

That setup seems reasonable and TC should provide the session data no matter which backend webserver is accessing it.

Does it appear that new sessions are being created on every request, or only on those requests for which a "hop" has occurred? (by "hop" I mean that request is being processed by a different server from the server who processed the previous request on the same session).

One thing that comes to mind is that the actual JSESSIONID cookie values might be in an unexpected format when using TC. The reason I mention this is that some load balancers depend on the format the sessionID to determine how to make "sticky" routing decisions. Do you know how your LB implements it's sticky feature? All that said, even if the load balancer was doing simply round robin (ie. non-sticky), the session data should still be available to all backend servers.

My best guess is still that the session cookie values are getting lost someplace, or that TC clustering is not enabled (which is where Taylor's suggestions of using TC admin console would come into play)

Zeus wrote:

H Taylor,
The procedure I followed is as follows:
...

I hope I have been lucid.

Sounds right to me. I would first make sure you are certain the HTTP Sessions are being clustered using the Admin Console.

not to belabor the point, but definitely confirm the presence of the root, and two connected clients in the terracotta admin console first thing. If you see no connected clients, then the tomcat startup needs to be reviewed (in particular make sure the Xbootclasspath is correct). If there are clients connected, but no roots visible, make sure the <web-applications> includes the context path of your web application