Terracotta Discussion Forums

Just to let you know, we have reproduced the issue with the test code you provided. We'll let know when we have anymore information

Hi Paul -- Thanks for the detailed information. I can't think of what the Terracotta instrumentation would be doing to interfere with a custom Vector marshaller, but clearly something is wrong.

I'm downloading a trial version of Visibroker right now, it appears to be version vb7-06.06.00.C3.02. Any idea if that version is (or is not) suitable for recreating the problem? Which version of Visibroker are you using?

I can't seem to find anything about this case in the documentation -- sorry about that. I'll put a documentation bug in JIRA about it

Specifying a web application name of "ROOT" should enable Terracotta sessions for apps that use the default context (ie. "/" context path)

<web-applications>
<web-application>ROOT</web-application>
</web-applications>

not to belabor the point, but definitely confirm the presence of the root, and two connected clients in the terracotta admin console first thing. If you see no connected clients, then the tomcat startup needs to be reviewed (in particular make sure the Xbootclasspath is correct). If there are clients connected, but no roots visible, make sure the <web-applications> includes the context path of your web application

That setup seems reasonable and TC should provide the session data no matter which backend webserver is accessing it.

Does it appear that new sessions are being created on every request, or only on those requests for which a "hop" has occurred? (by "hop" I mean that request is being processed by a different server from the server who processed the previous request on the same session).

One thing that comes to mind is that the actual JSESSIONID cookie values might be in an unexpected format when using TC. The reason I mention this is that some load balancers depend on the format the sessionID to determine how to make "sticky" routing decisions. Do you know how your LB implements it's sticky feature? All that said, even if the load balancer was doing simply round robin (ie. non-sticky), the session data should still be available to all backend servers.

My best guess is still that the session cookie values are getting lost someplace, or that TC clustering is not enabled (which is where Taylor's suggestions of using TC admin console would come into play)

It sounds like the session cookie is not getting sent from one request to the next.

One tool I find very helpful for session cookies is the "web developer" add-on for firefox. It shows a nice view of the cookies that apply to a given page.
https://addons.mozilla.org/firefox/60/

It might be interesting to inspect the differences in the session cookie issued with and without clustered session. Of particular interest are the domain and path attributes of the cookie.

In the first post you mentioned you are using a load balancer, which one is that? I'm just trying to figure out if it is interfering somehow.

When you are testing requests between servers, are you going through the load balancer, or just changing the host in your browser manually?

I apologize, I did not look at all of the files you had attached. Your stack trace does not appear to involve the HttpSessionListener.sessionDestroyed() event callback.

I recommend attaching a debugger and breaking on the method com.terracotta.session.util.StandardSession.setInvalid() to observe when the session is being marked invalid. Another approach would be to download the terracotta source code and insert some log statements yourself. Some more information on building terracotta code can be found here:
http://www.terracotta.org/confluence/display/orgsite/Building+Terracotta

I also created a new feature request around this too
http://jira.terracotta.org/jira/browse/CDV-158

I apologize, I did not look at all of the files you had attached. Your stack trace does not appear to involve the HttpSessionListener.sessionDestroyed() event callback.

I recommend attaching a debugger and breaking on the method com.terracotta.session.util.StandardSession.setInvalid() to observe when the session is being marked invalid. Another approach would be to download the terracotta source code and insert some log statements yourself. Some more information on building terracotta code can be found here:
http://www.terracotta.org/confluence/display/orgsite/Building+Terracotta

I also created a new feature request around this too
http://jira.terracotta.org/jira/browse/CDV-158

I've added this as new feature request. Thanks for the suggestion

http://jira.terracotta.org/jira/browse/CDV-157

You are correct about the <additional-boot-jar-classes> section, it does not accept patterns.

I'll search our JIRA database (http://jira.terracotta.org) to see if there is a feature request concerning this, and if not, I'll add one and post a link to it here

I actually ran across a similar looking exception (IllegalStateException) just now. It was happening in the standard sessionDestroyed() session event callback. Apparently the Terracotta implementation disallows attribute access while a session is being actively invalidated (in this detail it appears we differ from the tomcat implementation -- I'm not sure who the servlet spec agrees with).

Anyway, I was wondering what the complete stack trace was when you got the IllegalStateException? If I knew that I could say whether you are experiencing the same issue.

-tim

Unfortunately there is not currently a way to dynamically instrument (for use in terracotta distributed object graphs) classes that are defined by the bootstrap classloader (in general that means any class that is part of the standard java APIs). Those classes must be pre-instrumented

I don't have a good suggestion, other than throughly testing the application to know for sure whether any bootstrap classes have been missed. In general I have not seen cases where more than handful of extra classes need to be added to the boot jar (above the ones that are preconfigured, like java.util classes, etc).

If your context is purely from a library standpoint, and you have no idea what the client usage of your shared objects will be, then the problem is currently unbounded with regards to bootstrap classes.

We're actively ooking into other techniques (java agent, jvmti, etc) to ease the burden here

I think Steve is asking whether any synchronization (ie. java keyword synchronized) anywhere in the call stack before the list.add()?

A more direct question is whether there is an synchronization in any of these 3 methods?
com.quartetfs.sample.pivot.Main.activePivotSetup()
com.quartetfs.sample.pivot.Main.doTest()
com.quartetfs.sample.pivot.Main.main()

If there is no synchronization present, the autolock expression in the terracotta configuration does absolutely nothing.

If there is synchronization, is the target of the synchronization the root list object, or something else?

You need something like this:

synchronized(pivots) {
pivots.add(..);
}

hope this helps
Tim Eck (terracotta)

This issue will also be tracked in this jira item
http://jira.terracotta.org/jira/browse/CDV-127

I'll be investigating this further, but to help us diagnose this problem, has that class (com.ra6.sphe.common.pojos.BlogPost) been evolving since you started sharing instances of it? What I mean by "evolving" is that the fields have been added/removed from it over time in your source code.

Class evolution isn't a problem for terracotta, but if it is in play here, it might help our investigation. thanks