Terracotta Discussion Forums

Sorry I forget to answer some of the question...

It should be possible to instrument classes in an EAR. We hook into every single classloader that exists in the VM.

The custom security provider might be an issue, but I don't think it would be the cause of the InvalidClassException. I'm not actually sure what you mean by that exactly. Do you mean something that deals with code permission, or something that deals with authentication/authorization (ala JAAS). Both could be problematic, just in different ways :-)

I'll have to ping someone else about the timeline for WLS9

Your analysis is correct and load balancing components that depend on specific cookie formats are bound to break (or require adjustment) with terracotta sessions.

It sounds like we have some old documentation out there. To adjust the serverid component, create a file called "tc.properties" and place it in the common/lib folder of the terracotta distribution. In that properties file, add this line:

Code:

session.serverid=.myJvmRoute

I'll throw something in JIRA about the documentation issue. If you happen to know the URL, posting it here would help.

Hi there,

WLS 9.2 is not a supported weblogic platform at the moment, we only officially suport 8.1. That said, it is conceivable that some things might work in 9.2+. I'm downloading the 900MB install right now to give it a try :-)

Is there a stack trace associated with that InvalidClassException? I'm mostly wondering if the exception happens in weblogic code or terracotta code.

I wonder if perhaps you have tc.jar in CLASSPATH, or as part of you application/ear? If so, you should remove it.

Good deal, glad to hear you got to the bottom of it. Out of curiosity -- to the best of your knowledge was the issue related to Terracotta at all?

Mostly I'm wondering if you are using terracotta for session management in your web application. In that case I wouldn't be surprised if the format of the JSESSIONID cookie generated by terracotta was somehow not compatible with the sticky routing logic in mod_jk.

Your post sounds like there is a NullPointerException (NPE) in there too? Is it the cause of the ServletException that you posted? Is the stack associated the NPE that you can post?

Few more questions:
- what terracotta version are you running?
- Struts version
- app server version
- java version
- your tc-config.xml

If possible, a sample struts app with some terracotta config that demonstrates the issue will gaurantee a speedy fix.

Do all the clients and/or the terrocotta server have to be running the same major and/or minor version of java? does the client and/server platforms have to be homoginous?

Nope. You can mix and match the versions for the most part. Each client java version will require a boot jar specific the version in use. We recommend that the server is always run with 1.5 VM. Finally, you'll run into problems if you try to access 1.5 specific type (e.g. ConcurrentHashMap) from a client running 1.4 (hopefully this last one is obvious).

I noticed the jsessionid's format changed with a '!' and channelId appended to it.

The number after the '!' is an unqiue identifier for the cluster node. That bit of information is useful for load balancers to make sticky routing decisions, and it let us detect when requests ping-pong between servers.

My understanding of how terracotta worked is that when I access an object in the shared heap, I accessing the same instance, but when I output the value of the object in a jsp page it had a different object id (com.package.SomeClass@objid(values...)) but the same values. Shouldn't that object id be the same on both clients?

I can understand why you think that the hashcode values would be the same between nodes, but that is not something DSO provides. Of course, if you have your own hashcode() method, then they should be consistent between VMs. Technically would have to modify native portions of the VM to force identity hashcodes to have particular values, there isn't a bytecode level mechanism to do something like that really.

, the terracotta adminstrator console lists the classes persisted in serverName --> DSO --> Classes. I imagine that's a good reference to use to setup the tc-config.xml instrumented-classes

That is a good place to start -- It is a list of the unique type names that have been distributed. Trimming the instrumentation to minimum set should help performance.

Out of curiosity, what is the benefit of a clustered connection pool versus local connection pools on each node? I'm mostly wondering if it is the actual database connections that need a "cluster" aspect, or is it the connection information (host. port, schema, user, etc)? If it is the latter, then there is definitely some opportunity for using DSO here

This particular issue is fixed in the 2.3.x release. You can get a 2.3.x build from here:

http://www.terracotta.org/confluence/display/orgsite/Download

I've been told but can't say for sure that if you disable JMX remote support, you might be able to workaround this issue. To do that you would need to remove anything that is setting system properties that start with "com.sun.management.jmxremote"

To use java 6 you'll need to remake the boot jar. That NoSuchMethodError is the consequence of using a 1.5 DSO boot jar with a 1.6 runtime.

To make a 1.6 boot jar, set TC_JAVA_HOME=<path to 1.6 VM>, then run the make-dso-boot-jar.bat script found in the bin directory of the terracotta install. The tool should report where it is writing the new boot jar. Once you've created the jar, you'll need to update your -Xbootclasspath/p to point to it.

Tomcat 6.0.x and java 6 should work, but I don't believe either are officially supported yet.

I agree with Steve, that lock sequence doesn't sounds like it should deadlock. We definitely want to reproduce this issue. To that end, what version of terracotta are you using, and which platform (linux, win32, etc)?

Any possibility you could post the code and tc-config file?

I'm also very interested in why you're exploring lock upgrades? I think our current mechanism (which is guaranteed to deadlock in the face of concurrent upgrades) might leave a lot to be desired. I'd lock to hear how you think upgrades would be useful to your use case

It's hard to say for sure without something concrete to run, but I think this has something to do with the fact that some of these classes are runtime generated.

One of the exceptions messages -- "Can't get ClassInfo for com/connectil/web/pages/Home; java.io.IOException: Class not found" is almost certainly coming from line 328 of AsmClassInfo (http://svn.terracotta.org/svn/tc/dso/trunk/code/base/aspectwerkz/src/com/tc/aspectwerkz/reflect/impl/asm/AsmClassInfo.java)

...and that is probably the result of our code trying to call Clasloader.getResourceAsStream("com/connectil/web/pages/Home.class")

It's possible that adding <exclude> expression to your terracotta config might let you get around this issue. I don't know what your terracotta config looks like, but if you have an <include>*..*</include> in there, you might want to try an approach of only including what you know you need.

Any chance you create a bare bones Tapestry app that creates an exception like this one? If so, please post and I'll dissect it.

I modified my sample DMI program (attached) to use some on-load scripts. I don't see an issue with on-loads being executed for objects involved in a distributed method invocation.

Out of curiousity, in your code snipped below, are you sure that the "c" field is what is null (as opposed to the get() returning null)?
Code:

  Hashtable<String, Object> table; //SHARED
  table.put("key name",o);
  
  table.get("key name").c;
  -> NPE

Perhaps providing a more concrete code sample will help resolve the issue. Please use the PM button (private message) if you do not wish to provide samples that are publicly viewable.

The parameters to a distributed method do not need to be roots (but they can be), and as "siyer mentioned, they must be shareable/instrumented types (you would get an exception if that wasn't the case).

I'm not 100% clear on what the issue you are experiencing. I've attached a small example of program using DMI. Perhaps you can contrast this program with yours to help resolve the issue. It is also possible you have found a bug, in which case it would be great if you could modify this sample in such a way to reproduce the problem.

In the example, there is a single shared object with a DMI method that takes an argument. Run many instances, and for at least one of them pass an argument of "caller".

hope this helps

I just updated the eclipse files for the datagrid and spider projects. If you don't have the M2_REPO classpath variable set in eclipse, you'll need to go ahead and do that

http://maven.apache.org/guides/mini/guide-ide-eclipse.html

hope this helps

Found the issue -- It appears to be a bug in our implementation of reflective array access. With the 2.2.1 and earlier releases, you can workaround this bug by disabling reflection support and re-creating your dso boot jar.

Put this in your terracotta xml config file:
<application>
<dso>
<dso-reflection-enabled>false</dso-reflection-enabled>
</dso>
</application>

And run the boot jar script: <TC_HOME>/dso/bin/make-boot-jar.[bat|sh] -f <path-to-terracotta-config.xml>

A bug fix will be present in our next major release (2.3.0)

Thanks again for the providing the full test case, it makes all the difference.