Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

Hello, I can see where each webapp needs to be configured in the tc-config.xml, but how do I setup multiple hosts (all with one webapp named ROOT) in one tomcat instance and have them be clustered with terracotta?

Also I've been trying to run one of our apps in the configurator and it keeps coming back with 'cannot share an instance of type org.apache.commons.collections.ReferenceMap...'. When I click ok and it restarts I get the same error. I can see where it's getting added to the tc-config.xml but it doesn't seem to have any effect. We never use commons collections directly, could that be coming from hibernate?

Thanks
Mike

Hi mfor:

Not sure about the intent of your first question - perhaps you could clarify.
If you mean how to have multiple clients participate in a Terracotta cluster, check this out: http://www.terracotta.org/confluence/display/docs1/Troubleshooting+Guide#TroubleshootingGuide-HowdoIuseTCSessionswithTomcat%3F

If you mean you want to cluster multiple applications deployed to the same Tomcat instance - currently the Tomcat instance can only point to a single set of HA terracotta servers - so the applications deployed to the same tomcat instance are all participating in the same cluster.

The second issue you describe with the configurator is a real bug and has been fixed - https://jira.terracotta.org/jira/browse/DEV-37. Not sure which version of Terracotta you are on, but its been fixed to be released in our next version (2.3 of the software). You can currently get the nightly builds off our download site.

Thanks.
Sreeni Iyer

Hi siyer,
thanks for your response.
For multiple hosts, I mean a tomcat instance with many domain names. ie a tomcat server.xml like:
Code:

<Server port="8005" shutdown="SHUTDOWN">
 <Service name="Catalina">
  <Engine name="Catalina" defaultHost="domain.com">
         <Host name="domain.com"  appBase="webapps-domain">
         <Context path="/" />
         </Host>
         <Host name="special.domain.com"  appBase="webapps-special.domain">
         <Context path="/" />
         </Host>
         <Host name="special2.domain.com"  appBase="webapps-special2.domain">
         <Context path="/" />
         </Host>
  </Engine>
  </Service>
 </Server>

each host is cluster and load balanced separately but using the same code base. Since each webapp is named ROOT would there be any issues using terracotta to cluster them? I haven't found anything in the documentation regarding this kind of setup.

Thanks again.
Mike

Hi mfor:
Thanks for the clarification.

It seems to me that Tomcat's Virtual Hosting capability won't interfere with Terracotta clustering.

In that the different "domains" within the same Tomcat instance, would use the same java system properties passed to the catalina invocation .i.e. tc.install-root to indicate where Terracotta is installed, bootclasspath modification to intercept the class loading process and tc.config which would result in the virtual "domains" retrieving configuration from and being bound to same Terracotta server. That is, all the webapps served up by the different "domains" within the same Tomcat instance would be part of the same cluster.

So this configuration is kind of similar to deploying multiple webapps to the same tomcat instance (although with virtual hosting you address multiple domains) - so the same possibilities (share amongst all) and constraints (can't make individual domains talk to different Terracotta servers yet) exist.

Having said that, I could post something back after actually trying this configuration out - in case I see anything contrary to current understanding.

Regards
Sreeni Iyer
Terracotta.

Hi Mike

Actually trying it out ;-) lead to the exception below - based on this, it seems that Terracotta would need source modifications in order to be able to support the Virtual Hosting capability supported by Tomcat. We could add support for this in one of our upcoming releases, if there is sufficient usage of this capability in the community - do you plan to deploy with multiple virtual hosts and what's the approximate time-frame ?

Code:

 SEVERE: An exception or error occurred in the container during the request processing
 
 java.lang.NullPointerException
         at com.tc.tomcat55.session.SessionValve55.makeWebAppConfig(SessionValve55.java:99)
         at com.tc.tomcat55.session.SessionValve55.createManager(SessionValve55.java:78)
         at com.tc.tomcat55.session.SessionValve55.findOrCreateManager(SessionValve55.java:70)
         at com.tc.tomcat55.session.SessionValve55.tcInvoke(SessionValve55.java:53)
         at com.tc.tomcat55.session.SessionValve55.invoke(SessionValve55.java:46)
         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
         at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
         at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
         at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
         at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
         at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
         at java.lang.Thread.run(Thread.java:595)

Hi Sreeni,
Thanks for the response, sorry to took so long for me to get back to you. I have the usual work distractions as I'm sure you can imagine.

To answer your question, yes we're hoping to use that setup in our production environment and that would be a deal breaker. We have about 40 webapps each load balanced to 3 servers. Each server is running between 3 and 12 webapps.

Before we can test a production setup I need to test one of our webapps in the configurator. I am using the nightly snapshot from about 3 weeks ago per your suggestion. I have our war imported and it starts and runs fine without terracotta session enabled. When I enable them and start the server, it appears ok until I go to the home page and I get an error saying it must include a type 'java.lang.ref.ReferenceQueue' in the boot jar. When I hit ok, it stops all the servers but then never come back up. It acts as if it ran out of memory. I've tried to increased the memory in bin/start-tc-server.bat with no affect. Any tips on getting our webapp to run?

Thanks again,
Michael

Actually on rerunning with the virtual host configuration with 2.2.1 final - we were unable to reproduce the exception, I reported based on an earlier version of the software.

i.e. We were able to successfully cluster sessions across Tomcat instances that supported the Virtual hosting capability - just that all applications shared a single context (like in your example), so there was a single SessionManager which included sessions from apps across domains (which might be fine from your application's perspective, if the goal is to get availability for all domains hosted on a given node).

So i would definitely recommend being on one of these 2.2.1 final or 2.3.0 stable builds - (Nightly builds are by definition less reliable in that they are just a build based off some snapshot in time, whereas releases marked stable, final - have been through some QA). So it's possible that the bug you encountered was very specific to the build you downloaded.

The other option would be to side step use of the Configurator (it is a productivity tool) and you can still cluster sessions simply by adding java options to your catalina.bat or startup.bat script - see http://www.terracotta.org/confluence/display/docs1/Troubleshooting+Guide#TroubleshootingGuide-HowdoIuseTCSessionswithTomcat%3F

Let us know, if this resolves your issue.
Regards,
Iyer.
Terracotta.

Hi

I'm still having no luck getting our webapp to run. This is what I've tried:

tomcat version 5.5.17

$ java -version
java version "1.6.0"
Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Client VM (build 1.6.0-b105, mixed mode)

I've setup tomcat with 2 tomcat_base directories, and I can get them to run without tc. When I tried to add tc I get the following error in catalina.out:
Code:

 java.lang.NoSuchMethodError: java.util.Arrays.copyOf([BI)[B
         at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
         at java.util.jar.JarVerifier.update(JarVerifier.java:183)
         at java.util.jar.JarFile.initializeVerifier(JarFile.java:320)
         at java.util.jar.JarFile.getInputStream(JarFile.java:386)
         at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:689)
         at sun.misc.Resource.cachedInputStream(Resource.java:59)
         at sun.misc.Resource.getByteBuffer(Resource.java:154)
         at java.net.URLClassLoader.defineClass(URLClassLoader.java:249)
         at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
         at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
         at java.security.AccessController.doPrivileged(Native Method)
         at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:299)
         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
         at sun.security.jca.ProviderConfig$3.run(ProviderConfig.java:234)
         at java.security.AccessController.doPrivileged(Native Method)
         at sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:225)
         at sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:205)
         at sun.security.jca.ProviderList.loadAll(ProviderList.java:264)
         at sun.security.jca.ProviderList.removeInvalid(ProviderList.java:281)
         at sun.security.jca.Providers.getFullProviderList(Providers.java:132)
         at java.security.Security.getProviders(Security.java:421)
         at com.tc.object.bytecode.hook.impl.ClassProcessorHelper.<clinit>(ClassProcessorHelper.java:77)
         at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)

I'm running tomcat from cygwin so I added the following to my catalina.sh

Code:

 CATALINA_BASE="."
 CATALINA_HOME="..\apache-tomcat-5.5.17"
 
 JAVA_OPTS="$JAVA_OPTS -Xbootclasspath/p:C:\java\_tomcat1\tc\dso-boot-hotspot_win32_150_09.jar -Dtc.install-root=C:\PROGRA~1\TERRAC~1\TERRAC~1.1"
 JAVA_OPTS="$JAVA_OPTS -server -Dcom.sun.management.jmxremote -Xms64m -Xmx512m -XX:MaxPermSize=256m"
 JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote"

I've put my tc-config.xml in the same directory as the boot jar, when I start the tc server, it seems to start fine, but tomcat doesn't connect.

By the way, can terracotta run with tomcat 6.0.10? How about java 6?
Thanks again
Michael

To use java 6 you'll need to remake the boot jar. That NoSuchMethodError is the consequence of using a 1.5 DSO boot jar with a 1.6 runtime.

To make a 1.6 boot jar, set TC_JAVA_HOME=<path to 1.6 VM>, then run the make-dso-boot-jar.bat script found in the bin directory of the terracotta install. The tool should report where it is writing the new boot jar. Once you've created the jar, you'll need to update your -Xbootclasspath/p to point to it.

Tomcat 6.0.x and java 6 should work, but I don't believe either are officially supported yet.

Hello again,

I rebuilt the boot jar as described and restarted. On startup I get the following:
Code:

 java.lang.ExceptionInInitializerError
         at com.tc.management.AbstractTerracottaMBean.<init>(AbstractTerracottaMBean.java:52)
         at com.tc.management.AbstractTerracottaMBean.<init>(AbstractTerracottaMBean.java:43)
         at com.tc.management.beans.logging.TCLoggingBroadcaster.<init>(TCLoggingBroadcaster.java:32)
         at com.tc.logging.TCAppender.<init>(TCAppender.java:28)
         at com.tc.logging.TCLogging.<clinit>(TCLogging.java:390)
         at com.tc.object.bytecode.hook.impl.DSOContextImpl.<clinit>(DSOContextImpl.java:43)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:597)
         at com.tc.object.bytecode.hook.impl.ClassProcessorHelper.createGlobalContext(ClassProcessorHelper.java:380)
         at com.tc.object.bytecode.hook.impl.ClassProcessorHelper.init(ClassProcessorHelper.java:274)
         at com.tc.object.bytecode.hook.impl.ClassProcessorHelper.defineClass0Pre(ClassProcessorHelper.java:407)
         at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
         at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
         at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
         at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
         at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
         at java.security.AccessController.doPrivileged(Native Method)
         at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
         at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
         at java.util.logging.LogManager$1.run(LogManager.java:163)
         at java.security.AccessController.doPrivileged(Native Method)
         at java.util.logging.LogManager.<clinit>(LogManager.java:156)
         at java.util.logging.Logger.getLogger(Logger.java:274)
         at sun.management.snmp.util.MibLogger.<init>(MibLogger.java:57)
         at sun.management.snmp.util.MibLogger.<init>(MibLogger.java:42)
         at sun.management.jmxremote.ConnectorBootstrap.<clinit>(ConnectorBootstrap.java:584)
         at sun.management.Agent.startAgent(Agent.java:135)
         at sun.management.Agent.startAgent(Agent.java:239)
 Caused by: java.lang.NullPointerException
         at java.util.logging.Logger.getLogger(Logger.java:275)

Any thoughts

Thanks
Michael

This particular issue is fixed in the 2.3.x release. You can get a 2.3.x build from here:

http://www.terracotta.org/confluence/display/orgsite/Download

I've been told but can't say for sure that if you disable JMX remote support, you might be able to workaround this issue. To do that you would need to remove anything that is setting system properties that start with "com.sun.management.jmxremote"

Also wrt tc-config.xml location

I've put my tc-config.xml in the same directory as the boot jar, when I start the tc server, it seems to start fine, but tomcat doesn't connect.

,

I would recommend simply specifying the -Dtc.config java system property to your catalina.sh. Something like -Dtc.config=<terracotta-server-host-name>:9510 would imply that your Tomcat instance would bootstrap the config from the Terracotta Server (e.g. -Dtc.config=localhost:9510 where your TerracottaServer is running on the same box as your Tomcat instance). And you would start the Terracotta server perhaps like this: start-tc-server.sh -f <location-of tc-config.xml>

Relying on the implementation's preference based on just the location of the tc-config.xml might get problematic over time, especially when there could be conflicts (e.g. multiple tc-config.xmls in different locations)

regards
Sreeni Iyer
Terracotta

Hello Sreeni and Tim,

Thanks to both of you for your responses.

I've successfully got one of our webapps running on two instances of tomcat clustered with terracotta on my computer (win xp using cygwin).
in my tomcat catalina.sh:
Code:

JAVA_HOME="c:\PROGRA~1\Java\JRE15~3.0_0"
 CATALINA_BASE="."
 CATALINA_HOME="..\apache-tomcat-5.5.17"
 JAVA_OPTS="$JAVA_OPTS -Xbootclasspath/p:C:\java\_terracotta\dso-boot-hotspot_win32_150_09.jar"
 JAVA_OPTS="$JAVA_OPTS -Dtc.install-root=C:\PROGRA~1\TERRAC~1\TERRAC~1.1"
 JAVA_OPTS="$JAVA_OPTS -Dtc.config=mfortin.omp.loc:9510"
 JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote"

Most of the issues I've had were caused by not following the instructions (duuh). We'll probably start some load testing within the next couple of days on our production servers. Now that I have it running it raise some other questions.

Do all the clients and/or the terrocotta server have to be running the same major and/or minor version of java? does the client and/server platforms have to be homoginous?

I noticed the jsessionid's format changed with a '!' and channelId appended to it. I assume that points to the client instance, correct? I haven't seen anything in your docs about this.

My understanding of how terracotta worked is that when I access an object in the shared heap, I accessing the same instance, but when I output the value of the object in a jsp page it had a different object id (com.package.SomeClass@objid(values...)) but the same values. Shouldn't that object id be the same on both clients?

After some testing I discovered what my original error was in the Configurator. I was trying to save a hibernate proxy and it contained classes that were in the instrumented-classes exclude list. I imagine it's not a good idea to persist hibernate proxies, right?

Lastly, the terracotta adminstrator console lists the classes persisted in serverName --> DSO --> Classes. I imagine that's a good reference to use to setup the tc-config.xml instrumented-classes or can I leave it as *..*? In other words, do I gain any performance by updating the instrumented classes?

Thanks again
Michael

Do all the clients and/or the terrocotta server have to be running the same major and/or minor version of java? does the client and/server platforms have to be homoginous?

Nope. You can mix and match the versions for the most part. Each client java version will require a boot jar specific the version in use. We recommend that the server is always run with 1.5 VM. Finally, you'll run into problems if you try to access 1.5 specific type (e.g. ConcurrentHashMap) from a client running 1.4 (hopefully this last one is obvious).

I noticed the jsessionid's format changed with a '!' and channelId appended to it.

The number after the '!' is an unqiue identifier for the cluster node. That bit of information is useful for load balancers to make sticky routing decisions, and it let us detect when requests ping-pong between servers.

My understanding of how terracotta worked is that when I access an object in the shared heap, I accessing the same instance, but when I output the value of the object in a jsp page it had a different object id (com.package.SomeClass@objid(values...)) but the same values. Shouldn't that object id be the same on both clients?

I can understand why you think that the hashcode values would be the same between nodes, but that is not something DSO provides. Of course, if you have your own hashcode() method, then they should be consistent between VMs. Technically would have to modify native portions of the VM to force identity hashcodes to have particular values, there isn't a bytecode level mechanism to do something like that really.

, the terracotta adminstrator console lists the classes persisted in serverName --> DSO --> Classes. I imagine that's a good reference to use to setup the tc-config.xml instrumented-classes

That is a good place to start -- It is a list of the unique type names that have been distributed. Trimming the instrumentation to minimum set should help performance.