Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

So... does this bother anyone that the session-configurator doesn't cluster out of the box on 64-bit Linux + Java 6?

I'm sorry, I lied about Townshend and DepartmentTaskList apps not working when clustering is turned off. They do indeed run when I'm running them without clustering on my 64-bit workstation.

I'm running terracotta-2.6.2 on my laptop now, and it looks like the Townshend and DepartmentTaskList apps cluster properly.

They don't work when clustering is turned off. Still, I'd expect the Cart app to work. I tried clustering wicket-examples-1.3.4 using the clustered-wicket-1.3 module as well, but it doesn't seem to cluster either when I try out their "Library" sample application.

I'm running Ubuntu Hardy Heron with the following kernel:
Code:

Linux asdf 2.6.24-19-generic #1 SMP Wed Jun 18 14:15:37 UTC 2008 x86_64 GNU/Linux

with this java version:
Code:

java version "1.6.0_06"
 Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
 Java HotSpot(TM) 64-Bit Server VM (build 10.0-b22, mixed mode)

Yesterday I downloaded terracotta-generic-2.6.2.tar.gz, untarred it, and ran sessions-configurator.sh which started up fine. I clicked on "Terracotta Sessions enabled" and started the sample Cart + DepartmentTaskList + Townsend webapps. They seemed to start up fine. When I click on the DepartmentTaskList 9081 or 9082 link, I get this:
Code:

 org.apache.jasper.JasperException: Exception in JSP: /displayUserList.jsp:54
 
 51: <!-- - - - - - - - - - - - - - - - - - -->
 52: <script>
 53: dsoEnabled = <%= System.getProperty("tc.install-root")!=null %>;
 54: cartSize = <bean:write name="displayUserListForm" property="numTasks"/>;
 55: server1 = "9081";
 56: server2 = "9082";
 57: currentServer = location.href.indexOf(server1) == -1 ? server2 : server1;
 
 
 Stacktrace:
 	org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:506)
 	org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
 	org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
 	org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
 	javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
 
 root cause
 
 javax.servlet.ServletException: Cannot find bean: "displayUserListForm" in any scope
 	org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:843)
 	org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:776)
 	org.apache.jsp.displayUserList_jsp._jspService(displayUserList_jsp.java:302)
 	org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
 	javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
 	org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334)
 	org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
 	org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
 	javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

Something similar happens when I click on the Townsend app links:
Code:

 org.apache.jasper.JasperException: Exception in JSP: /display.jsp:56
 
 53: ���<!-- - - - - - - - - - - - - - - - - - -->
 54: <script>
 55: dsoEnabled = <%= System.getProperty("tc.install-root")!=null %>;
 56: cartSize = <bean:write name="displayUserListForm" property="listLength"/>;
 57: server1 = '9081';
 58: server2 = '9082';
 59: currentServer = location.href.indexOf(server1) == -1 ? server2 : server1;
 
 
 Stacktrace:
 	org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:506)
 	org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
 	org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
 	org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
 	javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
 
 root cause
 
 javax.servlet.ServletException: Cannot find bean: "displayUserListForm" in any scope
 	org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:843)
 	org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:776)
 	org.apache.jsp.display_jsp._jspService(display_jsp.java:317)
 	org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
 	javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
 	org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334)
 	org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
 	org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
 	javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

When I click on either of the Cart links, the app displays properly with the drop-down menu of videos, but it does not cluster (i.e. the shopping cart does not keep its items across the two instances). The logs look unremarkable. Server:
Code:

 2008-07-09 17:56:24,715 INFO - Terracotta 2.6.2, as of 20080626-150605 (Revision 8952 by cruise@rh4mo0 from 2.6)
 2008-07-09 17:56:25,053 INFO - Configuration loaded from the file at '/opt/terracotta-2.6.2/tools/sessions/configurator-sandbox/tomcat5.5/tc-config.xml'.
 Creating boot JAR at '/opt/terracotta-2.6.2/tools/../lib/dso-boot/dso-boot-hotspot_linux_160_06.jar...
 Successfully created boot JAR file at '/opt/terracotta-2.6.2/tools/../lib/dso-boot/dso-boot-hotspot_linux_160_06.jar'.
 2008-07-09 17:56:33,449 INFO - Terracotta 2.6.2, as of 20080626-150605 (Revision 8952 by cruise@rh4mo0 from 2.6)
 2008-07-09 17:56:33,879 INFO - Configuration loaded from the file at '/opt/terracotta-2.6.2/tools/sessions/configurator-sandbox/tomcat5.5/tc-config.xml'.
 2008-07-09 17:56:33,908 INFO - Log file: '/opt/terracotta-2.6.2/tools/sessions/configurator-sandbox/tomcat5.5/logs/server-logs/terracotta-server.log'.
 2008-07-09 17:56:34,161 INFO - Statistics store: '/opt/terracotta-2.6.2/tools/sessions/configurator-sandbox/tomcat5.5/statistics'.
 2008-07-09 17:56:34,323 INFO - Statistics buffer: '/opt/terracotta-2.6.2/tools/sessions/configurator-sandbox/tomcat5.5/statistics'.
 2008-07-09 17:56:34,374 INFO - JMX Server started. Available at URL[service:jmx:jmxmp://0.0.0.0:9520]
 2008-07-09 17:56:34,826 INFO - Terracotta Server has started up as ACTIVE node on 0:0:0:0:0:0:0:0:9510 successfully, and is now ready for work.
 2008-07-09 17:56:34,891 INFO - Update Checker: Checking...
 2008-07-09 17:56:35,382 INFO - Update Checker: No updates found
 2008-07-09 17:56:35,384 INFO - Update Checker: Next check at Wed Jul 16 17:56:35 EDT 2008

Tomcat-9081:
Code:

 2008-07-09 17:56:35,306 INFO - Terracotta 2.6.2, as of 20080626-150605 (Revision 8952 by cruise@rh4mo0 from 2.6)
 2008-07-09 17:56:35,764 INFO - Configuration loaded from the file at '/opt/terracotta-2.6.2/tools/sessions/configurator-sandbox/tomcat5.5/tc-config.xml'.
 2008-07-09 17:56:37,740 INFO - Terracotta 2.6.2, as of 20080626-150605 (Revision 8952 by cruise@rh4mo0 from 2.6)
 2008-07-09 17:56:38,035 INFO - Configuration loaded from the file at '/opt/terracotta-2.6.2/tools/sessions/configurator-sandbox/tomcat5.5/tc-config.xml'.
 2008-07-09 17:56:38,161 INFO - Log file: '/opt/terracotta-2.6.2/tools/sessions/configurator-sandbox/logs/client-logs/9081/terracotta-client.log'.
 2008-07-09 17:56:41,188 INFO - Statistics buffer: '/opt/terracotta-2.6.2/tools/sessions/configurator-sandbox/statistics/9081'.
 Jul 9, 2008 5:56:45 PM org.apache.coyote.http11.Http11BaseProtocol init
 INFO: Initializing Coyote HTTP/1.1 on http-9081
 Jul 9, 2008 5:56:45 PM org.apache.catalina.startup.Catalina load
 INFO: Initialization processed in 1576 ms
 Jul 9, 2008 5:56:45 PM org.apache.catalina.core.StandardService start
 INFO: Starting service Catalina
 Jul 9, 2008 5:56:45 PM org.apache.catalina.core.StandardEngine start
 INFO: Starting Servlet Engine: Apache Tomcat/5.5.20
 Jul 9, 2008 5:56:45 PM org.apache.catalina.core.StandardHost start
 INFO: XML validation disabled
 Jul 9, 2008 5:56:46 PM org.apache.catalina.realm.JAASRealm setContainer
 INFO: Set JAAS app name Catalina
 Jul 9, 2008 5:56:49 PM org.apache.catalina.startup.HostConfig deployWAR
 INFO: Deploying web application archive Cart.war
 Jul 9, 2008 5:56:50 PM org.apache.catalina.startup.HostConfig deployWAR
 INFO: Deploying web application archive DepartmentTaskList.war
 Jul 9, 2008 5:56:52 PM org.apache.coyote.http11.Http11BaseProtocol start
 INFO: Starting Coyote HTTP/1.1 on http-9081
 Jul 9, 2008 5:56:52 PM org.apache.catalina.startup.Catalina start
 INFO: Server startup in 7532 ms
 [ERROR] [jsp] - Servlet.service() for servlet jsp threw exception <javax.servlet.jsp.JspException: Cannot find bean: "displayUserListForm" in any scope>javax.servlet.jsp.JspException: Cannot find bean: "displayUserListForm" in any scope
 	at org.apache.struts.taglib.TagUtils.lookup(TagUtils.java:934)
 	at org.apache.struts.taglib.bean.WriteTag.doStartTag(WriteTag.java:225)
 	at org.apache.jsp.display_jsp._jspx_meth_bean_write_0(display_jsp.java:351)
 	at org.apache.jsp.display_jsp._jspService(display_jsp.java:135)
 	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
 	at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334)
 	at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
 	at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
 	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
 	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
 	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
 	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
 	at com.tc.tomcat55.session.SessionValve55.tcInvoke(SessionValve55.java:63)
 	at com.tc.tomcat55.session.SessionValve55.invoke(SessionValve55.java:50)
 	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
 	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
 	at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
 	at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
 	at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
 	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
 	at java.lang.Thread.run(Thread.java:619)

Tomcat-9082:
Code:

 2008-07-09 17:56:35,486 INFO - Terracotta 2.6.2, as of 20080626-150605 (Revision 8952 by cruise@rh4mo0 from 2.6)
 2008-07-09 17:56:35,882 INFO - Configuration loaded from the file at '/opt/terracotta-2.6.2/tools/sessions/configurator-sandbox/tomcat5.5/tc-config.xml'.
 2008-07-09 17:56:40,156 INFO - Terracotta 2.6.2, as of 20080626-150605 (Revision 8952 by cruise@rh4mo0 from 2.6)
 2008-07-09 17:56:40,507 INFO - Configuration loaded from the file at '/opt/terracotta-2.6.2/tools/sessions/configurator-sandbox/tomcat5.5/tc-config.xml'.
 2008-07-09 17:56:40,642 INFO - Log file: '/opt/terracotta-2.6.2/tools/sessions/configurator-sandbox/logs/client-logs/9082/terracotta-client.log'.
 2008-07-09 17:56:45,298 INFO - Statistics buffer: '/opt/terracotta-2.6.2/tools/sessions/configurator-sandbox/statistics/9082'.
 Jul 9, 2008 5:56:51 PM org.apache.coyote.http11.Http11BaseProtocol init
 INFO: Initializing Coyote HTTP/1.1 on http-9082
 Jul 9, 2008 5:56:51 PM org.apache.catalina.startup.Catalina load
 INFO: Initialization processed in 2413 ms
 Jul 9, 2008 5:56:51 PM org.apache.catalina.core.StandardService start
 INFO: Starting service Catalina
 Jul 9, 2008 5:56:51 PM org.apache.catalina.core.StandardEngine start
 INFO: Starting Servlet Engine: Apache Tomcat/5.5.20
 Jul 9, 2008 5:56:51 PM org.apache.catalina.core.StandardHost start
 INFO: XML validation disabled
 Jul 9, 2008 5:56:52 PM org.apache.catalina.realm.JAASRealm setContainer
 INFO: Set JAAS app name Catalina
 Jul 9, 2008 5:56:54 PM org.apache.catalina.startup.HostConfig deployWAR
 INFO: Deploying web application archive Cart.war
 Jul 9, 2008 5:56:54 PM org.apache.catalina.startup.HostConfig deployWAR
 INFO: Deploying web application archive DepartmentTaskList.war
 Jul 9, 2008 5:56:55 PM org.apache.coyote.http11.Http11BaseProtocol start
 INFO: Starting Coyote HTTP/1.1 on http-9082
 Jul 9, 2008 5:56:55 PM org.apache.catalina.startup.Catalina start
 INFO: Server startup in 4829 ms
 [ERROR] [jsp] - Servlet.service() for servlet jsp threw exception <javax.servlet.jsp.JspException: Cannot find bean: "displayUserListForm" in any scope>javax.servlet.jsp.JspException: Cannot find bean: "displayUserListForm" in any scope
 	at org.apache.struts.taglib.TagUtils.lookup(TagUtils.java:934)
 	at org.apache.struts.taglib.bean.WriteTag.doStartTag(WriteTag.java:225)
 	at org.apache.jsp.display_jsp._jspx_meth_bean_write_0(display_jsp.java:351)
 	at org.apache.jsp.display_jsp._jspService(display_jsp.java:135)
 	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
 	at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334)
 	at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
 	at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
 	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
 	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
 	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
 	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
 	at com.tc.tomcat55.session.SessionValve55.tcInvoke(SessionValve55.java:63)
 	at com.tc.tomcat55.session.SessionValve55.invoke(SessionValve55.java:50)
 	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
 	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
 	at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
 	at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
 	at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
 	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
 	at java.lang.Thread.run(Thread.java:619)

The strange thing is, when I downloaded this for my 32-bit laptop (also running Hardy Heron), the Cart webapp clustered properly (don't know about the DepartmentTaskList or Townsend webapps). Is there something wrong with the download, or some compatibility issue with 64-bit Linux + Sun Java 6? Or am I missing something about how session-configurator.sh works?

After some poking around, I eventually came to the same conclusion about the single node reaping. Though it's correct, it seems odd that a single node faults-in session objects from the other nodes in order to reclaim memory. Wouldn't there be a lot of I/O between the TC server and the reaper node, particularly in large clusters? Presumably the other nodes are flushing their session objects to the TC server, then the TC server faults in the session state to the reaper node. It also would seem that there's the potential that the TC server would end up writing a lot of state to disk if it becomes memory constrained during this process. Has this been a performance issue?

I'm using tomcat 6.0.13 and terracotta-2.4-nightly-rev4149. I added an HttpSessionListener to my web app to observe session creation and destruction with Terracotta, and I'm using JMeter to create sessions with an HTTP request sampler. I am also using a load balancer to distribute the requests across a cluster of 5 tomcat servers.

I can see the sessions getting created in a distributed fashion. However, when the session expires on each of the servers, only one of the servers actually shows a "session destroyed" event, but for all the sessions created. For example, if I issue 10 requests to the load balancer, then two sessions are created for each server. But, when the sessions time out, all of the sessions are expired on a single server, so it looks like 10 sessions are destroyed on that server instead of two for each server.

At first I suspected some routing issue. However, I'm able to reproduce this behavior with the session-configurator.sh app on my desktop without a load balancer. I use JMeter to point to localhost:9081 and localhost:9082 and create ten sessions on each instance, and only on one of them twenty sessions are invalidated.

Is this expected behavior? If so, why isn't the session invalidation distributed? If not.. then is there a bug filed for this somewhere? It could be harmful to a webapp that relies on this event handler to be fired more than expected.

Well, AFAICT all the shell scripts for Terracotta are #!/bin/sh. The issue is that Ubuntu Feisty has /bin/sh aliased to "dash", which is stripped down in functionality. Debian Etch has /bin/sh aliased to "bash" (at least the two machines I have running). There's a bug filed for this in Ubuntu here:

https://launchpad.net/ubuntu/+source/dash/+bug/61463

Although it's been marked "rejected", there have been quite a few complaints about using dash even up to last week.

Actually, in start.sh, instead of
Code:

. "${TC_INSTALL_DIR}"/bin/dso-env.sh -q

It should just be:
Code:

set -- "-q"
 . ${TC_INSTALL_DIR}/bin/dso-env.sh

That should force "-q" to be sent to dso-env.sh and clear the existing args.

I ran into the same problem. I think the tomcat "start.sh" script has a bug -- the line where it says:

Code:

 . "${TC_INSTALL_DIR}"/bin/dso-env.sh -q"

expects to pass "-q" into dso-env.sh. Instead, it actually just inlines dso-env.sh and doesn't pass the "-q" as a param. If you notice in start.sh, the variable "PORT" is set to "$1", and "$1" is used as TC_CONFIG_PATH in dso-env.sh.

What should happen is this:
Code:

 eval "${TC_INSTALL_DIR}"/bin/dso-env.sh -q"

and export TC_INSTALL_DIR and TC_CONFIG_PATH.