[Logo] Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)
  [Search] Search   [Recent Topics] Recent Topics   [Members]  Member Listing   [Groups] Back to home page 
[Register] Register / 
[Login] Login 
Tomcat crashed due to session clients rejected by the Terracotta server array  XML
Forum Index -> Terracotta for Web Sessions
Author Message


Joined: 10/01/2012 00:22:29
Messages: 1

I have 2 Terracotta session servers (v3.70) running in active/standby mode.

There are 14 app servers running Tomcat 6.0 and connecting to the session servers.

The Tomcat server will crash sometimes and unable to recover. This has happened on prod a few times and I have been able to reproduce it on our test env. It seems load is a factor contributing to the failure but it's not the root cause. Sometimes the problem happens when the load is normal.

I believe this has something to do with the Terracotta setup when I checked the TC server and client log file.

I've attached the following info:
1. TC server log
2. TC client log
3. Tomcat thread dump
4. TC config

Below are some of the errors I found after a load test. The problem started to happen after 13:40 in the log file.

2012-10-01 13:41:45,960 [L1_L2:TCComm Main Selector Thread_R (listen] ERROR com.tc.net.protocol.transport.TransportHandshakeErrorHandlerForL1 - com.tc.net.protocol.transport.TransportHandshakeErrorContext: com.tc.net.protocol.transport.TransportHandshakeErrorContext: "Client Cannot Reconnect. ConnectionID(17.c8dd256f25c04e5eabe8bfff0c9ffbea.31348ea2-7275-420a-9e1f-e41b63c9fadc-13a1a1eea42)[] not found. Connection attempts from the Terracotta node at are being rejected by the Terracotta server array."Message Class: com.tc.net.protocol.transport.TransportMessageImpl
Sealed: true, Header Length: 32, Data Length: 426, Total Length: 458
 Filename terracotta-client and server log.zip [Disk] Download
 Filesize 887 Kbytes
 Downloaded:  333 time(s)

 Filename tc-config.xml [Disk] Download
 Filesize 4 Kbytes
 Downloaded:  393 time(s)

 Filename threadDump_10011350.zip [Disk] Download
 Filesize 396 Kbytes
 Downloaded:  342 time(s)



Joined: 01/23/2011 10:58:07
Messages: 489

That indicates that your Tomcat clients have been disconnected from the cluster due to a N/W of GC pause thats over the threshold.
This is default behavior of Terracotta heartbeating

More details on how this works and how it can be tuned are here


Karthik Lalithraj (Terracotta)
Forum Index -> Terracotta for Web Sessions
Go to:   
Powered by JForum 2.1.7 © JForum Team