1. Start up terracotta server
Code:
$ tar xzf terracotta-3.7.0.tar.gz
$ terracotta-3.7.0/bin/start-tc-server.sh
2. Make some requests to pull the config at port 9530
Code:
$ for i in $(seq 1000); do curl -s "http://localhost:9530/config" > /dev/null; done
3. Check terracotta's open sockets
Code:
$ lsof -i -P | grep $(pgrep java) | grep CLOSE_WAIT | wc -l
781
It seems that Terracotta's HTTP server isn't properly closing its TCP connections. The file descriptor limit can be reached easily, causing Terracotta to die:
Code:
2012-10-08 15:17:02,245 [WorkerThread(dso-http-bridge, 0)] ERROR com.tc.net.protocol.HttpConnectionContext - Exception thrown
java.net.SocketException: Too many open files
at java.net.Socket.createImpl(Socket.java:414)
at java.net.Socket.getImpl(Socket.java:477)
at java.net.Socket.getSoTimeout(Socket.java:1050)
at com.tc.server.TerracottaConnector$SocketWrapper.getSoTimeout(TerracottaConnector.java:156)
at org.mortbay.jetty.bio.SocketConnector$Connection.<init>(SocketConnector.java:183)
at com.tc.server.TerracottaConnector.handleSocketFromDSO(TerracottaConnector.java:39)
at com.tc.server.HttpConnectionHandler.handleEvent(HttpConnectionHandler.java:35)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:145)
Sometimes the terracotta server becomes unresponsive and the process crashes:
Code:
java.lang.RuntimeException: java.lang.AssertionError: Protocol is 2
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:293)
Caused by: java.lang.AssertionError: Protocol is 2
at com.tc.net.protocol.ProtocolSwitch.getReadBuffers(ProtocolSwitch.java:139)
at com.tc.net.core.TCConnectionImpl.getReadBuffers(TCConnectionImpl.java:794)
at com.tc.net.core.TCConnectionImpl.doReadFromBufferInternal(TCConnectionImpl.java:433)
at com.tc.net.core.TCConnectionImpl.doReadFromBuffer(TCConnectionImpl.java:306)
at com.tc.net.core.TCConnectionImpl.doReadInternal(TCConnectionImpl.java:290)
at com.tc.net.core.TCConnectionImpl.doRead(TCConnectionImpl.java:266)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:624)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:290)
Could someone please confirm this is an issue? The CLOSE_WAIT sockets will never die without the application (terracotta) closing them.