Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

Hi,

I'm using the nightly eclipse plugin on Mac OS X 10.5 with eclipse 3.3 and I'm having a slow startup time of the TC Server in order to run my clustered application.

If i run dtruss to see which calls are being made by the server, I can see a bunch of stat64 calls (to get file status), even after it says 'Terracotta Server has started up as ACTIVE node on 0.0.0.0:9510 successfully, and is now ready for work.' in the console, I still get a busy icon and can't work for some time.

Here's some output of dtruss:

Code:

stat64("/Users/cassio/terracotta/server-statistics/statistics-store.lock.db\0", 0xB0F90B34, 0x0)                 = 0 0
 stat64("/Users/cassio/terracotta/server-statistics/statistics-store.lock.db\0", 0xB0F90BA4, 0x0)                 = 0 0
 stat64("/Users/cassio/terracotta/server-statistics/statistics-buffer.lock.db\0", 0xB1194B34, 0x0)                = 0 0
 stat64("/Users/cassio/terracotta/server-statistics/statistics-buffer.lock.db\0", 0xB1194BA4, 0x0)                = 0 0
 stat64("/Users/cassio/terracotta/server-statistics/statistics-store.lock.db\0", 0xB0F90B34, 0x0)                 = 0 0
 munmap(0x19711000, 0x8000)               = 0 0
 munmap(0x196F9000, 0x8000)               = 0 0
 munmap(0x196C1000, 0x8000)               = 0 0
 munmap(0x1774000, 0x8000)                = 0 0
 munmap(0x1764000, 0x8000)                = 0 0

after the munmap calls start I can go back to use eclipse...

any ideas why this is happening ?

The only logging configuration I've done in the tc-config.xml file is:

Code:

<server>
       <data>/Users/cassio/terracotta/server-data</data>
       <logs>/Users/cassio/terracotta/server-logs</logs>
       <statistics>/Users/cassio/terracotta/server-statistics</statistics>
     </server>

Sometimes the server startup time is really fast, but most of the time its slow.

Ive also put the terracotta folder under privacy so spotlight doesnt index it...

Thanks

2 things:
1. You can always turn off the statistics-recording - that should get rid of the "stat64s"

2. Also make sure that you are not excessively instrumenting - i.e. instead of *..* in the config, be specific about what classes you want to instrument.

Hope that helps.

zeeiyer, how do i turn it off ?

simply removing the statistics tag makes terracotta assume the default location, which is the same directory the application is running.

Thank you.

I don't know of any way to disable the creation of the statistics database unfortunately.

Hard to say where the real problem is here. Can you give us a little more insight into exactly the steps you're doing in eclipse? I'm mostly wondering if you're starting a server using the "Terracotta" menu, or are you trying to run a dso application (of which a side effect is starting a TC server)?

Is the cpu getting used heavily when this slownees happens? If so, it would be good to get some thread dumps of the TC server process and or eclipse (if the eclipse VM process is the one chewing cpu).

Hi teck,

I'm using the Terracotta menu > Servers > localhost.

Then I start my application using the run as DSO Application.

I cant see 100% cpu spikes during this process, it must be something else.

I'll pm you a thread dump of the TC process.

Thanks,

thanks. After looking at some code and talking to the author of the eclipse plugin, a thread dump of the TC server is definitely want we want. Of course that thread dump will only be interesting when it is hanging. If you can, please take a few thread dumps during that time (that way we can tell if it is making any progress).

Also feel free to attach the dumps directly here (you can attach files). I don't think there is any senstiive information in thread dumps from the TC server process. That way more than just me can look at it.

Ok, I PM'ed you one dump, I'll post here the one I sent you plus another one.

This thread looks relevant

Thread t@81923: (state = IN_NATIVE)
- java.net.Inet6AddressImpl.getHostByAddr(byte[]) @bci=0 (Interpreted frame)
- java.net.InetAddress$1.getHostByAddr(byte[]) @bci=4, line=842 (Interpreted frame)
- java.net.InetAddress.getHostFromNameService(java.net.InetAddress, boolean) @bci=7, line=532 (Interpreted frame)
- java.net.InetAddress.getHostName(boolean) @bci=10, line=475 (Interpreted frame)
- java.net.InetAddress.getHostName() @bci=2, line=447 (Interpreted frame)
- com.sun.jmx.remote.socket.SocketConnection.<init>(java.net.Socket) @bci=112, line=60 (Interpreted frame)
- com.sun.jmx.remote.socket.SocketConnectionServer.accept() @bci=32, line=173 (Interpreted frame)
- com.sun.jmx.remote.generic.SynchroMessageConnectionServerImpl.accept() @bci=29, line=47 (Interpreted frame)
- javax.management.remote.generic.GenericConnectorServer$Receiver.run() @bci=66, line=340 (Interpreted frame)

The eclipse plugin detects when the server is running by polling for a JMX bean to become available in the TC server process. The above thread looks like it is accepting a connection (presumably from eclipse) and blocking trying to do a reverse DNS lookup.

I had a feeling this had to do with dns. Perhaps invoking TC passing an argument to use only IPV4 (look at Inet6) stack would solve this ?

Perhaps disabling the IPv6 stuff will help, but I'm not entirely sure how to fix this problem. That code that wants the hostname is part of JMX and a bit out of our control. I'm not sure if an entry in your /etc/hosts (or the OSX equivalent) will help speed things along.