Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

I'm trying to run our maven test cases ("mvn test") on my Vista x64 machine, and keep getting the error:

Code:

2008-08-12 11:45:09,913 WARN - We couldn't load l1 reconnect properties from the URL :http://localhost:9510/l1reconnectproperties
 Skipping this source and going to the next one.

The error just repeats over and over.

This is from a clean SVN checkout of our repository, with no modifications. The exact same checkout works fine on Linux and OS X.

I can confirm that the above URL times out. However, as soon as I CTRL-C, the URL returns valid info.

If I run "mvn test" again, everything works properly, because (I believe), everything is running.

While annoying, this works for running tests, since it works every other time. However, if I try to run my application, using maven, it tries to run the tests, then restarts TC for the actual application. So if I do the thing where I start, then kill the test cases (to get the test cases to run), then I get the "couldn't load l1 reconnect properties" error when I get to the actual application.

I'm stuck. Any ideas?

I've attached my tc-config.xml, for reference.

Eric

I notice in your tc-config.xml, you have:

<server host='localhost' name="localhost">

Note the single tick on host and double-quote on name. I have no idea if that's a problem, but it looks weird. Maybe an xml processor is behaving differently there on win vs mac/linux due to the default character encoding?

amiller wrote:

I notice in your tc-config.xml, you have:

<server host='localhost' name="localhost">

Note the single tick on host and double-quote on name. I have no idea if that's a problem, but it looks weird. Maybe an xml processor is behaving differently there on win vs mac/linux due to the default character encoding?

Nice thought, but no dice. Tried it both ways.

Any other thoughts? Any other info that I can provide that would be useful?

Eric

Can you take a thread dump of the server when this occurs. JMX might be initialized enough to allow the AdminConsole (tc:admin) to work.

gkeim wrote:

Can you take a thread dump of the server when this occurs. JMX might be initialized enough to allow the AdminConsole (tc:admin) to work.

I also tried upgrading to Java 1.6.0_07, but it didn't help.

The Admin console doesn't work. It won't connect.

I've attached a dump of the TC process. It seems to get stuck trying to write some file. Maybe I missed something and need to open up permissions on some directory?

I've also attached the log files.

It looks like it's trying to lock the Terracotta server statistics store at C:\work\gnip-svn\devourer\statistics\statistics-store.lock.db. It's possible it doesn't have permissions to write there. You can change this by adding a line similar to the db and log lines in your tc-config.xml:

Code:

        <server host='localhost' name="localhost">
             <data>%(user.home)/terracotta/test/server-data</data>
             <logs>%(user.home)/terracotta/test/server-logs</logs>
 -->       <statistics>%(user.home)/terracotta/test/server-stats</statistics>
       </server>

More docs here:
http://www.terracotta.org/confluence/display/docs1/Cluster+Statistics+Recorder+Guide

Still not working. It created a bunch of statistics-store* and statistics-buffer* files in the new directory, but I'm still getting the same "reconnect properties" error.

Can you take another thread dump and see if it's still waiting on a stats file or doing something else? If waiting on the stats file, can you try writing in that location?

Didn't we see this problem in July at JavaOne? We had firewalling issues on the local machines and the last thing they wrote to the logs before trying to make TCP connections was about statistics. It turned out to be firewalling and not filesystem permissions.

Just thought I would throw this in here.

Another dev commented to me:

- The server should hook on to the default dso-port, as it was missing in tc-config. Check the same with "netstat -a | grep 9510" while running the test.
- Check accessing "http://localhost:9510/l1reconnectproperties" through a web browser and see if the properties are retrieved properly from the server.

I think you did the latter and it was timing out?

Stack trace looks a little different:

Code:

"main" prio=6 tid=0x000000000109c800 nid=0xb4c runnable [0x000000000282e000..0x000000000282fc50]
    java.lang.Thread.State: RUNNABLE
 	at java.io.FileOutputStream.writeBytes(Native Method)
 	at java.io.FileOutputStream.write(FileOutputStream.java:260)
 	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
 	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
 	- locked <0x000000003fed6d40> (a java.io.BufferedOutputStream)
 	at java.io.PrintStream.write(PrintStream.java:432)
 	- locked <0x000000003fed6d08> (a java.io.PrintStream)
 	at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
 	at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
 	at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:85)
 	- locked <0x000000003fed6e00> (a java.io.OutputStreamWriter)
 	at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:168)
 	at java.io.PrintStream.newLine(PrintStream.java:496)
 	- locked <0x000000003fed6d08> (a java.io.PrintStream)
 	at java.io.PrintStream.println(PrintStream.java:757)
 	- locked <0x000000003fed6d08> (a java.io.PrintStream)
 	at org.mortbay.log.StdErrLog.info(StdErrLog.java:69)
 	at org.mortbay.log.Log.<clinit>(Log.java:63)
 	at org.mortbay.component.Container.add(Container.java:200)
 	at org.mortbay.component.Container.update(Container.java:164)
 	- locked <0x0000000041709630> (a org.mortbay.component.Container)
 	at org.mortbay.component.Container.update(Container.java:106)
 	- locked <0x0000000041709630> (a org.mortbay.component.Container)
 	at org.mortbay.jetty.Server.setConnectors(Server.java:158)
 	at org.mortbay.jetty.Server.addConnector(Server.java:132)
 	at com.tc.server.TCServerImpl.startHTTPServer(TCServerImpl.java:364)
 	at com.tc.server.TCServerImpl.access$400(TCServerImpl.java:68)
 	at com.tc.server.TCServerImpl$StartAction.execute(TCServerImpl.java:318)
 	at com.tc.lang.StartupHelper.startUp(StartupHelper.java:39)
 	at com.tc.server.TCServerImpl.startServer(TCServerImpl.java:351)
 	at com.tc.server.TCServerImpl.start(TCServerImpl.java:164)
 	- locked <0x0000000040cfc808> (a java.lang.Object)
 	at com.tc.server.TCServerMain.main(TCServerMain.java:28)

While it is spinning on "couldn't load...", I timeout on through the browser as well. As soon as I kill the maven process, the TC process seems to recover and finishes loading. I can see the properties in the browser and can successfully connect with the TC Admin tool. netstat behavior is the same, not listening while the error message is printing, listening after I kill mvn.

I tried running with my firewall off, with no change in behavior.

I am an administrator on this machine, so theoretically I can write to just about anywhere. I doubt it is a permissions issue, unless TC is trying to write somewhere really strange.

Other thoughts?

Eric

So, this is in the middle of starting the embedded Jetty server inside the TC Server. This server gets sockets from the DSO listen port (9510) once they're identified as HTTP requests.

You said at the beginning that it works on Mac and Linux. Have you tried a different Windows machine?

I would also try it with your IP and a different port and difffernt paths.

Code:

         <server host="%i" name="localhost">
             <dso-port>2222</dso-port>
             <jmx-port>3333</jmx-port>
             <data>target/terracotta/test/server-data</data>
             <logs>target/terracotta/test/server-logs</logs>
         </server>

amiller wrote:

So, this is in the middle of starting the embedded Jetty server inside the TC Server. This server gets sockets from the DSO listen port (9510) once they're identified as HTTP requests.

You said at the beginning that it works on Mac and Linux. Have you tried a different Windows machine?

I just did a remote desktop to my home machine, which I haven't tried this on before. I hadn't used maven there, so it is a clean slate. It is also a Vista x64 w/ 64-bit JVM.

Same issue there.

hhuynh wrote:

I would also try it with your IP and a different port and difffernt paths.

Code:

         <server host="%i" name="localhost">
             <dso-port>2222</dso-port>
             <jmx-port>3333</jmx-port>
             <data>target/terracotta/test/server-data</data>
             <logs>target/terracotta/test/server-logs</logs>
         </server>

Unfortunately, I still get:

Code:

2008-08-13 10:22:28,482 WARN - We couldn't load l1 reconnect properties from the URL :http://172.20.1.56:2222/l1reconnectproperties
  Skipping this source and going to the next one.
 2008-08-13 10:22:28,482 ERROR - We couldn't load l1 reconnect properties from any of the servers. Retrying.....
 2008-08-13 10:22:30,475 WARN - We couldn't load l1 reconnect properties from the URL :http://172.20.1.56:2222/l1reconnectproperties
  Skipping this source and going to the next one.

Eric

The server uses an embedded Jetty instance, as you may have gathered. Included in the standard distribution is slf4j-nop-1.0.2.jar, which acts to disable Jetty logging. For some reason, this library is not included as a dependency of the terracotta maven artifact. Clearly Jetty is having trouble logging here. To verify that using slf4j-nop makes the server go, check out and build (mvn clean install) the tc-maven-plugin from here (https://svn.terracotta.org/repo/forge/projects/tc-maven-plugin/branches/tc-2.6) adding this to the pom:

Code:

     <dependency>
       <groupId>org.slf4j</groupId>
       <artifactId>slf4j-nop</artifactId>
       <version>1.0.2</version>
     </dependency>

And copy the attached library to ~/.m2/repository/org/slf4j/slf4j-nop/1.0.2/slf4j-nop-1.0.2.jar.

I would do this but I have no Vista.