Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

Hi Paul,
Have you tried running sample codes from this guide:
http://www.terracotta.org/web/display/orgsite/Lucene+Integration

toonti wrote:

I am trying to use Compass 2.0.2 along with Terracotta 2.7.0 (and Lucene 2.3.2) and I find the following error when I tried to store the Lucene Index:

Storing a Lucene index is explained in the guide above.
In brief, you can do it by using a TerracottaDirectory index (it's similar to RAMDirectory index but improved and pre instrumented to work with Terracotta). There is no extra config to instrument or lock a TerracottaDirectory object.
Using directly a RAMDirectory or a FSDirectory brings some troubles: you need to configure it manually and it's not really simple. So it's better if we can avoid these classes.
Again, you can find more information in the guide above.

Hope this can help!
Chu

PS: I attach the samples' source code here. You can use them directly if you want to.

Hi Hardy,
I'm writing a document specifying all steps to use Lucene 2.3.2 with Terracotta. It should be done in the next few days.
Cheers,
Chu

Hi Hardy,
I think your problem is your tc-config can't find the Compass TIM. That's why you have to configure a lot of things for the Lucene classes!

What I do is:
+Download the Compass TIM (which you already did)
+Specify this TIM in the tc-config (you may have to specify the folder containing the TIM)
Code:

    <modules>
       <repository>/home/nam/Bureau/src/lib</repository>
       <module group-id="org.compass-project" name="compass" version="2.0.1"/>
     </modules>

Then it's ready to build a Terracotta Lucene application.
Then you may have to:
+Specify roots.
+Instrument your own classes.
+Instrument some Lucene classes if you are demanded to. I don't think you need to instrument RAMDirectory since you are using the TIM.
+Lock your own methods and not Lucene methods.

I attach here a tc-config which works well with my small Terracotta Lucene application. You can give it a look if you have any doubts.

Hope this will help!
Chu

hardy wrote:

I guess I am in the need more help situation. Is anyone working to get the Lucene integration working again for 2.3 or 2.4? Is the source for the current integration publicly available so that one could try to learn from it and modify it for the newer Lucene versions?

Any help appreciated :)

--Hardy

Hi,
Lucene 2.3.2 can work well with the Compass TIM.
You just need to use the TIM in the TC config like I did. Then you can use only the Lucene API in your application.
By the way, the Compass API has a class named TerracottaDirectory, which is improved to work with Terracotta. You should give it a look!
Hope this will help!
Chu

Hi Gautier,
Thanks for replying me!
I confirm that the spyder example can run after being built with maven.
However its scripts don't work for me, so I have to launch it with command line (I had to add some missing librairies):
Master:
Code:

$TC_HOME/bin/dso-java.sh 
 -classpath /home/nam/Documents/terracotta-definitive-guide-1.2/chapter_11_masterworker/spider/target/classes/
 :/home/nam/Documents/terracotta-definitive-guide-1.2/chapter_11_masterworker/workmanager/target/classes/
 :/home/nam/Documents/terracotta-definitive-guide-1.2/foo-commonj-1.0.1/lib/commonj-twm.jar
 :/home/nam/Documents/terracotta-definitive-guide-1.2/foo-commonj-1.0.1/lib/jericho-html-2.6.jar
 :/home/nam/Documents/terracotta-definitive-guide-1.2/foo-commonj-1.0.1/lib/commons-cli-1.1.jar
  -server -Xms64m -Xmx64m -Dtc.install-root=$TC_HOME 
 -Dtc.config=/home/nam/Documents/terracotta-definitive-guide-1.2/chapter_11_masterworker/spider/tc-config.xml 
 org.terracotta.workmanager.spider.StartMaster 
 --url http://www.google.com --depth 4 --external-links false -i 1:2:3:4:5

Worker:
Code:

$TC_HOME/bin/dso-java.sh
  -classpath /home/nam/Documents/terracotta-definitive-guide-1.2/chapter_11_masterworker/spider/target/classes/
 :/home/nam/Documents/terracotta-definitive-guide-1.2/chapter_11_masterworker/workmanager/target/classes/
 :/home/nam/Documents/terracotta-definitive-guide-1.2/foo-commonj-1.0.1/lib/commonj-twm.jar
 :/home/nam/Documents/terracotta-definitive-guide-1.2/foo-commonj-1.0.1/lib/jericho-html-2.6.jar 
 -server -Xms64m -Xmx64m -Dtc.install-root=$TC_HOME 
 -Dtc.config=/home/nam/Documents/terracotta-definitive-guide-1.2/chapter_11_masterworker/spider/tc-config.xml 
 org.terracotta.workmanager.spider.StartWorker 1

But the master and the worker don't work. They just keep waiting. I checked the worker's pipe but there was nothing in it.
Master:
Code:

Starting Spider: 
   start url 			= http://www.google.com
   max parse depth 		= 4
   follow external links 	= false
   list with routing IDs 	= [ 1 2 3 4 5 ]
 created PipeManager for Initiator: MASTER
 DefaultQueueListener has started - serving pipe: 1
 DefaultQueueListener has started - serving pipe: 2
 DefaultQueueListener has started - serving pipe: 3
 DefaultQueueListener has started - serving pipe: 4
 DefaultQueueListener has started - serving pipe: 5
 Submitting URL: http://www.google.com
 scheduled work #: 0

Code:

Worker:
 -- starting worker...
 -- kill it with ^C when finished
 created PipeManager for Initiator: WORKER
 registering a new work queue for routing ID: 1
 Starting worker with routing ID: 1

About the tc-lib framework, I've run through your site many times and can see a lot of your efforts. I've seen a lot of master/worker examples (of course, with Terracotta enabled) on the internet and I think your examples are the closest ones to a real production system.
Regards,
Chu

Hi all,
I'm trying to run some codes from http://svn.terracotta.org/svn/forge/projects/thebook/tags/1.2/ (chapter 11 Master Worker).
I think there are some small bugs in the source code:

+In the class org.terracotta.workmanager.DefaultWorkItem, Ecllpse tells me that the compareTo() method doesn't handle WorkException. The same thing in the class org.terracotta.workmanager.routing.RoutableWorkItem.
So I wrapped these codes in try/catch blocks.
+In the class test.Master, the method run():
Code:

	public void run() {
 ...
 		m_pipeManager.clearPipes();
 ...
 	}

There is no method clearPipes() in the class PipeManager. I think the author wanted to use the method reset() instead. So I changed it to:
Code:

	public void run() {
 ...
 		m_pipeManager.reset();
 ...
 	}

A little further, in the methods testWaitForAll(), testWaitForAny_ReturnSingleItem(), testWaitForAny_ReturnCollection(), Eclipse tells me that there are non handled InterupptedExceptions. So I wrapped these codes with try/catch blocks.

+Then in the tc-config.xml (located in the directory "test"), the root points to an non existing object:
Code:

             <root>
                <field-name>org.terracotta.workmanager.routing.PipeManager.m_pipes</field-name>
             </root>

So I changed to:
Code:

        <root>
           <field-name>org.terracotta.workmanager.pipe.PipeManager.m_pipes</field-name>
         </root>

Finally, when I run test.Master, I got this exception:
Code:

Exception in thread "main" java.lang.IllegalStateException: pipe manager has been shut down
 	at org.terracotta.workmanager.pipe.PipeManager.getPipes(PipeManager.java:89)
 	at org.terracotta.workmanager.dynamic.LoadBalancingRouter.__tc_wrapped_waitForAtLeastOnePipe(LoadBalancingRouter.java:72)
 	at org.terracotta.workmanager.dynamic.LoadBalancingRouter.waitForAtLeastOnePipe(LoadBalancingRouter.java)
 	at org.terracotta.workmanager.dynamic.LoadBalancingRouter.route(LoadBalancingRouter.java:34)
 	at org.terracotta.workmanager.dynamic.DynamicWorkManager.schedule(DynamicWorkManager.java:48)
 	at org.terracotta.workmanager.dynamic.DynamicWorkManager.schedule(DynamicWorkManager.java:56)
 	at test.Master.run(Master.java:43)
 	at test.Master.main(Master.java:100)

I can't see how to fix this.
Can somebody help me to run this example?
Thanks a lot.
Chu

Hi all,
Thanks for your helps.

hhuynh wrote:

I would resort to writing my own data structure, one that TC doesn't instrument by default and sort my data with it. The fastest way to see which classes are preinstrumented by Terracota is to open up its bootjar and look at the content.

Today I've tried to make my ArrayList non instrumented in TC.
First I tried to declared it "exclude" in the instrumented class section of the tc-config. It doesn't work since the class is contained in the boot jar, so it's instrumented everytime I launch an TC application.
Then I copied the ArrayClass to make MyArrayClass which is used by my application and will not be declared as instrumented in TC. However, the ArrayClass uses some other classes (pre instrumented in the boot jar too), so I needed to do the same thing for these classes.
Finally, I have copied 3 classes : ArrayList, Collections, Arrays. The performance is much better. Before, TC overhead made the test 6 times slower. Now it's only 1,7 times slower. I think it's acceptable.

steve wrote:

I'll throw my two cents in here.

If sorting a large list you should copy the list sort it and then replace it.

Do you mean I copy the main list to a local list, then I sort the local list and finally replace the main list by the local list?
If it's that, I tried it already but it didn't work since the main list is also local. If it's not what you want to say, please clarify this idea for me!

gbevin wrote:

Another reason might be that to be able to do the sorting, all the elements of the list need to be faulted in. If this is a shared list that is partitioning over many nodes, the latency of the network access to fault everything in could cause such a slowdown.

It could be a reason, but I think it's not the problem here. In my test, the collection is not shared (as I said above, my tc-config doesn't have the dso section). So the collection only lives in the JVM client and can't be flushed to the TC server.

Thanks all again for your helps.
Chu

Hi Denis,
Thanks for replying me! I see your points!

But it's not quite that! Because I run the test with a minimun tc-config and there is no dso config, which means there is no classe instrumented, no object shared, no lock, ... However, maybe the collection I use (TreeMap and ArrayList) is pre-instrumented in Terracotta, so you can be right about this point.

About the collection, ArrayList is not modified (added or removed) when it's being sorted. Except for the TreeMap, it's modified (re sorted) when being added new element. I think that's why TreeMap is not a good solution for my case.

My goal is to do this operation localy in one JVM without any cluster constrains (I need this run as fast as it can), then I get the result (the top 50 elements) to send back to the cluster (50 small elements don't take a lot of bandwiths so this is not the problem at the moment).

Chu

Hi all,
I'm having a big list (from 200 000 to a few milions elements). I've already implemened the Comparable interface for my elements.
My goal is to find the top 50 elements from that list. I've tried TreeSet and Collections.sort.

With Terracotta enabled, the running time for sorting is up to 6 times slower (even there is nothing clustered in my test).

Is this overhead normal? Can somebody explain me why? And if there is a better way to sort with Terracotta enabled, please tell me!
Thanks a lots.
Chu

PS: the sorting operation is run inside one JVM. I can't make it smaller for it's already a basic operation in my application.

Hi,
You just need to specify the TC server IP. So the <servers> section will look like this:
Code:

...
   <servers>
     <server host="10.10.1.129" name="My Server">
       <dso-port>9510</dso-port>
       <jmx-port>9520</jmx-port>
      ...
     </server>
    </servers>
 ...

Use the same tc-config to launch your clients too. It'll help them to connect to the TC server.
Chu

nandinio wrote:

and the clients would then use the common tc-config that lies on the terracotta server?

Yes they will. You'll have more information in this document - section System Configuration Section (I think that you read it already): http://www.terracotta.org/confluence/display/docs1/Configuration+Guide+and+Reference#ConfigurationGuideandReference-SystemConfigurationSection

I'm not sure about the other question since I'm a newbie in TC. Maybe somebody can give you the right answer.
Chu

Hi,
Your clients can retrieve the tc-config from the TC server.
To do that, when launching the clients, you specify this option:
Code:

-Dtc.config=TCserverIP:dsoPort

So your clients have to know at least the TC server IP and the DSO port.
Have I replied your question? Correct me if I'm wrong.
Chu

Hi Geert,
My mistake! Thanks for helping me again!
Regards,
Chu

Hi Geert,
Thanks for your quick reply.

I was misunderstading because of this phrase "Unsupported Classes in Terracotta v. 2.4 and 2.5".
Moreover, the list is not totally exact. For instance, the java.util.concurrent.LinkedBlockingDeque is figured there. But I see a recipe to use it : http://www.terracotta.org/confluence/display/howto/Recipe?recipe=linkedblockingqueue

So if you can update the list, it would be useful.
Regards,
Chu

Hi all,
I'm using TC 2.6.2.
I got this exception in launching my application.
Code:

Unshareable class     : java.util.concurrent.ConcurrentLinkedQueue
 
 Action to take:
 
 1) Change your application code
    * Ensure that no instances or subclass instances of java.util.concurrent.ConcurrentLinkedQueue
      are assigned to the DSO root: com.lingway.tc.proto.MyRoots.queryQueue

I thought this class is supported in TC 2.6 version.
Can somebody please confirm this fact for me?
Thanks in advance.
Chu