Terracotta Discussion Forums

Tim, many thank for the answer. We tried to avoid toArray in our sources but as you said correctly this is out of our control for all 3rd party libs / frameworks and we use a lot of them...

Can please enlight me concerning "express mode"? What exactly do I have to change or can you point me to some docs?

Jens

Hi All!

We face some issues with Terracotta DSO and it looks like that the Terracotta replacement of ArrayList creates a high thread contention. What I do not understand when looking at the stacktrace is why Terracotta creates locks for instances which are not supposed to be instrumented at all. Means: in our tc-config we never instrument Tapestry framework classes. Is there any solution to explicitly exclude classes from instrumentation?

Thanks in advance

Jens

"http-9080-447" daemon prio=10 tid=0x00002aaabdff8800 nid=0x4474 waiting for monitor entry [0x0000000064b6a000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at com.tc.object.bytecode.hook.impl.ArrayManager.getObject(ArrayManager.java:84)
    - waiting to lock <0x00000006b67dd5f8> (a java.util.WeakHashMap)
    at com.tc.object.bytecode.hook.impl.ArrayManager.arraycopy(ArrayManager.java:367)
    at com.tc.object.bytecode.ManagerUtil.arraycopy(ManagerUtil.java:1087)
    at java.util.ArrayList.toArray(ArrayList.java:306)
    at org.apache.tapestry5.func.AbstractFlow.toArray(AbstractFlow.java:217)
    at org.apache.tapestry5.ioc.internal.util.InternalUtils.realizeObjects(InternalUtils.java:1706)

I read about the master / worker pattern before, but as far I understood it this just guarantees one worker is running on one machine within the cluster to do a job. I need multiple workers doing the same job on *each* VM.

In the meantime I believe DMI is not the right choice here. Hopefully it is allowed for a newbie to make a feature request:

Why can't you support something like a "broadcast-method" via tc-config which means: regardless in which VM the method is called it gets propagated to all others. This would be a really transparent DMI without the need to instrument tons of classes and the need for roots. And this "real" callback should allow an easy decoupling from the application and the terracotta DSO world (I guess it is just a new message to be send from TC-SERVER via the socket to the client). At the moment I have to use libs or to write tools to get the job done, which is odd because now the code depends on terracotta which terracotta DSO assumes to avoid? For sure I can not see the consequences or problems arising here, but I think that's the way how DMI should work from my point of view. Am I wrong?

And yes I switched to a LinkedBlockingQueue impl at the moment...
Again, thanks for the input and help

Sorry, but journey still continues.

"I forgot to mention" that the service is created by Spring using CTOR injection, so the onLoad trick is not working here. Dead end reached?

after small modifications I got it working, thanks (HeavyNonPortable is now a member of DMI, otherwise the broadcastMethod has no effect when called on the second instance).

Unfortunately there is always a but: due to the TC-Config "on-load" entry and the "redundant" new call in the DMI member initialization the HeavyNonPortable class is instantiated twice, but the first instance is thrown away. I can live with this behaviour but nevertheless this seems to be strange...

To be honest the DMI is more difficult as expected at least from a newbie point of view :-)

Wow. Thanks for the answer in near realtime..

yes, i made it "transient" as I try to avoid the instrumentation of tons of classes required by the backend only, which only exist in the space of the VM and some of them are non portable. I am only interested in sharing the callback, not all backend classes. I can try to instrument them all, but I was looking for a solution in case of "non portable classes". All I need is a DMI callback to an instance which itself is *not* shared.
If "transient" is removed immediately the trouble starts with "instrument the HeavyNonPortableService" and add "UUID to the additional-boot-jar-classes" aso.

Any other solution here?

hello guys!

I have some trouble / misunderstanding concerning DMI. What I basically try is to avoid JMS as we use Terracotta 2.6.4 anyway. So in my special case each VM runs some backend functionality but I need a way to "control" all these different instances (one per vm) from a central place without JMS/JMX. This is where I thought DMI could be of help. So regardless on which JVM you call a backend method this call should be transparently propagated to all others "backends" in the cluster.

Please see sample project.
Backendprocess (this will do all the heavy backend work. one instance per vm):
Code:

 import java.util.UUID;
 
 public class HeavyNonPortableService
 {
     final UUID _nonPortableServiceExampleOnlyField;
 
     public HeavyNonPortableService()
     {
         _nonPortableServiceExampleOnlyField = UUID.randomUUID();
         
         System.out.println("HeavyNonPortableService created [" + _nonPortableServiceExampleOnlyField + "] instance=" + this);
     }
 
     public void serviceMethod(final String s)
     {
         System.out.println("serviceMethod [" + s + "] for [" + _nonPortableServiceExampleOnlyField + "] instance=" + this);
     }
 }

a simple service wrapper to introduce DMI. This service wrapper is shared but basically should do the DMI magic in each VM and pass the message (method call) to the backend part. Parts of it are stolen from the terracotta samples:
Code:

 public class ServiceWrapper
 {
     public class DMI
     {
         public void internalBroadcast(final String s)
         {
             _heavyNonPortableService.serviceMethod(s);
         }
     }
 
 
     private final HeavyNonPortableService _heavyNonPortableService;
     private final DMI _dmi;
 
 
     public ServiceWrapper()
     {
         _heavyNonPortableService = new HeavyNonPortableService();
         _dmi = new DMI();
     }
 
     public void broadcastMethod(final String s)
     {
         _dmi.internalBroadcast(s);
     }
 }

and this is the tc-config
Code:

 <?xml version="1.0" encoding="UTF-8"?>
 <con:tc-config xmlns:con="http://www.terracotta.org/config">
     <system>
         <configuration-model>production</configuration-model>
     </system>
     <servers>
         <server host="%i" name="local">
             <dso-port>9510</dso-port>
             <data>terracotta</data>
             <logs>terracotta/server-logs</logs>
             <statistics>terracotta/cluster-statistics</statistics>
         </server>
     </servers>
     <clients>
         <logs>/data/terracotta/logs/%(node.name)/</logs>
         <statistics>/data/terracotta/client-statistics/%D/%(node.name)</statistics>
     </clients>
     <application>
         <dso>
             <roots>
                 <root>
                     <field-name>ServiceWrapper._dmi</field-name>
                     <root-name>Schedule_DMI</root-name>
                 </root>
             </roots>
             <instrumented-classes>
                 <include>
                     <class-expression>ServiceWrapper$DMI</class-expression>
                 </include>
             </instrumented-classes>
             <locks>
                 <autolock auto-synchronized="true">
                     <method-expression>* ServiceWrapper.__INIT__(..)</method-expression>
                     <lock-level>write</lock-level>
                 </autolock>
             </locks>
             <distributed-methods>
                 <method-expression>void ServiceWrapper$DMI.internalBroadcast(..)</method-expression>
             </distributed-methods>
             <transient-fields>
                 <field-name>ServiceWrapper._heavyNonPortableService</field-name>
             </transient-fields>
         </dso>
     </application>
 </con:tc-config>

and finally the "main" to start the service on each machine
Code:

 /**
  * -Xbootclasspath/p:/terracotta-2.6.4/lib/dso-boot/dso-boot-hotspot_osx_150_16.jar
  * -Dtc.install-root="/terracotta-2.6.4"
  * -Dtc.config=localhost:9510
  */
 public class Sample_TerracottaDMI
 {
     public static void main(final String[] args)
     {
         final ServiceWrapper serviceWrapper = new ServiceWrapper();
         serviceWrapper.broadcastMethod("hello");
 
         try {
             Thread.currentThread().join();
             // Thread.sleep(60000);    // block, whatever
         } catch (final InterruptedException e) {
             e.printStackTrace();
         }
     }
 }

The following output is produced in case of two running applications
Code:

 VM1
 HeavyNonPortableService created [04bfb9a5-6b3b-4585-8836-95a208e7a469] instance=HeavyNonPortableService@f71ad4
 serviceMethod [hello] for [04bfb9a5-6b3b-4585-8836-95a208e7a469] instance=HeavyNonPortableService@f71ad4
 serviceMethod [hello] for [04bfb9a5-6b3b-4585-8836-95a208e7a469] instance=HeavyNonPortableService@f71ad4

as you can see vm1 recognizes two calls to the service method which is correct (one call from each started vm). But VM2 is in trouble:

Code:

 HeavyNonPortableService created [55bc94d1-4761-4b91-a667-ce38440118b3] instance=HeavyNonPortableService@851105
 Exception in thread "main" java.lang.NullPointerException
 	at ServiceWrapper$DMI.__tc_dmi_internalBroadcast(ServiceWrapper.java:7)
 	at ServiceWrapper$DMI.internalBroadcast(ServiceWrapper.java)
 	at ServiceWrapper.broadcastMethod(ServiceWrapper.java:24)
 	at Sample_TerracottaDMI.main(Sample_TerracottaDMI.java:11)

Thanks in advance

Jens