Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

First and foremost the reference:

http://www.terracotta.org/confluence/display/docs1/Concept+and+Architecture+Guide#ConceptandArchitectureGuide-roots
===============================================

Hello, my name is Quincy. I am doing some research in Agent Simulation. I am hoping Terracotta has the specs of concept and architecture that I want.

The purpose of me posting is to find out more of the simple specifics of what terracotta is and how it works. I will start off down the list of topics from the website in order. Just for a matter of simplicity I'll start with the first two.

======From site=======
Roots
A "root" is the top of a clustered object graph as declared in the Terracotta configuration. Object sharing in Terracotta starts with these declared roots. Any object that is reachable by reference from a Terracotta root becomes shared by Terracotta which means that it is given a cluster-wide object id and its changes are tracked by Terracotta and made available to the cluster.

Clustered Objects

When a root field is first assigned, the Java object assigned to that field and all object reachable by reference from that object become clustered objects. This means that each object is given a cluster-wide unique id and all its field data is added to the current Terracotta transaction. When the Terracotta transaction is committed, all of the object data is sent to the Terracotta server. All changes to clustered objects must be made within the context of a Terracotta transaction.
===================

I am going to parrot back what I understand. Please correct me.

All shared objects have a root.
No non-shared objects have a root.
Each client define roots.
When a client initializes roots, if the server doesn't have the root initialized then its initialized on the server. If any of the root's graph is changed on the client then the client tells the server to change the graph on the server in the same manner.
The clients do the program processing only. The servers are only dumby memory banks that provide a way of sharing data between clients.

Am I understanding so far? Thank you.

Almost perfect. The non-shared objects do not have roots. if you think of the TC server--dumb memory bank--as if it were network attached, then roots make lots of sense. In networked files, you mount a remote directory to a local mount point. In Terracotta, you mount the memory bank to a local object. From that mount point, all objects underneath are in the TC server as well.

The root concept allows us to support some local-only objects and some clustered objects. It also allows us to not require you to tell us about everything that could join the graph...just the top level mount-point / root object.

Hope this helps.

--Ari

the server also takes care of locking :)

Thanks again for your quick replies. On to the next topic.

==========Copy from source==========================
Virtual Heap

Terracotta virtual heap allows arbitrarily large clustered object graphs to fit within a constrained local heap size in client virtual machines. The local heap has a "window" on a clustered object graph. Portions of a clustered object graph are faulted in and paged out as needed. Virtual heap is similar in concept to virtual memory and is sometimes referred to as "Network Attached Memory."

Clustered objects are lazily loaded from the server as they are accessed by a client JVM. This happens by injecting augmented behavior around the GETFIELD bytecode instruction that checks to see if the object referred to by that field is currently instantiated on the local heap. If it isn't on the local heap yet, the client will request the object from the server, instantiate the object on the local heap, and ensure that the field refers to that newly instantiated object.

Conversely, less frequently used objects may be transparently purged from local heap by Terracotta, subject to the whims of the local JVM garbage collector. The amount of clustered object data kept in local heap is determined at runtime. Terracotta actively sets references to objects that have fallen out of the cache to null so that they may become eligible for local garbage collection. Because clustered objects may be lazily loaded, purged objects will be transparently retrieved from the server as references to them are traversed.
=========================================

Again, I'll parrot and please correct me if I'm wrong in anything.

This is a really cool idea. Though like a double edged sword there are ways it could be used for bad. In memory intensive applications the trick is to have the client hold onto the least about of huge objects as possible so there are a low amount of transfers from server to client.

Let me give an example that could result in a very slow program. Lets say we apply an operation to N amount of trees to create a new tree. One bad scenario would be where only one tree can fit on a client and the operation was the following:

Step 0: start at the root of tree 0.
Step 1: read the data of current node to determine which node in which tree to go to next.
Step 2: go to that next node. then go to step 1.

At each step the client would need to purge all its memory and reload all its memory. If each tree was very big, then the run time would be ridiculously long.

A question that I have is how exactly does terracotta decide this: "The amount of clustered object data kept in local heap is determined at runtime." Is it possible to have a client say "out of memory" if it only uses clustered memory and never adds to the clustered memory?

Thank you again for your time,
Quincy

Quincy,

You are right again, like a cpu cache that is too small for main memory, or is used in ways that do not align with the strategy of a cache (e.g. a for loop that does not make sequential access, blowing your cache lines etc) the local JVM can be thought of as a local cache as compared to the clustered memory pool, and in that sense, if you attempt to perform memory intensive operations against the clustered memory pool (which is generally speaking larger than the local client memory) you could thrash the local memory pool and exhibit poor performance.

Terracotta uses a combination of LFU+LRU to evict items from the clustered heap. The evictor starts to evict clustered memory when overall heap utilization reaches a threshold which is configurable. It can not do anything about local heap utilization so if your program has a memory leak, Terracotta will not magically save you.

OTOH, you can use this to significant benefit, for example, your clustered memory pool can be significantly larger than a single JVM. By bringing data into the local JVM and operating on it long enough to gain a benefit from the caching (in other words, your application exhibits strong Locality of Reference) you can use a divide and conquer algorithm to tackle a large problem, or you can partition your work and Terracotta will naturally assign data to the partitions.

Again, the best analogy to make is that of the cpu cache in your local system. I realize we are Java programmers and don't think about cpu caches all day :) but they are still important in computer science, and Terracotta is built on fundamental computer science principles that are essentially unchanged (e.g. the cpu cache model for improving performance by decreasing latency of repeated reads to the same or nearby memory addresses, the Java memory model for consistent and predictable concurrent access to shared objects across threads and locking semantics etc.)

When a client initializes roots, if the server doesn't have the root initialized then its initialized on the server.

Also, I wanted to clarify this statement.

It is actually the client who initializes the root, the server never has client code, as stated before it stores what amounts to raw data, it is the clients which convert this into objects (or objects into raw data).

So the sequence is that a client initializes a root, and then attempts to assign the initialized object (graph) to the root. If the root has not yet been initialized, the server will accept the new graph and it will become the new root, otherwise the graph will be discarded and the already existing root will continue to be in use (once initialized, a root can never change).

There are two places you can look to find out more about this behavior,

1) Alex Miller wrote a simple piece of code illustrating the effects of root initialization which generally are unsurprising, but can be a bit tricky in very specific situations: http://tech.puredanger.com/2007/08/08/hello-terracotta/

2) The Terracotta Gotchas Page talks about root initialization (specifically side effects of constructors during root initialization): http://www.terracotta.org/confluence/display/docs1/Gotchas#Gotchas-MultipleInitializationGotcha

Thanks, this thread is helping a lot. I have one question from your response,

The evictor starts to evict clustered memory when overall heap utilization reaches a threshold which is configurable.

Where can I configure this threshold?

Next topics we kinda already discussed a bit. Distributed Garbage Collection seems pretty straightforward from the guide's description. Locks and Transactions also seem straightforward with the knowledge acquired above. Locks/Transactions just have some syntax that could easily be forgotten unless seen in the gotchas.

One important thought is to remember that terracotta does not require the same program to be run on a root. Client A running program A can use a root that client B running program B is also using. So locking and sycro methods are on objects used in certain lines of code not on the lines of code themselves. I think this is a very flexible attribute of terracotta that makes it more usable than other parallel programming concepts.

Distributed Method Invocation (DMI)
Editted: I deleted this paragraph because I want to keep the thread as correct as possible.

Am I understanding the material? Thanks for your consistent input and replies.

Quincy

Hi Quincy. I think you have it fairly well.

Have you run through the cookbook examples? I think that will help you understand a lot.

http://www.terracotta.org/confluence/display/howto/Cookbook

I don't think I really understand well enough what you are saying about DMI - if you have a question could you rephrase it please? Note also there is a DMI example in the Cookbook for you to try out.

Here is the link that explains how to set the CacheManager thresholds.

http://www.terracotta.org/confluence/display/devdocs/tc.properties

Thanks for the links. I editted and played around with the dmi project and found a few questions.

What is the difference between join(millisec) and sleep(millisec) ? Also please answer the question in a comment below.

Thanks so much!

-Quincy

Code:

 import java.util.concurrent.CyclicBarrier;
 
 public class Main
 {
     public static Main instance = new Main ();
 	private final CyclicBarrier barrier = new CyclicBarrier(2);
 
     public void receive() throws InterruptedException
     {
         System.out.println("Waiting for dmi method calls...");
 
         /*
          * Is there a condition to test if a DMI is called?
          */
          for(int i=0;i<10;i++){
         	
         	System.out.println("Sleep.");
 // there can be a Hello printed inbetween Sleep and End Sleep
         	Thread.currentThread().sleep(10000);
         	System.out.println("End Sleep.");
         	Thread.currentThread().join(1000);
         }
         
     }
 
     public void send()
     {
         print("HELLO");
     }
  
     // this method gets called on the originating node from the send() method, and, 
     // since it is marked as dmi, that call is packaged up and executed on all
     // all other nodes that have this shared object resident 
     public void print(String message)
     {
         System.out.println(message);
         
     }
 
     public void run() throws Exception
     {
     	int arrival=barrier.await();
         if (arrival==0) receive();
         else send();
         
     }
 
     public static void main(String[] args) throws Exception
     {
         instance.run();
     }
 }

You are being misled by the code. Thread.currentThread.join() is an idiom that will block the current thread forever.

In the DMI cookbook example, that is being used because without it, there would be no live threads, and the JVM would exit.

It's probably a bit confusing - but "receive" does not actually receive anything. It just waits.

When a DMI call is made from an external node, the call is packaged up and "replayed" on all other nodes.

The actual call in a node where the DMI is replayed happens on separate thread. The thread is in fact an internal thread started by Terracotta, not by the application at all.

I'm going to go change the method name to "waitForever" since that is more appropriate.

I am trying to make my own example of a dmi and I'm failing terribly.

Here is the code that was thrown together. One node calls the distributed method but the other node does not get the call and hangs at 'waiting for dmi'. This code is meant to be run on two nodes at the moment so run it with 2 as an argument.

Thanks for your help.

Main.java
Code:

 package sim;
 
 import java.util.concurrent.BrokenBarrierException;
 import java.util.concurrent.CyclicBarrier;
 
 
 public class Main implements Runnable {
 
 	private final CyclicBarrier barrier;
 	final int participants;
 	private int arrival = -1;
 	private Agent[] agentlist;
 	
 
 	public Main(final CyclicBarrier barrierIN, Agent[] agentlistIN, int threads)
 	{
 		agentlist=agentlistIN;
 		barrier=barrierIN;
 		participants=threads;
 	}
 	
 	public void run()
 	{
 		System.out.println("Waiting...");
 		try {
 			arrival=barrier.await();
 		} catch (InterruptedException e) {
 			// TODO Auto-generated catch block
 			e.printStackTrace();
 		} catch (BrokenBarrierException e) {
 			// TODO Auto-generated catch block
 			e.printStackTrace();
 		}
 		System.out.println("Arrived:"+arrival);
 		
 		agentlist[arrival].put("String "+arrival);
 		
 		try {
 			barrier.await();
 		} catch (InterruptedException e) {
 			// TODO Auto-generated catch block
 			e.printStackTrace();
 		} catch (BrokenBarrierException e) {
 			// TODO Auto-generated catch block
 			e.printStackTrace();
 		}
 		
 		System.out.println(agentlist[0].get(0));
 		System.out.println(agentlist[1].get(0));
 		
 		if(arrival==0)waithere();
 		
 		System.out.println("SLeep");
 		try {
 			Thread.currentThread();
 			Thread.sleep(1000);
 		} catch (InterruptedException e) {
 			// TODO Auto-generated catch block
 			e.printStackTrace();
 		}
 		System.out.println("End SLeep, call dmi");
 		dmi("HelloWorld");
 	}
 	
 	public static void dmi(String msg)
 	{
 		System.out.println(msg);
 	}
 	
 	public void waithere()
 	{
 		System.out.println("Waiting for dmi:");
 		try {
 			Thread.currentThread().join();
 		} catch (InterruptedException e) {
 			// TODO Auto-generated catch block
 			e.printStackTrace();
 		}
 	}
 	
 	public static void main(String[] args) throws Exception
 	{
 		Roots roots=new Roots(new CyclicBarrier(Integer.parseInt(args[0])), Integer.parseInt(args[0]));
 		
 		new Main(roots.getBarrier(),roots.getAgents(), roots.getTheads()) 
 		.run();
 	}
 }

Roots.java
Code:

 package sim;
 
 
 import java.util.concurrent.CyclicBarrier;
 
 public class Roots {
 	private final CyclicBarrier barrier;
 	private final Agent[] agentlist;
 	private int threads;
 	
 	public Roots(CyclicBarrier barrierIN, int numOfAgents) {
 		barrier = barrierIN;
 		threads=numOfAgents;
 		agentlist=new Agent[numOfAgents];
 		synchronized(agentlist){
 		for(int i=0;i<numOfAgents;i++) agentlist[i]=new Agent();
 		}
 	}
 	
 	public CyclicBarrier getBarrier() {
 		return barrier;
 	}
 
 	public Agent[] getAgents() {
 		return agentlist;
 	}
 	
 	public int getTheads(){
 		return threads;
 	}
 	
 	public static void print()
 	{
 		System.out.println("HI");
 	}
 }

Agent.java
Code:

 package sim;
 
 import java.util.ArrayList;
 
 public class Agent {
 	
 	public ArrayList<String> inbox;
 	
 	public Agent()
 	{
 		inbox=new ArrayList<String>();
 	}
 	
 	public void put(String in)
 	{
 		synchronized(inbox){
 			inbox.add(in);
 		}
 	}
 	
 	public String get(int index)
 	{
 		synchronized(inbox){
 			return inbox.get(index);
 		}
 	}
 	
 	public void clear()
 	{
 		synchronized(inbox){
 			inbox.clear();
 		}
 	}
 }

tc-config.xml
Code:

 <?xml version="1.0" encoding="UTF-8"?>
 <con:tc-config xmlns:con="http://www.terracotta.org/config">
   <servers>
     <server host="%i" name="localhost">
       <dso-port>9510</dso-port>
       <jmx-port>9520</jmx-port>
       <data>terracotta/server-data</data>
       <logs>terracotta/server-logs</logs>
       <statistics>terracotta/cluster-statistics</statistics>
     </server>
   </servers>
   <clients>
     <logs>terracotta/client-logs/%D</logs>
     <statistics>terracotta/client-statistics/%D</statistics>
   </clients>
   <application>
     <dso>
       <distributed-methods>
           <method-expression>void sim.Main.dmi(String)</method-expression>
       </distributed-methods>
       <instrumented-classes>
         <include>
           <!--include all classes in the example package for bytecode instrumentation-->
           <class-expression>sim..*</class-expression>
         </include>
       </instrumented-classes>
       <locks>
         <autolock>
           <method-expression>void *..*(..)</method-expression>
           <lock-level>write</lock-level>
         </autolock>
       </locks>
       <roots>
         <root>
           <field-name>sim.Roots.barrier</field-name>
         </root>
         <root>
           <field-name>sim.Roots.agentlist</field-name>
         </root>
       </roots>
     </dso>
   </application>
 </con:tc-config>

You need to call DMI on a non-static method of an instance that is clustered.

By the way, please don't get hung up on DMI. It's really the least interesting of Terracotta features. Compared to DSO, DMI is akin to a parlor trick - DSO is the main act.

Thanks thats all I needed to know. I might as well post my code and what it does.
Its meant to be run with 2 as the program runtime argument.

Its interesting to think that the program could have an infinite loop if it was fast enough to do the second dmi before node 0 exited.

Output of Node 0.
Code:

 Waiting...
 Arrived:0
 String 0
 String 1
 Waiting for dmi:
 Looking For DMI calls:0
 HelloWorld
 Looking For DMI calls:2
 SLeep
 End SLeep, call dmi
 HelloWorld

Output of Node 1.
Code:

 Waiting...
 Arrived:1
 String 0
 String 1
 SLeep
 End SLeep, call dmi
 HelloWorld

Main.java
Code:

 
 package sim;
 
 import java.util.concurrent.BrokenBarrierException;
 import java.util.concurrent.CyclicBarrier;
 
 
 public class Main implements Runnable {
 
 	private final CyclicBarrier barrier;
 	final int participants;
 	private Agent[] agentlist;
 	private int DMIamount;
 
 	public Main(final CyclicBarrier barrierIN, int threads, Agent[] agentlistIN)
 	{
 		agentlist=agentlistIN;
 		barrier=barrierIN;
 		participants=threads;
 		DMIamount=0;
 	}
 	
 	public void run()
 	{
 		System.out.println("Waiting...");
 		
 		int arrival=-1;
 			try {
 				arrival=barrier.await();
 			} catch (InterruptedException e1) {
 				// TODO Auto-generated catch block
 				e1.printStackTrace();
 			} catch (BrokenBarrierException e1) {
 				// TODO Auto-generated catch block
 				e1.printStackTrace();
 			}
 		
 		System.out.println("Arrived:"+arrival);
 		
 		agentlist[arrival].put("String "+arrival);
 		
 		try {
 			barrier.await();
 		} catch (InterruptedException e) {
 			// TODO Auto-generated catch block
 			e.printStackTrace();
 		} catch (BrokenBarrierException e) {
 			// TODO Auto-generated catch block
 			e.printStackTrace();
 		}
 		
 		System.out.println(agentlist[0].get(0));
 		System.out.println(agentlist[1].get(0));
 		
 		if(arrival==0)waithere();
 		
 		System.out.println("SLeep");
 		try {
 			Thread.currentThread();
 			Thread.sleep(1000);
 		} catch (InterruptedException e) {
 			// TODO Auto-generated catch block
 			e.printStackTrace();
 		}
 		System.out.println("End SLeep, call dmi");
 		dmi("HelloWorld");
 	}
 	
 	public void dmi(String msg)
 	{
 		System.out.println(msg);
 		synchronized(this){
 			DMIamount++;
 		}
 	}
 	
 	public void waithere()
 	{
 		System.out.println("Waiting for dmi:");
 		
 		
 		while(DMIamount < 2){
 			
 			try {
 				Thread.currentThread().join(1000);
 			} catch (InterruptedException e) {
 				// TODO Auto-generated catch block
 				e.printStackTrace();
 			}
 			System.out.println("Looking For DMI calls:"+DMIamount);
 		}
 		
 	}
 	
 	public static void main(String[] args) throws Exception
 	{
 		int threads=Integer.parseInt(args[0]);
 		Agent [] agentlist=new Agent[threads];
 		for(int i=0;i<threads;i++) agentlist[i]=new Agent();
 		
 		Roots roots=new Roots(new Main(new CyclicBarrier(threads),
 										threads,
 										agentlist
 										));
 		
 		(roots.getMain() ).run();
 	}
 }

Roots.java
Code:

 package sim;
 
 
 
 public class Roots {
 	private static Main instance;
 	
 	public Roots(Main instanceIN) {
 		instance=instanceIN;
 	}
 	
 	public Main getMain()
 	{
 		return instance;
 	}
 	
 }

Agent.java
Code:

 package sim;
 
 import java.util.ArrayList;
 
 public class Agent {
 	
 	public ArrayList<String> inbox;
 	
 	public Agent()
 	{
 		inbox=new ArrayList<String>();
 	}
 	
 	public void put(String in)
 	{
 		synchronized(inbox){
 			inbox.add(in);
 		}
 	}
 	
 	public String get(int index)
 	{
 		synchronized(inbox){
 			return inbox.get(index);
 		}
 	}
 	
 	public void clear()
 	{
 		synchronized(inbox){
 			inbox.clear();
 		}
 	}
 }

tc-config.xml
Code:

 <?xml version="1.0" encoding="UTF-8"?>
 <con:tc-config xmlns:con="http://www.terracotta.org/config">
   <servers>
     <server host="%i" name="localhost">
       <dso-port>9510</dso-port>
       <jmx-port>9520</jmx-port>
       <data>terracotta/server-data</data>
       <logs>terracotta/server-logs</logs>
       <statistics>terracotta/cluster-statistics</statistics>
     </server>
   </servers>
   <clients>
     <logs>terracotta/client-logs/%D</logs>
     <statistics>terracotta/client-statistics/%D</statistics>
   </clients>
   <application>
     <dso>
       <distributed-methods>
           <method-expression>void sim.Main.dmi(String)</method-expression>
       </distributed-methods>
       <instrumented-classes>
         <include>
           <!--include all classes in the example package for bytecode instrumentation-->
           <class-expression>sim..*</class-expression>
         </include>
       </instrumented-classes>
       <locks>
         <autolock>
           <method-expression>void *..*(..)</method-expression>
           <lock-level>write</lock-level>
         </autolock>
       </locks>
       <roots>
         <root><field-name>sim.Roots.instance</field-name>
         </root>
       </roots>
     </dso>
   </application>
 </con:tc-config>

qhimqq wrote:

Thanks thats all I needed to know. I might as well post my code and what it does.
Its meant to be run with 2 as the program runtime argument.

Its interesting to think that the program could have an infinite loop if it was fast enough to do the second dmi before node 0 exited.

Output of Node 0.

I am not sure why you are saying this - I haven't really analyzed your code carefully enough. But Terracotta will short circuit any DMI loops - if you call a method that is marked as DMI while you are executing a method that was triggered because of DMI, the DMI will not be triggered.

That is to say, you cannot trigger a DMI call from a DMI call.