Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

Just to be clear, the Terracotta server provides high scale, low latency clustering capabilities. But the number of Terracotta servers in your environment is not related to the number of applications. Many applications can work separate of each other or together using the same Terracotta instance. Many nodes running the same application can share one Terracotta instance. It is a 1:many relationship both for apps and app nodes.

Can you please explain more how it is either cheating or how "one host name to do TCP communication" prevents you from "implementing complete distributed stuff"?

--Ari

BTW, you can approximate a transient attribute with a code change, I think. Instead of passing the attribute reference directly to session, use a containing object that has a reference to the real attribute. Then set the reference field as "transient" and you are ok.

The customer documented in the story partitioned the problem all the way up the stack meaning that based on user ID, you were routed to an application pool in each datacenter and the application pool could be clustered by 1 TC Server in 1 data center, even though the application pool was spread out geographically. Example: Say users were cut up into 10 pools, and each pool contained 200 application servers, there would be 10 TC Servers--one for each pool of users--and each would have 200 app severs connected to it, even though the TC Server could be in a different data center than the app server nodes.

Generally speaking, partitioning an app with Terracotta can be done in steps:

1. Partition your data (map striping and lock striping) so that you reduce contention and chattiness. If that is insufficient, our users go to step 2.

2. Partition your traffic via an application-specific load balancer. For example, if you need 2 TC Servers for a given workload, then the load balancer would take some KEY from a request (such as session ID or user ID) and mod by the number of TC instances. Then send the request to an app instance clustered by either TC-A or TC-B. This impacts your app but gets you linear scale.

3. For some, the latency of the extra hop to the app-specific load-balancer is too much so they collapse the load-balancer down onto the app server and run multiple copies of the app on each app server node and the load balancer balances via localhost or other local interfaces to avoid the added latency. This is complicated but scales linearly _and_ at low latency.

Soon, we will have active-active where we will transparently split data evenly across Terracotta Servers. Stay tuned.

I would really like to answer "yes" to this, but I think I see where you are going. Short answer: Terracotta cannot speed up your program...in fact, clustering adds overhead to applications.

The only way I could answer "yes" to this new question is if you presented a business problem that could be chopped into subsets using algorithms such as Master / Worker.

Specifically:

if your program does
int j = 0;
for( int i = 0; i < 1,000,000; i++ ) {
j =i;
}

Terracotta will not speed up that for-loop because the speed of that loop is dependent on the compiler, the HW, and the other software your OS is running at the moment.

If you however chopped up a problem such as a search for "cometta" in every web page on your site, you could reduce such a problem into sub-problems where you ask a single instance of your application to search just 1 page and then farm out the list of pages to a bunch of nodes all using Terracotta to coordinate the work across JVMs.

Yes. Different apps can share data because our solution doesn't serialize your objects.

Here is an example:
http://unserializableone.blogspot.com/2006/10/share-precious-heap-memory-accross.html

And, while you are reading Hung's blog, you might want to take a look at the stand-alone Spring app:
http://unserializableone.blogspot.com/2007/07/ive-heard-about-spring-for-awhile-and.html

--Ari

Terracotta _can_ speed up your application but it is use case specific. Here's the deal:

1. We provide "add a brick" scalability. If you take a Java application installed on one node, you can go to 2 or more nodes under a load balancer with little effort. And, since we are fine-grained inside the application memory, this simple approach to scaling will actually work quite well.

2. We provide "restartability" and "no single point of failure." This is achieved by virtue of the fact that your application's important objects are stored in your JVM, our server's JVM, and our server's disk. Our server is then clustered to eliminate the chance for data loss as well as downtime.

In many cases these capabilities are used to deflect load off of the database since data in Terracotta is durable (like a database) and many application instances can attach to the dataset and work together (like a database). The difference is Terracotta is 10 - 100X faster than a database in working with fine-grained Object Oriented in memory data such as customer objects, sales order, financial transaction objects, etc.) Take a look at the following online training session:
http://feeds.feedburner.com/~r/TerracottaVideoPodcasts/~3/147146202/i=tcvideo%252Fonline_training%252Fterracotta_hibernate%252FTerracottaHibernate.mov

It is located Here:
http://www.terracotta.org/confluence/display/orgsite/Online+Training

Basically, if your application is performing slowly because it is bottlenecked on the database or on network replication (messaging, JGroups, etc.) Terracotta can make each node faster than without us. (Conversely, if your application is highly optimized and has no bottlenecks, we will likely not improve a single instance's throughput.)

--Ari

Terracotta works at the JVM-level. So, yes a stand-alone Spring app can be spread across JVMs. The reason the documentation spends so much time on containers is that containers are harder to get set up that stand alone JVMs.

All you will need is a application that can be clustered, a valid tc-config.xml, a TC server instance to connect to, and dso-java.sh in $TC_HOME/bin. dso-java.sh will eventually run $JAVA_HOME/bin/java for you, but not before hooking TC's libraries into that JVM's startup routine.

--Ari

Nans,

sorry for the delays in getting back to you. I do not know much about what you are trying to get done with Teck's help but I can answer your question about the significance of the boot.jar file.

The bootjar is not too complicated. It is designed to allow for certain classes to be guaranteed to be instrumented by Terracotta. With these certain classes we might not get the chance to do our bytcode manipulation unless it is done in this manner. When our software suggests that you add something to the bootjar, it is best to go ahead and do it, even though it may not be the proper thing to do.

Go ahead and try it. And if that doesn't work, you could try marking the javax.naming.InitialContext reference as transient. The interface does not give me enough detail to determine if this is safe. So, unless another Terracotta engineer would like to step in here I don't see harm in trying these things.

--Ari

Ah,

this probably does not relate to Terracotta. The only problem I have seen everyone with this architecture faces is that directories can get very deep and then the file system starts to get slower (since many file systems actually linearly scan for files in a particular directory).

So, just chop image files into multiple directories. The usual trick here is to convert the file name of the image to hex and chop that name two characters at a time making directories for each 2-character slice. Then, at some tunable level of depth (2 to 4 directories deep is plenty), write the file. This will deliver a reasonably bushy directory structure that the file system will not have problems supporting.

Hope this helps. Feel free to email me offline at ari AT terracotta DOT org since, unless I am misunderstanding, this is not Terracotta-specific.

If I am misunderstanding, please correct me here on the forum.

Thanks,

--Ari

Not sure I understand. Are you asking if Terracotta should be used to store the images? Or are you asking if Terracotta can help with the coordination?

By the way, what is the need for coordination? I am not sure I see why you need to coordinate image access.

More explanation would be helpful. Thanks,

--Ari

isn't this already coming in 2.5? Or soon thereafter?

--Ari

Yes. The regular approach for this sort of problem would be using a relational database as a back-end storage and retrieve data from there in case of L1 cache miss. The documentation on the site says that Terracotta works faster that a relational database, so there might be a win here, unless I misunderstand something. Both relational database and terracotta can be "sharded" or "partitioned", so that's not a factor. Also there could be some session "stickyness" on application level so subsequent requests for the same customer will hit same L1 machine, but that's again could be done for both terracotta- and RDBMS-based implementations.

I think you are over-simplifying the problem quite a bit. Here's the issue: you are looking for an analogy in a database-based architecture to Terracotta and if one exists then nullifying the Terracotta approach as being no better than the database on that particular dimension. Just because there is an analogous approach across 2 different architectures doesn't mean those analogies perform the same nor does it mean they are as easy to implement.

Analogies:
1. A cache miss is a cache miss. This is not true. A cache miss to the DB fires up JDBC, SQL over the network, to the database's query cache and disk block cache which in turn respond with a row or rows that get(s) marshaled into a Java object. A cache miss to Terracotta translates into a network call for the object itself AND any associated objects that might be smart to pre-fetch based on application access patterns. For example, a customer object w/ a reference to sales orders can grab both in 1 TC call w/o a complicated OR-mapping relationship having to be predefined. Terracotta can do that lookup in O(1) since all objects are uniquely IDed. The database lookup is either O(lg n) or O(n) depending on the existence of an index / sort on the appropriate column. Sure, both technologies hit the network, but the latency is usually greater than that of a network call for both and Terracotta's added latency turns out to be _significantly_ lower. As measured when including batching, Terracotta's latency is on the order of 20 microseconds (sub-millisecond) for a single object when loaded as part of a batch of, say, 1000 objects. And, Terracotta's latency is on the order of less than 10 millis when loading a single object. Thus, hitting the Terracotta server is ok. (See my blog at blog.terracottatech.com for measuring latency of a single object-write using Terracotta.)

2. A read is a read is a read...this is incorrect. With Terracotta, we have established that the DB is slower on a cache miss. But what about on a cache hit. TC is plugged in to the heap, so you read at memory-speeds. The DB cannot do this. If you allow application developers to hold references to your cached DB objects (and thus read at memory-speed from Java objects), how could you _ever_ notify them of changes? YOu cannot. This is because you need them to call yourCacheInterface.get() so that you can make sure your copy of the db record in cache is consistent with the DB itself. Or you need them to call put() on a write so that you can make sure the DB is consistent with their changes. Either way, a cache hit is overhead when a DB is in the mix. So a cache hit with Terracotta is memory-speed. So what? Simple evidence of the power: Hibernate with TC as 2nd level cache as opposed to just EHCache, reads at 10X the read-rate of a simple cache. Hibernate in disconnected mode reads _several orders of magnitude_ faster than Hibernate 2nd level cache. (See our online training for a test case.)

3. Sticky session / user affinity works for both DB and Terracotta. This is not really true. Sticky session with caching of the DB still does not allow memory-speed reading of Java objects. You must do a get() from a SessionManager or other context-specific cache which will perform as discussed in #2 above. Furthermore, however, Terracotta is fine-grained pushing only deltas only where needed. If nothing changes during a request, nothing is pushed. If the read / write ratio of the session is pretty low (i.e., most requests read) then having the object in many JVMs is not a problem for Terracotta. For the database, every JVM to which the user is balanced _must_ re-get the appropriate DB records just in case the DB changed. Or, optimistic caching must be employed where transactions significantly complicate the application as well as lower performance.

Also, if the read / write ratio is lowish _but_ only small changes are made relative to the scope of the session context (for example, we had one user with 15MB sessions and 10KB of change on every request). Terracotta's ability to find and push the deltas makes it uniquely viable even when not sticky.

This is all a long-winded way to hopefully succeed at justifying the following assertion. Architectures tend to have analogies. The name of the game is not to find the analogy and then stop there. The name of the game is to find the analogy and weigh the implementation advantages / disadvantages.

Yes, there is a database analogy to many things that Terracotta can do. But, Terracotta can do those things in pure Java, and in many cases without your having to compute lots of things like _deltas_ or _relationships of objects_. So Terracotta is simpler to code, I assert. As you stated you have not tried it so it makes sense that you don't necessarily see this value at the moment.

But Terracotta is object-oriented and matches the application representation of data on a field-for-field basis. This means that it performs most operations in constant time (algorithmically speaking, of course. We are not a real time system with guaranteed latencies otherwise we would not support Java

). The database is disk-block oriented which means that it is always marshaling blocks and block-caches into responses. This also means that its ability to bind rows together as related is weak and, thus, its ability to batch up operations into fewer network calls is essentially non-existent. It pretty much has to fetch the rows you request and no more or less. A Terracotta-based application can be 10 - 100X faster than its database-back equivalent (again, see the Hibernate online training session for support).

Thanks! Great questions. perhaps we can take this dialog out of forums and into email as I can see you are thinking through things in a very methodical and complete manner and will likely have more questions. I can always post back a summary of follow-on discussion for everyone to see. My email is ari AT terracotta DOT org.

Well, if you are using a Weblogic version we support _and_ an official Terracotta Integration Module (formerly config modules), then having to manually exclude parts of BEA might be a bug.

You should file it in JIRA, and please include the exact version of WLS, TC, and your tc-config if possible.

(Note that having to manually exclude classes is not the issue here...that may occur under any use case. The issue is in having to exclude BEA classes which TC's integration module should have accounted for already. In this case, my guess is that BEA Portal has more classes than straight WLS and this is why you might have had to add them manually)

Thanks

Interesting. I am not sure why this is happening but I do know what that error means. It means you have a sequence of events where a client thinks it is REconnecting but the server thinks this client is connecting _for the first time_.

The easiest way forward is to shut down ServerA, SliderA, ServerB, SliderB, clean out the Terracotta data directory on serverA (if there is anything there, you can just RM the entire data directory) and restart ServerA first, then SliderA, then SliderB. It already looks like SliderB is talking with ServerA as you pointed out.

As you move from 1 machine to multiple, it is sometimes easier to not run the TC Server from the Eclipse Plug-in. You can keep running your app from the plug-in and using your Eclipse-plugin as you otherwise would. Just start the server at the command line (shell prompt) by hand and then tell our Eclipse-plugin not to start a server...

the reason this helps is that you can be certain you aren't running multiple server instances, etc.

Let us know if you need help figuring out how to get the server started on its own (<$TC_HOME>/bin/start-tc-server.sh).

Someone correct me if I am wrong...but as I understand it, the clients can get config from the server but not the other way around. This too might be a source of confusion.