Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

Currently trying to decide on Terracotta for Cache vs. Infinispan for building my remotely accessible distributed data grid (note, I am looking for distributed, not replicated grid). From what I know, Infinispan is truly distributed (same data is not on all copies), which is great. However, for remote access, it only offers RESTful interface. I know how Terracotta works, but can you elaborate on whether Terracotta for Cache is distributed, not replicated?

Thanks,

Eric

Eric,

If you understand Terracotta then you know about L1 and L2 layers, yes?

For us, distributed vs. replicated is not necessary. The 2 are just load balancing modes to us and access patterns to the data, not a configuration mode or coding pattern or any of that.

For us L1 cache holds what a particular JVM has been asking for. L2 array holds all the data.

Let's model a replicated cache and a distributed cache using a sticky vs. round-robin load balancer PLUS an active/passive L2 array vs. an active/active. The concepts are slightly different but we will do it nonetheless.

REPLICATED CACHE / data grid:
definition -- All data is on all nodes. Any change on any node is updated on the others either synchronously or asynchronously, depending on your choice.

With Terracotta -- round robin load balancer will send random cache requests to random L1 nodes. This means eventually we should assume all nodes have random data and could have _all_ the data if they have enough RAM. Should this be a replicated cache? It sort of is. Every change made on a node would be forced to be sent to many if not all other nodes, but at least with Terracotta it would be asynchronous (but transactional and ordered which is a fancy way of saying this node won't wait for others to update but they will guaranteed update before they read the cache entry he changed).

The L2 twist -- The L2 array could be an open-source array where 1 active and 1 passive work together to store all the data to disk, transparently. This array would be replicating all your cache. But, the array could be the paid FX array which runs both stripes (active, active) and mirrors (active, passive). In this case, your data would be replicated in the mirror group and distributed in the stripes. 2 stripes == 50% of the data transparently partitioned to each node. 4 stripes == 25%. You get the idea.

Summary -- So, a round-robin load balanced app could be considered to be replicated if it is using OSS L2's and round-robin load balancing. But what is it doing when using the FX L2 array? Definitely partitioning the data evenly and transparently.

DISTRIBUTED:
Without going into the same long discussion of distributed, what we would do is use a sticky load balancing algorithm somehow load balancing cache key lookups in an intelligent fashion to L1 nodes. Then we have our cache distributed amongst our L1 nodes w/ zero overlap and zero update overhead etc. The same L2 OSS vs. FX decision is available to us here as well.

Here's what I would say on the matter. Ignoring Infinispan's lack of age, I would go with Terracotta (obviosuly) for a different reason. I would use Terracotta because the notions of distribution and replication are more in my control in a more fine-grained fashion. I can:
1. round-robin access to my cache if I don't care about locality or cannot load balance. Consider the situation where you sticky load balance 90% of the workload but some types of work cannot be intelligently load balanced for some reason. In terracotta those key, value pairs that are being accessed by the rogue workload can get at the data in ANY L1 node on the fly. That data will just flush out of the L1 a little while later when no longer needed.

2. Terracotta OSS vs. FX array allows me to keep things simple in the L1 tier and just do all my scaling / tuning / partitioning at the L2 tier which needs no configuration or tuning (it manages the data split on its own).

I can write a RESTful service in front of Terracotta's L1 clients, a POJO / network-attached-memory library, I can do both at the same time! So I am free to support multiple interfaces, etc.

Definitely more ways to keep things simple and flexible and not permeate the notion that I am using a "data grid" throughout my app layers and architecture.

But the choice is of course yours. I think the JBoss guys are pretty smart and that they are on a good path with Infinispan to correct some of the weirdness of JBossCache.

--Ari

So the fundamental question is, if I have a 10 node grid with each node running a 2GB JVM, can I cache a total of 10GB unique data with 2 copies or only 2GB unique data with 10 copies?

Eric

neither. Terracotta has virtual memory so you are limited by the combination of size of disk on the L2 array and read/write rate of the app.

With 10 nodes @ 2GB each, I would say you could have:

1. Round-robin load balancer 2GB of write-heavy data, 100GB+ of read-only data
2. Sticky load balancer >>2GB write-heavy, and unlimited data read-only.

And, no matter your load balancing strategy or capability in your use case, using the FX paid array will get you further, in a linearly scalable fashion.

--Ari

BTW, if you have 10 nodes w/ 2GB each, if your use case is write-heavy, consider buying the FX array and using 2 nodes as L1 and 8 nodes as L2. Otherwise, write-heavy use cases need lots of tuning (and some read-only ones do as well).

I bring this up because most people think to run 8 L1 nodes and 2 L2 nodes which in a data grid is not always right. Data grids are sometimes more I/O intensive than compute-intensive and thus you need more nodes for I/O (L2) than for compute (L1).

Cheers,

--Ari

It seems the issue is resolved , Please let know if need more information.