Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

It is actually possible. Check out Quartz Where. It allows you to specify type of machine and resources on which to run the job.

--Ari

yes. We do this with customers every day, actually.

You want to store their entire message as the value in the Ehcache element and use a sequence as the key. The reader and writer can agree on the sequence using RMI or other lightweight protocol at runtime. Just don't share the sequence in Ehcache as that will be highly contended between reader and writer.

You get ordering only because of key naming convention. Put another way, the data structure is a map and unordered but the keyspace is ordered itself so you end up get()ing values in the order they were put() w/o any sort of communication between reader and writer. reader can poll for the next value (spinning to block) or you can signal somehow Peer to peer when there are new keys to read. NOTE: don't signal every key's presence or you will not scale either.

Last is a question: how fast do you want this to go? HOw big are the messages? And how much RAM do you have relative to the use case? I worry about your statement that you have lots of messages and not enough memory. Example:

1TB of messages / second and 1GB of RAM would never work, right? It is just logical. How could it? The disk would have to be so fast as to store 999 GB of messages / second and no such technology exists. Well, this is an absurd example but you get the idea. If you need more disk bandwidth than you can get, don't bother going down this path. That said, adding more RAM is cheap now and Ehcache can use BigMemory to store hundreds of gigs of messages w/o needing a large Java heap to do it. I would be very very cautious about the ratio of messages / second vs. disk bandwidth vs. available memory.

Hope this all makes sense.

--Ari

I think you cross-posted this against another thread no?

--Ari

Quartz based on Oracle RDBMS? This is likely the #1 deployment mode for the technology and there are millions of users. I think it is stable, yes.

Also worth recalibrating here:

operations / millisecond is a good thing to observe in a single JVM. But once a network is involved that ratio will go smaller than 1:1 in some cases. I would expect milliseconds / operation and if the system delivers more than 1 operation / millisecond I would be happy (on gigabit).

NOTE: you will definitely be able to get Ehcache on server arrays to do more than 1 operation per millisecond if you leverage locality of reference and/or eventual consistency mode.

--Ari

1. why not get a histogram of the heap to see what's in it near the end of the test. Kinda hard to tell otherwise.

2. RMI is not built to perform. Server Arrays are. Check out www.terracotta.org for more info.

--Ari

I would do one of two things, neither of which is ideal:

1. move from HibernateSearch to simple Ehcache Search. Nothing against Hibernate Search (loads of respect to Emanuel too) but we don't support it yet. Both HibernateSearch and Ehcache Search use Lucene internally so you get the same behavior from a different part of your existing stack of libraries. Just note, you cannot search a Hibernate 2nd level cache, only a POJO cache of your own making.

2. Take the Lucene TCDirectory we implemented available on our forge and mutate it to use Toolkit maps or something like that. Let me explain: TCDirectory requires DSO-version of Terracotta. You can't mix DSO-mode and standard mode so don't try to go there. That said, all the TCDirectory needs is a clustered map, available in the Toolkit via standard mode. Contribute back the TCDirectory and we'll host it again for others to get at / maintain. Now, you can use our clustered Lucene instead of NFS-based clustered Lucene (NFS is unreliable in write-heavy scenarios anyways).

If either of the above two options is not palatable (I don't really like the 2nd one too much myself as it hasn't been tested and you are responsible for some of the code yourself), I suggest working with us and JBoss to build Terracotta-transport as a plugin for Hibernate Search. We've talked about it with JBoss many times. What is your timeframe as this approach would be best for all?

--Ari

someone will look into this and post back to you shortly. Sorry for the delay :)

--Ari

oh: make sure to use Ehcache and Quartz _without_ DSO. Use them in serialization mode.

Thanks :)

--Ari

you are bottlenecked on the single queue. Are you really using DSO? If yes, please stop.

The best pattern with our products and OSS projects is Ehcache to store the work data, in eventual mode plus quartz or quartzwhere ("where" is paid-only at the moment) to fire the work. People stopped doing LBQ + DSO perhaps 2.5 years ago now.

Are you using a single LBQ with DSO? If yes, this is why you are not getting linear scale out, BTW. In DSO, the producer and consumer on a single LBQ fight with each other for the internal locks on the queue itself (to read and write cause information to be sent across the wire...it is not a messaging layer, it is a consistent shared queue and you have to think of it differently).

If you want LBQ + DSO to scale, you need many queues, one per consumer. The producer writes to a queue targeted at a single consumer. Also, make sure the producer never reads from the queue. And last, try to send strings (JSON, XML workload, etc.) down the queue as these are immutable and the queue will fly.

2 problems:

1. if the same class keeps showing up as needing instrumentation, after you add it to tc-config, you need to add it via bootstrap loader, not via tc-config. Add it to bootstrap classes then make bootjar again pointing at the tc-config you added it to.

2. You shouldn't use sessions in DSO mode. Just move to regular mode. DSO is for very advanced use cases and makes pretty much no sense for 99.9% of use cases on web sessions, for sure.

Thanks,

--Ari

you cross-posted this in multiple areas of the site.

I answered it elsewhere.

Thanks

--Ari

Sorry folks. One of our smart developers lent a hand to our ops team and found a broken JDBC URL causing part of our Tomcat / maven system to queue up data till it OOMEd.

All fixed, once and for all. Should stay up now w/o flakiness.

--Ari

ATG was working on joint integration w/ Ehcache and Terracotta up until Oracle acquired them. Not sure where they got.

Sorry :(

--Ari

JGroups is a JBoss Redhat product. Have you tried asking someone on their forums 1st?

--Ari