Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

Does this link help?

http://www.ehcache.org/documentation/configuration/cache-size#built-in-sizing-computation-and-enforcement

--Ari

very close. a rwLock is a lock that supports multiple readers and one writer at a time.

When you get the readLock() you get a HANDLE to the readLock, not a locked readlock.

Try lock.lock(); with your reference to rwLock's readlock before you unlock().

Cheers,

--Ari

Hmmm,

Definitely take a look at Examinator for hints as to how to implement SpringSecurity w/ session replication. I think the problem might be you are walking down the dependency graph into your DAO.

Examinator, IIRC, only had username and authentication-level / role as strings inside it under the Spring Security graph when visualized using the Terracotta dev console.

--Ari

Oh my no. DOn't try the examinator. It is not up-to-date. But definitely look at the deployment descriptors and config in there and see if there are some flags.

A quick scan of the Javadoc over at Springsource shows almost all of the security framework extends (or what have you) java.io.Serializable interface.

Can you just tell us what exact reference is failing to serialize? You may have to implement the serialization methods since much of security is an interface so there is no concrete serialization impl (as the docs say...for security purposes). If you scanned the source to Examinator that might be what I would look for 1st...a concrete impl.

--Ari

I went through release notes:
http://docs.terracotta.org/confluence/display/release/Terracotta+3.5+Release+Notes

And don't see any repaired deadlocks in the comms layer.

What I do see on closer examination of your stack trace is that it seems you might have been disconnected from the cluster, not deadlocked.

GC pause? Healthchecker kicked in?

I would look for a smoking gun log entry around long GC being detected and disconnect / reconnect activity right before this assumed-deadlock.

--Ari

3.5.0 is not the latest minor on the 3.5-line, right? Can you not apply all the updates at least? To 3.5.4, etc?

http://docs.terracotta.org/confluence/display/release/3.5.4+Downloads

Not sure I understand.

Not using Shibboleth, I get.

But are you saying you need TreeMap whether or not -XX:... works?

TreeMap experimental impl from 1.6.x support in DSO is definitely not gonna get added any time soon based on what I heard from TC guys. Can you work around the issues w/ what you get in off-the-shelf DSO as is?

--Ari

Good luck with that.

--Ari

BTW, I am basing this advice on only a very cursory scan of your post. It looked to me like you had clients waiting on a lock that one client thinks it has and yet the server is reporting the lock is recalled and server-owned at the moment.

FWIW,

Greediness is important to know here:
1. GREEDY READ == all clients can hold a greedy read lock at the same time. It means no writer is present and holding the lock in the cluster. Also, if a writer on that lock appears, readers must yield.
2. GREEDY WRITE == I have the lock and I am holding on to it locally even if I may not be using it at the moment. READERS must wait for me to yield the lock till they are allowed into the critical code segment it protects.
3. RECALL == server saying to a client "hey, greedy dude, give up your lock right now." It is almost lock preemption but not truly in that it will not forcibly take the lock. It just raises a flag with the client that after a lock release, you must yield the lock.

So a lock can be stuck in attempt to recall due to bad GC on a client holding that lock, for example. Then you would tune HA settings in your tc-config.xml. But I thought I saw the clients waiting and the server having it yet waiting at the same time which sounds 3.4.0-ish to me.

--Ari

can you upgrade right away? I am not sure because I am not part of the TC day-to-day team but I think those guys might not be supporting 3.4.0 anymore.

There are several updates on the 3.4 line. And the 3.5 line has been GA over 1.5 years. And the 3.6 line is out and GA and very stable as well.

I think there might have been a few races you want to upgrade away from, to be blunt.

--Ari

yes but it will lead you down a path of more complexity that can be avoided. It is not simply "DSO or standard" here. Behind that DSO decision is a DSO-only path. DSO apps cannot do anything but DSO. And DSO is not as feature-rich as the other products. The others can do all sorts of things like XA, rollback w/o XA, rejoin-mode (for clients that take long GC naps and need to reconnect), and more.

Best to stay on a serialization-based path if possible.

Thanks,

--Ari

this is a cross-post with another forum request. Pick one thread and let's chase this down as a group.

Thanks for your patience,

--Ari

SpringSource's tcServer has serialization-based session replication. I am not keeping up to date on this stuff but would be surprised if Spring Security cannot be set into a serialization-friendly mode for Tomcat, tcServer, and Terracotta.

Webflow, MVC, etc. can all be set to store state in session as serializable objects but will not necessarily do so by default. Perhaps something similar is going on with Security? (FWIW, our example Examinator app we used to showcase on the site used Spring Security and had no problems serializing it.)

--Ari

What version of Terracotta is this? Looks to me on a 60 second glance that the lock is held by a client and simultaneously recalled by the server which is a violation of the locking state machine inside the core.

Let's start with version and go from there.

--Ari

The shibboleth team is using an older integration mechanism to Terracotta. Discussions have led to the conclusion that Ehcache-based integration instead of direct clustered collection support would be a better way to go. But no work is being done on the new integration concept.

So no, Treemap is not going to be supported any time soon. Unfortunately, I think the Shibboleth team plan to cluster using another technology go forward and that message is taking time to permeate the community.

Sorry for the bad news :(

Feel free to press them to work with Ehcache as these sorts of tight-coupling and fragile implementation issues go away if Shibboleth changes the Terracotta implementation approach just a bit.

--Ari