[Logo] Terracotta Discussion Forums
  [Search] Search   [Recent Topics] Recent Topics   [Members]  Member Listing   [Groups] Back to home page 
[Register] Register / 
[Login] Login 
[Expert]
Bug with Terracotta + ObjectOutputStream?  XML
Forum Index -> Terracotta Platform
Author Message
anodos

journeyman

Joined: 12/11/2006 13:33:29
Messages: 41
Offline

I have a service that uses Terracotta to cache information. Remote clients invoke this service using Spring's HTTPInvoker (which, in turn, uses Java serialization to transmit parameters and return values). This service will often directly return an object from its cache (which is a DSO). My expectation is that the object will be serialized to the client, and the client will deserialize it as a normal local (non-DSO) object (the clients neither know or care that Terracotta is being used by the service).
The problem: sometimes the client will receive an object with fields set to null. On the server, these objects are fine, but once they get deserialized on the client, they can contain null fields. I believe what is happening is that the null fields were paged out on the server side and Terracotta did not fault them back in during the serialization process.
Is this a known bug? How can I work around it?

Tim Eck

journeyman
[Avatar]
Joined: 01/25/2007 08:57:02
Messages: 47
Location: San Mateo, CA
Offline

Sounds like you've found a bug, and your analysis seems sound. I'm going to look into further.

I don't know the actual mechanics of how java serialization retrieves field values during the serialization process, but it is almost certainly not through bytecode which DSO has had a chance to enhance (thus it can see unresolved reference fields).

As a short term workaround, I would think if you did readObject() and writeObject() manually, then the field access in those methods would trigger DSO to resolve references that may have been paged out. I mostly suggest it to help give us more data about the problem, not as a "solution".

We've have similar problems with the native implementation of Object.clone().

I'll report back when I have more info, and more than likely a link to the bug report
anodos

journeyman

Joined: 12/11/2006 13:33:29
Messages: 41
Offline

Further testing on my side seems to confirm my original hypothesis. I've created a test that starts a Terracotta server and two clients. One client puts several thousand new objects into a map, then the other client gets and serializes them. If the serializing client does not access the objects, then all reference fields in the serialized objects are null. On the other hand, if I change the serializing client to access a couple of fields before serializing, then those fields will not be null (they will contain their proper values), though the unaccessed fields will remain null.
anodos

journeyman

Joined: 12/11/2006 13:33:29
Messages: 41
Offline

I've created a temporary workaround. It's not ideal, as it forces you to use a custom ObjectOutputStream (which isn't always a possibility), and it happens to be slower (from my tests, it takes rougly twice as long to serialize with this method), but its easy and it works. Behold FaultingObjectOutputStream:

Code:
public class FaultingObjectOutputStream extends ObjectOutputStream {
   public FaultingObjectOutputStream(OutputStream out) throws IOException {
     super(out);
     enableReplaceObject(true);
   }
 
   @Override
   protected Object replaceObject(final Object obj) throws IOException {
     try {
       faultInFields(obj, obj.getClass());
       return obj;
     } catch(IllegalAccessException e) {
       throw new IOException(e);
     }
   }
   
   protected void faultInFields(final Object obj, final Class<?> clazz) throws IllegalAccessException {
     for(final Field field : clazz.getDeclaredFields()) {
       if(!Modifier.isStatic(field.getModifiers())) {
         field.setAccessible(true);
         if(!field.getType().isArray()) {
           field.get(obj);
         } else {
           // Fault in every array element, just to be on the safe side...
           final Object array = field.get(obj);
           if(array != null) {
             for(int i = Array.getLength(array) - 1;i >= 0;i--) {
               Array.get(array, i);
             }
           }
         }
       }
     }
     if(clazz.getSuperclass() != null) {
       faultInFields(obj, clazz.getSuperclass());
     }
   }
 }
anodos

journeyman

Joined: 12/11/2006 13:33:29
Messages: 41
Offline

Actually, most of the "overhead" of FaultingObjectOutputStream is coming from the fact that it is faulting in field references, meaning it also has more data to serialize (the resulting stream is almost double in size).
steve

ophanim

Joined: 05/24/2006 14:22:53
Messages: 619
Offline

Tim entered a jira for the issue.

https://jira.terracotta.org/jira//browse/CDV-244

Just curious what you are building over there?

Cheers,
Steve

Want to post to this forum? Join the Terracotta Community
Tim Eck

journeyman
[Avatar]
Joined: 01/25/2007 08:57:02
Messages: 47
Location: San Mateo, CA
Offline

Good deal. I think we collectively know and agree on what the bug is here. Steve already posted the link to the bug. I've confirmed the issue in a test case here.

There is a race condition with that custom ObjectOutputStream, but you're not likely to observe it. Even though you're using reflection to read/resolve all of the fields of the objects being serialized, there is still the chance that the DSO memory manager could page out those references before they were committed into the serialization stream. Our fix for this problem will obviously need to make sure this race is dealt with.

anodos

journeyman

Joined: 12/11/2006 13:33:29
Messages: 41
Offline

Yes, and another problem with my solution is: what kind of locking should it do when accessing these objects? Right now, this solution seems to be working, but I can't quite figure out why... in my test case I'm not locking against anything while serializing the graph, yet Terracotta is still faulting in the values without complaining? Is this a cool feature or is my test code just getting lucky? I'd like to avoid taking out some coarse grained lock while serializing, and I'd also like to avoid putting locking logic into FaultingObjectOutputStream.


steve

ophanim

Joined: 05/24/2006 14:22:53
Messages: 619
Offline

I would recommend at least using read locks while serializing. Otherwise you are not guaranteed to have a consistent/stable view (same is multi-threaded single jvm apps). But, that said, reading without locks will still resolve stuff properly. You won't get any exceptions.

The problem you are seeing is likely due to the fact that serialization uses Unsafe to do stuff to the object. We actually intercept reflection stuff so that wouldn't have been a problem but we need to add support for the Unsafe calls.

The race condition tim refered to only happens if objects are cleared by the memory manager due to a low memory condition while serializing is going on. You can even turn the memory manager off if your object graphs fit in memory.

Hope this helps a little. Lots of good questions.

Cheers,
Steve

Want to post to this forum? Join the Terracotta Community
anodos

journeyman

Joined: 12/11/2006 13:33:29
Messages: 41
Offline

steve wrote:
... Just curious what you are building over there? 

We use Acegi security in our product (which is essentially Spring security), which has an Access Control (ACL) package. However, the JDBC implementation provided by Acegi has two limitations: 1) it is slow (not because of the implementation, but because it is based on JDBC), 2) it has to be ported to different DBs. Moving the implementation to Hibernate solves #2, but causes the service to become even slower. In our system, the ACL service gets heavy use - almost every object accessed or returned by other services in our system checks access against the ACL service. Returning a large collection full of other objects can be particularly slow.
We have reimplemented the ACL package using transactional maps (which we adapted from Commons Transactions in order to synchronize with surrounding Spring driven JTA transactions) and Terracotta. So far, it is much faster. This FaultingObjectOutputStream solved our last remaining issue in our test environment, so we are now going to put this into our QA environment and see what happens.
If I had the time, I'd contribute the Spring driven transactional map and Terracotta ACL implementation. At the moment, I'm swamped. :)
kbhasin

consul

Joined: 12/04/2006 13:08:21
Messages: 340
Offline

Hi anodos,

I am just curious, have you thought about clustering the org.acegisecurity.acl.basic.cache.EhCacheBasedAclEntryCache where either org.acegisecurity.acl.basic.cache.EhCacheBasedAclEntryCache.cache (of type net.sf.ehcache.Cache) or net.sf.ehcache.store.map (of type java.util.Map) could be Terracotta roots. There is also a possibility of creating a TerracottaBasedAclEntryCache. We have internally cluetered EHCache and are working on packaging the cluster-configuration as a Configuration Bundle.

This is a perfect example of something to put on Terracotta Forge so others in the community can benefit and it also highlights the flexibility of Terracotta. If and when you have some time, we would be happy to set up a forge project to host this solution.

Here is the link to Terracotta Forge if you are curious:

http://www.terracotta.org/confluence/display/orgsite/Forge

Regards,
Kunal Bhasin.

Regards,

Kunal Bhasin,
Terracotta, Inc.

Be a part of the Terracotta community: Join now!
anodos

journeyman

Joined: 12/11/2006 13:33:29
Messages: 41
Offline

Actually, we are using Acegi's new (still beta) ACL package (org.acegisecurity.acls as opposed to org.acegisecurity.acl). We thought about clustering the cache only, but with our current ACL arrangement, we don't really need the ACLs to be stored in the DB since Terracotta persistence provides the required level of service. But the main reason we decided to reimplement MutableAclService using transactional maps and Terracotta is because we are evaluating this approach for possible use in other services.
kbhasin

consul

Joined: 12/04/2006 13:08:21
Messages: 340
Offline

Ah! I haven't followed Acegi progress for a while and didn't know they were rewriting the ACL package.

~ Kunal.

Regards,

Kunal Bhasin,
Terracotta, Inc.

Be a part of the Terracotta community: Join now!
 
Forum Index -> Terracotta Platform
Go to:   
Powered by JForum 2.1.7 © JForum Team