| Author |
Message |
![[Post New]](/forums/templates/default/images/icon_minipost_new.gif) 04/26/2007 14:53:51
|
anodos
journeyman
Joined: 12/11/2006 13:33:29
Messages: 41
Offline
|
I have a service that uses Terracotta to cache information. Remote clients invoke this service using Spring's HTTPInvoker (which, in turn, uses Java serialization to transmit parameters and return values). This service will often directly return an object from its cache (which is a DSO). My expectation is that the object will be serialized to the client, and the client will deserialize it as a normal local (non-DSO) object (the clients neither know or care that Terracotta is being used by the service).
The problem: sometimes the client will receive an object with fields set to null. On the server, these objects are fine, but once they get deserialized on the client, they can contain null fields. I believe what is happening is that the null fields were paged out on the server side and Terracotta did not fault them back in during the serialization process.
Is this a known bug? How can I work around it?
|
|
|
 |
![[Post New]](/forums/templates/default/images/icon_minipost_new.gif) 04/26/2007 16:08:39
|
Tim Eck
journeyman
Joined: 01/25/2007 08:57:02
Messages: 47
Location: San Mateo, CA
Offline
|
Sounds like you've found a bug, and your analysis seems sound. I'm going to look into further.
I don't know the actual mechanics of how java serialization retrieves field values during the serialization process, but it is almost certainly not through bytecode which DSO has had a chance to enhance (thus it can see unresolved reference fields).
As a short term workaround, I would think if you did readObject() and writeObject() manually, then the field access in those methods would trigger DSO to resolve references that may have been paged out. I mostly suggest it to help give us more data about the problem, not as a "solution".
We've have similar problems with the native implementation of Object.clone().
I'll report back when I have more info, and more than likely a link to the bug report
|
|
|
 |
![[Post New]](/forums/templates/default/images/icon_minipost_new.gif) 04/26/2007 16:20:15
|
anodos
journeyman
Joined: 12/11/2006 13:33:29
Messages: 41
Offline
|
Further testing on my side seems to confirm my original hypothesis. I've created a test that starts a Terracotta server and two clients. One client puts several thousand new objects into a map, then the other client gets and serializes them. If the serializing client does not access the objects, then all reference fields in the serialized objects are null. On the other hand, if I change the serializing client to access a couple of fields before serializing, then those fields will not be null (they will contain their proper values), though the unaccessed fields will remain null.
|
|
|
 |
![[Post New]](/forums/templates/default/images/icon_minipost_new.gif) 04/26/2007 16:45:23
|
anodos
journeyman
Joined: 12/11/2006 13:33:29
Messages: 41
Offline
|
I've created a temporary workaround. It's not ideal, as it forces you to use a custom ObjectOutputStream (which isn't always a possibility), and it happens to be slower (from my tests, it takes rougly twice as long to serialize with this method), but its easy and it works. Behold FaultingObjectOutputStream:
Code:
public class FaultingObjectOutputStream extends ObjectOutputStream {
public FaultingObjectOutputStream(OutputStream out) throws IOException {
super(out);
enableReplaceObject(true);
}
@Override
protected Object replaceObject(final Object obj) throws IOException {
try {
faultInFields(obj, obj.getClass());
return obj;
} catch(IllegalAccessException e) {
throw new IOException(e);
}
}
protected void faultInFields(final Object obj, final Class<?> clazz) throws IllegalAccessException {
for(final Field field : clazz.getDeclaredFields()) {
if(!Modifier.isStatic(field.getModifiers())) {
field.setAccessible(true);
if(!field.getType().isArray()) {
field.get(obj);
} else {
// Fault in every array element, just to be on the safe side...
final Object array = field.get(obj);
if(array != null) {
for(int i = Array.getLength(array) - 1;i >= 0;i--) {
Array.get(array, i);
}
}
}
}
}
if(clazz.getSuperclass() != null) {
faultInFields(obj, clazz.getSuperclass());
}
}
}
|
|
|
 |
![[Post New]](/forums/templates/default/images/icon_minipost_new.gif) 04/26/2007 17:00:02
|
anodos
journeyman
Joined: 12/11/2006 13:33:29
Messages: 41
Offline
|
Actually, most of the "overhead" of FaultingObjectOutputStream is coming from the fact that it is faulting in field references, meaning it also has more data to serialize (the resulting stream is almost double in size).
|
|
|
 |
![[Post New]](/forums/templates/default/images/icon_minipost_new.gif) 04/26/2007 17:20:03
|
steve
ophanim
Joined: 05/24/2006 14:22:53
Messages: 619
Offline
|
Tim entered a jira for the issue.
https://jira.terracotta.org/jira//browse/CDV-244
Just curious what you are building over there?
Cheers,
Steve
|
Want to post to this forum? Join the Terracotta Community |
|
|
 |
![[Post New]](/forums/templates/default/images/icon_minipost_new.gif) 04/26/2007 17:28:41
|
Tim Eck
journeyman
Joined: 01/25/2007 08:57:02
Messages: 47
Location: San Mateo, CA
Offline
|
Good deal. I think we collectively know and agree on what the bug is here. Steve already posted the link to the bug. I've confirmed the issue in a test case here.
There is a race condition with that custom ObjectOutputStream, but you're not likely to observe it. Even though you're using reflection to read/resolve all of the fields of the objects being serialized, there is still the chance that the DSO memory manager could page out those references before they were committed into the serialization stream. Our fix for this problem will obviously need to make sure this race is dealt with.
|
|
|
 |
![[Post New]](/forums/templates/default/images/icon_minipost_new.gif) 04/26/2007 19:08:34
|
anodos
journeyman
Joined: 12/11/2006 13:33:29
Messages: 41
Offline
|
Yes, and another problem with my solution is: what kind of locking should it do when accessing these objects? Right now, this solution seems to be working, but I can't quite figure out why... in my test case I'm not locking against anything while serializing the graph, yet Terracotta is still faulting in the values without complaining? Is this a cool feature or is my test code just getting lucky? I'd like to avoid taking out some coarse grained lock while serializing, and I'd also like to avoid putting locking logic into FaultingObjectOutputStream.
|
|
|
 |
![[Post New]](/forums/templates/default/images/icon_minipost_new.gif) 04/26/2007 19:18:38
|
steve
ophanim
Joined: 05/24/2006 14:22:53
Messages: 619
Offline
|
I would recommend at least using read locks while serializing. Otherwise you are not guaranteed to have a consistent/stable view (same is multi-threaded single jvm apps). But, that said, reading without locks will still resolve stuff properly. You won't get any exceptions.
The problem you are seeing is likely due to the fact that serialization uses Unsafe to do stuff to the object. We actually intercept reflection stuff so that wouldn't have been a problem but we need to add support for the Unsafe calls.
The race condition tim refered to only happens if objects are cleared by the memory manager due to a low memory condition while serializing is going on. You can even turn the memory manager off if your object graphs fit in memory.
Hope this helps a little. Lots of good questions.
Cheers,
Steve
|
Want to post to this forum? Join the Terracotta Community |
|
|
 |
![[Post New]](/forums/templates/default/images/icon_minipost_new.gif) 04/26/2007 19:20:50
|
anodos
journeyman
Joined: 12/11/2006 13:33:29
Messages: 41
Offline
|
steve wrote:
... Just curious what you are building over there?
We use Acegi security in our product (which is essentially Spring security), which has an Access Control (ACL) package. However, the JDBC implementation provided by Acegi has two limitations: 1) it is slow (not because of the implementation, but because it is based on JDBC), 2) it has to be ported to different DBs. Moving the implementation to Hibernate solves #2, but causes the service to become even slower. In our system, the ACL service gets heavy use - almost every object accessed or returned by other services in our system checks access against the ACL service. Returning a large collection full of other objects can be particularly slow.
We have reimplemented the ACL package using transactional maps (which we adapted from Commons Transactions in order to synchronize with surrounding Spring driven JTA transactions) and Terracotta. So far, it is much faster. This FaultingObjectOutputStream solved our last remaining issue in our test environment, so we are now going to put this into our QA environment and see what happens.
If I had the time, I'd contribute the Spring driven transactional map and Terracotta ACL implementation. At the moment, I'm swamped. :)
|
|
|
 |
![[Post New]](/forums/templates/default/images/icon_minipost_new.gif) 04/26/2007 23:59:50
|
kbhasin
consul
Joined: 12/04/2006 13:08:21
Messages: 340
Offline
|
Hi anodos,
I am just curious, have you thought about clustering the org.acegisecurity.acl.basic.cache.EhCacheBasedAclEntryCache where either org.acegisecurity.acl.basic.cache.EhCacheBasedAclEntryCache.cache (of type net.sf.ehcache.Cache) or net.sf.ehcache.store.map (of type java.util.Map) could be Terracotta roots. There is also a possibility of creating a TerracottaBasedAclEntryCache. We have internally cluetered EHCache and are working on packaging the cluster-configuration as a Configuration Bundle.
This is a perfect example of something to put on Terracotta Forge so others in the community can benefit and it also highlights the flexibility of Terracotta. If and when you have some time, we would be happy to set up a forge project to host this solution.
Here is the link to Terracotta Forge if you are curious:
http://www.terracotta.org/confluence/display/orgsite/Forge
Regards,
Kunal Bhasin.
|
Regards,
Kunal Bhasin,
Terracotta, Inc.
Be a part of the Terracotta community: Join now! |
|
|
 |
![[Post New]](/forums/templates/default/images/icon_minipost_new.gif) 04/27/2007 08:10:14
|
anodos
journeyman
Joined: 12/11/2006 13:33:29
Messages: 41
Offline
|
Actually, we are using Acegi's new (still beta) ACL package (org.acegisecurity.acls as opposed to org.acegisecurity.acl). We thought about clustering the cache only, but with our current ACL arrangement, we don't really need the ACLs to be stored in the DB since Terracotta persistence provides the required level of service. But the main reason we decided to reimplement MutableAclService using transactional maps and Terracotta is because we are evaluating this approach for possible use in other services.
|
|
|
 |
![[Post New]](/forums/templates/default/images/icon_minipost_new.gif) 04/27/2007 10:23:07
|
kbhasin
consul
Joined: 12/04/2006 13:08:21
Messages: 340
Offline
|
Ah! I haven't followed Acegi progress for a while and didn't know they were rewriting the ACL package.
~ Kunal.
|
Regards,
Kunal Bhasin,
Terracotta, Inc.
Be a part of the Terracotta community: Join now! |
|
|
 |
|
|