[Logo] Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)
  [Search] Search   [Recent Topics] Recent Topics   [Members]  Member Listing   [Groups] Back to home page 
[Register] Register / 
[Login] Login 
[Expert]
Messages posted by: shortmem  XML
Profile for shortmem -> Messages posted by shortmem [17] Go to Page: 1, 2 Next 
Author Message
I searched all around about Wicket + TC session clustering but everything I've found seems to be related to the old DSO session clustering.

My question is quite simple actually...Should I use HttpSessionStore or SecondLevelCacheSessionStore (the default)?

As far as I know, SLCSS delegates page resolution to DiskPageStore, which (de)serializes pages from/to the disk in order to avoid excessive RAM consumption.

I think that DiskPageStore is meaningless for failover unless the files are written on some sort of NFS, which is not my case.

If I use HttpSessionStore, everything will be put directly into the HttpSession provided by the container (the Terracotta HttpSession impl). Does WebSessions work more or less like DSO, in the sense that keys are kept in memory and data is faulted to/from the TC server, keeping RAM in check?

florescu wrote:
We have implemented a lot of clustering infrastructure on top of terracotta, eg a specialized map-reduce implementation with sophisticated recovery when nodes fail, a task channel (somewhat like your master-worker), clustered locks, barriers, cluster membership & control services, etc.

ehcache is just that, a distributed cache, and cannot be the answer...
Should we infer that Terracotta is NOT going to support DSO going forward and Java 1.6 is the last supported version? If the answer is yes, that is very disappointing, I have pushed for terracotta adoption at all my jobs for the last 6 years (including introducing terracotta to one of your 9 big customers listed on your home page...) 


We're on the same page. I'm trying to rework a map-reduce framework with terracotta express and ehcache. The code resembles pretty much working with hibernate: load some stuff, update it, remember that the object must be re-serialized (persisted) so that everyone can see it...To be fair, ehcache key locking, which I'm using to control some metadata state, is fast and reliable, but the lack of Terracotta transactions makes the code non-elegant in some situations.

It's been a while that questions about DSO are replied with "Go with express. It's easier and faster". Well for those who have been following Terracotta from it's roots (no pun intended) these comments seem at least ...suspicious. I would like to know when did brute force serialization became faster than delta-only serialization in general.

As I've said in another post, I understand why express/ehcache is being chosen in favor of DSO. I do believe that express has better scalability, since one of the main flaws of DSO is keeping all keys in memory for a map structure, however it is annoying to read "Go with ehcache/express". You sound like salesmen...:(

Anyway, I could use DSO with jdk 7 with update 3. I think that after update 3 they pushed String from alt-rt.jar (which is loaded in the boot-classpath with XX:+AggressiveOpts) into rt.jar.
This is very sad.

I understand the comercial reasons behind this decision as well as new the direction the product has taken to take advantage of today's cheaper hardware.

DSO was the most exciting technology I came across and it can do wonders for medium scale deployments. Plus, it was fun to watch some flame wars between Ari and Cameron Purdy on the blogosphere. In fact it can be painful to deal with DSO's intricacies, but like any other technology, once you get used to it and fully understands its caveats, it becomes easy and productive.

I feel lucky to have had the opportunity to implement it when no other technology could achieve what DSO could (TreeCache/PojoCache were conceptually the closest contenders, but they weren't nearly as good and powerful) and at the same time I feel sad for the kids that are starting to play around distributed computing. Probably they will never experience the fun of network attached memory and, more importantly, they'll never touch some of the concepts DSO has brought to the understanding of distributed systems.

rajoshi wrote:
Which version of terracotta you are using , in the current version by default it's shared across VM's. 


Rajoshi I tested in both 3.7.2 and 3.7.4. The test is very simple: Just call await in one or more nodes, and launch another an call signalAll(). The previous nodes remain blocked.

The examples I found work only with DSO:

Code:
  protected Condition getCondition(final Lock lock, String name) {
     Assert.assertTrue(Terracotta.isManaged(lock));
     return Terracotta.lookupOrCreateRoot(name, new Callable<Condition>() {
       public Condition call() throws Exception {
         return lock.newCondition();
       }});
   }
 


I think that, if possible, the returned lock from writeLock() should be a Terracotta specialization with a getCondition(String name). Even in the same node newCondition() will return distinct conditions (as is expected).
Hi, is there any way to get a clustered condition when using express mode?

The following code

Code:
 ReadWriteLock lock = toolkit.getReadWriteLock("TheRWLock");
 
 Lock wl = lock.writeLock();
 
 Condition cond = wl.newCondition();
 


Seems to produce a cluster aware condition (org.terracotta.locking.TerracottaCondition), however there's no way to share it across distinct VMs.

njain wrote:
Absolutely.

Recommended approach is to use a combination of quartz and ehcache. Use Clustered Quartz for scheduling the work and distributed ehcache for sharing the state.

Quartz where which is an enterprise feature gives you finer control on scheduling.  


This is an old question, but I'd like to know how to do this properly in express mode. I believe the answer refers to working with asynchronous jobs, whoever I don't think it's that simple if one needs to actually collect the results of a distributed computation (like joining the threads in a cluster). I see that terracotta seems to be integrating with Hadoop, whoever I'd like to know if there's another, more light-weight approach.

In the "old" way, which I think still works if I configure a DSO cluster, it was straightforward to collect and merge the results if the state were to be shared across Work instances and queues, due to DSO magic.

In the express mode there's a messy logic involved in order to update the state of work instances, plus error handling, joining etc is not trivial to implement using a clustered cache.

I'm currently trying to implement this using a cache based solution, and usually the involved patterns are more or less like this:

Code:
        //copy values to a mutable list to shuffle it
        List keys = new ArrayList<>(cache.getKeys());
         
         //shuffle them to avoid sequential bottleneck on lock-acquisition
         Collections.shuffle(keys);
 	
 	for (Object key : keys) {
 		cache.acquireWriteLockOnKey(key);
 		try{
 			Element element = cache.get(key);
 			
 			if(element!=null){
 				Work w = (Work) element.getObjectValue();
 				
 				if(w.getState()==State.PENDING){
 					w.setState(State.RUNNING);
 					//return serialized copy - overhead in re-searializing the whole state contained in w
 					cache.replace(new Element(key, w));
                                         //submit work to queue ...
 				}
 			}
 		}finally{
 			cache.releaseWriteLockOnKey(key);
 		}
       }
 

rajoshi wrote:
Can you please raise a community JIRA for this here:
https://jira.terracotta.org/jira/browse

 


Jira raised: https://jira.terracotta.org/jira/browse/CDV-1651

rajoshi wrote:
Can you please raise a community Jira for this here with all the details :

https://jira.terracotta.org/jira/browse 


Posted https://jira.terracotta.org/jira/browse/CDV-1650
Well ok...hacked in the TerracottaSessionManager code...

Code:
  public boolean killSession(String browserSessionId) {
     SessionId id = this.idGenerator.makeInstanceFromBrowserId(browserSessionId);
     if (id == null)
     {
       id = this.idGenerator.makeInstanceFromInternalKey(browserSessionId);
     }
     return expire(id);
   }
 
 private boolean expire(SessionId id) {
     boolean result = false;
     SessionData sd = null;
     boolean locked = false;
     try {
       sd = this.store.find(id);
       if (sd != null) {
         if (!isSessionLockingEnabled()) {
           id.getWriteLock();
         }
         locked = true;
         expire(id, sd);
         result = true;
       }
     } finally {
       if ((sd != null) && (locked)) {
         id.commitWriteLock();
       }
     }
     return result;
   }
 


It seems that there's no SessionData associated with the id, but since the id keeps showing in dev-console I assume it's is dangling somewhere in the sessionData (a ConcurrentDistributtedMap) keyset:

Code:
  public Set<String> getSessionIds(int batchSize) {
     int count = 0;
     HashSet result = new HashSet();
     for (String id : this.store.getAllKeys()) {
       result.add(id);
       if (count++ >= batchSize) {
         break;
       }
     }
     return result;
   }
 



Is this not possibly a memory leak? Perhaps the code should force the removal of these ids from the store:

Code:
 private boolean expire(SessionId id) {
     boolean result = false;
     SessionData sd = null;
     boolean locked = false;
     try {
       sd = this.store.find(id);
       if (sd != null) {
         if (!isSessionLockingEnabled()) {
           id.getWriteLock();
         }
         locked = true;
         expire(id, sd);
         result = true;
       }else{
            //FORCE REMOVE
           this.store.remove(id);
        }
     } finally {
       if ((sd != null) && (locked)) {
         id.commitWriteLock();
       }
     }
     return result;
   }
 
 
OK I managed to get a standalone APP to connect to jmx-server and attempt to invalidate the sessions.

Unfortunately I can't use this code in my web-application since I had to put the whole tc.jar in this standalone app classpath to use the jmxmp protocol. Is there any other way to connect programmatically to the JMX server?

Code:
 import java.io.IOException;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
 
 import javax.management.MBeanServerConnection;
 import javax.management.MBeanServerInvocationHandler;
 import javax.management.ObjectName;
 import javax.management.remote.JMXConnector;
 import javax.management.remote.JMXConnectorFactory;
 import javax.management.remote.JMXServiceURL;
 
 //copied from the tim inside sessions.jar
 import com.terracotta.session.management.SessionMonitorMBean;
 
 public class TCJMXSessionHelper {
 
 	String connectorUrl = "service:jmx:jmxmp://172.16.100.2:9520";
 
 	String appPattern = "org.terracotta:appName=localhost/myapp";
 
 	private JMXConnector getJMXConnector() throws IOException {
 		return getJMXConnector(connectorUrl);
 	}
 
 	JMXConnector getJMXConnector(final String url) throws IOException {
 		return JMXConnectorFactory.connect(new JMXServiceURL(url));
 	}
 
 	List<SessionMonitorMBean> getSessionMbeans(final JMXConnector connector) throws IOException {
 
 		final List<SessionMonitorMBean> ret = new ArrayList<>();
 		final MBeanServerConnection connection = connector.getMBeanServerConnection();
 
 		final Set<ObjectName> names = connection.queryNames(null, null);
 
 		for (final ObjectName on : names) {
 			if (on.toString().startsWith(appPattern)) {
 				ret.add(MBeanServerInvocationHandler.newProxyInstance(connection, on, SessionMonitorMBean.class, false));
 			}
 		}
 
 		return ret;
 	}
 
 	public void purgeSessions(final Iterable<String> ids) throws IOException {
 
 		final JMXConnector connector = getJMXConnector();
 
 		try {
 			final List<SessionMonitorMBean> mbeans = getSessionMbeans(connector);
 
 			for (final SessionMonitorMBean bean : mbeans) {
 				try {
 					for (final String id : ids) {
 						bean.expireSession(id);
 					}
 					break;
 				} catch (final Exception ex) {
 					// ignore go to next bean
 				}
 
 			}
 
 		} finally {
 			connector.close();
 		}
 
 	}
 
 	public void purgeSessions(final String... ids) throws IOException {
 		purgeSessions(Arrays.asList(ids));
 	}
 
 	public void purgeStaleSesssions() throws IOException {
 
 		final JMXConnector connector = getJMXConnector();
 
 		try {
 			final List<SessionMonitorMBean> mbeans = getSessionMbeans(connector);
 
 			beanLoop: for (final SessionMonitorMBean bean : mbeans) {
 				final int step = 100;
 				final long total = bean.getActiveSessionCount();
 
 				try {
 					for (long j = step; j <= total + step; j += step) {
 						final Set<String> ids = bean.getSessionIds(step);
 
 						if (ids == null || ids.isEmpty()) {
 							break beanLoop;
 						}
 
 						for (final String id : ids) {
 							final Map<String, String> map = bean.getSessionAttributes(id);
 
 							if (map == null || map.isEmpty()) {
 								if (!bean.expireSession(id)) {
 									System.out.println("Why can't TC expire " + id + " ????");
 								}
 							}
 						}
 					}
 					break;
 				} catch (final Exception ex) {
 					// ignore go to next bean
 				}
 			}
 		} finally {
 			connector.close();
 		}
 	}
 
 }
 
 


What are the conditions for the expireSession method to return false?

When I create a session, it is successfully expired via this jmx helper, however these "blank" sessions can't be expired!!! Even in dev-console (which I believe uses a similar code) they can't be expired at all!
Hello, I wonder if there is a way to programmatically interact with Terracotta sessions.

My application has 2 requirements that has given me a lot of headaches, but I somehow managed to handle them:

  • No concurrent login
  • Admin must have an interface to manually expire a session


    I'm using spring-security and my app is deployed under jboss-as 4.3.

    For the first requirement, I had to create a cluster-aware SessionRegistry so that a spring would work in a clustered environment. Plus, I had to extend the SessionInformation object in order to notify the cache once an object was mutated (Unfortunately spring code mutates the objects, it's not up to the service only - it assumes everything is in memory, but to reflect the changes on EhCache I must deal with Serialization caveats).

    For the second requirement I created a JMX service that delegates invalidation to the JMX Manager of my application (jboss.web:host=localhost,path=/myapp,type=Manager) so that I can invoke the expireSession method by passing a sessionID.

    The problem is: Even though the application is behaving correctly, the sessions are NOT expired in terracotta.

    To test this behavior I did the following:

    1)Logged in with a user in my app
    2)Tried to log again in the same node (Spring Concurrent login worked)
    3)Tried to log again in from another node (Spring Concurrent login worked)
    4)Expired the session
    5)Repeat 1-3 (OK)
    6)I still can see the session created on step 1 on dev-console plus some empty sessions

    From my understanding it seems that the JBoss Session Manager does NOT play nicelly with Terracotta's sessions. Is there any out of the box jboss service that can handle this?

    Also, I can see TONS of "empty" sessions on dev-console (once I select the id the Attribute/Value table is empty). I believe some of them are actually created by spring (concurrent login control replaces the initial session after a successful login), but I don't understand why they're hanging there forever. After a week there are more than 80000 objects even if no one uses the Application in the weekend.

    Is there any programatic way to actually access these dead sessions? My real sessions are tagged with some attributes, so I could filter out the empty sessions from the real ones in order to remove them.
  • Hello, I just checked that terracotta 3.7.3 officially supports java 7 update 7!!!

    I'm currently running both my app servers and terracotta 3.7.2 with update 10. Both run without problems as long as I don't use the G1 collector.

    It seems that there's a G1 bug with jitted code of trove library (I tried 3.0.0 to 3.0.3). Someone else has alredy suffered from it:

    G1 BUG

    I tried to fill a bug report but oracle's bug database is not working and they'd probably would leave as low priority.

    I didn't actually see TC crashing when using G1 collector, just my application code every time a tight-loop calls put on TObjectIntHashMap.

    The bug is very easy to reproduce in a standalone test case, just create four or so Threads (each with it's own private map), calling map.put for about 1 million times.

    I'm sharing this because on TC 3.7.3 release notes there are no warnings about possible erratic G1 behavior and TC seems to make use of trove internally.
    If you do not want object state to be passed around, then you'll have to go with JGroups.

    I think there might be a JMS based solution, but I don't think it would perform as well as the raw jgroups because it would be topic based: every node would have subscribe to receive the notification to mantain coherence around the cluster, not to mention that the JMS overhead is probably higher than the JGroups stack overhead, which is 'closer' to the network.

    If you configure it for invalidation mode (see bellow), it will replicate only the CacheKeys, which contain references only to the id of your entity as well as some internal hibernate stuff, which are more or less lightweight:

    Code:
     public class CacheKey implements Serializable {
     	private final Serializable key;
     	private final Type type;  (has no fields for simple types, e.g. primitives and Strings)
     	private final String entityOrRoleName;
     	private final EntityMode entityMode; (enum like class)
     	private final int hashCode;
     ...
     }
     


    Config for invalidation mode (just turn off replicatePuts and updatesViaCopy):

    Code:
     <cache
      		name="some.domain.Entity"
      		maxElementsInMemory="5000" 
      		eternal="true"
      		overflowToDisk="false"
      		memoryStoreEvictionPolicy="LRU">
      		<cacheEventListenerFactory
      			class="net.sf.ehcache.distribution.jgroups.JGroupsCacheReplicatorFactory"
     properties="replicateAsynchronously=true,replicatePuts=false,replicateUpdates=true,replicateUpdatesViaCopy=false,replicateRemovals=true" />
      	</cache>
     


    You'll also have to configure the Jgroups stack itself to form a cluster. There are plenty of docs about it!
    I would like to know a bit more about how ehcache-terracotta handles classloading.

    I could get around most of the deployment issues of my app, but there was one that left me wondering how the ehcache-terracotta/terracotta-toolkit behave.

    Once we clustered our terracotta caches, everything was fine until we begun the round of HA tests :)

    I killed one of the app nodes and redeployed without any issues. The problem started to occur once we tried to use it by accessing something I knew other node was putting in the second level cache.

    We started to get serialVersionUID errors, hibernate`s CacheKey class was the culprit:

    Our app uses hibernate 3.3.2.GA whereas the server lib pointed to an older version. This older version was used by one of the jboss esb`s internal components, so we decided it would not be safe just replace it with the version we`re using.

    So we moved all 'old' hibernate dependencies from the jboss lib directory and placed inside the esb component, cleaned up and restarted terracotta and all app nodes.

    The deployment structure now was like this:

    Code:
     instanceName/
                 lib/ <no hibernate jars, tc-toolkit and tc-session jars>
                          
                 deploy/
                       myEar.ear/
                                lib/<app and hibernate 3.3.2.GA jars/deps, ehcache-core and tc jars>
                                    
                       someesb.esb/<jboss hibernate moved here>
     


    The problem persisted!!! I don't know how, but it seems that the terracotta engine picks up the very first CacheKey class loaded by the App Server and sticks to it! Since the ESB is a service it's always deployed and started first.

    Later I placed the hibernate 3.3.2.GA and its minimal dependecy set (slf4j) upon the lib dir. Another exception:

    org.hibernate.cache.CacheKey != org.hibernate.cache.CacheKey,

    after trying to access data from a restarted Node. SerialVersionUID problem gone, just another classLoader issue on the way.

    Solution? Remove the hibernate jars from our lib, place on the server lib and voila!

    Code:
     instanceName/
                 lib/ <tc-toolkit,tc-session jars, hibernate 3.3.2.GA jars/deps>
                          
                 deploy/
                       myEar.ear/
                                lib/<app, ehcache-core and tc jars>
                                    
                       someesb.esb/<jboss hibernate moved here>
     
    It's a bad idea to use native SQL through hibernate.

    As far as I know, it will use a 'conservative' approach and evict everything, even unrelated stuff.

    Even if your cache is not clustered, it's a terrible approach.

    Use direct jdbc and evict only the affected regions manually if necessary.
     
    Profile for shortmem -> Messages posted by shortmem [17] Go to Page: 1, 2 Next 
    Go to:   
    Powered by JForum 2.1.7 © JForum Team