Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

Screen shot 2010-08-31 at 12.32.15 PM.png

We recently deployed WebSessions (w/ TC 3.3) with JBoss 4.2x we noticed disk usage of [jboss_home]/server/[server]/work directory explode to 25GB per node.

As far as I can see the DGC is keeping up, DGC total times are less then 1 second and 90% of the time less then 500ms.

With Sessions, the total live object count has grown from about 25k to 125k.

I've attached a screenshot of the runtime statistics from the dev console.

Is this normal, if not can someone point me to what I should be tuning?

ps. I have young gc turned on, but am planning on turning it off, it's not really doing much.

Is your terracotta server configured to write in your jboss work directory? Do you know which files are contributing to that 25GB? On the jboss/client side of things the only file we write is our log file. Something can trigger some excessive logging but until you know what files are taking up the space I wouldn't want to guess/speculate about anything

Hi Tim,

As far as I know terracotta isn't configured to write in my jboss work directory unless thats the default, but I don't see any reference to it in the client log.

It's not the terracotta log, those are taking up a trivial amount of disk space.

The files that are taking up the space are

/work/jboss.web/localhost/[mywebcontext]/[filter-name]-filestore/

which has hundred of directories like

9670/1778
9884/1343

etc

which then contain another directory named for the jsessionID

uvKJQsAes3x1fBmv60G7.node3
K03QZwFFBmLIzZkL6amd.node3

etc

and finally a file called pm-null which looks like a serialized version of the actual session.

Our session stuff shouldn't be writing files like, we don't use disk on the jboss/client side for persisting sessions.

I don't really know what those files are. If you can it'd be interesting to know if the files still creating even when terracotta is removed from this application.

The site has been live for over two years, this only happens when terracotta sessions is active (we use it otherwise for ehcache) for over 6 months now...

hmm, well I'd want to figure out who is writing those files before anything else. I know our session code isn't explicitly doing that but I guess it is within reason to suggest the presence of terracotta is influencing some other code to do it.

I'd need something that demonstrates the issue though. If you can figure out a reasonably simple test case I'd be happy to debug it. If Terracotta is at fault we'll get a bug entered and scheduled.

I did a bit more digging, the files are actually being generated by Wicket.

Specifically the DiskPageStore.

Code:

 private static File getDefaultFileStoreFolder()
     {
         final File dir = (File)((WebApplication)Application.get()).getServletContext()
             .getAttribute("javax.servlet.context.tempdir");
         if (dir != null)
         {
             return dir;
         }
         else
         {
             try
             {
                 return File.createTempFile("file-prefix", null).getParentFile();
             }
             catch (IOException e)
             {
                 throw new WicketRuntimeException(e);
             }
         }
     }

Not sure how this interacts with Terracotta however... I remember older versions has a TIM Wicket plugin, is this also the case with Sessions?

There is a tim-wicket but it our session stuff doesn't use it in anyway.

From the code posted there I don't see how we'd be affecting it. Our session could possibly interfere with request attributes but not context attributes.

Anyhoo, if you get further and have a WAR that will show the issue my offer to debug it is standing.

thx

Hi,
Here you can find this is a tim-wicket.but session is very stuff. so you do not use it in anyway.

_______________________________________________________________
Want to get-on Google's first page and loads of traffic to your website? Hire a SEO Specialist from Ocean Groups seo specialist

teck wrote:

There is a tim-wicket but it our session stuff doesn't use it in anyway.

From the code posted there I don't see how we'd be affecting it. Our session could possibly interfere with request attributes but not context attributes.

Anyhoo, if you get further and have a WAR that will show the issue my offer to debug it is standing.

thx

Thanks Tim, I'm going to take you up on that offer. Just to recap:

Wicket serializes session data using it's DiskPageStore which creates a files located at

[jboss server home]/work/jboss.web/localhost/[mywebcontext]/[filter-name]-filestore/

In the normal course of operation these files get cleaned up, but with WebSessions many of the files seem to be orphaned.

The platform I tested this on is:

Centos 5.5
JBoss 5.1 (two nodes)
Apache 2.2 using mod_balance/proxy
Terracotta 3.3.0
Wicket 1.4.10

I was able to recreate the issue by deploying the attached war and simulating a number of sessions. In the course of these sessions I would rotate in and out the two nodes. This resulted in several hundred 'hanging' sessions.

I suspect that the session is sufficiently replicated so that the user continues to seamlessly browse, but not enough that it gets evicted when the session invalidator runs on the sessions that have failed over and exceeded the session timeout. (But I don't really understand how it works so this is pure speculation.)

I've included the war which has a basic Wicket application and 3 pages. Hitting the homepage will generate a session, the other two pages were for my script to flip between. I have the session timeout set to 3 minutes. You will need to set a system property ${tcServerConfigURL} which is how I configure the location of the terracotta server.

I've included the war, sources and the JMeter JMX script I used to simulate the sessions.

Thanks for you help, let me know if I can provide any more information.

Jeremy

Bump :)

Tim, is this something that you'll be able to look at?

We have upgraded to 3.4 and continue to see this issue. Would be great if someone from Terracotta could chime in.

J

To follow up on this.

The issue was because in addition to the functionality which Terracotta provider with it's Web Sessions functionality Wicket also includes a SecondLevelCacheSessionStore via DiskPageStore.

DiskPageStore will serialize Pages and some additional Metadata to disk, up to about 10mb per session. As a result, when a session fails over from node A to node B, the primary session data is migrated via Terracotta however the secondary data is not. If node A is restarted and the session never returns to node A the unbind/cleanup methods are never called for that session because after a restart it's servlet container is never aware of the session in the first place. As a result it leaves the data on the disk that Terracotta was never aware of.

When doing a rolling deployment with a couple of thousand of users on a node you can see how a lot of data was being left orphaned.

We looked at two solutions, replicating the session cleanup across all nodes, therefore a session expiring on node 2 would also try to cleanup the corresponding data on node 1 but this wasn't entirely ideal and in theory would still leave data on the disk if no servlet container was aware of the session.

We ended up implementing an Ehcache backed disk page store this seems to provide missing failover which didn't exist in the pure Terracotta Sessions implementation due to Wickets SecondLevelCacheSessionStore. As well as ultimately relying on Ehcache for the cleanup.

jelevy wrote:

teck wrote:

There is a tim-wicket but it our session stuff doesn't use it in anyway.

From the code posted there I don't see how we'd be affecting it. Our session could possibly interfere with request attributes but not context attributes.

Anyhoo, if you get further and have a WAR that will show the issue my offer to debug it is standing.

thx

Thanks Tim, I'm going to take you up on that offer. Just to recap:

Wicket serializes session data using it's DiskPageStore which creates a files located at

[jboss server home]/work/jboss.web/localhost/[mywebcontext]/[filter-name]-filestore/

In the normal course of operation these files get cleaned up, but with WebSessions many of the files seem to be orphaned.

The platform I tested this on is:

Centos 5.5
JBoss 5.1 (two nodes)
Apache 2.2 using mod_balance/proxy
Terracotta 3.3.0
Wicket 1.4.10

I was able to recreate the issue by deploying the attached war and simulating a number of sessions. In the course of these sessions I would rotate in and out the two nodes. This resulted in several hundred 'hanging' sessions.

I suspect that the session is sufficiently replicated so that the user continues to seamlessly browse, but not enough that it gets evicted when the session invalidator runs on the sessions that have failed over and exceeded the session timeout. (But I don't really understand how it works so this is pure speculation.)

I've included the war which has a basic Wicket application and 3 pages. Hitting the homepage will generate a session, the other two pages were for my script to flip between. I have the session timeout set to 3 minutes. You will need to set a system property ${tcServerConfigURL} which is how I configure the location of the terracotta server.

I've included the war, sources and the JMeter JMX script I used to simulate the sessions.

Thanks for you help, let me know if I can provide any more information.

Jeremy

I found frostwire best .Hope you can solve your problem.