Terracotta Discussion Forums

Dear all,
I build a SOAP application with Terracotta and Glassfish.

The request and repsonse SOAP is about 10K bytes long.

I am using a 4GB with Q8200 4-core CPU in CentOS 5.3.

The average performance is about 800-1000 response per second(TPS).

I used SOAP UI to send 1000 threads concurrent to the Glassfish to get such result.

Is this a resonable result?

BTW, in my SOAP, I need some task which would take about 1-4 secs to process additional work. So I use Threads.sleep(1000) to simulate the work.

Then the response go down to only 500 TPS.

If I sleep(2000) the response is only about 300 TPS. Is that reasonable, too?

After that I use Glassfish Loadbalancer plugin with SUN web server 7.
I use sleep(1000) then the response only go to about 150.
Then I found the reason is when The threads in SOAP UI exceeed 256. Then some request will fail with "java.net.SocketException:Connection reset".
I am wondering how to configue SUN web server 7 to get better throughput.

Thank you for all your help!!

Eric

Sorry, but we would need more info before replying...

At first, I would ask, why use Terracotta if you are running only 1 JVM? More importantly, can you step back up to a higher level? What is the app for? How many JVMs are you targeting? What latency and throughput are you targeting?

--Ari

Thank you for your help.
My application is similar to chatting program.
Each client send request to server by SOAP.
There would be about 3 Glassfish server and 1 SUN Webserver 7(load-balancer) and 1 Terracotta server.
Basically, we hope the throughput would be at least 1200 request/response pair per second from the point of client view.
We are very thanks for your Terracotta. But we don't know if the performance result is correctly configured.

The main issue for my SOAP application is that the process will need about 1-4 seconds to make some additional call from outside server. The whole process throughput decreased.

I use SOAPUI to simulate concurrent request.
I open 4 condifition 128, 256, 512 or 1024 threads(use diffrent condition in each test) in my client to send concurrent request to the servers.
And in the Glassfish SOAP application, I used Thread.Sleep(1000) to simulate the waiting outside response.

There are no latency and the throughput are
(All with Terracotta server on)
1.One Glassfish server no Load balancer.
a. no Thread.Sleep(1000) , the result is about 600 reuest/response per second.
client
b. with Thread.Sleep(1000),
client with 128 threads the result is about 110 request per second.
client with 256 threads the result is about 230 request per second.
client with 512 threads or more the result is about 410 request per second.

2. Two Glassfish server with Load balancer.
a. no Thread.Sleep(1000) , the result is about 1000 reuest/response per second. I dont need to use more than 128 thread in my client to reach such throughput.
But the problem reuest thread more than 256 then some request will show "java.net.SocketException:Connection reset".
b. with Thread.Sleep(1000),
Unfortunately, if I wait 1 seconds to simulate the process. the SUN webserver7 will result in "java.net.SocketException:Connection reset". in my client console.
If I issue more than exactly 256 threads.
I also configured the SUN Webserver7 but still can not slove this problem.
I am wondering why????
Then the throughput is no more than 200, sigh.

From above, I know the application server's working thread is hanged to process additional request just because it has to wait additional 1 seconds to response back to client.(Normally each request take about 10-15 ms to complete).

I want to know if my current throughput good or too bad.
And what's wrong with my Loadbalancer. Do you have any suggestion?

Thanks

ari wrote:

Sorry, but we would need more info before replying...

At first, I would ask, why use Terracotta if you are running only 1 JVM? More importantly, can you step back up to a higher level? What is the app for? How many JVMs are you targeting? What latency and throughput are you targeting?

--Ari

I finally found some more results.

As previously stated, my throughput based on Terracotta running.

I made some more test without Terracotta connected.

The results are

A.One Glassfish without Terracotta and no Load-Balancer.
1.No Thread.Sleep(1000)--> about 800 TPS.
With request client thread less than 128.
2.With Thread.Sleep(1000)-->
if request client thread around 800-100 then about 800TPS, too.

B.One Glassfish without Terracotta and through the SUN WebServer 7 Load Balancer.
1.No Thread.Sleep(1000)--> can reach about 800 TPS.
With request Client Thread less than 128, too.

2. The most interesting thing that I think is the bottleneck.
The client can request more than 512 thread at this time.
and reach about 800 then the throughput is about 750-800 TPS.

Finally, I am wondering why if I use Load Balancer with Glassfish and the
Terracotta Server connected. The system will be hanged.

Why???

It looks like the Terracotta is restricted by the connection.

In my SOAP applicaiton, I think the read cache action should not lock the whole processing.

I attched some part of my code related to Terracotta.

Hope you will give me some help.

Thanks.

A.The SOAP code.
@WebService(serviceName = "messaging.asmx", portName = "MessengerSinkSoap12", endpointInterface ="com.xxx.Messaging" wsdlLocation = "WEB-INF/wsdl/MessagingServer/messaging-server.wsdl")
@BindingType(value = javax.xml.ws.soap.SOAPBinding.SOAP12HTTP_BINDING)
public class MessagingServer implements MessengerSinkSoap {
// UserManager to communicate with Terracotta Server
UserManager userManager = UserManager.getInstance();

public RegisterResponse messagingRegister(RegisterRequest registerRequest) throws SoapenvFault {
checkDeviceAddress(registerRequest.username, registerRequest.deviceAddress);
Threads.Sleep(1000);
}

public boolean checkDeviceAddress(String username, String deviceAddress) throws SoapenvFault {
DeviceInfo deviceInfo = userManager.getCurrentDevice(deviceAddress);
// If the deviceAddress exist, check if the username is matched with the clientInfo's username
// if matched, no futher action, if not, unregister the user
// If the deviceAddress not exist, no action
if (deviceInfo == null) {
deviceInfo =
new DeviceInfo(deviceAddress, username);
// Update Terracotta Cache
userManager.registerDevice(deviceInfo);
return false;
} else {
if (deviceInfo.getUserName().equalsIgnoreCase(username)) {
// No update action
return false;
} else {
// username mismatched
deviceInfo.setUserName(username);
// Update Terracotta Cahce
userManager.registerDevice(deviceInfo);
return true;// indicate deviceAddress updated

}

}
}
}
}

B. The Terracotta related code
public class UserManager {

static class UserManagerHolder {

static UserManager instance = new UserManager();
}

public static UserManager getInstance() {
return UserManagerHolder.instance;
}
private final ConcurrentStringMap<DeviceInfo> devices;

public UserManager() {
this.devices = new ConcurrentStringMap<DeviceInfo>();
}

public DeviceInfo getCurrentDevice(String deviceAddress) {
// synchronized (devices) {
return this.devices.get(deviceAddress);
//}
}

public void registerDevice(DeviceInfo deviceInfo) {
/*
synchronized (devices) {
if (!devices.containsKey(deviceInfo.getDeviceName())) {
return;
}
devices.put(deviceInfo.getDeviceName(), deviceInfo);
}
*/
devices.putNoReturn(deviceInfo.getDeviceAddress(), deviceInfo);
}
}

public class DeviceInfo {

private String deviceAddress;
private String userName;

public DeviceInfo(String deviceAddress, String name) {
this.deviceAddress = deviceAddress;
this.userName = name;
}

public String getDeviceAddress() {
return this.deviceAddress;
}

public String getUserName() {
return this.userName;
}

public synchronized void setUserName(String name) {
//public void setUserName(String name) {
this.userName = name;
}

@Override
public String toString() {
return userName;
}
}

Not sure I understand this at all. Sorry, but can you explain more about your test and environment? All the discussion about threads is not helping me understand what you are trying to achieve.

--Ari

Thank you. Let me detail in here.

As previously stated. I used SOAPUI to access my Register service.
In my client PC, the SOAPUI is configured to spam N threads simutaneously to simulate N connection at the same time. Where N is range from 64, 128, 256, 512, 768, 1024, etc..

My Glassfish server is very simple as my code except that some additional action whcih call outside service and wait that service response. So I use Thread.Sleep(1000) to simulate the situation.
In the SOAP service, it will access Terracotta and make some update action.
I checked the DevConsole and found the access time less than 1 ms.
And the lock seems work fine.

The test environment is in LAN. Glassfish Server AP1 and AP2.
Glassfish Load-Balancer Plug-in with SUN WebServer7 is LB1.
Each Server is 4GB RAM with 4-core CPU on CentOS 5.3.
Glassfish is V2U2 as Terracotta need.
Terracotta server is 3.0.1
JDK is 1.6 update 13

1-3 client is 4GBRAM 2-core CPU on Windows XP with SP3.
JDK is 1.6 update 13, too.
SOAPUI is 2.5.1

1. Start Terracotta Server.
2. Start 1 or 2 Glassfish server
3. Start Load-Balancer
4. Start Client
The client use 64, 128, 256, 512, 768 or 1024 threads in each test cycle.
Then I compared each different thread's throughput.

I found that if NO Terracotta Server (with/without Load-Balancer) the throughput is almost the same for each different threads input.
Except that if I put Thread.Sleep(1000) will has a less throughput.
But the most important thing is the client can send more than 256(that is 512 or 768 or 1024) threads simutaneously . That also indicated that the whole system can accept more than 256 concurrent requests.

In another case, if Terracotta server is connected.
Without Load-Balancer, if I did not put Thread.Sleep(1000) in my code. The throughput is about 600 for different request N threads. In this case, only 1 Glassfish server is counted.

But the most curiously one is, if the Terracotta server is connected and the Load-Balancer is used. No matter 1 or 2 Glassfish server on. When the clients(1 or more) total threads just more than 256 threads, then the client will get some error said that the server can not response. The throughput at most around 120 TPS.

I don't know why. What kind of info. you need to help me slove this issue?

Thank you!!

I don't need a very powerful result. But I dout why the Load-Balancer is used and the Terracotta server is connected will limit the total connection?

BTW, I forgot to said that if my code used Thread.Sleep(1000) then this problem will occurred.

If I did not put Thread.Sleep(1000) then it worked fine.
The load-balancer will give me the added throughput as I want.
That means is about 1200 TPS with Terracotta server connected and Load-Balancer used no matter N threads I requested.(more than 512).

I think maybe there some resource locked. But I dont know why and where is locked.

Thank you!!

ok. looks like I asked the same question (more detail) twice, sorry. I am still lost though.

Let me try to play back what I am hearing:

1. You can do 1000 tps on a single Glassfish instance w/o Terracotta
2. When you add Terracotta, you get 500 TPS from the same Glassfish instance
3. If you add Sun's load balancer, then the system hangs and outputs no responses to your SOAP service.

Is this correct?

--Ari

1. You can do 1000 tps on a single Glassfish instance w/o Terracotta
Case1. No Thread.Sleep(1000) for each service no waiting one second.
Then the result is 800 tps.
Case2. If I add Thread.Sleep(1000), each request is hang 1 second.
Then the result is about 800 TPS.

2. When you add Terracotta, you get 500 TPS from the same Glassfish instance
Case1. No Thread.Sleep(1000) for each service no waiting one second.
Then the result is 600 tps.
Case2. If I add Thread.Sleep(1000), each request is hang 1 second.
Then the result is about 400 TPS.

3. If you add Sun's load balancer, then the system hangs and outputs no responses to your SOAP service.
Wrong,
Case1. No Thread.Sleep(1000) for each service no waiting one second.
Then the result is 600 tps./1 Glassfish server
if I used 2 Glassfish server it will be around 1200 tps.

Case2. If I add Thread.Sleep(1000), each request is hang 1 second.
Then the result is about 120 tps. and the max. concurrent connection threads is at most 256. If more than 256 threads, there will be some requests said that the server is unreachable. But still some requests will get correct response. The total throughput si about 120 tps, too.

This is the case I really need to be sloved.

Thanks.

Oh,

I get it now. Let's go to the next step:

In scenarios:

1. No Terracotta. When you don't add Thread.sleep(1000) how much CPU is used? And how does GC look? (Do you use jstat, verbose GC on the command line, or jconsole to watch GC? Which one? And how does it look?) How big is the heap, how many Full GC's occur? How long do they run? How much overall time does GC take?

2. With Terracotta, how does CPU and GC picture change? If you don't know, this is your first step. YOu must figure out how the test is utilizing your HW.

3. As for With the load balancer, clearly you are saturating the Sun load balancer because the system scales to 1200 tps (2X throughput) when there is no sleep(1000) but breaks down and starts throwing errors when the sleep is there. I guess, you could have a highly contended lock inside Terracotta and since you introduce a 2nd JVM running your app, the lock contention could start kicking in too. Are you running the Terracotta dev console? What do the speedometers look like at the bottom? Can you gather statistics during your entire run w/ load balancer and without? Export the results and attach the zip file here. Perhaps this will helps us both figure this out.

BTW, if you are not running our dev console (in $TC_HOME/bin/dev-console.sh) you should start. It can tell you everything about the CPU, the heap, and Terracotta in one dashboard view.

--Ari

ari wrote:

Oh,

Thank you.
I did not notice anything about Gargabe Collect. But I have the CPU usage.

1. No Terracotta. When you don't add Thread.sleep(1000) how much CPU is used?
For 1 Glassfish server the CPU is about 55-70%*4=270% of 400(4 core CPU.)

And how does GC look? (Do you use jstat, verbose GC on the command line, or jconsole to watch GC? Which one? And how does it look?)
Don't know the performance of GC.

How big is the heap?
Heap is 2GB for Glassfish.

how many Full GC's occur? How long do they run? How much overall time does GC take?
I did not collect GC info. I will try it tomorrow.

2. With Terracotta, how does CPU and GC picture change? If you don't know, this is your first step. YOu must figure out how the test is utilizing your HW.
I will get these result next monday. Thank you.

3. As for With the load balancer, clearly you are saturating the Sun load balancer because the system scales to 1200 tps (2X throughput) when there is no sleep(1000) but breaks down and starts throwing errors when the sleep is there. I guess, you could have a highly contended lock inside Terracotta and since you introduce a 2nd JVM running your app, the lock contention could start kicking in too. Are you running the Terracotta dev console? What do the speedometers look like at the bottom? Can you gather statistics during your entire run w/ load balancer and without? Export the results and attach the zip file here. Perhaps this will helps us both figure this out.
I will do this. But the issue occurred with only 1 Glassfish server running. If 2 Glassfish servers runnging, the problem occurred, too.
The Heap for Terracotta is 496M, is this enough?

BTW, if you are not running our dev console (in $TC_HOME/bin/dev-console.sh) you should start. It can tell you everything about the CPU, the heap, and Terracotta in one dashboard view.

I'll give you these info next Monday.

Thank you!

BTW, why are you synchronizing on a CSM? I don't think you should be doing that just to call containsKey(). Where else do you grab a synchronizer on the collection? (It is a concurrent collection like ConcurrentHashMap.)

--Ari

Oh!!

I just modified some code from your site example.
I just in case it did not lock the object.

So, I don't need put syncronize on the method or object, right?

I can modify it.

Thank you!!

FYI. When you need to scale up your soapUI test to larger concurrently running requests than can run on a single machine, please consider http://soapui.pushtotest.com

ari wrote:

BTW, why are you synchronizing on a CSM? I don't think you should be doing that just to call containsKey(). Where else do you grab a synchronizer on the collection? (It is a concurrent collection like ConcurrentHashMap.)

--Ari

Dear Ari,

I did not synchronize on CSM but on the Object's field(DeviceInfo's setXXX method).

If I did not syncronize then the Terracotta will ask me to do that just because it showed "Terracotta can not find the lock bonudary".

Any suggestion to modify my code?

Thanks!