Latest posts for the topic "How to reduce disk writes"

How to reduce disk writes

philipwu — GMT

What else can i do to reduce disk writes? I've already set the persistence mode to be temporary-swap-only in the tc-config.xml Attached is my status dump

Re:How to reduce disk writes

philipwu — GMT

attached is an image of the disk throughput

Re:How to reduce disk writes

ari — GMT

Actually, Given that I just finished the last chapter of the Terracotta book, yes I can tell you what's wrong. The book helps explains how to "visualize" your application. First: have you used the snapshot visualization tool? You gathered this nice zipfile for us, but have you taken a look at it yourself. You really must. Because here is what I see without knowing anything about your use case: 1. Your L2 has too small a heap. Thus, even in temporary-swap-only mode it cannot keep objects in RAM so it is spilling them to disk. 2. Your L2's small heap also causes lots of GC activity on that box so it is going slower than it needs to 3. You have very poor locality of reference and your objects are resident in all 3 L1's at once. You need some sort of load balancing / workload partitioning strategy or a different domain model if you are going to get more scale from your use case. 4. Your heap on your application JVMs is also too small and you are getting throttled there by GC. This is also causing your app JVM to flush data it cannot hold on to over to the Terracotta server. You need more memory on your app nodes if you want to avoid them thrashing on the network. 5. BTW, note that you are writing 12MB to the disk per sec on your TC server and writing a few MB/sec on each NIC interface on each app node. It suggests to me that you are doing lots of writing to objects too. Not sure. Anyways, I don't want to suggest that I have all the answers. I don't know your code at all. We can definitely help. But just so you understand: a) run the visualization tool that is available by pressing the "view" button while you record data or after you have a recorded stats sesssion. Answer for yourself why your app is doing so much broadcasting, faulting / flushing to and from the TC server, and why you are under such memory pressure on each node. b) temporary-swap mode still writes to disk. You can't stop us from writing to disk. It is how we keep our server from OOMEing. Since you have too little memory in your app and we have too little memory in our server, all that disk I/O is showing up to save both your JVM and ours from out of memory. Make sense? If not, consider "Talk to an Expert". Cheers, --Ari

Re:How to reduce disk writes

ari — GMT

BTW, I should point out to anyone who might read this thread that phillipwu generated the stats and graphs with Terracotta 2.6. Everything I figured out was done loading Phillip's zip-output from his performance test run into my own instance of the Terracotta admin console. This sort of thing cannot be done with Terracotta 2.5 or earlier. Sorry if I got your hopes up. Cheers, --Ari

Re:How to reduce disk writes

philipwu — GMT

Thanks Ari, that was a very good and detailed and thorough explanation! Just what i needed to hear. You mentioned a Terracotta book. Is that free? Can you provide a link to it so that I can read up on it too? I will look into giving expanding the heap size setting on all my nodes to see how it will improve things. I would definitely recommend anybody else considering using Terracotta to download 2.6+ because this Cluster Stats Recorder implemented in 2.6+ is really amazing. Provides a deeper understand of what's going on under the hood and anybody with performance tuning in mind or debugging can't go without.

Re:How to reduce disk writes

ari — GMT

Sadly, the book is not free. You can get it these two ways: 1. Amazon @ http://www.amazon.com/Definitive-Guide-Terracotta-Hibernate-Scalability/dp/1590599861/ref=pd_bbs_1?ie=UTF8&s=books&qid=1209736722&sr=8-1 2. Buying the "Talk to an Expert" service. BTW Phillip, I don't want to get your hopes up too much. I was excited that I could deduce potential performance issues from your zipfile. But I do not know your app. I really just wanted to target you at certain hotspots: 1. GC (looks like a sawtooth in your mem graphs) 2. Broadcasts / sec and Broadcasts / txn (2 broadcasts per txn) a few more things I noticed at the time (CPU is mighty idle, for example). Can't remember it all. But there is still work to be done. GC tuning might not be a magic bullet. Thanks for the kind words, BTW! Cheers, --Ari

Re:How to reduce disk writes

philipwu — GMT

Hi, I like the theory that you had put forth about spilling memory from clients over to the TC server. However, increasing the heap size on all the clients and TC server did not seem to help. Attached I've included my latest test run with TC server having 4096 Mb of RAM and each of the clients having 3096 Mb of heap. In my previous run the TC server was 1024 Mb heap and each of the clients were 256 Mb heap (or whatever the default was). I think more info about my test run is in order. I am using DMI to invoke my worker nodes to do some work. Each node effectively writes to a shared DSO adding several hundred Documents. One of these nodes, reads from the DSO as more Documents get added, asynchronously. Everytime it reads the Document, it is removed from the DSO to keep its size from exploding. The size of these Documents can be huge. I've seen 100 Documents use up about 5 Mb. In this run, 46165 Documents are being pulled in from the 3 nodes. Records are added to the DSO in batches of 20. At first I thought it was the Documents themselves being written to disk on the TC server. But I had explicitly set the temporary-swap-only as the persistence mode, so that can't be it. What else can be causing high disk throughput besides memory spilling over from the clients? Many thanks so far

Re:How to reduce disk writes

gkeim — GMT

One more thing that you might find useful is that thread dumps you take, via the AdminConsole, while statistics are being recorded are also included in the database. So, it's constructive to take cluster-wide thread dumps at critical points in the recording. These thread dumps are displayed in the SVT charts as small icons that can be double-clicked to bring up a separate window containing the thread dump text.

Re:How to reduce disk writes

ari — GMT

Phillip, What is your goal? I am confused. Do you want to avoid disk i/o altogether? Or do you want your application to go faster? By asking us to focus on disk i/o we cannot do much. I noticed in this second run, that your disk i/o is almost all gone. The application does a lot of i/o on the L2 at the beginning and at the end. Your application doesn't do a full GC till the end of the test. Your Terracotta server dropped from 100% to 50% cpu utilization in this second test run. Anyways, you did not fix all the things I suggested you fix. You only increased your heaps. Let me explain: 1. See picture 1...Your application is flushing tens of thousands of objects back to our server every second. Why? The application doesn't have memory pressure yet it is dumping objects back. This is slowing you down for sure. You are likely doing this because you don't have the client part of Teracotta tuned properly. You might have set prefetch really high, for example, and are not touching all the document data or something. Need your tc-config and java properties and tc properties, etc. Can't tell... 2. Your tc server is doing 100's of broadcasts per sec. This is very inefficient. How are you distributing load and documents amongst your JVMs? Clearly they are sharing objects, but did you intend for this to happen? 3. You are sending 100's of objects from our server to your JVM per second. Again, why is this happening if your JVMs have so much memory? Are your documents all loaded at the beginning of the test? The cluster does lots of work at the beginning and end of the test. What's going on? I return to my question: do you want to simply stop disk I/O on the tc server (which is mostly gone in this 2nd run, anyway) or do you want to speed up performance? Your slow performance is not due to disk i/o. The TC server is rarely waiting for disk i/o according to your test output. Your test is slow because you are moving thousands of objects back and forth around the cluster. *Your JVMs are not using the entire CPU. There is a bottleneck in your app. It is most likely all the network i/o to push objects around the cluster.* You need to involve us more (tell us what your app is meant to do, tell us how you are generating load and routing requests to your cluster, and most importantly tell us what your goals are). At least that's my opinion. Would you rather get on a phone call? This back and forth is too slow and I am afraid to help you we will need to understand everything you are doing. --Ari

Re:How to reduce disk writes

ari — GMT

BTW, If you want to eliminate disk i/o altogether, I would defer to someone else at Terracotta as to whether or not that is even possible. Does anyone at TC know if we can avoid _all_ disk i/o? --Ari

Re:How to reduce disk writes

ssubbiah — GMT

Even in "temporary-swap" mode we write to disk when we are not able to fit all the objects in memory at the L2. So if you want to reduce the disk activity give L2 more memory. The problem with giving more memory is that then GCs take longer time. I dont know if this is acceptable or not for you. One thing that we found to work well for us in the past is instead give the OS a lot of memory for disk cache. Your L2 GCs faster while you are still not hitting the disk for reads. Writes will go to disk eventually though. Another trick is to run DGC (our distributed Garbage collection) more often so you avoid swapping out of garbage to disk. You GC them while they are in memory thus reliving the memory pressure. This will help only if you are creating a lot of Garbage which seems to be true for your use case. Also there are some cachemanager properties that you can tweak to keep more objects in memory. As a first step, I would suggest that you enabled the cachemanager logging in tc.properties to see how many objects are getting swapped out. Code:

l2.cachemanager.logging.enabled = true

But like Ari said, if Disk IO is not your bottleneck, reducing it will show no improvement in ur throughput.

Re:How to reduce disk writes

philipwu — GMT

HI Ari, thanks again for the reply. I'm still trying to learn how to interpret these graphs so please forgive me as I come to grips with figuring out all these stats. Ultimately, performance is my main convern. I am trying to see what kind of performance I can get with my particular use case and see how I can tune it to make it faster. I had believed that if I could tune the TC server to reduce disk writes then I would see some performance gains. What i am confused about is that you mentioned that my 2nd test run had nearly little disk IO, but looking at the disk throughput graph i was still seeing chunks of 10Mb/s occuring on L2. How was is that you came to this conclusion? The high number of broadcasts you are seeing is done on purpose. I am effectively trying to stream documents in batches. So the node that is processing the documents has the ability to request all the nodes to send out more documents, until all 46165 documents have been sent. Since the batch size is 20, it can process those documents really fast and sends a signal to the cluster to write more Documents to the DSO, so that it can process even more documents and this goes on and on until completion. Thus, the high number of broadcasts per second. I'm already planning to do a consultation some time in the middle of May, so I am planning on bringing this topic to the table aswell.

Re:How to reduce disk writes

ari — GMT

Phillip, How are things going? Did you figure out the graphs? Let me know. I will get ona call with you and see what's going on if you like. --Ari