[Logo] Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)
  [Search] Search   [Recent Topics] Recent Topics   [Members]  Member Listing   [Groups] Back to home page 
[Register] Register / 
[Login] Login 
[Expert]
Write Behind for Big file (2GB)  XML
Forum Index -> Ehcache
Author Message
babusona

neo

Joined: 10/23/2012 17:13:55
Messages: 3
Offline

We are considering to use distributed cache to speed up write to file repository. Most of our files are 15KB range. But few of the files will be at 2-3GB size.

Does ehcache support caching such large file?
steve

ophanim

Joined: 05/24/2006 14:22:53
Messages: 619
Offline

We don't really test with files that big so I don't know for sure but... I've heard of a number of people fragmenting large entries into smaller entries and wrapping them in a stream like class for easily getting them in and out of the cache. Does that make sense?
I could give more detail if I'm being too abstract.

Want to post to this forum? Join the Terracotta Community
babusona

neo

Joined: 10/23/2012 17:13:55
Messages: 3
Offline

I am new in ehcache so did not get it quite. Can you store stream in ehcache? Could you give more details?
steve

ophanim

Joined: 05/24/2006 14:22:53
Messages: 619
Offline

Off the top of my head I might create a class called BigFileStorageManager that is created with a reference to a cache as it's parameter. The class would have 2 methods called

store(Key, BigFileStream)
BigFileStream retrieve(Key)

What it would do is break your multi-gig file into smaller chunks and give them specialized key names derived from the original (i.e. myKey-1of100, myKey-2of100 etc). The stream would be an abstraction that as bytes are requested from it pulls the section of the file that is needed from the cache and returns then proper portion without the user of the stream knowing it's happening (basically making the stored entry look like one big stream.

Does that help?
Cheers,
Steve

Want to post to this forum? Join the Terracotta Community
babusona

neo

Joined: 10/23/2012 17:13:55
Messages: 3
Offline

Thanks much Steve! I got it now. What's the recomended size of of the smaller chunks?
steve

ophanim

Joined: 05/24/2006 14:22:53
Messages: 619
Offline

I would say start small, maybe 10k-100k and then keep upping it until your not seeing the performance/latencies you want.
Cheers,
Steve

Want to post to this forum? Join the Terracotta Community
 
Forum Index -> Ehcache
Go to:   
Powered by JForum 2.1.7 © JForum Team