Terracotta Discussion Forums (LEGACY READ-ONLY ARCHIVE)

We are considering to use distributed cache to speed up write to file repository. Most of our files are 15KB range. But few of the files will be at 2-3GB size.

Does ehcache support caching such large file?

We don't really test with files that big so I don't know for sure but... I've heard of a number of people fragmenting large entries into smaller entries and wrapping them in a stream like class for easily getting them in and out of the cache. Does that make sense?
I could give more detail if I'm being too abstract.

I am new in ehcache so did not get it quite. Can you store stream in ehcache? Could you give more details?

Off the top of my head I might create a class called BigFileStorageManager that is created with a reference to a cache as it's parameter. The class would have 2 methods called

store(Key, BigFileStream)
BigFileStream retrieve(Key)

What it would do is break your multi-gig file into smaller chunks and give them specialized key names derived from the original (i.e. myKey-1of100, myKey-2of100 etc). The stream would be an abstraction that as bytes are requested from it pulls the section of the file that is needed from the cache and returns then proper portion without the user of the stream knowing it's happening (basically making the stored entry look like one big stream.

Does that help?
Cheers,
Steve

Thanks much Steve! I got it now. What's the recomended size of of the smaller chunks?

I would say start small, maybe 10k-100k and then keep upping it until your not seeing the performance/latencies you want.
Cheers,
Steve