My understanding was that the disk persistence would be used when the off heap becomes full. I was expecting the same performance for the insertion of 1.5million records. However, it took, ~7 minutes!!
It completely baffles me... Can someone explain or point me in the right direction?
The disk persistence settings do not work in that way. If you run with localTempSwap on it will put everything to disk initially. The offheap and heap will act as read caches for that disk layer. This is why you don't see the write throughput you expect.