while running search query on a distributed EHCache (say CACHE) , sometimes getting below xception. Here Keys will be like 123_sdfsdfksdfsdf .And Query is
java.lang.NullPointerException
at com.tc.object.ClientObjectManagerImpl.lookupExistingObjectID(ClientObjectManagerImpl.java:415)
at com.tc.object.TCObjectServerMapImpl.updateLocalCacheIfNecessary(TCObjectServerMapImpl.java:301)
at com.tc.object.TCObjectServerMapImpl.getAllValuesForKeyFromServer(TCObjectServerMapImpl.java:520)
at com.tc.object.TCObjectServerMapImpl.getAllValuesUnlocked(TCObjectServerMapImpl.java:356)
at com.terracotta.toolkit.collections.ConcurrentDistributedServerMapDsoArray.unlockedGetAll(ConcurrentDistributedServerMapDsoArray.java:210)
at org.terracotta.collections.ConcurrentDistributedMap.unlockedGetAll(ConcurrentDistributedMap.java:231)
at org.terracotta.cache.TerracottaDistributedCache.getNonExpiredEntries(TerracottaDistributedCache.java:284)
at org.terracotta.cache.TerracottaDistributedCache.getNonExpiredEntriesUnlocked(TerracottaDistributedCache.java:165)
at org.terracotta.cache.TerracottaDistributedCache.unlockedGetAllTimestampedValue(TerracottaDistributedCache.java:659)
at org.terracotta.modules.ehcache.search.ClusteredResultsList.faultBatch(ClusteredResultsList.java:85)
at org.terracotta.modules.ehcache.search.ClusteredResultsList.faultIfNecessary(ClusteredResultsList.java:70)
at org.terracotta.modules.ehcache.search.ClusteredResultsList.indexAccessed(ClusteredResultsList.java:51)
at org.terracotta.modules.ehcache.search.ClusteredResultsList.access$000(ClusteredResultsList.java:18)
at org.terracotta.modules.ehcache.search.ClusteredResultsList$Itr.next(ClusteredResultsList.java:208)
at org.terracotta.modules.ehcache.search.ClusteredResultsList$Itr.next(ClusteredResultsList.java:190)
at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1010)
at com.subscription.impl.SearchUtility.executeAndGetSessionKeys(SearchUtility.java:61)
at com.subscription.impl.ServiceIdUserSubscriptionImpl.getMyPartitionUsers(ServiceIdUserSubscriptionImpl.java:95)
at com.jmx.ServiceMonitor.getPartitionedUserCount(ServiceMonitor.java:23)
at com.tasks.cleanup.ServiceHealthChecker.logCacheStats(ServiceHealthChecker.java:146)
at com.tasks.cleanup.ServiceHealthChecker.run(ServiceHealthChecker.java:87)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Also, can you post the ehcache.xml you're using? Specifically, I want to check if the cache being searched is strong consistency. I believe this corresponds to a known issue that's been fixed with terracotta 3.7.3 and above, just want to confirm the details to make sure.
The settings xml seems consistent with the known issue. Can you confirm the version of terracotta and ehcache you're using and/or retry with a later version to see if the fix works for you?