What indicates that a cached table is not performing optimally under Spark's MEMORY_ONLY storage level?

When evaluating whether a cached table is performing optimally under Spark's MEMORY_ONLY storage level, an indication that the setup may not be efficient is when the size on disk is greater than zero. This circumstance suggests that, instead of utilizing memory exclusively for caching the data as intended by the MEMORY_ONLY storage level, some portion of the data is spilling over to disk. This can occur if there's not enough memory available to hold the entire dataset, which means Spark has resorted to writing some of the data to disk, thus reducing the performance benefits of caching.

Ideally, under the MEMORY_ONLY setting, all data should fit in memory without any need to store it on disk. If any data is found on disk, this is a clear sign that performance could be lacking, primarily because accessing data from disk is significantly slower than accessing it from memory. Therefore, when the size on disk exceeds zero, it indicates a suboptimal caching scenario where the expected performance advantages of using in-memory caching are compromised.