data skipping algorithm works without Databricks

Question

Delta log stats are not captured in non Databricks environment. Can data skipping algorithm works without Databricks ?

Answer 1

Hello @M, Mathan ,

Thanks for the question an using MS Q&A platform.

Z-Ordering is a technique to co-locate related information in the same set of files. This co-locality is automatically used by Delta Lake on Azure Databricks data-skipping algorithms to dramatically reduce the amount of data that needs to be read.

For more details, refer to Z-Ordering (multi-dimensional clustering).

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators