Microsoft Open-Sources Haystack - An Indexing Subsystem for Apache Spark
cloudblogs.microsoft.com
cloudblogs.microsoft.com
- Build indexes on your data (e.g., CSV, JSON, Parquet).
- Maintain the indexes through a multi-user concurrency model.
- Leverage these indexes automatically, within your Spark workloads, without any changes to your application code for query/workload acceleration.
GitHub: https://github.com/microsoft/hyperspace Spark + AI Summit 2020 presentation: https://databricks.com/session_na20/hyperspace-an-indexing-s...