I think this is a bit dramatic. Deletion vectors are a feature of the open sourced toolchain. Just the bit that is open source is well, Delta Lake, the Java/Scala library. You can still use that to resolve issues and launch your own Spark system as you please. The only stop-gap here is the Delta Lake support for Delta-RS hasn't made it over the line yet. All of the tools you mentioned here depend on the kernel implementing that. So it's not that you're experiencing vendor lock-in, on the contrary it's just that the open source kids on the block are far behind, and there's not that many resources being dedicated to delta-rs if I am honest. Still open source. Still a good duck.
Some problems are not purely technical, some problems are esoteric, this being one of them. Mark my words, this is the beginning of many such cases that will continue to cause more and more havoc and only prove to be bad for Delta Lake and Databricks in the long run.
Good staff. I recently used duckb with databricks to save a few bucks everyday and it works just fine
Send me a message, would love for you to share a guest post with your experience
Duckdb can read DeletionVector.
Stop telling me to use tools I don't want to use. I won't do it.
I think this is a bit dramatic. Deletion vectors are a feature of the open sourced toolchain. Just the bit that is open source is well, Delta Lake, the Java/Scala library. You can still use that to resolve issues and launch your own Spark system as you please. The only stop-gap here is the Delta Lake support for Delta-RS hasn't made it over the line yet. All of the tools you mentioned here depend on the kernel implementing that. So it's not that you're experiencing vendor lock-in, on the contrary it's just that the open source kids on the block are far behind, and there's not that many resources being dedicated to delta-rs if I am honest. Still open source. Still a good duck.
Some problems are not purely technical, some problems are esoteric, this being one of them. Mark my words, this is the beginning of many such cases that will continue to cause more and more havoc and only prove to be bad for Delta Lake and Databricks in the long run.