weep moan, wail all ye Databricks and Snowflake worshipers
Excellent breakdown
Creating namespace was successful...
spark.sql(" CREATE NAMESPACE IF NOT EXISTS s3tablesbucket.test_namespace")
But when I create table...
spark.sql(" CREATE TABLE IF NOT EXISTS s3tablesbucket.test_namespace.test_table( id INT, name STRING, value INT ) USING iceberg")
getting below error...
py4j.protocol.Py4JJavaError: An error occurred while calling o34.sql.
: java.lang.NoClassDefFoundError: software/amazon/awssdk/services/s3/model/S3Exception
at software.amazon.s3tables.iceberg.S3TablesCatalogOperations.initializeFileIO(S3TablesCatalogOperations.java:111)
Could u pls suggest?
Did you find any solution to it?
I’m wondering what is actually S3 Table, a new S3 optimized towards Iceberg?
So far I could use normal S3 with Glue Data Catalog and EMR. What will be the gain in case of S3 tables, speedup?
I need to play with it a bit
Hi, thank you for writing such a great blog.
You mention "Who’s going to use Databricks and write to a S3 Table? No one.".
But what do you think about consuming from a S3 Table with Databrick or Snowflake? Is that possible?
Think about it. They added EMR support to Sagemaker too.
Very disappointing that they chose to release only Spark as the supported engine, and all the functionality is basically abstracted behind a .jar :(
Integrating query engines like Daft is going to be a pain.
How dissapointing, just as you say, aws being aws
This is avery good writeup
Thanks for the great write-up. We are heavily using AWS tools and were in the middle of moving to use Iceberg, so this article comes in just at the right time.
Excellent breakdown
Creating namespace was successful...
spark.sql(" CREATE NAMESPACE IF NOT EXISTS s3tablesbucket.test_namespace")
But when I create table...
spark.sql(" CREATE TABLE IF NOT EXISTS s3tablesbucket.test_namespace.test_table( id INT, name STRING, value INT ) USING iceberg")
getting below error...
py4j.protocol.Py4JJavaError: An error occurred while calling o34.sql.
: java.lang.NoClassDefFoundError: software/amazon/awssdk/services/s3/model/S3Exception
at software.amazon.s3tables.iceberg.S3TablesCatalogOperations.initializeFileIO(S3TablesCatalogOperations.java:111)
Could u pls suggest?
Did you find any solution to it?
I’m wondering what is actually S3 Table, a new S3 optimized towards Iceberg?
So far I could use normal S3 with Glue Data Catalog and EMR. What will be the gain in case of S3 tables, speedup?
I need to play with it a bit
Hi, thank you for writing such a great blog.
You mention "Who’s going to use Databricks and write to a S3 Table? No one.".
But what do you think about consuming from a S3 Table with Databrick or Snowflake? Is that possible?
Think about it. They added EMR support to Sagemaker too.
Very disappointing that they chose to release only Spark as the supported engine, and all the functionality is basically abstracted behind a .jar :(
Integrating query engines like Daft is going to be a pain.
How dissapointing, just as you say, aws being aws
This is avery good writeup
Thanks for the great write-up. We are heavily using AWS tools and were in the middle of moving to use Iceberg, so this article comes in just at the right time.