2. Installing packages via requirements.txt: we’re shipping a UX that will make it easier to manage and install these packages without going through so many steps.
3. Circular reference: fixing today.
4. Spark configuration error messages: we’re adding a better alert message instead of just showing a scary error message.
5. Install Spark locally by default: we’ll add this to the Docker image so you can use Spark locally when running Mage using Docker.
6. Improve scratchpad UX: we’ll add in the UI a disclaimer for the user notifying them about using scratchpads only for throw away code.
7. Data validation messaging: sorry this sounded misleading, we’ll fix the product messaging around this to be more clear.
8. Integrations: there are companies running pipelines in production using many of the integrations you mentioned in the article (e.g. Spark, BigQuery, Snowflake, DBT, etc).
9. Document best practices: we just shipped a brand new documentation UI/UX (https://docs.mage.ai/); we care a ton about documentation. We’ll work on adding an entire section for documenting data engineering best-practices.
Excellent review.
That means a lot coming from you, thanks.
Interesting stuff! Going to have a go at it
🚀🧙♀️
1. Issues installing via pip: fixing today.
2. Installing packages via requirements.txt: we’re shipping a UX that will make it easier to manage and install these packages without going through so many steps.
3. Circular reference: fixing today.
4. Spark configuration error messages: we’re adding a better alert message instead of just showing a scary error message.
5. Install Spark locally by default: we’ll add this to the Docker image so you can use Spark locally when running Mage using Docker.
6. Improve scratchpad UX: we’ll add in the UI a disclaimer for the user notifying them about using scratchpads only for throw away code.
7. Data validation messaging: sorry this sounded misleading, we’ll fix the product messaging around this to be more clear.
8. Integrations: there are companies running pipelines in production using many of the integrations you mentioned in the article (e.g. Spark, BigQuery, Snowflake, DBT, etc).
9. Document best practices: we just shipped a brand new documentation UI/UX (https://docs.mage.ai/); we care a ton about documentation. We’ll work on adding an entire section for documenting data engineering best-practices.
Thank you so much for writing this and sharing it!