All of my long-form thoughts on programming, leadership, product design, and more, collected in chronological order.
Background In this series of posts, I will be going through data warehouse design. We'll start with some application tables and build out Type 2/Slowly Changing Dimensions (SCD). A lot of attention is paid to the engineering aspect of data engineering...
I wrote earlier about building data pipelines using foreign data wrappers. Some of the benefits include: Development costs: You don't need any additional infrastructure or developer resources since this is a SQL based pipeline. Velocity: This goes ha...
My recent focus has been on data warehouse design and helping organizations extract value from their data. Given the recent news surrounding Iceberg, I wanted to spend a little time exploring the open source table formats. There is a lot of news recently,...
I generally prefer to do all development in a Docker container. So, when it came time to work with Nextjs, I was looking for a way to dockerize my Nextjs app. Most of the Nextjs tutorials generally have you install node, etc. on your local machine; some of...
Setting up MWAA in the AWS console is fairly straightforward. However, what is not very obvious is how to enable email notifications for task/DAG failures. While not obvious, it is fairly straightforward. This post will show you how to set your email provi...
Background In a previous post, I talked about utilizing foreign data wrappers to move data between our production database (application/microservice) and our data warehouse. The high level architecture diagram looked something like this. We are now going...
In this post, I'll be going over how to ETL Postgres data into Redshift using minimal code and no infrastructure other than our source (RDS Postgres) and target (Redshift) databases. We will be using Foreign Data Wrappers and SQL to achieve this. Here is t...