Combine Dataform with Typescript for Improved Workflows
Dataform pipeline code that is fully typed with unit tests
Jan 29, 20258 min read
Command Palette
Search for a command to run...
Latest articles
Writing data quality tests in Dataform
Ensure your BigQuery tables have reliable data with Dataform
Nov 24, 20248 min read
A Practical Introduction to Google Cloud Dataform
Building data pipelines in Dataform
Nov 4, 20249 min read
Partition and cluster an existing BigQuery table
Sometimes it so happens that we create or are using a table with data that is non-partitioned but we need to convert this into a partitioned table. A typical use-case is old tables that start accumulate data over time. Quite often, we need the same d...
Oct 17, 20242 min read
Using Google Cloud Run to process batch jobs
Cloud Run helps you deploy containerized workloads at scale. Using it as a backend for web server use cases is quite well known, but it also can be very useful for large batch jobs that tend to be CPU heavy, especially if the job can be divided into ...
Mar 24, 20246 min read
Getting answers from data using PySpark
Concise list of ways to process nested data
Oct 3, 202312 min read
Parquet format - A deep dive : Part 4
Demystifying how Parquet works internally
Jul 29, 20233 min read
Parquet format - A deep dive : Part 3
Demystifying how Parquet works internally
Jul 23, 20235 min readAutomatically add partitions in Athena
Using S3 and Athena is great for data storage and retrieval using queries. But when I first started using it, one common problem that came up fairly quickly is: How can I add new partitions automatically? The issue was this: Partitioned data was gett...
Dec 18, 20225 min read