Combine Dataform with Typescript for Improved Workflows
Dataform pipeline code that is fully typed with unit tests
Latest articles
Writing data quality tests in Dataform
Ensure your BigQuery tables have reliable data with Dataform
A Practical Introduction to Google Cloud Dataform
Building data pipelines in Dataform
Partition and cluster an existing BigQuery table
Sometimes it so happens that we create or are using a table with data that is non-partitioned but we need to convert this into a partitioned table. A typical use-case is old tables that start accumulate data over time. Quite often, we need the same d...
Using Google Cloud Run to process batch jobs
Cloud Run helps you deploy containerized workloads at scale. Using it as a backend for web server use cases is quite well known, but it also can be very useful for large batch jobs that tend to be CPU heavy, especially if the job can be divided into ...
Getting answers from data using PySpark
Concise list of ways to process nested data
Parquet format - A deep dive : Part 4
Demystifying how Parquet works internally
Parquet format - A deep dive : Part 3
Demystifying how Parquet works internally
Automatically add partitions in Athena
Using S3 and Athena is great for data storage and retrieval using queries. But when I first started using it, one common problem that came up fairly quickly is: How can I add new partitions automatically? The issue was this: Partitioned data was gett...