The first step
Working mostly on Data Engineering, Machine Learning and Bayesian analytics.
Search for a command to run...
Working mostly on Data Engineering, Machine Learning and Bayesian analytics.
No comments yet. Be the first to comment.
Dataform pipeline code that is fully typed with unit tests

Ensure your BigQuery tables have reliable data with Dataform

Building data pipelines in Dataform

Sometimes it so happens that we create or are using a table with data that is non-partitioned but we need to convert this into a partitioned table. A typical use-case is old tables that start accumulate data over time. Quite often, we need the same d...

Cloud Run helps you deploy containerized workloads at scale. Using it as a backend for web server use cases is quite well known, but it also can be very useful for large batch jobs that tend to be CPU heavy, especially if the job can be divided into ...

Several reasons, some practical and others personal. The more pratical reasons revolve around why having a blog for software engineers might be a good idea. The "ghost who codes" is a nice take on this line of reasoning. The other one is that it is quite useful to maintain a "commit log" of your thoughts if you will, and to be able to refer back to it if needed. Other practical reasons I've found center around the benefits of journalling. Though I think this blog will probably be technical in nature, I have a feeling forcing your thoughts into the written form will encourage the ability to communicate better. And good communication, I believe, is one of the most important skills to have for an engineer working in a team.
As for personal reasons, being able to share knowledge and know that it could perhaps be useful to others is a great feeling. I've been fortunate to do this in a professional capacity once before at Crio.do and certainly treasure those memories. And lastly, well to put it bluntly, writing a public blog is uncomfortable territory for me. This is my fourth attempt at trying to start a blog, the other attempts having fallen to "char log kya kahenge" syndrome. Every time I've had the feeling of wanting to publcily share something cool that I've come across or done, inevitably I would find myself convincing me out of it. Heck, as I'm writing this I'm imagining deleting everything and going back to youtube xD Note to future me - don't delete this blog!
Well, with that out of the way, let's dive into it. The first post is about the parquet format. I hope you find it useful.