Yes, I thought to myself, how do I ensure that? This was a question posed recently to me and I realized I don’t quite have an answer. And I really didn’t want to use Terraform as the solution. We could use Javascript with Dataform and use objects that are passed around. That would certainly help.

But Typescript is just oh so much nicer. Is there a way to use it?

On first thought, it seemed quite simple. Typescript compiles to Javascript. Dataform has a package.json so clearly it supports node modules. I can probably just install anything I want, and then simply use the compiled Javascript files for my workflow. Time to wrap this one up no? Not so fast.

Turns out there are a host of problems and I will go through some the things I bumped into in this little experiment. But if you want to skip all of that and just get the code, here’s a preview into what the final thing looks like.

A peek into the final result

As you can see, there are typescript files under src folder, javascript under the build folder and finally the dataform files under definitions. This allows us to clean typed definitions, chain functions with autocomplete, get type hints, the whole deal (well, almost as you will soon see).

The repository with code is here: https://github.com/raghuveer-s/combine-dataform-with-typescript/tree/main

Now let’s breakdown this journey.

Problems and solutions

Missing type definitions

The very first thing you might look for is type definitions when working with a greenfield Typescript project. But unfortunately, there is no convenient @types/dataform provided by Google in npm. If you read the Dataform docs on using Javascript or look at the dataform core reference (https://cloud.google.com/dataform/docs/reference/dataform-core-reference), its clear we can use types in the Typescript code. It also does exist to some degree in the source code in the core package (https://github.com/dataform-co/dataform/tree/main/core) and the protobuf files.

First problem solved, we can use a mix of the documentation and the source to extract the type definitions we need: https://github.com/raghuveer-s/combine-dataform-with-typescript/blob/main/src/global.d.ts

Node modules are supported.. but not really?

Dataform documentation mentions that you can install packages through package.json (https://cloud.google.com/dataform/docs/install-package#install-package). This sounds great. Node modules embedded along with ETL transformations? Sign up me up so I says!

But unfortunately, this was yet another road bump. YMMV greatly here.

When I was experimented with installing even the simplest of modules, I ran into compilation issues. Here’s the usecase I used (building off the brewery example as always, refer here: https://raghuveer.me/a-practical-introduction-to-google-cloud-dataform):

• The daily location sales has a column which is the total sales in INR (Indian Rupees).

• So, how about I enrich this by getting the latest exchange rate and have the total sales in USD (US Dollar)?

A little contrived, but I think it makes sense in the context of sales reports.

To do this, I downloaded a really small module called currency-exchange-rates (https://www.npmjs.com/package/currencies-exchange-rates) and attempted using it, but even before I could check if the module’s currency conversion still works, I kept running into compilation issues. A couple of things I discovered while debugging:

• Dataform only uses CommonJS.

• Overcoming this hurdle through bundlers is still a no go, as there seems to be a strict subset of modules that are allowed in the end.

Going down the rabbit hole

What struck me as odd is I tried it with the module mentioned in the documentation (postoffice, for which the source doesn’t seem to exist on Github) but even then it resulted in compilation error. Iirc, the error was a message which said it failed to find node:url. This was interesting, if this is the inbuilt url package and Dataform complains with compilation errors, then I have to ask if dependencies are severely restricted in some way. I half-confirmed this hypothesis by installing is-buffer library which has no dependencies and no compilation errors surfaced.

Needing more information, I decided to check things locally using dataform compile first. I tried bundling everything into bundle.js file wondering if that might help, but I kept running into VMError.

Browsing the source, what I can gather is that Dataform uses vm2 (https://github.com/patriksimek/vm2) as a way to execute untrusted Javascript code. This means code and modules are executed in an isolated context, and the support for even for built in modules maybe restricted, let alone support for any module from npm. This constraint also explains the need to use CommonJS because vm2 is built on top of node vm (https://nodejs.org/api/vm.html).

So it would seem the only way to avoid the problem is.. by not using most modules from npm in the core code. Which is where I stand right now. All the modules I’ve used in the Typescript repo so far are for testing, lint, building etc.

Honestly, this one’s a bummer. I do hope I’m in the wrong here and I just misunderstood the documentation somehow. Or if someone can suggest a workaround, that would be awesome. Technically I guess you could this by stitching together a pipeline with Workflows, Cloud Functions and so on.

I really hope Dataform can do this in the future. In my opinion, there is a strong case for working with node modules and enriching workflows directly in Dataform. But for now, we soldier on.

Global scope

includes/ folder can have Javascript files with constants and utility functions (https://cloud.google.com/dataform/docs/reuse-code-includes), but using them with module.exports as mentioned in the docs makes them available in the global scope.

In other words: Using them from other files does not strictly need a require(). Referencing the file name allows you to use the exported constant or function. You can see an example of this here: https://github.com/GoogleCloudPlatform/bigquery-utils/blob/09082672f8a35037f79a54d166cf17fca7792c6d/dataform/examples/dataform_assertion_unit_test/definitions/tests/test_date_assertions.js

I find this a bit confusing. I would rather use the require() instead to reference modules that have Javascript code. Turns out this is works just as a regular require() where we specify the path of the module.

This is both interesting and important to keep in mind.

Folder structure

Dataform has a strict folder structure requirement. It needs definitions/, and after that includes/ for any utility functions.

And now this one’s one me, but I don’t like my typescript code’s folder structure being coerced like this, I want to have my tables in a tables folder, my ETL in a transformations folder, my tests in a tests folder and so on. Which means I need to map my folder structure to what Dataform wants and modify import paths if necessary to have it respect Dataform’s requirements in the final .js files.

Any good build tool and some scripts should be able take care of this one.

Path referencing and manipulation

Now we get to the glue that makes all of this work.

To smoothly interoperate between Typescript land and compiled Javascript, we can use tsconfig.json. Depending on what you are trying to do, there are a few options available:

• paths config in tsconfig.json to remap imports to the correct location. I used this to remap all paths with a prefix of @includes/ to resolve to modules in the includes/ directory.

• Use declare to declare the variable or function to the Typescript compiler. This is useful, for example, to make functions accessible globally with any imports. It can be used when path manipulation is tricky.

• And lastly, you have typeRoots which is useful if you want to add more organization to your code or you have more than one type definition file. For example, you could create a @types folder for your type definitions and add typeRoots in tsconfig.json with the folders that are interesting to you.

I used paths configuration to help with the module resolution. You could use a mix of the above to varying effect.

The build tool I used is rollup (https://rollupjs.org/guide/en/) with the replace() plugin to transform the @includes paths into ./includes and wrote a tiny script to move the files I want from the build output into a structure Dataform finds acceptable. Depending on how you structure your project, this can technically mean, that the files have (or may have) incorrect paths in the build directory. For the final code in this repository, this is infact what happens. It can be corrected, but my reasoning was that I’m never going to use the artifacts in the build/ directory apart from being a temporary location (and I got lazy) to copy over from into Dataform’s folder structure so I cheated a little and let it be.

This is the most important section in this experiment. Depending on how you organize your project, your tsconfig.json and rollup.config.json will be the files that need tweaking.

Building the code

With most of the heavy lifting being done with tsconfig.json, rollup.config.json, the code can be built as a regular Typescript project.

You could also chain dataform compile to this and check locally if the compilation is valid! This way a lot of the work is shift left, we spend less time going online and checking if things work in the Dataform Workspace code editor.

Now coming to the original question of how do you ensure BigQuery tables created have the same configuration. As a first step, we can create a “BigQuery Incremental Table Config” and ensure that it has the properties that you want (partition, clustering spec etc etc) and use this in the publish() method.

Next, we could even think of enforcing rules for publish() methods, for “daily sales data”, to get called only with this incremental table config. I certainly think this can tweaked further to harden the transformation logic and make it resilient to regressions or errors.

The key take away is that we now have programmatic way of checking what we are doing with our data transforms.

Putting it all together

The source code as mentioned is here: https://github.com/raghuveer-s/combine-dataform-with-typescript/tree/main.

The end result is we have our Dataform workflows in full typed code with unit tests, which in my opinion does make it much more maintainable. It’s a proof of concept and is a bit rough around the edges, but I intend to use it as a base and experiment with it in future projects. Let me know if it was useful!

Dataform tries to tackle this problem through "Assertions". And since it is Dataform, you can validate data with SQL or Javascript.

Jump to the code here: https://github.com/raghuveer-s/brewery-dataform/tree/assertions-data-quality/definitions

Wait. Assertions? You mean like unit tests?

Kinda, sorta. Unit tests in SQL do exist, but that is different from tests for Data Quality.

For SQL unit tests, we fix the inputs, apply the transformation, and set expectations on what the output should look like. For the most part, it tests the actual transformation. Which is very similar in principle to unit tests in code, where we want to test the behavior of the function.

However, data quality tests set expectations about the nature of the data itself. For example, let’s say in a table called user_first_visit, you’d expect every row to be a user and none of those users are repeated, after all you can have more than one first visit. You’d also expect user id never to be null. We’re not talking about the output of the data transformation, but rather the nature of the output of the data transformation. In general, this probably will be closely tied in with the expectations that the consumer of the data has.

A key point where the difference between SQL unit tests and data quality tests comes into focus is that, data quality tests can break without you ever changing data pipelines. As in, the tests are not strictly Hermetic. For example, let’s say you have a validation on a user’s mobile phone column which checks for number of digits to ensure a valid mobile number. If your company expands into different geographies, it may happen that this particular validation breaks.

In short, the idea behind good data quality is that we capture the “correctness” of the data itself.

Assertions in Dataform

Once again we’ll use the brewery data seen in the Introduction to Dataform post. In the brewery data, we have the daily sales data job which runs every day. The table has just 3 columns : location, daily sales and date. If I imagine myself as a brewery owner for a moment, what I would expect of my daily sales data is:

• The daily sales data must be generated for every location, which makes location a unique value in the context of that day.

• Date must make sense. I.e, if I was looking at yesterday’s daily sales as a report, it must only have data with yesterday’s date.

• Daily sales of each location is > 0.

This is the “business expectations” part of data quality. We talk about the first two in the inbuilt assertions section.

The last one is interesting. Sales of a location “must be greater than 0”? What if there were no sales at that location? Then technically it’s not a “data” issue right? Well.. maybe. Maybe the data pipeline is indeed broken, but maybe something happened at the data source, maybe the internet was down, maybe this maybe that. The only thing we know for a fact is: It’s weird if sales is 0 all of a sudden. Strictly speaking, this falls under data anomaly detection. And the cool thing about Dataform is that it integrates very well with BigQuery. And BigQuery has anomaly detection out of the box (https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-detect-anomalies). We’ll touch upon this in the manual assertions section.

Inbuilt assertions

Data quality checks on the generated data is very useful, the earlier we know about quality issues, the better. The inbuilt assertions can be set in the config block, which means the same SQLX you use for table / incremental / view workflow objects just need a bit of tweaking.

Out of the box, Dataform offers:

• Null checks

• Uniqueness checks

• Custom row level conditions

For us this, this looks like:

config {
  type: "incremental",
  // .. the other configurations
  // Note: Remember the partitions filtering caveat mentioned below
  assertions: {
    nonNull: ["Location"],
    uniqueKey: ["Location"],
    rowConditions: [
      'd < TIMESTAMP_TRUNC(CURRENT_TIMESTAMP(), DAY)'
    ]
  }
}

SELECT
  Location as location, 
  TIMESTAMP_TRUNC(Brew_Date, DAY) d, 
  SUM(Total_Sales) AS daily_location_sales
FROM
  ${ref("brewery_partitioned_clustered")}
WHERE
  TIMESTAMP_TRUNC(Brew_Date, DAY) >= timestamp_checkpoint
GROUP BY
  location, d

pre_operations {
  DECLARE timestamp_checkpoint DEFAULT (
    ${when(incremental(),
    `SELECT MAX(d) FROM ${self()} WHERE d is not null`,
    `SELECT TIMESTAMP("2023-01-01")`)}
  )
}

Note: If partition filters are mandated using requirePartitionFilter in the config block, the inbuilt assertions will not work. This is because internally, Dataform uses Views to implement assertions. And since a View is basically just a query, it requires the partition filter to be specified, and Dataform has no way to know ahead of time which partition you want for the View.

For example, the uniqueness constraint above gets converted into this query by Dataform:

Manual assertions

This is my preferred way of writing assertions because I like the idea of separating my code which does the assertions and the code that does the data transformations. In this style, assertions are just another Action in Dataform. This means you can utilize the full power of BigQuery SQL. And since it’s just another Action, you can have independent tags for assertions, dependencies become much more readable in my opinion, and even specify assertions in a different release configuration with its own schedule if needed.

Since we’re looking at anomaly detection, we will use BigQuery’s machine learning capabilities. Let’s keep it simple and use k-means.

Let’s create the model in BigQuery first.

CREATE OR REPLACE 
MODEL `website-code-395711.brewery.sales_anomaly`
OPTIONS(
  MODEL_TYPE='kmeans',
  KMEANS_INIT_METHOD='kmeans++',
  NUM_CLUSTERS=10
)
AS
SELECT
  location, daily_location_sales
FROM 
  `brewery.daily_area_sales`
WHERE 
  d IS NOT NULL;

Then, we need the anomalous data. For this, I just manually inserted daily sales data with 0 value for the location “Koramangala”. And for those of you who know the scene in Koramangala, Bengaluru, you can most definitely agree that a brewery having beer sales of 0 is definitely weird 😁

Next, run the anomaly detection as an assertion. Create a new folder called “assertions” under “definitions” folder, and then the sales_anomaly.sqlx.

config {
  type: "assertion",
  tags: ["daily", "anomaly_detection"]
}

SELECT
  *
FROM
  ML.DETECT_ANOMALIES(
    MODEL `brewery.sales_anomaly`,
    STRUCT(0.01 AS contamination),
    (SELECT location, daily_location_sales FROM `brewery.daily_area_sales` WHERE d >= TIMESTAMP('2024-01-01'))
  )
WHERE
  is_anomaly IS TRUE

If done right, you should see something this.

This is great, we can use all of BigQuery’s cool stuff directly in Dataform. But still, something is not quite right. Assertions are “binary” in some sense, that is they can say “all good” or scream “Alert!”, but anomalies fall somewhere in the middle which says “Hey, something’s odd.”. Unfortunately, this type of alert levels is not something doable out of the box in Dataform and needs more effort.

Writing Assertions in Javascript

Since Javascript is a first class citizen in Dataform, you can use it write assertions as well. It becomes extremely easy to create test assertions by applying programming patterns, for example, if we have the same test assertion to apply for different tables, we can just do so using for loops. You can also bring the benefits of having node modules into the mix. For example, I can imagine using json schema validators if you are storing event data in your tables. We’ll cover using Javascript in Dataform with some examples in a future article.

A word on Data Quality

So far, we’ve described the why and how of Data Quality tests but to a lesser extent, the what to test for. A brief overview of the so-called “six pillars of data quality” should give a bit more direction in that regard:

• Accuracy: The degree to which data about the real world object or event is captured. Eg: Filling out a tourist visa application and you put a space accidentally in your first name, leave the middle name blank, and fill the last name correctly. It is easy to imagine that this may cause problems as you try to explain yourself at the immigration counter. Closely related to precision. Eg: Event timestamps till millisecond granularity.

• Completeness: How “complete” is the data being stored. Say for user data, do we have everything that adequately describes the user for our needs? Eg: A name can be first name, last name. But you could also capture middle name, birth name (Will your system use it though?).

• Consistency: Multiple data stores must not differ in the data stored. Eg: Same userId must not have different birth dates in two different systems.

• Timeliness: Freshness of the data. Eg: Must not store events timestamped in the future, must not have “stale” data (definition of stale comes from business logic).

• Uniqueness: Unique constraints in the world must be respected in the data as well. For example, unique email ids. There are some edge cases here though. For example, Mobile numbers are unique, but they can be reassigned to a different person if the original holder decides to cancel their mobile subscription.

• Validity: Semantic or syntactic checks. Eg: Email must have “@” in it.

Conclusion

We’ll close off with a quick overview of the pros and cons.

Pros:

• If you’re using BigQuery, data quality assurance becomes extremely easy.

• Assertions are written in SQL or Javascript. SQL is the lingua franca of the data domain, Javascript brings node modules with it should you need it. Both are widely used and powerful, making it a strong choice for writing data quality checks.

• Assertion failures are just like workflow execution failures. Which means if you have observability built on Google Cloud Logging, you can easily reuse that for alerts on data quality.

Cons:

• Rows that break the assertions are accessible through a view. But since this is created on every run of the assertions, if the next assertion is run before you get the chance to view the existing assertion failures, the failing rows may be lost.

• While we can alert on data quality failures, we cannot configure different alerting levels. I’ve often found that some data quality checks are more important than others, having a way to configure this does not exist out of the box.

Overall, Dataform assertions offer a simple and flexible way to get started on ensuring the quality of your data in BigQuery. However, it’s simplicity does have some tradeoffs. It’s a great tool in the data quality toolbox, but it does have its limitations compared to more powerful tools out there.

Note: We use Dataform v3 which changes things a little from the v2 version.

Architecture

Let’s start with some key pieces that we work with in Dataform.

Going on by one:

• Dataform Repository: The place to store the pipeline code. Every Dataform project needs to have a Repository as a top level object. We can connect the Dataform Repository to a Git Repository.

• Dataform Workspace: Online editor where we can edit the code.

• SQLX: Dataform has extended SQL a little to make it easier to define pipelines, it’s dependencies and so on. Eventually it all gets compiled into SQL.

• Javascript: We can use Javascript along with SQLX. Even node modules can be installed, which is pretty cool!

• Action: The smallest executable unit. Most of the time it’s SQLX code.

• Workflow: A collection of Actions. This is the pipeline, essentially. When run, Dataform calls it “creating a workflow invocation”.

• Release configuration: The result of compiling the SQLX files in the codebase. The compilation can be parameterized by passing in variables.

• Workflow configuration: A workflow configuration is the set of Actions chosen to be executed.

• BigQuery: Provides source and destination tables.

Creating the pipeline step-by-step

A word on permissions before we begin

There are two principals who needs permissions:

1. Dataform service account (which is created automatically when repository is created in the next step).

2. The user administering dataform settings.

Refer to these documents for more:

1. https://cloud.google.com/dataform/docs/required-access

2. https://cloud.google.com/dataform/docs/connect-repository

But if you don’t want to read the docs and are in a position to grant yourself additional roles (lucky you), then add roles/bigquery.dataEditor, roles/bigquery.user, roles/secretmanager.secretAccessor to the dataform service account, roles/dataform.admin to the user and keep cracking on.

Create and connect the repository

A “Dataform repository” is a top level concept. It hosts the code that makes up the pipeline. It can be connected to a Git Repository.

To connect to a remote Git repo, just click on the “Connect with Git” and follow the UI. You will need to create a Secret and store it as well which allows you to connect securely to the Git repository. Both of these steps need adequate permissions.

Optional: Create a development workspace

The docs define a Workspace as “an editable copy of your repository”. It’s more easily understood as an online editor UI for code. It allows users to work independently directly on the UI online, commit their changes to git, execute workflows and so on. I’ve found it useful for quick prototyping and visualizing the pipeline DAG.

Local development environment

For Dataform, I use VSCode. There’s even a Dataform extension that comes in useful for a good development environment.

To get started, follow these steps:

1. Search and install the Dataform extension for VSCode. This provides syntax highlighting and on the fly compilation which can detect errors early in the development cycle.

2. Create a local folder for the code and step into it.

3. Install the dataform package through node: npm i -g @dataform/core.

4. Execute dataform init . <gcp_project_id> <bigquery_dataset_location>. Use dataform help to see what other options there are.

My dataform core version is 3.0.7 (the release notes can be tracked here: https://cloud.google.com/dataform/docs/release-notes) and as of this version, it creates three things:

1. definitions folder: Home for the .sqlx and .js files.

2. includes folder: Home for common Javascript constants or functions.

3. workflow_settings.yaml: These are default settings for the Dataform workflow. It consists of the dataform-core version being used, as well as the default project, dataset and location of the BigQuery data. These variables can be overridden. We will go in more detail a bit later.

Quick note: This step differs based on the Dataform version being used. I’m using v3.0.7, which uses yaml as part of the setup. Prior versions would have had dataform.json, package.json and node_modules in the setup. For more: TODO

Data

The dataset we’ll use is the Brewery dataset again (https://www.kaggle.com/datasets/ankurnapa/brewery-operations-and-market-analysis-dataset/). For our purposes, let’s compute the daily sales of beer grouped by area. From a pipeline perspective, this follows the classic flow: source → transform → sink executed at a daily cadence with some scheduler.

Code

We have everything we need to begin, let’s look at code. Here is the Git repository for reference: https://github.com/raghuveer-s/brewery-dataform/tree/basics-with-workflow-settings

First, let’s take a moment and look at the different pieces of a SQLX file:

1. A config block. This is a section that which defines type of workflow object, dependencies, tags, options for BigQuery table and more.

2. SQL operations for data transformations.

3. Pre and post operation blocks.

And that’s it. Now let’s map to our objective of computing daily sales.

Steps:

1. Create a declaration workflow object (Dataform defines different types of workflow objects, refer: https://cloud.google.com/dataform/docs/sql-workflows).

   1. All this does is tell Dataform where a source table is and keeps a reference to it, which means all you need is a config block with details that point towards this table.

   2. While this step is optional, I often prefer to create declarations. It helps keep the rest of the code consistent, and best of all, it can be visualized as a DAG in the workspace online.

      

   3. Example: https://github.com/raghuveer-s/brewery-dataform/blob/basics-with-workflow-settings/definitions/brewery_source.sqlx

2. Create a SQLX file:

   1. Since we are talking about data pipelines here, the result of our data transformation needs a home. We can have a table gets completely populated with this result or we can have table that has new data appended to it. The later is called “incremental” workflow object, the former is called “table”. Again, refer here: https://cloud.google.com/dataform/docs/sql-workflows

   2. In the config block, specify:

      1. Workflow object type as incremental.

      2. Tags. These will be useful when we look at “workflow configurations” in the section below.

      3. BigQuery options to the table that is going to hold the result of the transformations.

   3. What follows is the actual SQL that do the work. In this case, we just sum up the sales that happened in a day and group it by area. Pretty simple.

   4. Example: https://github.com/raghuveer-s/brewery-dataform/blob/basics-with-workflow-settings/definitions/daily_area_sales.sqlx

The incremental table code has another section as well. We talked about config block and SQL, Dataform allows optional pre_operations and post_operations blocks as well. As one might expect, the code in this is run before and after the SQL transformations.

There are three methods that can be seen in the code: when(), incremental() and self(). These are some of the so-called “context methods”. when() provides conditional capabilities, incremental() returns true if the table being built is incremental or not, and self() is used to reference the current table being built.

So taken together, what we are doing is just establishing a checkpoint to be used in the where clause, and we start appending latest data if the table exists and is incremental. If not, then we take all the data in the source starting from 2023-01-01.

For more about pre and post operations, refer here: https://cloud.google.com/dataform/docs/table-settings?hl=en#execute-sql-before-table . And for context methods, there are a few more that can be used, refer here: https://cloud.google.com/dataform/docs/reference/dataform-core-reference#itablecontext.

Commit and push. If you have created a workspace online, then you may also see this error on the right complaining that a table does not exist:

Don’t worry about it. We are creating an “incremental” table. Since we cannot increment something if it didn’t already exist, the error just highlights that. If you go to the non-incremental tab, it all looks good. After you execute this workflow for the first time, the error will disappear.

All that’s left to do is run this. We could do this by selecting the “Start Execution” button on top and following the UI:

But let’s be a bit more formal and execute the code by creating release configurations and workflow configurations.

Create a release configuration

You can create a release configuration by the UI or through an API. The API is of course much more powerful and makes a great case for using it with CI/CD but we’ll reserve that later.

Release configuration is easily understood if you think of it like a function.

release_configuration = compile(code, config).

The “release” part of a release configuration is the “compilation result” of the code. Give it a name (release Id) and set the frequency at which these compilation results are to be generated. My settings are quite simple: “production” for the release id and “never” for the frequency.

Since a release configuration is the compiled result of SQLX, if you are using the release configuration as the source for pipeline executions (aka, workflow invocation in Dataform lingo), it means you must create a new compilation whenever a change happens. This is where the API comes in handy. Otherwise, you can just set a frequency and it gets compiled. For this example, since we chose “never”, the compilation result has to be created manually every time a change in the code needs to be reflected in the actual pipeline being executed.

The “configuration” part arises from that you can pass parameters to this compilation. This is where workflow_settings.yaml comes into play again.

We don’t need to override anything in this simple example, but let’s say we have a tablePrefix variable , we could use it store compilation results for staging and production for example. You can even pass custom variables using vars, for example, something like max_lookback. Taking the two together, we could say for staging environment, max_lookback is 1 month, and for production environment max_lookback is 1 year. And then, wrap this in the pre_operations block. Variables in workflow_settings.yaml are referenced using the dataform.projectConfig object. We’ll explore this in a future post.

Create a workflow configuration

Similar to the release configuration, it is easy to mentally map a workflow configuration to a function like so:

workflow_configuration = create_workflow(release_id, config)

The “workflow” part here is just the release configuration which has the compiled code.

The “configuration” part is how often you want to run this workflow (aka, “create the workflow invocation” in Dataform lingo) and which parts of the compiled code that needs to be executed.

The second part is more interesting. A Dataform “action” is the smallest executable unit in a workflow. In this example, the only Action we have is daily_area_sales . But we can have multiple actions in a single workflow. Actions can even depend on each other. An Action can be either selected directly in a workflow configuration or through tags (this is specified in the config block, refer to the SQLX file to see it).

Taken along with the ability to parameterize a release configuration, this becomes very powerful as you can specify, configure and execute pipelines in a very flexible way.

And that’s it! Dataform will schedule the execution of this pipeline nightly and you get daily aggregated sales data in the table.

Concluding remarks

And with that, a simple pipeline in Dataform is ready. There’s still a lot to talk about of course. In the upcoming posts, we’ll go a bit further in depth on things like variables, action dependencies, data quality tests, CI/CD and observability.

I’ve created a simple helper script that comes in handy for just that: https://github.com/raghuveer-s/example-code/blob/main/partition-bq-table/partition-bq-table.sql

The script does try to preserve some table and column metadata before partitioning, cluster, and copying the data into a new table. For the most part, you should be able to just change the variable names and use it.

Some final points to consider:

• There are some assumptions baked in the script, such as the existence of common dataset where it creates the stored procedures needed.

• You probably want to change the region (I have used region-eu in the code). The script makes use of the INFORMATION_SCHEMA and this needs a qualifier (https://cloud.google.com/bigquery/docs/information-schema-intro#syntax). Note for my future self: This really should be parameterized.

• If you want to preserve the same table name for the partitioned table, add a couple of commands for changing table name with ALTER TABLE RENAME followed by DROP TABLE of the old table. Note that, changing a table name does come with some limitations in BigQuery (https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#alter_table_rename_to_statement).

The first half talks briefly about Cloud Run Jobs, and in the second half we will explore a very simple use case with sample code.

Use cases

In a nutshell: Long running jobs! Many use cases that I can think of fall neatly in the data processing or ML category:

• You have a transactional database and need a way to compute aggregations of data without overwhelming the database.

• Download and process data from a third party API.

• Preprocess data stored on GCS and you cannot use Spark (for whatever reason).

How does it work?

Cloud Run Jobs can be divided into the following components:

• Container. The actual workload to be executed.

• Job. It is a higher level construct that references the container workload to run, specifies number of Tasks (refer below), accepts parameters during job creation and so on.

• Task. This is the basic unit of parallelism run for a Cloud Run Job. A Task is an executable container instance. Cloud Run can spin up multiple tasks to parallelize your workload. Understanding this is the key to understanding how Cloud Run Jobs work.

In essence, a Cloud Run Job is a template that you fill in which is then later executed by the system. The mandatory fields are:

• Job name.

• The container image url.

• GCP region.

• Number of tasks.

• Task timeout.

• Task retries.

For a full list, it is informative to glance at the gcloud CLI command: https://cloud.google.com/sdk/gcloud/reference/run/jobs/create. You can see it accepts several other options such as container arguments, cpu and memory allocated for the container instance and so on.

Understanding Tasks

Tasks are the most important aspect of working with Cloud Run Jobs when it comes to parallel processing.

There are two key points to note:

1. Task parallelism. The total number of tasks specified may be different from the total number of Tasks being executed in parallel.

2. Task independence. Since there are many tasks being executed in parallel, a question that tends to come up is: Are all these tasks executing the same container or can we somehow get each task to execute something different?

Let's get a little deeper into these two concepts and then work through it with a very simple example following that.

Task parallelism

For the sake of explanation, let us say that we have a dataset of 100 users and we want to analyze the user records. We can specify the number of tasks to be 100, but it may be that we can specify only 10 tasks to be executed in parallel.

But why only 10 and not all 100 in parallel? Well for one, there are regional limits enforced by Cloud Run (https://cloud.google.com/run/quotas). Other reasons could be more technical, say for example, we wouldn't want to overwhelm databases downstream by slamming them with workloads from parallel tasks.

Regardless, the important thing to remember is: These two can be different numbers.

Task independence

We know that Tasks are container instances being executed in parallel. For large batch jobs or long running jobs, it would be perfect if we divide our work into smaller pieces and somehow tell each Task to execute these pieces independently, in parallel.

Cloud Run lets you know which task (which container instance) it is currently executing and the total number of tasks it has through environment variables:

• CLOUD_RUN_TASK_INDEX: The task index in this Job. With this, we can know the index of the task which is being executed.

• CLOUD_RUN_TASK_COUNT: The number of tasks in this Job that are being executed in parallel at any given time.

Note that "maximum number of tasks" is different from "maximum number of tasks running in parallel". The image above should demonstrate the difference. Referring to the resource limits page again, we see the total number of tasks spawned at once can be in the thousands range but the number of tasks being executed in parallel is in the hundreds range. The tasks must wait for their turn.

Let's now look at how we can use these two variables to retrieve some data and process it independently, we achieve our goal.

Example: Calculating daily user revenue

Here is our scenario:

• We are running an e-commerce website with a transactional db.

• We have a dataset of 100,000 users from a Users table.

• There is another table Purchases with records that reference a user whenever a user purchases something on the website.

• For simplicity, the user ids are sequential integers starting from 0.

• The coding language is Python.

Our goal: Create a job computes the total revenue from a user for that day. Store it in another table called Revenue.

Code

Note: The code is a contrived example and far from production grade code, but it is enough to demonstrate the two key pieces : CLOUD_RUN_TASK_INDEX and CLOUD_RUN_TASK_COUNT.

def compute():
    task_index = int(os.environ["CLOUD_RUN_TASK_INDEX"])
    task_count = int(os.environ["CLOUD_RUN_TASK_COUNT"])

    user = os.environ["MYSQL_USER"]
    password = os.environ["MYSQL_PASSWORD"]
    db_name = os.environ["MY_DATABASE"]

    num_users = 10000
    batch_size = num_users / task_count
    start_user_id, end_user_id = [int(batch_size * task_index), int(batch_size * (task_index + 1) - 1)]

Use task_index to retrieve a set of data points. Since the user ids are incremental in our case, we calculate the batch size based on the total number of tasks that can run in parallel at any given time, which can be used to retrieve the set of user ids to work on.

# Retrieve user ids
user_ids = session.execute(
    text(f"""
        SELECT user_id
        FROM Users
        WHERE user_id >= {start_user_id} and user_id <= {end_user_id}
    """)
)

Here we see that we're using a simple select query based on the user id range retrieved above. It should be apparent right away that this is a potential bottleneck, and as mentioned above, one way to mitigate this to some extent is to select the task count environment variable judiciously. The same idea applies when we want to compute and store the result in the database. In other words, any service that is dependent on cloud run task execution must be able to handle the incoming load.

The full code is at: https://github.com/raghuveer-s/example-code/tree/main/cloud-run-jobs

Creating the job

This step is quite intuitive with the google cloud console. You can also do the same with the CLI or SDK if it suits your workflow.

In GCP, you can use the artifact registry (or the older container registry) as a registry for your container. Refer to this link: https://cloud.google.com/artifact-registry/docs/docker/pushing-and-pulling for more on how to do this.

Towards the bottom of the screen, you can specify the total number of tasks that can execute in parallel, i.e, task count.

Once you are ready, just click the create job button and Cloud Run will do the rest.

Summary

1. Cloud Run jobs are well suited for large batch jobs. In this case we chose to do data processing, but it can be any long running divisible computation.

2. Use the task index to retrieve the subset of work we wish to perform computation on. Use task count to control number of parallel executing tasks.

3. Be mindful of services upstream and downstream if any.

4. Cloud Run has resource limits which vary by region. Refer to this page: https://cloud.google.com/run/quotas to know more about the limits on number of maximum tasks, and number of parallel tasks.

Dataset and workflow

The dataset we will use is US Financial news data hosted at Kaggle. You can find the source for it here: https://www.kaggle.com/datasets/jeet2016/us-financial-news-articles. In it, each record is a JSON separated by a newline, i.e, the so-called "JSON newlines" format (https://jsonlines.org/). Picking a record in this dataset, we can see that it this structure: https://github.com/raghuveer-s/example-code/blob/main/spark-json-processing/schema/raw_data_schema.json. This is a fairly lengthy schema, but it has some nice nested structure with fields and arrays going more than just a level down which makes it convenient to explore several use cases with it.

We will attempt to duplicate a process which is a fairly common:

1. You obtain a dataset.

2. An internal or external user wants some question answered from it.

3. You as a Data Engineer, must either answer it or facilitate getting the answer in some way.

Since we have financial news, let's try to think of some questions that we can ask from it. For the purposes of the article, the questions might be a bit contrived since we want to get a reasonable technical workout as well, but it should be more or less similar to the kind of questions real world users would like to know.

A few questions that comes to mind are:

• Give me the count of unique authors.

• What is the most popular and least popular article in a news website?

• How many times does an external domain point to the domain of a news organization?

• Give me a list of the most mentioned people.

• What is the "sentiment score" of an article?

We will proceed by grouping one or more of these questions under how we intend to process it from a technical perspective.

Processing use cases

For all the cases below, a common variable df is being used. This loads the dataset from the environment variable DATA_PATH.

df = spark.read.json(path=f"{DATA_PATH}/*.json", schema=None)

Flatten nested record to a single row

One of the most common tasks is to process data for analytical processing. Generally speaking, this means the sink probably has a flat(-ish) schema in a columnar store such as BigQuery, ClickHouse, etc.

Processing a nested value

Let's start with the first couple of questions:

• Give me the count of unique authors.

• What is the most popular and least popular article in the news website?

For this, we can attempt to flatten the schema into this:

url, domain, author, title, socials, text, published_at

Looking at the schema, to get values for some the columns above, we have to walk down the path of nested data. This can be easily done in Spark.

flattened_df = (df
    .select(
        F.col("thread.url").alias("url"),
        F.col("thread.site").alias("domain"),
        F.col("author"),
        F.col("title"),
        F.col("thread.social").alias("socials"),
        F.col("text"),
        F.to_timestamp(F.col("published")).alias("published_at")
    )
)

Now that it is flattened, we can easily count the number of unique authors.

# Count of unique authors
c = flattened_df.select("author").distinct().count()

Implementation notes:

• You can walk down the path to get the deeply nested columns.

• distinct() and count() will probably "shuffle" your data during processing, so make sure you keep that in mind.

The code is here: https://github.com/raghuveer-s/example-code/blob/main/spark-json-processing/unique_authors.py

We used only author, what other answers can we get from data with this schema? Off the top of my head:

• Get across how many domains an author gets published in.

• Use it for training summarizers since we have both title and text.

• Word clusters for articles over date published to grasp what makes headlines over time.

In a typical workflow, we will have a broad list of use cases and from that a schema will be derived.

Coming back to our original two questions, for the second question, the final schema we want might look like this:

domain, most_popular_article, most_popular_article_num_shares, least_popular_article, least_popular_article_num_shares

But what is the definition of popularity? Looking at the schema, there seems to be data on the number of times an article gets shared on social media. Let's use that as an indicator and see what comes up.

We have to break this down into multiple steps:

1. Get the total number of shares for each record.

2. Get the most popular and least popular from this.

3. Perform a join to combine into a single dataset.

Step 1

# Get the total shares for an article

social_shares_df = (df
    .select(
        F.col("thread.url").alias("url"),
        F.col("thread.site").alias("domain"),
        F.col("thread.social.gplus.shares").alias("gplus_shares"),
        F.col("thread.social.pinterest.shares").alias("pinterest_shares"),
        F.col("thread.social.vk.shares").alias("vk_shares"),
        F.col("thread.social.linkedin.shares").alias("linkedin_shares"),
        F.col("thread.social.facebook.shares").alias("facebook_shares"),
        F.col("thread.social.stumbledupon.shares").alias("stumbleupon_shares"),
    )
    .withColumn("total_social_shares", F.col("gplus_shares") + F.col("pinterest_shares") + F.col("vk_shares") + F.col("linkedin_shares") + F.col("facebook_shares") + F.col("stumbleupon_shares"))
    .drop("gplus_shares", "pinterest_shares", "vk_shares", "linkedin_shares", "facebook_shares", "stumbleupon_shares")
    .groupBy("url", "domain")
    .agg({
        "total_social_shares" : "count"
    })
    .withColumnRenamed("count(total_social_shares)", "total_shares")
)

Implementation notes:

• agg() can take either a dict or a list of columns. If it is a dict, then we specify the column that we want to aggregate on as the key, and the aggregate function as value.

• Using aggregate in this way results in a column name with the function name and the original column, this can be renamed with withColumnRenamed()

Step 2

# Get the maximum and minimum shared articles for a news website

most_popular_df = (
    social_shares_df
    .groupBy("domain")
    .agg(
        F.first("url").alias("most_popular_article"),
        F.max("total_shares").alias("most_popular_article_num_shares")
    )
)
most_popular_df.createOrReplaceGlobalTempView("most_popular_df")

least_popular_df = (
    social_shares_df
    .groupBy("domain")
    .agg(
        F.first("url").alias("least_popular_article"),
        F.min("total_shares").alias("least_popular_article_num_shares")
    )
)
least_popular_df.createOrReplaceGlobalTempView("least_popular_df")

Implementation notes:

• This time agg() uses a list of columns. Personally, I like this style a lot better. It reads better and aliasing comes right in the same line.

• createOrReplaceGlobalTempView() creates a table managed by Spark. For a concise explanation on how it works and some ideas on how to use it, refer here[1] and here[2].

• Why are we doing this? It's to demonstrate the spark.sql() API in the next step.

Step 3

final_df = spark.sql(
    """
    select * 
    from global_temp.most_popular_df t1
    left join global_temp.least_popular_df t2
    on t1.domain = t2.domain
    """
)

Implementation note:

• Do we need to use the spark.sql() API here? No. The coolest thing about Spark is that you can use the functional API or the SQL API. I like mixing and matching it depending on the use case and how it maps to my mental model of the data.

The code is here: https://github.com/raghuveer-s/example-code/blob/main/spark-json-processing/popular_article.py

And here are the final results:

Well, that was rather disappointing after all that effort. It's quite unlikely the results above represent reality which means our dataset does not have reliable data in this regard. If we want to answer this question, we probably need to look for supplemental data from other datasets, if it exists.

Luckily though, we caught this pretty early, and we didn't build even more use cases on top of this answer, on the assumption such an answer exists. This type of exploration which seeks out answerable questions, interesting patterns, outliers and so on from a dataset is called Exploratory Data Analysis. It forms the bedrock on what we can and cannot answer from our data.

Processing nested arrays

News articles are often shared on sites that link to the news websites. Let's say you are doing link analysis, a question you might want answered is:

• How many times does an external domain point to the domain of a news organization?

The final schema that we want would be something like this:

external_website, news_website, num_links

We need to modify the flat schema from the previous section and add a single extra column:

url, domain, domain_rank, author, title, external_link, text, published_at

But to get external_link, we need to get it from external_links which is an array. We need to explode this array. This works quite similarly to a JOIN. For each of the existing records, we take the record values and attach each of the values in the external_links array in turn, thereby creating new records.

flattened_df = (df
    .select(
        F.col("thread.url").alias("url"),
        F.col("thread.site").alias("domain"),
        F.col("thread.performance_score").cast("int").alias("performance_score"),
        F.col("author"),
        F.col("title"),
        F.explode("external_links").alias("external_link"),
        F.col("text"),
        F.to_timestamp(F.col("published")).alias("published_at")
    )
)

Implementation notes:

• explode() can take an array or a map. In the next section, we look at how to explode an array of structs.

From this, we need to count the number of times an external website pointed to our news website.

# Count number of times an external website points to the news website

links_df = (
    flattened_df
    .select(
        F.expr("parse_url(external_link, 'HOST')").alias("external_website"),
        F.col("domain").alias("news_website")
    )
    .groupBy("external_website", "news_website")
    .agg(F.count("*").alias("num_links"))
    .orderBy(F.desc("num_links"))
)

Implementation notes:

• F.expr() is a middle ground between the functional API and SQL API, it lets you use SQL expressions inside select statements. This is particularly convenient when you want to quickly apply SQL functions on the columns such as the parse_url() function shown above.

Here are the top 10 results:

It is pretty much what we would expect. News articles are shared on web syndication websites for news content and social media websites.

Notice though, that Thomson Reuters is repeated twice as unique websites due to the www domain, and quite clearly throws off our count. This is an important aspect of Data Cleaning which we look at briefly in the next section.

The code is here: https://github.com/raghuveer-s/example-code/blob/main/spark-json-processing/num_times_external_domain.py

Process a nested array of structs

News articles often are about or mentioned people. This is captured in the dataset in the path entities.persons[] . This array contains a struct which has the name of the person and a sentiment expressed by or about this person.

One question that might be interesting is:

• Give me a list of the most mentioned people.

The schema we want for this might look like:

person, mentioned_count

most_mentioned_df = (
    df
    .select(
        F.explode(F.col("entities.persons")).alias("person_struct")
    )
    .withColumn("person", F.col("person_struct.name"))
    .drop("person_struct")
    .groupBy("person")
    .agg(F.count("*").alias("mentioned_count"))
    .orderBy(F.desc("mentioned_count"))
)

most_mentioned_df.show(20)

The code is here: https://github.com/raghuveer-s/example-code/blob/main/spark-json-processing/most_mentioned.py

Running this, we get this output:

Clearly, there are repeated names. This can lead to incorrect counts. Should we resolve this? And if yes, how to resolve this? And to what degree?

Importance of data cleaning

Data cleaning is a whole topic on its own. As we can see from the above result, it is quite evident that before we pass on this dataset for further usage, we must clean it.

But, what exactly does it mean to clean your data? Spelling mistakes? Unwanted characters? Duplicated records? Unwanted records? Corrupt data? There are so many ways of looking at this, which naturally means there must be a corresponding great number of ways we can "clean" the data. Producing and maintaining high quality datasets is difficult and time consuming. Data Engineers must often grapple with how good is "good enough" for the use case being looked at.

Let's demonstrate a rough data cleaning procedure for this particular case. We note that:

• We are dealing with a person's name here, which generally has a first name and last name.

• Since it is the US, when there a person's name mentioned in isolation without reference to first or last name, it will most likely be the last name keeping in mind this is a news dataset which means there most likely will be a degree of formality involved. This means we can "group" together the references of a person who has the full name and only the last name.

• We can ignore spelling mistakes and unwanted characters for simplification.

The code is here: https://github.com/raghuveer-s/example-code/blob/main/spark-json-processing/most_mentioned_cleaned.py

Here are the top 20 results:

Doesn't look too bad. The duplicates problem from before is gone and the results for Donald Trump add perfectly. Let's explore a bit more. This is what we get when filtering for just "trump":

Now things get a bit murkier. But before the question "how to resolve it", we must ask "do we really need to resolve it"? Percentage wise, mentions for donald j. trump over donald trump comes to 0.3%. Is it worth the effort to clean it even further? And remember, this is just for one column, and one value inside that column. If we really do decide for this use case that we need a higher accuracy, then we will likely need to use a NER model (example using Spacy[3]), preferably trained on a dataset of famous personalities in the US but this arguably requires more effort than our quick and dirty approach.

The key takeaways are:

• There are different paths to clean data: Removing nulls, duplicates, syntactical correctness, semantic correctness and so on.

• It is a balancing act of effort, time and data reliability.

Using UDFs

Let's look at a slightly more complicated use case. Each news article has person, location and organization entities with a sentiment. This naturally prompts the question:

• What is the "sentiment score" of an article?

Let's make some assumptions on the dataset and define how to compute a sentiment score. Keeping it simple, for any entity we will map the "sentiment" to a number. A positive sentiment is +1, negative is -1 and no sentiment is 0. We will also use TextBlob[4] which is a text processing package that has sentiment polarity built into it. Our sentiment score is the average of both of these numbers.

This goes beyond what is built into Spark by default. But Spark provides a mechanism by which we can define our own functions that can be executed on the dataset. This is the user defined function, shortened to UDF.

The final schema we want is:

url, sentiment_score

We can get this pretty quickly from what we have already learned.

flattened_df = (df
    .select(
        F.col("thread.url").alias("url"),
        F.col("entities.persons").alias("persons"),
        F.col("entities.locations").alias("locations"),
        F.col("entities.organizations").alias("organizations"),
        F.col("text"),
        F.to_timestamp(F.col("published")).alias("published_at")
    )
)

From this, we need to:

• Process the sentiment values from the persons, locations and organizations column.

• Use a sentiment analyzer on the text column.

def sentiment_score_1(persons:List, locations:List, organizations:List) -> float:
    def fn(s:str):
        if s == "positive":
            return 1
        elif s == "negative":
            return -1
        else:
            return 0

    s1 = sum(map(fn, [p.sentiment for p in persons])) / max(len(persons), 1)
    s2 = sum(map(fn, [l.sentiment for l in locations])) / max(len(locations), 1)
    s3 = sum(map(fn, [o.sentiment for o in organizations])) / max(len(organizations), 1)

    return (s1 + s2 + s3) / 3

def sentiment_score_2(text) -> float:
    return TextBlob(text).sentiment.polarity

# Creating an user defined function
sscore1 = F.udf(sentiment_score_1, FloatType())
sscore2 = F.udf(sentiment_score_2, FloatType())

# Using udf in the transforms
final_df = (
    flattened_df
    .select(
        F.col("url"), 
        sscore1(F.col("persons"), F.col("locations"), F.col("organizations")).alias("sscore1"),
        sscore2(F.col("text")).alias("sscore2")
    )
    .withColumn("sentiment_score", (F.col("sscore1") + F.col("sscore2")) / 2)
    .drop("sscore1", "sscore2")
    .orderBy("sentiment_score", ascending=False)
)

One key point to remember about UDFs is that they are essentially the same as your sql functions, which means they act on the columns you specify, for that row.

The code is here: https://github.com/raghuveer-s/example-code/blob/main/spark-json-processing/sentiment_analysis.py

Summary

In conclusion, we took a high level view on how Data Engineers are tasked to help answering data questions, how to translate this into PySpark, and a few points on Data Exploration and Data Cleaning.

Where to go from here? There are so many directions. Taking these scripts into production using AWS or GCP is a natural progression from what we just talked about. In terms of utility, Data Cleaning, enforcing Data Quality, and in general producing reliable datasets will be both useful and highly regarded by your teams and team members. And of course, the last section progresses neatly into Machine Learning pipelines and ML Ops.

The full code is here: https://github.com/raghuveer-s/example-code/tree/main/spark-json-processing

References

[1] Types of tables in Spark : https://medium.com/@subashsivaji/types-of-apache-spark-tables-and-views-f468e2e53af2

[2] How does createReplaceTempView() work in Spark? : https://stackoverflow.com/questions/44011846/how-does-createorreplacetempview-work-in-spark

[3] Custom NER using Spacy : https://www.machinelearningplus.com/nlp/training-custom-ner-model-in-spacy/

[4] TextBlob : Text processing library (https://textblob.readthedocs.io/en/dev/index.html)

parquet-mr consists uses maven multi modules approach, the modules that I found myself spending the most time on are: parquet-common, parquet-column, and parquet-hadoop. The codebase follows object oriented principles beautifully so there are many classes. But, the responsibility of each is very specific and it all comes together very elegantly to perform the operations required.

Note: This is a simplified class diagram with many classes missing, and some liberties taken with the associations, inner classes and so on. The overall structure roughly falls along the same lines.

Writing data

Here is a rough class diagram for the write process.

There are various classes that handle writing data into pages, column chunks and so on.

Each column of data has a ColumnDescriptor which describes its path, type and holds the maximum repetition level and definition level. This along with ColumnChunkPageWriter for each column form the collection of writes for each column through which data is written. This is maintained in the ColumnWriteBase subclasses which itself is generated for the specific column, from a provider through ColumnWriteBaseStore and its subclasses.

The actual writing itself happens through InternalParquetRecordWriter<T> in the parquet-hadoop module. During writing, we must keep in mind the encoding in which data is written, this happens through subclasses of ValueWriter , and since the record itself can have a nested structure, it needs to be serialized-deserialized appropriately which is done through ColumnIO and its subclasses. Finally, subclasses of Statistics maintain the metadata through which optimizations are performed for the read side.

Reading data

Class diagram for the read process.

As you can see, there are prats of the diagram that are structurally very similar to the write side of the equation, except it performs the read operations.

Skipping over the counterparts which are mirrored from the write side, the new claases here are primarily RecordMaterializer and RecordReaderImplementation. RecordMaterializer has subclasses converts the read data into the appropriate format.

For example, Avro uses AvroRecordMaterializer, Protobuf uses ProtoRecordMateralizer and Spark uses ParquetRecordMaterializer.scala. RecordReaderImplementation is used to assemble the records and read its data, it constructs the automaton in the previous post used to read the shredded records, and holds a reference to the materializer.

Summary

And with that, we come to the end of the four part series. There's of course a lot more that I had to skip out, I highly recommend browsing through the parquet-mr codebase to get a sense for what this file format can do.

Personally, reading the code in parquet-mr was a difficult but rewarding experience. I do not claim to have understood it perfectly, but it was certainly a fun trip navigating through it and trying to figure things out. Shout out to the devs for this incredible project!

Record assembly algorithm

The basic idea is that we need to assemble the original record from the flattened data using its repetition level and definition level.

NOTE: What is described below is based on Dremel's original paper and parquet-mr. Spark implements a vectorized parquet reader based off parquet-mr, but the code flow is slightly different from what is described here and is mentioned briefly towards the end.

Let's revisit some of the key points from part 2 that will be useful to reassemble the record:

• Each data record has a tree structure with potentially multiple levels. The leaf of this structure represents the unique columns.

• Data can potentially repeat.

• Data can be optional.

• Repetition level indicates at what "repeated" field the value as repeated.

• Definition level is used to capture the level up to which optional values exist (mainly so that we can capture the levels at which a value or NULL occurs).

• The final data had flattened the tree structure into columns and associated them repetition and definition levels.

In simple terms: We have a multiple lists of leaf nodes with some numbers attached to it. Therefore, to reconstruct the original hierarchy we need three things:

• Access specific leaf nodes inside a list. Each list is just your column data.

• Jump from list to list.

• Place the nodes at the correct levels.

All we need now is to specify how to "access" lists of data, "jump" across lists and "place" the nodes.

Finite state machine

Let's visualize some of the columns and how such a jump might look like. Going with the example data from part 2 if we try to jump across columns while simultaneously trying to reconstruct the original hierarchy we can imagine doing something like this:

Even in this simplified example, rules for parsing seem to be emerging:

• Go through the columns in the same order as it is in the schema.

• When a "0" is encountered in the repetition level, move to the next field.

• Otherwise drain all the data which has a common repetition level.

In the above example, we go from Links.Backward to Links.Forward column with RL of 1, and drain all these fields of the same RL. Once that is over, we move to RL of 0. At which point we observe its value and move to the next field.

And what about definition level? Recall that DL helps know if there are NULL values along the path of the node, and if there are optional values along the path. Which means, with DL we can know the depth of the node in the record we are building, especially given that we know the maximum definition level depth directly from the schema.

Summarizing and drawing parallels from the previous section:

• Use schema to get the column order, maximum definition level.

• Use RL to go from column to column.

• Use DL at runtime and max definition levels to place the node.

The full state machine for the example data, taken from the Dremel paper:

Notable optimizations

When dealing with large dataset, we need to think of improving compute time. Most of the tricks below try to accomplish this in one way or another.

Predicate pushdown

This is a fairly common optimization borrowed from databases. The idea is that instead of gathering the results and then filtering, we attempt to pushdown the conditions so that the data is filtered at the lower layers before gathering them.

Fun fact: I was curious how old this technique is and a casual search on google scholar led to this paper[1] which references predicate pushdown to one of Jeffrey Ullman's[2] book from 1989!

Filtering row groups

Parquet can use the block metadata which allows it to skip over whole pages of data. It does this mostly through two important interfaces called ColumnIndex and OffsetIndex. Briefly, the metadata has statistics such as min, max values, null counts which allows parquet to scan through the data and skip pages. The original issue link[3] and a corresponding google doc[4] make for interesting reads.

Partial aggregation (Spark)

If the requested query has aggregations, then can choose to perform partial aggregations by using Parquet's column statistics instead of returning rows to Spark and aggregating upstream. This way, compute is saved and data movement also reduces.

Vectorized reader (Spark)

Spark has a vectorized reader implementation (enabled through spark.sql.parquet.enableVectorizedReader configuration setting) to read parquet data which according to the docs is based off the code in parquet-mr but it's Spark own implementation of the read operation. The basic principle is that in the vectorized approach it reads columnar data in batches, instead of the non-vectorized approach which follows a more iterative style approach to loading data. But there is a lot more to vectorized style of data processing and it can go quite low level. For more, refer to this talk[5] and this video[6] (both are from Databricks).

Summary

In short, we covered how recreate the flattened record from the repetition level and definition levels captured earlier. We also saw some tricks parquet uses to speed up its reads, the driving idea being to find ways to only process as much data as required and no more.

References

[1] Query Optimization by Predicate Move Around

[2] Jeffrey Ullman

[3] PARQUET-922 Jira issue

[4] PARQUET-922: SortColumnIndex Layout to Support Page Skipping

[5] Recent Parquet Improvements in Apache Spark

[6] Enabling Vectorized Engine in Apache Spark

Dremel was built at Google, it is web scale analytical system for performing ad-hoc queries on nested data. Parquet takes a page from Dremel and uses its record shredding and record assembly algorithms to store and retrieve nested data efficiently.

Record shredding is how data is broken into columnar data, and then written into parquet files. The reverse procedure (record assembly) is covered in part 3.

Nested data

Before beginning, let's take a step back and look at the structure of the incoming data. Speaking generally, it can have the following characteristics:

• Nested sections

• Repeatable sections

• Optional or required sections

This can be understood better when represented with a schema.

Schema

A schema is a structural view of the data. Let's use the same example as in the Dremel paper:

Minor note: The Dremel paper uses Protobuf[1] to represent the schema but you can use other protocols like Thrift[2] as well.

Looking at the above schema in a different way, we can think of it as a of group of nodes of size n >= 1 which are optional / required / repeating. The parquet format requires that we attempt to convert this into a flat structure.

All the values of this structure are present in the leaves of the tree. Looking at it another way, the data can be flattened if we know the full path to the leaf + we know the depth of the node + we are able to represent optional / required nodes + we can represent if the repetition of fields. In other words, if we can reduce each path to a column and are able to generate this metadata for the nested incoming data, then along with the actual values themselves we have everything we need to convert the nested data into a flattened form.

In the above example, if we look at the schema, Name.Language.Code is part of Name node which can repeat, Language node which can also repeat and Code is required. Thus for each Name.Language.Code value that exists in the data, if we can somehow capture the levels in the tree at which it repeats, and whether there are any optional along with the value then in principle we have everything we need to convert to and fro. This is what repetition levels and definition levels do.

To understand this better, let's go over them with an example.

Repetition level

When we have nested data with multiple levels and there can be data at some level that is a) optional and b) repeated (ie, a list) we need to know one thing : At what level should I start create a new list for this value? In the Dremel paper this idea is defined as: "It tells us at what repeated field in the field’s path the value has repeated.". In the above example, let's look at the Name.Language.Code path (column) again.

Observations:

• Going by the schema, Name and Language are groups which can be repeated. They do in fact repeat in record 1. Name node repeats three times, Language node repeats twice in the first name node, and once in the last one. And it occurs just once in record 2.

• Code is a required field. Which means if Language exists, Code must exist as well as per the schema otherwise it is an invalid document.

Let's try to flatten this. Appending a number for simplicity and going in order, we have the following observations:

The above observations can be encoded by a single number called repetition level with the following rules:

• For a new record, use the number 0. The list is created implicitly as well.

• For an existing record, if the list needs to be created, use the level at which the list is going to be created.

• For an existing record, if the list does not need to be created, use the level at which the element is being added.

Rewriting the table above with repetition level this time, we get for Name.Language.Code:

Definition level

What is remaining is that for nested data, we need a way to represent optional values along the field path, particularly if it happened to be NULL.

If the data is present, this is the "normal" case. If it is not, then we need a way to tell where along the field path this data became NULL.

Putting it another way: Definition level is mostly useful when we have a) nested structure + b) optional fields that leads to NULL values. We will be able to recreate the missing pieces in a record if we know to be missing.

Let's take two examples this time. Name.Language.Code and Name.Language.Country. Here, Code is required but Country is optional. Name and Language are repeated groups which can be optional.

As before, for Name.Language.Code we have:

For Name.Language.Country , we have:

Observe the slight difference between the two tables, and that is only because Code is a required field. And for a required field, the definition level does not apply, therefore the DL value is reduced slightly.

Flattened structure

The full flattened structure for all the fields can be found in Dremel's paper. For brevity, here are couple additional example for Links node, ie, Links.Forward and Links.Backward.

Links.Forward

Links.Backward

Record shredding

With the key concepts above, we can look at the record shredding algorithm.

To recap, we want a way to convert nested data to flat data with additional metadata that helps preserve this structural information. Repetition levels and definition levels precisely do this.

First, lets referring to the Dremel's paper again for the pseudocode:

procedure DissectRecord(RecordDecoder decoder, FieldWriter writer, int repetitionLevel): 
    Add current repetitionLevel and definition level to writer
    seenFields = {} // empty set of integers
    while decoder has more field values
        FieldWriter chWriter = child of writer for field read by decoder
        int chRepetitionLevel = repetitionLevel

        if set seenFields contains field ID of chWriter
            chRepetitionLevel = tree depth of chWriter
        else
            Add field ID of chWriter to seenFields
        end if

        if chWriter corresponds to an atomic field
            Write value of current field read by decoder using chWriter at chRepetitionLevel
        else
            DissectRecord( new RecordDecoder for nested record read by decoder, chWriter, chRepetitionLevel)
        end if
    end while
end procedure

What we are basically trying to do is walk down the nested record recursively preserving the level information. Let's take a very high level view and break down the logic:

• RecordDecoder decodes binary records.

• Each of the fields have a writer associated with them.

• As we traverse down the tree, we maintain the repetition level and definition levels keeping in mind how they work from the previous sections.

• If we encounter a node that does not have children (either NULL or some primitive type) we store the repetition level and definition level, otherwise we recurse further down.

• Once we are done with the write procedure, the data is written to column chunks using the appropriate encoding mechanism.

And that's it! In part 4, we will go over some of the code, classes involved in making this happen. If you want to check out the code right away, navigate to the write and writeGroup method of GroupWriter.java in parquet-hadoop module in the parquet-mr library. It provides an example reference implementation.

Summary

In this article, we looked at how parquet converts nested records into flat data with some additional metadata to preserve the nested structure. In the following articles, we will look at how to recreate the original record from this data and will take a peek under the hood in parquet-mr and spark codebase on how this happens as well.

References

[1] Protobuf: https://developers.google.com/protocol-buffers/

[2] Apache Thrift: https://thrift.apache.org/

To begin, there were few objectives that I wanted to meet:

1. What are the ideas / algorithms? that went into this data storage format?

2. How is data written / read?

3. Where is the codebase that puts it all together?

This series of articles is the output of the above questions. This article presents an overview and the following articles go deeper into it, I hope you'll find it of some utility.

Parquet overview

How is data stored?

Is parquet row-oriented or column oriented or hybird?

Broadly, there are three types of databases: Row-oriented, column-oriented and hybrid. There is good material on the web that explain them in greater detail, refer here[1] and here[2]. Though parquet is typically called as a columnar store, in my opinion, it is not strictly columnar. It follows a hybrid approach. The similarity comes from that both of these approaches do read data a column at a time, however traditional columnar stores tend to store each attribute or column as files independently of other columns. Parquet instead stores columns together as "row groups". A diagram can help in understanding the differences a bit better.

Row-oriented

Columns in a record are arranged one after another, this allows you to seek to a particular record quickly through index structures but if you want to retrieve lots of specific column data quickly as is common in analytical workloads, then we would have to seek to the next record, take that column values, move to the next one and so on.

Column-oriented

The columns are stored together and typically stored independently of other column values. This allows for fast retrieval of vast amounts of data but tradeoffs advantages of row-oriented structures such as fast updates.

Hybrid

Hybrid does not have a strict definition as far as I can tell, it is implementation dependent. Parquet does it by grouping together column values like in the column oriented db, however they are not stored independently in files as it is traditionally done. Instead think of it as the columns of multiple records being packed together, so what we have is a group of records aka a "row group " with columns packed inside these row groups as chunks of data.

Physical layout and term definitions

The diagram for parquet's physical layout from https://parquet.apache.org/docs/file-format/.

![Parquet file format image](https://parquet.apache.org/images/FileLayout.gif align="middle")

Now for the terms used in describing parquet format. Most of the definitions listed here correspond to the image above and come from the parquet-format[3] library. There are a couple extra ones that come from code in parquet-hadoop module and parquet-column in parquet-mr[4] which I thought to add in as well.

• Column chunk: A row is a bunch of columns with each column having a value. A column chunk is a chunk of values for that particular column.

• Page / Data page: Column chunks are split into pages.

• Row group: A group of column chunks.

• Page header: Has metadata for this page. For example: number of values, repetition and definition levels which are used in the record shredding process, statistics such as min max which are useful in filtering, among others.

• Footer: Contains metadata. Notably, file metadata and block metadata.

• File metadata: Contains schema

• Block metadata: Contains row count, list of column chunk metadata, path to this column, etc.

Encoding and compression schemes used

Parquet has a variety of encoding and compression schemes. Data is encoded using Run-Length encoding + bit packing hybrid. Another option that parquet can use for encoding is Delta encoding.

Parquet can store compressed data in its data pages. Snappy is the most popular compression scheme but parquet by default also supports gzip and zstd out of the box.

How is data written?

Record shredding

Sidebar: Whoever invented the term "record shredding" needs a raise, it sounds a lot cooler than record splitting.

We need a way to convert data stored as records into flat columnar data. And this data need not be flat, it can have structure. For example, if the data can be nested, can have array values, fields in records are optional and so on.

So how do we convert this into flat columnar data? This done using Dremel's record shredding algorithm[5]. The basic idea is that nested data is a tree of nodes at different levels, and what we do is associate each node with some metadata that lets us know if it is repeated, if the path to the node has NULL values and so on so that we do not lose the record structure information.

Writing data into parquet is documented in part 2 of this series.

How is data read?

Record assembly

Reading the data from parquet back into records requires a reconstruction of the original record structure. This is a little bit more complex than the record splitting process but at a high level it uses the metadata from before that we use to encode the record structure during the record shredding process. What we need is a way to arrange the nodes in relation to each other so that the original record can be reconstructed.

This process is documented in part 3 of this series.

Reading parquet data in spark vs. parquet-mr

When it comes to implementation of reading parquet data, there are two libraries: the spark way and the parquet-mr way.

Parquet-mr itself has two APIs, an older api that used to read all the data, and filter records during assembly and a newer API that skips over pages by looking at the metadata in each block which is much faster. Spark uses a vectorized approach to read data, which means it reads columnar data in parquet files in batches for each column. It is based off parquet-mr's implementation to some extent. Spark has a fallback mechanism to parquet-mr's code which can be set in the spark configuration. There are other tricks which are used to improve reading speed: Projection pushdown, predicate pushdown, bloom filters among others.

Grokking the source code

In part 4, we will look at some of the key classes, methods used in spark and parquet-mr along with the code flow. This is useful when you want to navigate the code base.

References

[1] https://levelup.gitconnected.com/data-storage-and-reporting-understanding-columna-vs-row-storage-formats-8cae46347fa3?gi=59f9c06236f7

[2] https://www.youtube.com/watch?v=Vw1fCeD06YI

[3] https://github.com/apache/parquet-format

[4] https://github.com/apache/parquet-mr

[5] https://storage.googleapis.com/pub-tools-public-publication-data/pdf/36632.pdf