Wistia Case study

GX monitors data pipelines and validates pre-ingestion and post-ingestion

About Vimeo

Vimeo is one of the world's leading service providers of HD streaming video, allowing over 200M professionals, teams, and organizations to create, collaborate and communicate. It offers HD video hosting for video creators, as well as professional help for marketers for creating and sharing promotional videos through Vimeo Create. The BI Engineering team at Vimeo works along with other internal teams like CRM, finance, sales, and marketing analytics with the goal of integrating streaming data from different advertising platforms to enable more accurate analytics. In doing so, the team deals with many of the typical issues data teams encounter when working with a large amount of data from a variety of sources. Here is how Vimeo uses Great Expectations to overcome their data quality challenges. Also, if you’d like to learn more about the innovative work being done by the Engineering team at Vimeo, we recommend checking out the Vimeo Engineering Blog.

The Challenge

One of the major challenges the BI engineering team faced had to do with detecting problems caused by service interruptions or upstream schema changes. In a typical scenario, those types of events would result in data not being properly processed by the data pipeline. This in turn would cause the downstream tables to contain missing data or data that was out-of-date. Without proper monitoring, this issue would potentially remain undetected, sometimes for days, until it resulted in a broken report that was noticed by stakeholders. Additional challenges were faced during data pipeline migrations, which involved migrating hundreds of DAGs while ensuring the data was kept up-to-date and consistent. The BI engineering team built an in-house validation framework to address some of these needs, but there was still a need for a more powerful tool, one that could be used to validate the data pre-ingestion, and post-ingestion, without having to depend on the data being loaded into a database before validation could be done.