Continuous Integration at Scale Streaming 50 Billion Events per Day for Realtime Feedback

34 Views

Thanks! Share it with your friends!

You disliked this video. Thanks for the feedback!

Published Aug 27, 2025

Pure Storage runs a lean engineering workforce, with only 5% of the team dedicated to QA. As a result, the company has invested heavily in automated testing for a continuous integration and release cycle. As the products and teams grow, the number of tests have exploded, requiring an automated solution to prioritize, classify, and understand failure root causes.

Ivan Jibaja (FlashBlade engineer lead) offers an overview of Pure Storage’s streaming big data analytics pipeline, which uses open source technologies like Spark and Kafka to process over 30 billion events per day and provide real-time feedback in under five seconds. This pipeline is supported by Pure Storage’s FlashBlade as a shared storage solution, which enables a streaming use case as well as on-demand batch analytics. Ivan explores the use case for big data analytics technologies, the lessons learned from this project, and the underlying elastic infrastructure that provides flexible scaling, agility, and simplicity across multiple application clusters.