500TB of audit logs for 500$/mo and still searchable in real time

Audit logs, payments, or transaction logs,  … in the modern world, we are forced to store vast amounts of data for auditing and compliance purposes. When data volume grows, a question arises - should we store it on-premise with some “cheap disk arrays” or pay extra for cloud storage? The answer should be - let’s store it cheaply in the cloud! 

Let me show you how we managed to store over 500TB of audit data, with a cost under 500$/mo and a fast response time no matter how big the query time range is. This is the story of locating, classifying, and effectively storing 0.5PB of customer data. We’ve managed to utilize Google BigQuery technology in a rather unique setup that allowed us to store the entire dataset for less than 500USD/month and allows the support & compliance team to execute complex queries against the entire dataset with a response time of 11s.