Behind the scenes: moving big data into the cloud

1024 683 Spil Games

Pruthviraj (Pruth) Shivanna works in Business Intelligence as a data engineer. He has been at Spil Games for a little over a year, working in the team migrating the company’s massive data warehouse from Vertica to Google BigQuery.

 

Why is data engineering so important to a games publisher like Spil Games?
Sherlock Holmes cries impatiently in The Adventure of The Copper Beeches: “Data! Data! Data! I can’t make bricks without clay.” Gaming is a competitive business, and you need the best tools and information to back your decisions.

This article in text format would be less than 5kB. Compare that to the 20TB of plain text information we collected in the last year from our games (4 billion times the size of this article). As we gather more data than ever, how we do we derive insight from it?

Data Engineering is all about automating the process of collection, and putting into context and presenting the information in a way that helps answer key business questions.

I understand you are currently migrating the data warehouse into the Google cloud. What does that involve?
We’re moving away from a legacy data warehouse to a more advanced and server-less alternative on the Google Cloud platform.

What are the advantages of moving to the cloud?
The biggest advantages are scalability and reliability. That gives us the flexibility to support analytics for more than 100 games at a moment’s notice without the need to maintain the hardware. We should see no more midnight alerts or delayed reports to business users.

What is your biggest lesson learnt in the migration to Google?
When technical issues cause delays in delivering reports, people who depend on those reports are immensely affected. Effectively, the decision-making process is delayed. That’s why we took extra care to ensure the stability of the pipeline in the new setup and that’s the biggest take away.

Can you give some examples of the kind of thing people do with the data you supply?
The examples below are just a drop in the ocean. Our Reporting team takes our data and creates a whole range of reports for different teams within the business.
• Marketing gets user acquisition reports to understand how much of the marketing spend actually resulted in mobile game installs.
• Finance creates executive reports based on the data.
• Live Ops gets information to help manage games day-to-day, including installs, DAU, ARPDAU, total collections and churn. We also provide near real-time user behavior data such as people’s progression through levels and a bunch of other metrics.
• Advertising use data to optimize revenue by user segmentation, attribute revenue to games in portals and so on.

What accomplishment are you most proud of in your time at Spil Games?
Absolutely our new BI (business intelligence) setup in the cloud. Each day, the batch process in the old setup would take around 10 hours to complete due to the constraints of legacy technology, and if there were any technical issues, stakeholders had to wait another full day to get the reports. After we migrated to the cloud, the daily process is less than 2 hours and we have immediate turn-around if there are issues.

Where are you from originally and how did you come to be working in The Netherlands?
I am from India and back in 2016 I did a trip across Europe and frankly fell in love with The Netherlands. When I went back to India I saw a job ad for a data engineer at Spil Games and I applied immediately. I pestered Rogier (now my manager) continually until he was convinced to hire me.

How do you feel about living in the Netherlands?
It feels like home and the best thing is, of course, the people. Everybody is so open minded and accepting of others and it also helps that the company is a really cool family. And to top it all, you can roam the whole country on a bike.

Leave a Reply