TALKS WITH EXPERTS

Welcome back to the 7th episode of our knowledge-sharing series Talks with experts. In today’s episode, we will discuss the Game Analytics with the manager of the department Peter.

Peter, can you tell us what exactly are the responsibilities of a game analysts here in Pixel Federation?

Well, basically it’s everything that revolves around the data. We have a designated analyst for each of our projects. As you know, you can collect qualitative and quantitative feedback for your games. Our Community Managers collect the voice of the people,the qualitative feedback, but it can be biased, as usually, the least satisfied players are the most vocals ones. That’s where we, the data analysts, come to the equation to look at the data and provide the objective view of the changes and the actual response of the players, how they interact with the features etc.

We provide our Game Designers with quantitative feedback, so they actually know what are the effects of the changes that they are making in the games. Apart from game analysts, we have two analysts working directly with our marketing team helping them to optimize our user acquisition channels. And we also have our own data engineering team that is responsible for the infrastructure, the data warehousing, tools and looking for ways to improve the effectivity of Data Analyst department.

Can you name and describe some of the tools you use?

We are processing more than 10 000 events per second from our players. As conventional databases wouldn't be sufficient to process such a large amount of data, we are using Big Data technologies. We have our own Hadoop cluster and we are processing data using Spark (the distributed computing framework) and after that, it depends on the personal preference of every analyst, whether they choose to use Python, R or Excel. Apart from this, we use Power BI for our dashboarding and reporting tools in order to serve data to the whole company so everybody can look into the data about our performance, basic KPIs etc.

We all heard about the A/B testing and how it can deliver meaningful results and answers to the questions we ask. Can you tell us, how exactly does it work?

A/B testing is a methodology that measures the actual impact of one specific change in the game. It is a controlled experiment where you divide the players into two equal groups, then you introduce the specific change only for one group and you let them interact with the same game. This way you can be sure that there are no other changes or factors impacting the player.

Recently we made a major change in our A/B testing methodology and ditched the frequentist approach which is one of the most widespread testing methods right now. We came to this decision because it is not really intuitive and it is hard to explain the results to the people without statistical or mathematical background. We switched to Bayesian methodology instead and so far it seems great. If you are interested in the topic, don't miss out this great blog from my colleague Victor - Why we completely changed the A/B testing methodology

Recently we heard a lot A/B/C tests. Can you explain us what it’s about? Especially the C?

Well the A/B test is just a term. This C can represent a test with more than 2 groups. You can divide the players into more categories, then you can test for example different versions of a new feature. But in such case, you have to keep in mind that more group divisions will shrink the group size and you will have to run the test for a longer time to achieve some meaningful results. The C can also refer to the control group - the group that you are trying to measure the impact of the feature of the change against. Imagine you have some baseline, that can be for example how the game is performing right now. Then you introduce the change in the game only for a subset of the players, and the players that are actually not affected by the change is the control group that you are controlling the experiment with.

What are the disadvantages of the A/B or A/B/C testing?

The A/B testing sound almost too good and if it was that easy, we would test basically every change possible. But there are some obstacles. Sometimes it is not possible to roll out the changes in the game only to a subset of the players because they are active in their player circles and talk about their experience with the game on the social media. If you introduce some important change only to a subset of the players, the rest of them will be unhappy and subsequently our community managers will be unhappy too...

Another thing is that you really need a lot of players in order to get some meaningful results from the testing, so the A/B tests can be quite costly. The important thing is to test the features that you expect to have a large impact on the game, not just some cosmetic changes.

To wrap this all up, do you remember some recent case study that had an impact on one of our live projects?

Recently we were preparing a launch of a new feature in our game TrainStation. The feature was a minigame designed to keep our players busy while they are waiting for their trains to return from the quests. The hypothesis was that we will roll out this feature only for the new players and it should boost the retention. But the results of the A/B test were quite surprising! It turned out that the minigame actually lowered the retention of the new players as they were not familiar with the core loop of the game yet, the minigame confused them and subsequently they stopped playing because of this reason. In the end, we rolled out this feature only for seasoned players that are already well aware of the gameplay and in general appreciate new features as some kind of bonus.

Other interesting case study is our recent test of a new monetization mechanics in TrainStation. So far, the best monetization feature is a monthly Special offer that includes ingame currency and a special train. Not long ago we introduced smaller Flash offers of discounted gems throughout the month but we were afraid this could negatively impact the monetization of Special offers. So we decided to A/B test this hypothesis. In the group A the Flash offer disappeared 3 days before arrival of the Special offer while in the group B only one day before the Special offer. We were measuring whether it has a negative impact on the bigger monetization mechanics of Special offer, but the hypothesis was not confirmed and it turned out that that it is completely ok to let the smaller offers run for longer.

So that is how the Game Analytics department works in Pixel Federation. And now it's your turn. What testing tools and methodologies do you use and which one works the best for you? Tell us about your approach. Don't hesitate to ask further questions and discuss this week’s topic in the Facebook group called Free to play game developers. Please feel free to invite your fellow game developers as well :)