Today I’m getting interviewed by Erkki Lassila, who is a Doctoral student at the University of Oulu. It’s an interesting talk about the different topics around analytics in gaming, and how it’s seen as a driving force for free to play game development.

Topics that we cover:

  • Could you briefly describe the change in game development? (from premium to free-to-play / before and after the analytics)
    • Games became a live service, that was the big change
  • In your opinion, is there a certain moment in history, or technological advancement, when this new revenue logic of free-to-play became viable in game development?
    • Facebook canvas days around 2009, which a) blew up the market, b) removed middlemen, c) game server and data warehousing/processing become cheaper.
  • Could you explain the revenue logic of free-to-play?
    • It’s based on engagement, people stick around for years and eventually they spend on their “hobby”. They understand the value proposition of the game and they feel that they have a relationship with the developer and the game.
  • The role of analytics and metrics in game development process?
    • Understanding what a large number of players are doing in your game.
  • What are the most important metrics? Why these ones?
    • Engagement metrics, which include retention numbers, participation in different features. If people aren’t coming back to your game, these engagement metrics will reveal what is broken.
  • What kind of information these metrics are providing and how this information is used in practice? (examples)
    • Tutorial funnel is the usual suspect,
    • Session length on the first day of gameplay. Like, are there significant drops on retention day-1 for people who played 5 minutes, 10 minutes, 15 minutes? Where are most people stuck?
    • Lots of these things can be figured out before the game launches. Lots of playtesting, and common sense on what are the realities in the market, can reveal issues and you don’t need analytics
    • Going from there, looking at the engagement on different features, based on how long the player has been in the game
  • In which development phase these are the most interesting?
    • Whenever you have a few hundred players playing the game. Then it just gets more interesting as you get more players.
  • In terms of the game development process, how do you know if you have succeeded in your work? (the role of metrics)
    • Once you have a solid game with engagement, you can move forward with the game and open it up for people
    • The common sense here is that you shouldn’t lose the players that are coming back on day. For day 1 and day 3, a ratio of 0.7, which indicates the slope. The better the ratio, meaning the closer the Day-1 and Day-3 numbers are, the less people will be dropping off later as well. But you still need to go to Day-7 and Day-30 and see that the ratio there is similar.
  • Could you specify the main focus areas of free-to-play metrics? (for example user acquisition, user retention, and monetization?)
    • As I mentioned, user retention and all the other engagement metrics are the most important thing. If they’re not working, it doesn’t really matter how much focus you put on your UA metrics and monetization metrics, until you figure out your engagement problems.
  • How do you use monetization metrics?
    • Once you have a solid game with engagement, you can figure out monetization
    • The old saying was that you can’t slap on monetization later. I would rather say that you can’t splat on engagement. It’s much harder if the core gameplay, the meta-game, and the live ops aren’t working.
    • For monetization, looking at LTV is important. It helps you drive the right kind of UA. Here you need cohort analysis, where you break down players on where they are coming from. Important dimensions are country and device. Tier 1 countries usually have more disposable income and will more likely have a higher LTV, but also higher marketing costs.
    • Should you roll out the game to a global audience before you’ve figured out LTV and user acquisition costs? No, unless you have confidence in the engagement, that you can keep players in the game for months and years.
    • One place where free-to-play often breaks is that the dev team hasn’t accounted for people getting to the end of the game very early. Basically this happens so that someone spends money, buys everything and suddenly they are at the end of the game. The question to ask here is: how much money will a player need to spend to have everything that they need to beat the hardest content in the game?
  • How can you try to influence the behavior of (potential) users in these different areas? (examples)
    • In the industry, we often talk about conversion. From App Store view to installing, from install to tutorial complete, from active player to active spender.
    • Each of these requires their own methods, starting with getting the attention of the prospective gamer on the App Store, then getting them to play long enough so that the game becomes a hobby.
    • The common question here should be “What is the need that my game is providing a solution to? What is the need that my $9.99 bundle is providing a solution to?”
    • It goes back to understanding the target audience and catering to their needs as gamers. That’s fundamentally what it’s all about.
  • How would you define the criteria for killing a game development project?
    • It’s very hard for developers to kill their game. I was recently talking with Round Zero, and one dev had brought their game like six times to a UA test until they called it quits.
    • It becomes harder, the more time that you put into the game.
    • It’s also about the faith in finding a solution to fix the game.
    • I’m usually very keen on seeing constant progress with the updates you are doing. If you aren’t, you should ask yourself: Do I feel confident that I know why I can’t improve the engagement metrics? Has there been something that we overlooked in the dev process? What don’t we know yet about the target audience and what are game doesn’t provide to them?
    • Ask for advice, don’t try to force it on yourself to just push on.
  • If there are opposing views or conflicting ideas about the direction to be taken in game development project or decisions to be made, how these kinds of situations are handled? (Do analytics/metrics play a role in these situations)
    • Early in the project when you still don’t have data or lots of it:
    • These usually come up with you don’t fundamentally know the market situation, the target audience and the game you are building for that combination of the market situation and target audience.
    • Come to an agreement first on the market and the audience. Then argue based on those facts.
    • Later on:
    • You can rely more on data, but if you are seeing retention fall off at day-60 or 90, it probably has something to do with the game design, either what the player can achieve in the end game, for example running out of player goals, things to achieve. Harder content to beat, new items or characters to collect. If the loops stop being interesting, go back to the drawing board and design your way out of it.
  • How analytics relate to future risks and predictions? Could you describe how?
    • Spending interest flatlines at a certain part of the game. You can observe this by player level. On a higher level, just look at the cumulative spend of your active player base. You can see if people who’ve already spent money in the game started dropping out, which could be a sign that they are moving to other games.
  • Who generates the required algorithms for analytics?
    • Most teams have dedicated analytics people. These people can often become bottlenecks for decision making, as you are waiting for data
    • Often, it’s good to have an early start to decide what you’d want to measure from your game. Put your design folks and analytics peeps into a room and set a goal for them, that they need to know what are the top ten dashboards that will show the health of the core game, metagame, and live ops. Note! You also need to have regular session time, retention and monetization dashboards. So the earlier you get these planned, the better.
  • How decisions are made and what they are based on before any consumer telemetry data is available?
    • As I mentioned earlier, the dynamics of F2P really revolve around understanding the market dynamics and the target audience. You should have a sense as a developer if the game that you make will have an audience. Analyze the market, play other games, look at the download and grossing charts.
    • When designing the game, you will need to model out how the metagame and the core gameplay will provide experiences to the player, that fit the way that people like to play mobile games.
  • Could you give an example of how non-financial user behavior is translated into financial figures?
    • To make predictions about the future, you really have two toolsets at your disposal:
    • 1) is the previous cohort data that you have for the game, from the same geo, for example, LTVs that they have achieved.
    • 2) the modeling from the game economy and understanding of the players will desire to spend when they reach day-60, day-90, and so on. Or if you want to look at your end game when players reach halfway through the available game, how motivated are they spend to reach further into the game.
  • Do you see raw behavioral data having any value as such?
    • It comes down to understanding what the players are doing, what they like about the game and what they dislike about the game. Just looking at numbers all day is very hard to read, if you don’t have a feel on what your players are doing.
    • One way to get into the heads of players is to conduct player surveys. Lots of sentiment questions will reveal many aspects of the player behavior that raw data will not.
  • How do you calculate/decide the amount of investments/resources needed for collecting, handling and storing raw data?
    • It feels to me that it’s always underestimated, but the common sense approach is to have at least one person who is setting up the data engineering part, which basically is coding the event collecting inside the game, then the pipelines to the data warehouse. And you have a person who prepares the data for analysis and dashboards.
    • Keep your entire product and design team up to date on what is being looked at and why so that there is alignment.
  • Is there anything else you would like to add or highlight for the purposes of the study?
    • Analytics will never make a good game, but you need to be informed about the analytics once you have thousands or millions of users. Otherwise, you can’t properly keep the business going.
  • How would you define creativity in Free-to-play?
  • How would you define a good game?