Thinking about Tracking Design, Part 3: What should you track?

Designing a tracking system is about dealing with trade-offs as you try your best to navigate complexity

Jul 17, 2023

Note: This is the third part in my series on tracking design, which I’m writing v e r y s l o w l y

You can see the previous parts here:

Let’s say you are in your first job leading a data team, and you are now responsible for setting up a new frontend tracking system. One of the key questions you will face is exactly what to track.

Will you only use the ‘out of the box’ events from your tracker of choice, or will you also capture custom events?
Do you want to capture all user interactions, or will you focus only on a sub-set?
What level of detail do you want to capture as parameters; do you want to just know that a button has been pressed, or do you want to know exactly which button?
Do you want to track anonymously, or do you want to capture as much identity information as possible?

These are all big questions, and basically no consensus exists as to what is the right approach, and when you try to find out more by searching on the internet you will encounter very strongly-held opinions backed with precious little evidence. So if you’re the man or woman who has to make a practical decision, how should you do so?

In this post, I want to walk you through what I think are the key questions to ask, based on my perspective as someone who has been through this process several times before.

The good news is that you can ignore the yelling data influencers, because there is no right answer, or at least no answer that’s always right … there’s only the answer that’s right for you, and for your business.

Possible approaches

In my previous post on ‘what is tracking anyways?’ I discussed the kind of events that are typically captured by a tracker: page/screen views, link clicks, form fills and time-related events. So let’s assume that you will have some information available out of the box, therefore for you as a decision-maker the choice is about what else to track, if anything.

Let’s start by weighing your options. Broadly speaking, there are two main approaches to the question of what to track:

Track every user interaction, to make sure you don’t miss anything that is potentially useful
Track only a limited set of interactions, to make it easier to derive meaningful business insights from the data

Basically, those are the two poles, and most tracking plans will fall somewhere in between.

The advantage of the first approach is that if you capture everything … you’ve got everything. Theoretically you can answer any question that comes your way. Which is great!

The downside is that analysis can become a bit ‘needle-in-the-haystack’, in that it can be hard to find the information you really need from the massive flood of information. Secondly, it can be more expensive to store and process a lot of events, particularly if you have a very popular website and/or mobile app.

The advantage of the second approach, of focusing your tracking only on capturing the events that have the most business value, is that it is easier to perform analysis on the data that you have; you’re not having to filter away lots of marginal random events to get at the good stuff.

The problem, however, is that while it’s pretty easy to assess which events have high business value in the short-term it’s not so easy to guess which events will be useful in the medium- to long-term. Trust me on this one, I’ve certainly been burned before by being asked to do an analysis only to sheepishly reply, “uh, we can’t do that because we aren’t tracking that yet.”

What are your business needs?

When deciding what to track, it’s important to start by thinking through the relevant business use cases for frontend tracking. After all, you’re doing this to help the business, not because it’s a fun challenge and a cool tool (although it definitely helps that it’s that as well).

As I’ve discussed previously, there are two fundamental questions that you are trying to answer with frontend tracking:

1. Where do your users come from?

2. What do they then do once they arrive?

When deciding what to track, it helps to look at those questions and then map them to your overall business strategy in order to help you make the decision.

Let’s look again at Soundcloud as an example (as I have through the series); what is Soundcloud’s business strategy? Soundcloud has several revenue streams:

Creator subscriptions (buying the ability to post more content)
Listener subscriptions (buying a quasi-Spotify experience)
Advertising
Mastering services
Events

If I were to create a tracking strategy for Soundcloud, it would focus first and foremost on the events that were most directly related to the generation of revenues.

Let’s look again at the example page that I’ve shown before for how this would play out in practical terms:

Player events: I would certainly want to know player behavior, because this would include the amount of ads triggered, and it would also include people listening to the short clips that exist as promos for the premium subscription service. More generally I would want to know player behaviour because I would want to know how sticky the service is - what share of people’s attention are we getting?
Try Go+: Are people clicking on this button and going into the subscription funnel? How far do they progress down the funnel? Where do they drop out and can we identify any friction points?
Engagement events: Are users interacting in any significant way with the content besides listening? Do they comment, like, repost? Knowing this information and cross-referencing it with the information about the commercial events can help you to understand what actions lead to actual cash money and help you to tweak the user experience accordingly

What are your resources?

In an ideal world you would have unlimited developer resources at your fingertips … but that’s not likely, so you have to work with what you have.

Generally speaking, tracking implementation (or instrumentation, as it’s often called) is not particularly complex software development, but the amount of effort is greater than zero. It doesn’t just magically happen. So part of your tracking plan involves answering these questions:

Who is going to do the actual implementation work?
How much experience do they have?
Are they completely swamped or will they be able to fit this in sometime soon?

What I’ve experienced is that if the frontend developers have an insane roadmap it can be difficult to get space for tracking development, as it’s not something with a clear and quick ROI like developing a new customer-facing feature. Even if you’ve agreed with product management that tracking should be part of your ‘definition of done’ for new features, it can be hard to get the time and space to implement tracking on existing features if capacity is very low.

With this in mind, when designing your tracking plan, think carefully about the resources you can tap into - it might not always be feasible to take the ‘track everything!!!’ approach that I discussed above. Or, alternatively, you can take such an approach but break it down into phases where the tracking progressively gets richer and more complex.

Future Proofing

Story time: when I was transitioning from being an analyst to being a data leader, I was tasked with designing a mobile analytics setup for a mobile app that we were building at Neugelb.

Thinking with my analyst hat on, I came up with an elegant solution for naming each unique action:

feature_noun_verb

What does that mean? It meant that literally each action had a unique event label in Google Analytics, so to go back to the Soundcloud screenshot above, if you wanted to name the event for the click on the ‘Try Go+’ button, the event would be named something like:

navigation_try_goplus_click

As an analyst, this meant that it was very easy when querying the data warehouse to batch events by feature and by common actions; in this case I could look at this event in terms of navigation (the overall feature), as part of events related to Go+, or in terms of click events in general.

Very easy for generating insights!

But can you spot the problem with this approach?

If you guessed “it’s not very scalable”, award yourself with a cookie! (I can’t give it to you, so you’ll have to go to the kitchen yourself)

This approach was totally fine when we had a limited feature set on the app, but as we added more and more features it became harder and harder to maintain - it became more and more confusing for the developers and bugs crept in more frequently, and eventually we needed to refactor the events to follow a more scalable naming convention (and here I have to give credit to Asis Ybarra, now of Glazed Analytics, for coming up with a much better approach).

So this is another crucial thing to consider when designing your tracking: how easily can you add new features under this planned structure? How much work will adding tracking to these new features be? Ideally you want to design your tracking in such a way that most of the pain happens upfront, and implementation gets easier over time, i.e. that if you need to add tracking for new features six months down the line it should be pretty straight-forward.

Considering trade offs

To wrap things up, I’d like to reiterate the point that designing tracking at most companies is about managing trade-offs. The biggest trade off is between depth (how much information you capture) and effort (how many resources you want to put into implementation), but there is also a trade off between making it easy for your analysts to distinguish between events and making it easy for your developers to maintain the project. These trade offs also have consequences in other areas: you might, for example, need to devote more time and energy to doing data modelling to make it easy to extract insights.

The final point I’d like to make is that this is not a one size fits all process - there’s no single right way to decide what to track! This is a very situationally dependent process; it really depends on your resources, your business needs, and, of course, your own personal skill level and experience.

One last (music) thing

Every post ends with one of my dj mixes, since I’ve been a dj for (gulp) 26 years now. This time I’m going to share a particularly mad one, Darkside Generation, my mix of 1993 darkside hardcore. This was one of the most creatively fertile, even insane, periods in the history of rave music, as UK producers moved past the hands-in-the-air euphoria of the early rave era to compete to produce the sickest, most twisted sounds imaginable.

Yes, this is a little, uh, frisky, by most peoples’ standards, but, well … I love this stuff!