As part of our expert interview series, Fuzy was lucky to sit down with Tom Wilbur, a senior product and data leader, who has led teams on the leading edge of data science and product analysis at Vast (acquired by Vroom) and Indeed.
In this interview, you’ll get his experienced perspective on the importance of data and insights for product development, as well as some words of wisdom for developing product science capabilities within your own company.
What roles and experiences have most shaped your thinking in data science and product development?
In the early 2010s, I worked at a startup called Vast, where I was the Product Director. Vast’s business model was about creating insights from data for consumers making big or important purchases – a house, a car, a vacation package, that sort of thing. Especially in the automotive space, we ran a lot of white-label marketplaces, where we matched users to inventory––connecting people to items that matched what they wanted. That was the first time (at scale) I was directly making product decisions from data using common methods of analysis like split testing and such. I found that to be an incredibly powerful lever as a product manager.
After Vast, I worked at Indeed for eight years, where I held a couple of roles that were really formative to how I feel about using data to drive product decisions. First, I started a new group and product area as the director of Analytics Product, building products to help people make better decisions using Indeed's amazing data about the worldwide job market. We built internal tools to help product managers and product analysts, business leaders, and technical teams better access their data and better organize it.
We also did a lot to democratize the use of data across the company, making it accessible to everyone in the company, not just highly technical people. And in line with what I did at Vast, we also built out campaign analytics systems and other tools to help our customers make better decisions with data when using our product.
I also spent a good portion of my time at Indeed running platform teams, especially engineering platforms and data platforms, figuring out how to build the systems that help technical teams be more effective in their work.
So, I got to see the full spectrum across those jobs––having responsibility for the product team that is creating the core, deep, data platform at a company, to building the actual applications that help people make decisions, to running websites where consumers are actively making decisions, and you're able to see and leverage that data.
More than anything else, I learned that the ability to track your business outcomes and users’ behavior, and to use that data to inform your decisions as a product manager is an incredible advantage to have. It's a way to be more successful in whatever opportunity you're pursuing.
Can you elaborate on your philosophy about the role and importance of data and measuring outcomes to make decisions that advance product development?
One of the core things I believe about effective product management is that you need to start from your business outcomes. The entire product development team––not just the product manager, but also the engineering team and your product designers––should be looking at everything through the lens of what that outcome is. What is the user behavior that creates value for that user or for their business?
I also feel very strongly about the importance of measuring outcomes over outputs. But we do often run into the challenge that it's very hard to measure outcomes. A lot of times those outcomes are not taking place in our system. For example, at Indeed, the outcome we want is for someone to get a job. That happens when somebody signs an offer letter, far downstream of Indeed’s involvement. As product managers, we made our product decisions based on the behavior we could measure (like clicks, applies, and messages between an employer and a job seeker), but we simply couldn’t see some of the long-term outcomes that were most interesting to us. But that’s not a reason to throw up your hands and give up - we kept our eye on that and kept working toward it. It was a big part of the company's data strategy: to move our business closer to that end outcome and to work hard at building instrumentation and visibility into that outcome.
What are some of the key challenges in being able to gain visibility into outcomes?
Sometimes, like in the Indeed example I gave previously or like when car shoppers actually purchase a car, the core event that people come to the site for isn't visible to the product team because it's offline––or it's in a different system. Making product decisions on that outcome is hard because you likely won’t know how people get to that end state when you first start. Especially when you're an early stage company, as you start building out parts of your experience, you may not yet be implementing in software the parts that are farther downstream. That means you might have to instrument and attempt to draw inference from predictive upstream events. I think that's what most product teams end up doing, whether it's predicting a path through your pages or some sort of a sales funnel. These techniques are all about identifying those upfront events and using them to infer downstream success.
It sounds like it might also be difficult to gather and analyze outcomes data because each outcome is highly dependent on the industry and the application itself, like finding a job or selecting a car. How does this level of specificity and complexity affect the process?
I do think it's very specific. At Vast, we specialized in cars, real estate, travel, and vacation rentals. So much of the value––where the money is––is often in big verticals, and they are very different. In fact, that ended up being a key strength for Vast; we had some competitive differentiation. There was a time when Google was moving into cars and was seen as a potential competitive threat, but they were coming at it from a very horizontal perspective. They thought they could apply a generic inventory database and generically present people with the information. We found business value in knowing exactly how that works in automotive, and in particular, we focused on used cars, which is a very different market from new cars.
So, yes, it’s specific, but that’s your competitive differentiation, your “secret sauce.” And it does mean that as a product manager, I need to make sure that specificity is present in my analytics, either by intentionally representing it or by training a model to reflect it.
One of the important (and highly valuable) initiatives we did at Indeed to help more people at the company better use our data was to uplevel our expertise and understanding of that “secret sauce.” We created a Business Intelligence team that could help teams across the company run analyses, build dashboards, and develop specific datasets for their needs. As that type of expertise grew, we formed a Product Science team, embedding data science-capable analytics experts into product teams. This was very successful, but of course, as it was a solution based on the expertise of highly-skilled and highly-paid people, it was hard to scale and very expensive. But we didn’t have another option––no solution was available at that time that could programmatically uncover those sorts of insights into business outcomes.
Where do you think current solutions fall short in connecting product data to business outcomes and meeting the needs of product leaders?
When I've looked at most off-the-shelf solutions, they're often effective at helping companies capture data, but they're not good at helping them understand what data to capture.
Where I’ve worked in the past, we used a couple common off-the-shelf things, and it’s easy to get into the habit of continuously adding logging before you actually know what you're trying to get to. Instead, if you can see the big picture and work backwards from that end outcome, that can be very powerful. However, what we've learned in so many ways with the modern data science stack and the deep learning and AI paradigms is that sometimes it’s a matter of letting your technology connect some of those things for you. You don't have to perfectly build the whole map to get from the individual data to the exact outcome––but you still have to know what that outcome is so that you can train models, etc.
I think when most existing tools are purchased by product teams, they find themselves collecting a whole bunch of data, and they're not really sure what it tells them or how to use it. They might get a few really basic facts, like a conversion funnel, but I think it often falls short of the promise of modeling your whole data set and getting massively valuable insights.
What that means is that a lot of companies end up building it themselves, and that is expensive. Especially if you're a small company, or if you're a small team inside a big company, and you're trying to get going, it's really hard to invest in all of that upfront. In fact, it's probably a bad idea to start by building out a whole bunch of measurement infrastructure. You've got to validate your product hypothesis in a faster way.
Ideally, I think you’d find the middle ground (which Fuzy is doing a great job of targeting), where you get a set of insights using their view of how to draw those insights out, because that’s unlikely to be the specialty of the engineering team you put together to go build an early-stage product.
More product leader interviews this way 👉
Connecting the dots to uncover deeper insights: An interview with product expert Ethan Handel
Why outcomes data builds better teams and products: An interview with product expert Adam Dahlgren