As W. Edwards Deming famously said, 'Without data, you're just another person with an opinion.' In product development, data can be a guiding light, informing decisions, uncovering insights, and driving innovation. But is data always the answer? In this article, we'll explore when collecting data is valuable, and when it may not make sense—or could even have negative consequences.
The Right Time to Rely on Data
Data is a powerful tool for decision-making. It helps you understand the current state of your product and its users: What does the landscape look like right now? What’s happening with your product? What matters most to users at this moment? Additionally, data closes the loop on learning and drives continuous improvement (e.g., What did we learn from this user test? What changed after we introduced a new feature?). You can even use current data to make informed predictions about the future, such as forecasting user behavior based on trends (e.g., predicting increased mobile-only usage of your app).
Below are crucial scenarios where leveraging data is essential to driving your product’s success:
- Identifying customers needs to address them effectively and find inspiration for product innovation.
- Understanding market trends to inform product-related decisions.
- Validating new concepts, ideas, and features before making significant investments, especially when there’s considerable risk involved.
- Monitoring user engagement and retention to be able to quickly spot opportunities and challenges.
- Testing different feature options or solutions to select the one users prefer.
- Ordering the Product Backlog based on user feedback, engagement metrics, market demand, or the results of the validation of your ideas (note that a combination of data sources is often used to order the Product Backlog).
- Optimizing product performance and key metrics.
Generally speaking, if you want to bring transparency to your product's reality, you need to gather data and insights to form a clear picture of the current situation. If you aim to improve something, first understand the current state, then introduce changes and collect data to evaluate their impact. This is a simple pattern for continuous improvement—or continuous learning about the best actions you can take to maximize the value of your product. In particular, data plays an important role in product discovery and validation to help us navigate the land of unknowns and uncertainties.
When Data Is Not Helpful
While relying on evidence is often better than sticking to opinions or beliefs, data isn’t always the reliable friend you expect it to be. It’s helpful, but we shouldn’t always follow it blindly. Sometimes, we don’t need data at all—or we may need it later. So, when is it better to set data aside or disregard it, even if it’s already available?
Let’s take a look at the following examples:
- Regulatory obligations: You don’t need to ask users whether they want it or not because you simply have to deliver the functionality to comply with legal requirements.
- Low risk, high reward: If there’s a low risk of making the wrong decision and the predicted value is high (or sufficient), data may not be necessary. You just do it and later measure the outcomes.
- Cost of inaction: When the cost of delaying a decision is higher than the cost of making a wrong one, avoid analysis paralysis and gathering data if it requires significant time investments. Instead, focus on quickly delivering something that will help you learn.
- Significant context changes: If the context has shifted dramatically, comparing current data to historical data may lead to misleading insights. In such cases, the product development rules have changed, but data gathered now can still be useful for future learning.
- Biased data collection: If you suspect the data is biased—whether due to poorly formulated survey questions or involving the wrong group—it may be better to disregard it.
- Encouraging creativity and innovation: When brainstorming breakthrough product ideas, allow creativity to flow without immediately asking for evidence. Validation can come later.
Sometimes, the best approach is to simply JDI (just do it). This applies when legal obligations are present, and it can also work when the risk is low, and the perceived value is high. When your product situation changes significantly, relying on historical trends for decisions can lead to mistakes. Creativity needs space for free exploration, sometimes joining non-obvious ideas. Don’t limit yourself or your team by asking too early: "What is the evidence that it’s possible or makes sense?"
As a rule of thumb when using data, maintain a healthy level of skepticism, especially when predicting the future. The future is uncertain, and disruptive events can completely change your product development landscape. Remember, there’s always a risk of false positives (something seems true when it’s actually false), false negatives (something seems false when it’s actually true), and biases that can mislead you.
Summary
Data can be incredibly valuable, enabling Evidence-Based Management in product development. It helps us learn and make informed decisions. However, it’s important to recognize that in some cases, data may not be necessary, or it should be set aside. Biased or incorrectly gathered data can actually undermine transparency.
Time is also a critical factor in this process, and it can change everything. What may not be true or effective today could shift completely next week, next month, or even tomorrow. That’s why collecting data correctly—as much as needed and when needed—is essential for Product teams. Conversations like "Do we need any data or evidence?", "What kind of data will be helpful?", and "What are the risks?" should happen regularly within teams.
Hello! I'm Joanna. I work as a trainer, consultant, mentor, and coach in Action Learning. I'm passionate about helping individuals, teams, and organizations tackle complex challenges. Enjoying my content? Want to learn more? You're warmly invited to get in touch with me: