Data and Metrics Collection

What is it?

Data and Metrics Collection is how we track what matters. It’s the systematic gathering, monitoring, and interpretation of data that tells us whether our products and services are achieving user needs and delivering meaningful mission impact.

At Rise8, we don’t collect metrics for metrics’ sake. We focus on what the Lean Analytics crowd calls the “One Metric That Matters” (more commonly referred to as One Mission Metric that Matters in our Rise8 context) — the critical measures that give us clarity, not noise. Whether your team is measuring deployment frequency, service reliability, or user task success rate, our user and system behavior metrics are tightly coupled to how we then also measure our mission outcomes and help us continuously learn.

The term "data and metrics" typically refers to quantitative measures, but we also seek signal through qualitative insights, especially when quantitative indicators raise questions. The combination helps us build better hypotheses, smarter experiments, and stronger evidence for what's working — or not.

Why is it important?

As highlighted in books like Lean Startup, The Four Disciplines of Execution, and thought leader posts across the product world on North Star Metrics, OKRs and KPIs, effective teams use data as a compass. Here’s why that matters for us:

Diagnose and optimize – Data shows where we're winning and where we’re wasting effort.
Support pivot/persevere decisions – Is our current approach working? Metrics reveal if we should double down or try a different path.
Drive decisions with evidence, not anecdotes – Good metrics challenge bias and keep us honest.
Align execution to strategy – Clear metrics tie back to agency mission, product vision, and our customers’ real-world outcomes.
Enable transparency and trust – Regular reporting helps stakeholders and delivery teams rally around a shared source of truth.

If we can’t measure it, we can’t manage it. But just as important — if we measure the wrong thing, we may manage in the wrong direction.

How do we do it?

Our approach borrows from the best of modern product and platform thinking:

Identify What Matters

We collaborate with stakeholders to define the mission outcomes that matter most — those tied to user goals, delivery health, and business/mission value. These include:

Mission Outcome Metrics: e.g., lives protected, critical healthcare delivered, crisis aversion rate, dollars saved/re-invested, assets managed, civilian call center efficiency
Health Metrics: e.g., lead time for changes, failed deployment rate, mean time to restore, user satisfaction
Usage Metrics: e.g., adoption curves, churn patterns, completion rates

We prioritize leading indicators that help us course-correct early, not just lagging results that are too late to change.

Start with Questions

Before jumping into dashboards or automation, we begin by asking: - What behavior change do we want to see in users? - What mission impact are we hoping to achieve? - What assumptions are we trying to validate?

These questions help us frame our data needs around real-world outcomes. Early on, we often track these metrics manually or with simple tools — spreadsheets, surveys, observation — so we can quickly learn what matters. If a low-fidelity solution helps us assess outcomes in prod, we start there. Then, once we’ve validated what’s important, we invest in automation for greater speed and scale.

Instrument & Automate

We lean on tools like Prometheus, Grafana, and feature flag analytics to automate collection where feasible. This reduces manual burden and ensures we capture high-fidelity data in near-real time.

Data isn’t useful until it’s interpreted. We visualize and review metrics weekly, monthly, and at key product moments (launches, pivots, retros). Dashboards and reports are made accessible to stakeholders to drive shared understanding and accountability.

Continuously Improve

We don’t treat our metrics as static. As our products evolve, so do the questions we ask. Every iteration is an opportunity to refine how we measure success — and to stop measuring what no longer matters.

Want to get better at defining the right metrics for your product or service? Start with this question:
“What is the smallest amount of measurable learning, that shows we’re heading in the right direction for mission impact?”

Then test, learn, and adapt — just like we do with everything else.