The Vanity Trap: Why Download Count ≠ PMF
1 million downloads. Zero retention. This is your reality.
User count is the easiest metric to game. Push notifications, viral loops, referral bribes. Indian startups especially fall for this. Jio's free data era trained users to download everything. You're not special.
The hard metric: Will users come back? Specifically—will they come back frequently and stay long?
Framework 1: The Time-Spent Hierarchy
Michael Seibel from YC ranks engagement signals in priority order. Follow this.
Tier 1: Weekly Retention + Average Session Length
- Weekly retention (W1) > 25% signals genuine use.
- Session length > 3 minutes shows real engagement (not accidental opens).
- Combined: W1 > 25% AND session > 3 min = potential PMF.
Example: Unacademy hit 8-minute daily sessions before Series A. This predicted willingness to pay. Downloads meant nothing.
Tier 2: Frequency (Session Count)
- 4+ sessions per week per active user = habit formation.
- Track: Average sessions/user/week for top 20% cohort.
- If only 5% of users return weekly, your product isn't sticky.
Tier 3: Depth (Actions per Session)
- Count distinct actions: Posts created, items saved, messages sent.
- Benchmark: Pinterest averages 8+ actions/session. Quora averages 4.
- Your product should have 3+ core actions users repeat.
- If users open your app but don't act, engagement is false.
Framework 2: Cohort Analysis (The Non-Obvious Test)
Don't measure average retention. Measure cohort-by-cohort.
Scott Belsky calls this "messy middle analysis." Your early cohorts are biased (friends, power users). Week 4 cohorts show true PMF signal.
Run this test:
- Split users by signup week (8-10 cohorts minimum).
- Track W1 retention (day 7 return rate) for each cohort.
- If W1 retention is flat across cohorts, you've found product-market fit.
- If it declines (early cohorts 35% retention, recent cohorts 15%), you're selling, not iterating.
Indian context: Many startups see this decline and blame "market saturation." It's actually product decay. You changed UX, or word-of-mouth quality dropped.
Framework 3: NPS Without Revenue (The Murky Metric Done Right)
NPS is a lagging indicator. But it's better than nothing for pre-revenue products.
Measure correctly:
- Ask: "Would you recommend this to a friend who needs [specific problem]?"
- Track weekly. Need 30+ responses to detect signal.
- NPS > 40 in cohorts of 50+ users = PMF adjacent.
- Below 20? You don't have it yet.
But here's the trap: Free product NPS is inflated. Free users rate generously. Discount by 15-20 points mentally.
Better: Track NPS for your engaged users only. If engaged users (W1 returners) have NPS > 50, your product solves a real problem.
Framework 4: The Monetization Readiness Signal
You can test willingness to pay without a payment system.
Run a fake paywall test:
- At a key action (download, share, unlock), add a screen: "This feature requires ₹99/month."
- No actual charge. Just measure: What % clicks "Purchase"?
- If < 5% of engaged users would pay, engagement is speculative.
- If > 15%, you have monetization PMF.
Indian founders often skip this. Then Series A hits, monetization fails, investors panic. Test it early.
What to Track Weekly (Your Metrics Dashboard)
1. D1 Retention (day 1 return): Baseline signal. Target > 20%.
2. W1 Retention (week 1): True stickiness signal. Target > 25%.
3. Session frequency (active users with 3+ sessions/week): Target 25%+ of DAU.
4. Session length (median): Target > 2.5 min.
5. Action depth (median actions/session): Baseline against competitor benchmarks.
6. Cohort stability (does W1 retention plateau across signup weeks?): Plateau = PMF signal.
7. Core loop completion (% users completing your main workflow): Target > 40% of signups.
8. Fake paywall conversion: Target > 10% for paid PMF.
The Non-Obvious Insight: Engagement Cliff at Week 2
Most B2C products see 60% drop from D1 to D7. This is normal.
But if you see a cliff at D14 (week 2), you've got a cold-start problem. Users try your product, like it, then forget it exists.
This means: Your product doesn't create habit. Notifications don't help (they annoy). You need a stronger value loop.
Indian startups miss this. They add more notifications instead of fixing the core loop. Bad.
Avoid These Mistakes
- Don't measure "MAU." Meaningless. One login = counted. Measure W1 retention instead.
- Don't trust early cohorts. Your co-founder's college friends skew signals. Wait 4 weeks minimum.
- Don't mix acquired and organic users. Paid cohorts have different retention profiles.
- Don't obsess over NPS. Track it, but weight engagement metrics 3x higher.
- Don't forget the India context. Low-income users have different engagement patterns. Segment by device, data cost, network type.
Actionable Checklist
This week:
- Define your core loop (the 1-3 actions that define product value).
- Measure W1 retention by cohort (last 4 cohorts). If flat, you have PMF signal.
- Run a fake paywall test. Measure conversion.
- Calculate average session length. Benchmark against 3 competitors.
This month:
- Segment W1 retention by: acquisition source, device, geography, user segment.
- Identify which segments have > 25% W1 retention. Double down there.
- For segments < 10% retention, run user interviews. Why did they churn?
- Ship 1 experiment to improve session length by 30 seconds.
Founding insight: Investors don't bet on revenue in pre-revenue startups. They bet on these engagement metrics. Get them right, and money follows.