A/B Testing

A/B testing, also called split testing, runs two or more versions of a product feature simultaneously with different user groups. Version A is the control (current state). Version B is the variant (new change). You measure a specific metric—conversion rate, click rate, retention—and determine which version wins based on statistical significance.

Statistical significance means the difference isn't due to random chance. Most startups use a 95% confidence level: if p-value is below 0.05, the result is valid. Sample size matters heavily. Testing with 100 users gives unreliable results; 10,000 users gives stronger signals. Running tests for at least 7 days removes day-of-week bias.

A/B testing reduces guessing. Instead of arguing whether a button should be blue or green, you test both and data decides. It works for landing pages, email subject lines, pricing, onboarding flows, and push notification copy. The cost is low—mostly engineering time—but the clarity is high.

India's internet behaviour differs from the West. Users on 3G networks favour lighter pages—A/B tests must compare load times, not just conversion. Many Indian startups skip A/B testing due to low traffic; with 500 daily active users, reaching statistical significance takes months. Startups like PhonePe and Paytm run thousands of tests yearly because they have volume. Early-stage companies with under 5,000 monthly users should focus on qualitative feedback and usability testing instead.

Regional preferences matter: Hindi-language buttons convert better in tier-2 cities than English. Payment method preferences vary—UPI adoption in metro cities far exceeds rural areas. Testing locally relevant messaging (Diwali discounts, festival language) drives better results than generic variants. Indian regulators don't restrict A/B testing, but darkpatterns (deceptive UX variations) face scrutiny under consumer protection laws.