Auto Reply WA A/B Testing

Tidak yakin mana yang lebih bagus? TEST!

A/B testing memungkinkan kamu compare dua versi dan pilih yang terbukti lebih efektif.

Apa yang Bisa Di-A/B Test?

🧪 ELEMENTS TO TEST:

📝 COPY:
- Greeting style (formal vs casual)
- CTA wording
- Emoji usage
- Message length
- Tone of voice

⏰ TIMING:
- Response delay
- Follow-up timing
- Best hours to send

📐 STRUCTURE:
- Menu vs free text
- Number of options
- Order of information
- With/without images

💰 OFFERS:
- Discount percentage
- Free shipping vs discount
- Urgency messages
- Social proof

A/B Test Framework

┌────────────────────────────────────┐
│         INCOMING MESSAGE           │
└──────────────────┬─────────────────┘
                   │
                   ▼
┌────────────────────────────────────┐
│     RANDOM ASSIGNMENT (50/50)      │
│  ┌─────────────┬─────────────┐     │
│  │  Variant A  │  Variant B  │     │
│  │    (50%)    │    (50%)    │     │
│  └─────────────┴─────────────┘     │
└──────────────────┬─────────────────┘
                   │
        ┌──────────┴──────────┐
        ▼                     ▼
┌───────────────────┐  ┌───────────────────┐
│   Send Version A  │  │   Send Version B  │
└─────────┬─────────┘  └─────────┬─────────┘
          │                      │
          ▼                      ▼
┌───────────────────┐  ┌───────────────────┐
│   Track Metrics   │  │   Track Metrics   │
└─────────┬─────────┘  └─────────┬─────────┘
          │                      │
          └──────────┬───────────┘
                     ▼
┌────────────────────────────────────┐
│     ANALYZE & DECLARE WINNER       │
└────────────────────────────────────┘

Implementation

Basic A/B Test System:

javascript

const experiments = {
    'greeting_test': {
        name: 'Greeting Style Test',
        variants: {
            A: {
                message: 'Halo! Selamat datang di Toko ABC. Ada yang bisa kami bantu?',
                weight: 50
            },
            B: {
                message: 'Hai kak! 👋 Welcome to Toko ABC! Mau lihat-lihat atau langsung order nih? 😊',
                weight: 50
            }
        },
        metrics: ['response_rate', 'conversion', 'satisfaction'],
        status: 'running',
        startDate: '2026-02-01',
        minSamples: 100
    },
    'cta_test': {
        name: 'CTA Wording Test',
        variants: {
            A: { message: 'Mau order? Ketik ORDER', weight: 50 },
            B: { message: '🛒 Order sekarang! Ketik BELI', weight: 50 }
        },
        metrics: ['click_rate', 'order_completion'],
        status: 'running',
        startDate: '2026-02-01',
        minSamples: 200
    }
};

const userAssignments = new Map(); // Track which variant each user sees

function assignVariant(userId, experimentId) {
    const key = `${userId}_${experimentId}`;
    
    // Return existing assignment if already assigned
    if (userAssignments.has(key)) {
        return userAssignments.get(key);
    }
    
    const experiment = experiments[experimentId];
    const variants = Object.entries(experiment.variants);
    
    // Weighted random selection
    const totalWeight = variants.reduce((sum, [, v]) => sum + v.weight, 0);
    let random = Math.random() * totalWeight;
    
    for (const [variantId, variant] of variants) {
        random -= variant.weight;
        if (random <= 0) {
            userAssignments.set(key, variantId);
            
            // Log assignment
            trackAssignment(userId, experimentId, variantId);
            
            return variantId;
        }
    }
    
    return 'A'; // Fallback
}

function getExperimentMessage(userId, experimentId) {
    const experiment = experiments[experimentId];
    
    if (experiment.status !== 'running') {
        // Return winner if experiment ended
        return experiment.winner?.message || experiment.variants.A.message;
    }
    
    const variantId = assignVariant(userId, experimentId);
    return experiment.variants[variantId].message;
}

Tracking Events:

javascript

async function trackEvent(userId, experimentId, eventType, data = {}) {
    const variantId = userAssignments.get(`${userId}_${experimentId}`);
    
    await db.experimentEvents.insert({
        userId,
        experimentId,
        variantId,
        eventType, // 'impression', 'click', 'conversion', 'bounce'
        data,
        timestamp: new Date()
    });
}

// Usage in bot
client.on('message', async msg => {
    const userId = msg.from;
    
    // Track which greeting variant was shown
    const greeting = getExperimentMessage(userId, 'greeting_test');
    await msg.reply(greeting);
    await trackEvent(userId, 'greeting_test', 'impression');
    
    // Later, track response
    if (userRespondedPositively(msg)) {
        await trackEvent(userId, 'greeting_test', 'positive_response');
    }
    
    // Track conversion
    if (userCompletedOrder(msg)) {
        await trackEvent(userId, 'greeting_test', 'conversion');
    }
});

Analysis:

javascript

async function analyzeExperiment(experimentId) {
    const events = await db.experimentEvents.find({ experimentId });
    
    const results = {};
    
    for (const variantId of ['A', 'B']) {
        const variantEvents = events.filter(e => e.variantId === variantId);
        
        const impressions = variantEvents.filter(e => e.eventType === 'impression').length;
        const responses = variantEvents.filter(e => e.eventType === 'positive_response').length;
        const conversions = variantEvents.filter(e => e.eventType === 'conversion').length;
        
        results[variantId] = {
            impressions,
            responseRate: impressions > 0 ? (responses / impressions * 100).toFixed(2) : 0,
            conversionRate: impressions > 0 ? (conversions / impressions * 100).toFixed(2) : 0
        };
    }
    
    // Calculate statistical significance
    const significance = calculateSignificance(
        results.A.impressions, results.A.conversions,
        results.B.impressions, results.B.conversions
    );
    
    return {
        results,
        significance,
        winner: determineWinner(results, significance)
    };
}

function calculateSignificance(n1, c1, n2, c2) {
    // Z-test for two proportions
    const p1 = c1 / n1;
    const p2 = c2 / n2;
    const p = (c1 + c2) / (n1 + n2);
    
    const z = (p1 - p2) / Math.sqrt(p * (1 - p) * (1/n1 + 1/n2));
    
    // Convert to p-value
    const pValue = 2 * (1 - normalCDF(Math.abs(z)));
    
    return {
        zScore: z.toFixed(3),
        pValue: pValue.toFixed(4),
        isSignificant: pValue < 0.05 // 95% confidence
    };
}

Real Examples

Test 1: Greeting Style

VARIANT A (Formal):
"Selamat datang di Toko Fashion ABC. 
Ada yang bisa kami bantu hari ini?"

VARIANT B (Casual):
"Hai kak! 👋 Welcome di Toko ABC! 
Lagi cari apa nih? 😊"

RESULTS (after 500 impressions each):
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Metric          │ A (Formal) │ B (Casual)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Response Rate   │ 45%        │ 62% ✅
Order Rate      │ 8%         │ 12% ✅
Avg Rating      │ 4.1        │ 4.4 ✅
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

WINNER: Variant B (+38% response, +50% orders)
Significance: p < 0.01 ✅

Test 2: CTA Button

VARIANT A:
"Mau order? Balas 'ORDER'"

VARIANT B:
"🛒 ORDER SEKARANG!
Ketik 'BELI' untuk checkout langsung!"

RESULTS:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Metric          │ A          │ B
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Click Rate      │ 15%        │ 22% ✅
Completion      │ 65%        │ 58%
Net Orders      │ 9.75%      │ 12.76% ✅
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

WINNER: Variant B (+31% more orders)

Test 3: Response Timing

VARIANT A: Immediate response (< 1 second)
VARIANT B: Delayed response (3-5 seconds, feels more human)

RESULTS:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Metric          │ A (Fast)   │ B (Delayed)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Satisfaction    │ 4.2        │ 4.3
Completion      │ 72%        │ 70%
Feel "Personal" │ 35%        │ 52% ✅
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

INSIGHT: Slight delay makes bot feel more personal,
but doesn't significantly impact conversion.

Dashboard View

📊 A/B TEST DASHBOARD

Active Experiments: 3

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
GREETING TEST
Status: Running (Day 5 of 14)
Samples: 423/500 (85%)

      A: ████████░░ 42% response
      B: ██████████ 58% response ✅

Current Leader: B (+38%)
Significance: p=0.02 (significant)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

CTA TEST
Status: Running (Day 3 of 7)
Samples: 156/200 (78%)

      A: ███████░░░ 15% click
      B: █████████░ 22% click

Current Leader: B (+46%)
Significance: p=0.08 (not yet significant)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Best Practices

Test Design:

✅ DO:
- Test ONE variable at a time
- Run until statistically significant
- Have clear success metrics
- Document hypothesis
- Equal traffic split (50/50)

❌ DON'T:
- Change test mid-way
- End too early
- Test multiple things at once
- Ignore statistical significance
- Forget to track all metrics

Sample Size:

📊 MINIMUM SAMPLES:

For 5% improvement detection:
- ~400 per variant

For 10% improvement detection:
- ~200 per variant

For 20% improvement detection:
- ~100 per variant

Rule of thumb: Wait for 95% confidence (p < 0.05)

Quick Test Ideas

🧪 HIGH IMPACT TESTS:

1. Greeting: Formal vs Casual
2. Emoji: With vs Without
3. Length: Short vs Detailed
4. CTA: Question vs Command
5. Urgency: With vs Without
6. Social proof: Include vs Skip
7. Timing: Immediate vs Delayed
8. Menu: Buttons vs Text
9. Personalization: Name vs Generic
10. Follow-up: 1 day vs 3 days

FAQ

Berapa lama harus run test?

Minimum 1-2 minggu atau sampai statistical significance (p < 0.05), whichever longer.

Bagaimana kalau hasilnya sama?

Keep the simpler version. Jika tidak ada perbedaan, pilih yang lebih mudah maintain.

Bisa test lebih dari 2 variant?

Bisa (A/B/C/n test), tapi butuh sample size lebih besar. Recommended mulai dengan 2 dulu.

Kesimpulan

Test, don't guess!

Start testing, start improving!

Setup A/B Testing →

Artikel Terkait

• Statistik Auto Reply
• Cara Mengukur Performa Bot
• Kesalahan Auto Reply WA