![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
Does A/B testing worth it?
Sep. 7th, 2014 01:12 pmA/B testing (and multivariate testing) has its appeal:
1) Try two different combination of a web page
2) See what converts [into goals, such as registrations or sales] better.
3) Keep the best performing version of the page.
But it has problems too. One of this problems is that A/B testing adds complexity to your system.
Not only you have to create two different versions of the page, but you also have to add testing scripts, setup goals, measure the performance of every variation.
Extra complexity to your system means more bugs in your system and slower development of other features.
Today I got an ironic example of A/B testing flaws.
In order to setup A/B testing on my website we registered account on Visual Website Optimizer (VWO).
Today I tried to pay on VWO in order to upgrade to premium membership.
I was NOT able to.
Here's how it went:
1) I tried to pay with American Express.
Got "Failed to authorize: Authorization Failed"
Got verification call from AmEx. Confirmed that transaction is valid (most likely that verification was triggered by the fact that Visual Web Site optimizer is based in India, not the US).
4) Tried to pay with PayPal.
Got no errors in PayPal popup, and no messages on Visual Website Optimizer.
But transaction did NOT complete.
VWO checkout page lists non-working phone number in India: +91 9868221372 ("The number you are calling is either switched off or not available at the moment")
"Contact Us" link points to:
5) Actual Contact Us page shows 3 phones.
I tried two of them:
+1 844-822-8378
+1 415-349-0105
Both lead to answering machine.
Granted, it's Sunday today. Hopefully they would resolve it on Monday.
I know that VWO team "eats their own dog food" and run A/B tests on their web site themselves.
That adds complexity and extra stress to their engineering process up to the point where they are not able to diagnose and fix basic bugs (wrong phone numbers on checkout page, non-working links, failing billing system).
I know that testing is still important and A/B testing occasionally makes sense. But sacrificing quality of your system in order to constantly run A/B tests is clearly an overkill.
1) Try two different combination of a web page
2) See what converts [into goals, such as registrations or sales] better.
3) Keep the best performing version of the page.
But it has problems too. One of this problems is that A/B testing adds complexity to your system.
Not only you have to create two different versions of the page, but you also have to add testing scripts, setup goals, measure the performance of every variation.
Extra complexity to your system means more bugs in your system and slower development of other features.
Today I got an ironic example of A/B testing flaws.
In order to setup A/B testing on my website we registered account on Visual Website Optimizer (VWO).
Today I tried to pay on VWO in order to upgrade to premium membership.
I was NOT able to.
Here's how it went:
1) I tried to pay with American Express.
Got "Failed to authorize: Authorization Failed"
Got verification call from AmEx. Confirmed that transaction is valid (most likely that verification was triggered by the fact that Visual Web Site optimizer is based in India, not the US).
4) Tried to pay with PayPal.
Got no errors in PayPal popup, and no messages on Visual Website Optimizer.
But transaction did NOT complete.
VWO checkout page lists non-working phone number in India: +91 9868221372 ("The number you are calling is either switched off or not available at the moment")
"Contact Us" link points to:
https://vwo.com/%23contactus 404 Page Not Found
5) Actual Contact Us page shows 3 phones.
I tried two of them:
+1 844-822-8378
+1 415-349-0105
Both lead to answering machine.
Granted, it's Sunday today. Hopefully they would resolve it on Monday.
I know that VWO team "eats their own dog food" and run A/B tests on their web site themselves.
That adds complexity and extra stress to their engineering process up to the point where they are not able to diagnose and fix basic bugs (wrong phone numbers on checkout page, non-working links, failing billing system).
I know that testing is still important and A/B testing occasionally makes sense. But sacrificing quality of your system in order to constantly run A/B tests is clearly an overkill.
