Does A/B testing worth it?
Sep. 7th, 2014 01:12 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
dennisgorelikA/B testing (and multivariate testing) has its appeal:
1) Try two different combination of a web page
2) See what converts [into goals, such as registrations or sales] better.
3) Keep the best performing version of the page.
But it has problems too. One of this problems is that A/B testing adds complexity to your system.
Not only you have to create two different versions of the page, but you also have to add testing scripts, setup goals, measure the performance of every variation.
Extra complexity to your system means more bugs in your system and slower development of other features.
Today I got an ironic example of A/B testing flaws.
In order to setup A/B testing on my website we registered account on Visual Website Optimizer (VWO).
Today I tried to pay on VWO in order to upgrade to premium membership.
I was NOT able to.
Here's how it went:
1) I tried to pay with American Express.
Got "Failed to authorize: Authorization Failed"
Got verification call from AmEx. Confirmed that transaction is valid (most likely that verification was triggered by the fact that Visual Web Site optimizer is based in India, not the US).
4) Tried to pay with PayPal.
Got no errors in PayPal popup, and no messages on Visual Website Optimizer.
But transaction did NOT complete.
VWO checkout page lists non-working phone number in India: +91 9868221372 ("The number you are calling is either switched off or not available at the moment")
"Contact Us" link points to:
5) Actual Contact Us page shows 3 phones.
I tried two of them:
+1 844-822-8378
+1 415-349-0105
Both lead to answering machine.
Granted, it's Sunday today. Hopefully they would resolve it on Monday.
I know that VWO team "eats their own dog food" and run A/B tests on their web site themselves.
That adds complexity and extra stress to their engineering process up to the point where they are not able to diagnose and fix basic bugs (wrong phone numbers on checkout page, non-working links, failing billing system).
I know that testing is still important and A/B testing occasionally makes sense. But sacrificing quality of your system in order to constantly run A/B tests is clearly an overkill.
1) Try two different combination of a web page
2) See what converts [into goals, such as registrations or sales] better.
3) Keep the best performing version of the page.
But it has problems too. One of this problems is that A/B testing adds complexity to your system.
Not only you have to create two different versions of the page, but you also have to add testing scripts, setup goals, measure the performance of every variation.
Extra complexity to your system means more bugs in your system and slower development of other features.
Today I got an ironic example of A/B testing flaws.
In order to setup A/B testing on my website we registered account on Visual Website Optimizer (VWO).
Today I tried to pay on VWO in order to upgrade to premium membership.
I was NOT able to.
Here's how it went:
1) I tried to pay with American Express.
Got "Failed to authorize: Authorization Failed"
Got verification call from AmEx. Confirmed that transaction is valid (most likely that verification was triggered by the fact that Visual Web Site optimizer is based in India, not the US).
4) Tried to pay with PayPal.
Got no errors in PayPal popup, and no messages on Visual Website Optimizer.
But transaction did NOT complete.
VWO checkout page lists non-working phone number in India: +91 9868221372 ("The number you are calling is either switched off or not available at the moment")
"Contact Us" link points to:
https://vwo.com/%23contactus 404 Page Not Found
5) Actual Contact Us page shows 3 phones.
I tried two of them:
+1 844-822-8378
+1 415-349-0105
Both lead to answering machine.
Granted, it's Sunday today. Hopefully they would resolve it on Monday.
I know that VWO team "eats their own dog food" and run A/B tests on their web site themselves.
That adds complexity and extra stress to their engineering process up to the point where they are not able to diagnose and fix basic bugs (wrong phone numbers on checkout page, non-working links, failing billing system).
I know that testing is still important and A/B testing occasionally makes sense. But sacrificing quality of your system in order to constantly run A/B tests is clearly an overkill.
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
no subject
Date: 2014-09-17 12:47 am (UTC)Вот это вот "with minimal IT help" - это совершенно порочный подход как для использования A/B в частности, так и для работ по юзабилити в целом.
Отвечая же на вопрос поста - для владельцев vwo a/b тестирование безусловно worth, бизнес-модель, деньги приносящая.
А вот заказчику тестирования, полагаю, намного больше бы помогла консультация среднего юзабилиста за те же деньги*.
P.S.
Вообще же A/B часто применяют неправильно. Это инструмент когда всё в интерфейсе в целом уже нормально, но есть некоторые моменты которые трудно рассчитать (аналогично тому как в игре "Сапер" бывают ситуации когда приходится действовать наугад). И вот A/B позволяет выбрать из нескольких хороших вариантов лучший.
А многие разработчики считают что можно из плохого интерфейса - путем нескольких последовательных итераций A/B - сделать хороший.
Но это неверно, ибо A/B способно указать разве что "локальный максимум", а не действительно лучший вариант.
------
* - фраза была написана до того как я сходил на сайт. Теперь-то (после того как заглянул) точно очевидно, что ему нужен юзабилист в первую очередь.
no subject
Date: 2014-09-17 02:20 am (UTC)Как найти правильного юзабилиста?
> что ему нужен юзабилист в первую очередь.
"Ему" - это кому: www.vwo.com или www.postjobfree.com?
no subject
Date: 2014-09-19 03:02 pm (UTC)Вот не знаю, у меня проблема ровно обратная - найти заказчика :)
"Ему" - это кому: www.vwo.com или www.postjobfree.com?
Ну, на самом деле обоим, но с VWO там чуть сложнее. Дело в том что часть из свистоперделок, портящих удобство пользования - может работать на увеличение продаж.
В расчете что потенциальный клиент посмотрит и впечатлится - мол, какой навороченный и модный сайт, наверняка ребята профессионалы.
(Для студий веб-дизайна хорошо знакомая диллема.)
no subject
Date: 2014-09-19 03:48 pm (UTC)no subject
Date: 2014-09-20 12:46 am (UTC)Финансово - там в принципе может зависеть от многих переменных, смотря какой проект/схема работы и пр.
Если упрощённо - от 7$ почасовая ставка.
no subject
Date: 2014-09-20 01:24 am (UTC)no subject
Date: 2014-09-25 12:25 am (UTC)no subject
Date: 2014-09-20 01:38 am (UTC)What do you mean under "testing interfaces"?
Do you mean UI?
no subject
Date: 2014-09-20 02:46 am (UTC)Exactly.
no subject
Date: 2014-09-20 03:04 am (UTC)That it works according to plan?
Something else?
no subject
Date: 2014-09-25 12:32 am (UTC)Ну, такая формулировка - это ближе к функциональному тестированию.
Основная задача юзабилити-тестирования - проверить, что всё сделано в соответствии с принципами юзабилити и законами эргономики, опционально выдать рекомендации по изменению.
Чтобы использование пользователем интерфейса - было комфортным и эффективным.