What are A/B tests?
The A/B test
is a research method which compares at least two slightly different versions of
a design or product to find out which feature is better. A/B tests are mostly
used to evaluate changes introduced on a website or related tools, e.g.
marketing newsletters. The test allows you to find out which version is more
suitable for the customer and which is more effective. It is most often
determined by the conversion rate - the higher it is, the more effective the
content or product is. You need to see what will convince users more to the
desired action.
A basic example of A/B test is sending the same newsletter with two different topics and the same sender name to check which one will be opened and clicked more often. This way you decide which topic is better.
https://www.webhostingsecretrevealed.net/wp-content/uploads/2015/01/ab-testing.png
How to schedule an A / B test?
When you’re
scheduling an A/B test you must start with
determining what problem you want to solve. Then you must analyze the existing
data. To know what exactly to test, you should know how much time a visitor
spends on your website and how quickly it leaves it. Data analysis will help
you find the element that is worth working on.
Assess how
many users you need to test to get reliable results. Divide your test group
into parts and determine how long the
test should take. Prepare your pages for testing using the appropriate
software. When you have prepared every step which was mentioned before - run
the tests. The condition for the correct execution of the test is that both
versions work at the same time, randomly hitting the users. Half of the
visitors see version A (that's your control group, you show them the current
version of what you are testing) and the other half see side B (variable).
Simultaneously, data is collected showing whether the responses in group B have
changed in response to the new variant of the side.
In the end
verify the results. Has version B turned out to be more effective than A and
actually caused the assumed increase? If so, you already know what the mistake
was and which way you should go. If not, you should repeat the test, checking another
element that may be responsible for discouraging users from visiting your website.
By testing
your content, you can understand what attracts users attention. Such detailed
and precise knowledge can significantly reduce the costs of an advertising
campaign. If all elements are equally effective in attracting new users, even a
shorter one will be effective. Test everything but skillfully prioritize. Start
with the foundational elements that directly influence the success of the
website, then pay attention to the minor ones. Also, remember not to optimize
smaller functions at the expense of larger ones.
Questions
1. Have you ever heard about A/B
testing?
2. Do you think this way of testing is good
enough for making changes in the department
of e-marketing in company?
Sources
1. https://pl.wikipedia.org/wiki/Test_A/B
2. https://charzynska.pl/testy-a-b-poradnik-dla-poczatkujacych/
3. https://poradnikprzedsiebiorcy.pl/-testy-a-b-dowiedz-sie-o-nich-wszystkiego-i-wybieraj-dobrze
I’ve never heard about such a way of testing but it sounds really interesting to me. On the other hand, as I’m a software developer I have to be sure that the programs which I create will be always displayed in the same style and look for every user so I can’t do A/B tests there. Despite that, I think that such testing may have very great potential as it’s collecting info about user behavior. It not only can allow determining which newsletter header will be better but I do believe that it can also reveal some visitor models and schemes. In my opinion, all data related to user/visitor behavior is becoming very valuable.
ReplyDeleteYou're right - data about a users is valuable from the company's perspective, but it is important to remember about data quality. It's interesting that sometimes AB tests are also carried out on websites, then for some visitors it looks different than for the rest :)
Delete
ReplyDeleteTo be honest I was never really interested in user behavior analysis, hence I was not aware of existence of AB tests, however if I was ever to open my own company I would definitely take using it into consideration. Would it seem enough for me to base my whole internet marketing on? Probably not. It seems like a fairly simple way to receive some data about efficiency of your site or campaign, however I would probably treat it as a sort of entry point to a larger and more detailed study. Maybe combined with some sort of questionnaire, where users could share their suggestions and comments it could create a more complex study, therefore being more effective in my opinion.
AB tests are carried out in order to changing the already existing marketing rules. Visitors/receivers don't know that a given "thing" is being tested, so the questionnaire proposal, though interesting, doesn't match the AB tests. After the test, it is checked whether the differences between the results are statistically significant. If so - it is worth making a change.
DeleteI've never heard of such a testing method. Although, it seems quite interesting and in my opinion it can be very useful. As a user I think this kind of a test may give decent results, but to be honest this kind of testing may be not enough. As someone mentioned above, a/b testing seem to be a good option to gain some smaller information or preferences and it seem to be a good study combined with other types of getting information (such as questionnairies or or asking for comments)
ReplyDeleteYes, I've heard about it. I read few articles about A/B testing 3-4 years ago. It's an interesting method that has some pros and cons. There are a lot of parts of the website when it can be useful. On the other hand, we never know the result if we combine few changes.
ReplyDeleteNo, I don't. There are possible aspects where we have to make decisions based on the correlation between changed elements.
AB testing is using mainly for examinig small changes. When we want to compare multiple changes we should try with MVT (Multivariate Test).
DeleteYes, I’ve heard about A/B testing and I think it’s a very useful method when it comes to attracting customers. Conducting A/B tests provides us with vital info about customers’ preferences and it also lets us know what parts of our content we still need to work on.
ReplyDeleteYes, I think this way of testing is good enough for making changes in the department of e-marketing in company, however I think we should not rely on A/B testing only, but we should take into consideration many other factors.
I agree that when we want to test more complex changes we should use more advanced methods :)
DeleteThis is the first time I have heard of this testing method. This method sounds interesting, it is a pity that it was not discussed in one of my subjects during my engineering studies. As other users have mentioned, in my opinion this method is good for examining more basic factors and in order to extract more information it is worth combining it with other methods.
ReplyDelete1. Have you ever heard about A/B testing?
ReplyDeleteI heard something about this method but I didn't know that it's called A/B. This method is useful because when you have to choose one of You simply choose one that you like. In other methods when you have multiple options you need to think harder about answer.
2. Do you think this way of testing is good enough for making changes in the department of e-marketing in company?
For small changes like layout or colour change I think yes, but for bigger changes probably You need second methodology to be sure answers are right
I have heard about something very similar but I did not know that this kind of testing is called that way. Maybe there are few test like that but are named differently. :D
ReplyDeleteI don't think it would change a lot, since it is already in use. I have heard about google using it to place their paid campains in a place where people look first. If it comes to websites, there are a lot of rules that says about placement of important things, one of them is that people always look first on the left side of a screen, since we are used to reading from a left side. So to me it might be a good test but I would not relay on it.
I've heard that Google is using that kind of test too. I find it very interesing that such a big company is using such a simple test :)
DeleteI think I've heard of this type of test before, although I'm not sure, at least it was good to recall. The very specificity of such a test seems very interesting and unconventional to me, after all, the testing process takes place in real time on two different versions of the same product. This is slightly different from the usual standards for testing one product using, for example, usability tests on a small sample of testers.
ReplyDeleteI think that in the face of today's times and a slightly difficult approach to product testing, this should be a sufficient method to take care of an already finished website design or other invention which is still destined for development. By using good tools for tracking user movements, you can probably extract a lot of useful information about their behavior and needs.
Yes, I have heard about something like A/B tests somewhere before. It is pretty simple way to test which of the product verion or option is more likely to please customers or which one will succeed more likely, generally speaking. It can be widely used in e-commerce or product pricing. In my personal opinion this methodology is simple enough to support uncomplicated issues. While using A/B testing there's only two available options and focusing only on two narrows possibilities. Sometimes the best solution is to seek the alternative third option.
ReplyDeleteYou're right - it's quite simple way to test product's option, but in some point it can be difficult. I think dividing test group into parts to make test efficient is one of the most difficult steps during AB testing.
Delete1. I heard about this testing method when I was learning about testing reports in BI. It was used to create best layout of reports. I know this method is used in many different subjects in business and can be much more useful in other areas of business.
ReplyDelete2. It is good tool for testing many software changes. However there are many better ways to test it nowadays for example Quality function deployment method. In my opinion A/B testing could be a good testing tool for small changes in company. As we know there are very complex problems in business and A/B method can be too limited.
You've got a point there - AB testing is only useful for small changes. Furthermore, we can ctes multiple versions of a particular element. When we want to test multiple changes, we should rather do a "Multivariate Test".
Delete1. I have heard about this kind of test many times, but I am not a fan of it. I'm an individualist, so I don't like it when an opinion or a path is imposed on me. In this kind of test we can't express our opinion, but we are forced to choose between one and another. Is this expressing our opinion? I don't think so, although it is probably legitimate in some cases.
ReplyDelete2. Certainly, it is a good method among people who do not have their own opinion and prefer to choose between one or the other. Shouldn't the e-marketing department collect reliable opinions rather than imposing their own opinions? It seems to me that people should have the freedom of choice and should determine for themselves, whether they like the solution to an issue, and if not, what their opinion is in this case.
The idea of AB testing is exploring users/customers behaviour by (for ex.) sending them an one email (one email per each group), so there are no option to choose which option is better. Results are evaluated with statistical methods :)
DeleteI have never heard about such testing methods, however i find it very interesting. I see lots or pros and cons behind that, in my specific case i would use it to test which variant of layout of our products fits best our customers. Unfortunately i would not say such test is enough to change way of company marketing. To change marketing patch of products company should relay on more than just one test checking only two variants. For sure its a good base to start but for sure it shouldn’t be a core of marketing.
ReplyDeleteInteresting Article. Hoping that you will continue posting an article having a useful information. Facial Recognition
ReplyDelete