Statistical Fundamentals and Growth Marketing

This week I started Statistical Fundamentals for Testing from CXL, I enrolled in Growth Marketing Mini Degree from CXL, this is my fourth week of studying, and here’s a quick summary of what I’ve learned so far.

I started with Growth Marketing Foundation by John Mcbride, the basic course discussed the difference between Growth Marketing and Traditional Marketing which is a really interesting topic and I chose it as the topic for my assignment in the second week: Growth Marketing Vs Traditional Marketing (please check it out in my previous post, I would love to hear your thoughts on it).
The course also includes other topics like User-Centric Marketing, a really interesting topic which is mainly about user research and testing followed by Identifying Growth Channels.

The second course is Running Growth Experiment, the first course is quite theoretical but the second is more advanced and practical, especially the A/B Testing Mastery. I had solid background knowledge of A/B testing, but after studying A/B Mastery, I found that I know nothing. The course is advanced and that is what I like about CXL so far, that you can find content that is not available elsewhere.

Growth Marketing & Statistics

This week was pretty tough at work so I didn’t learn much, but I was lucky enough to finish Statistical Fundamentals for Testing.

The course started with an introduction to sampling, introducing the first concept of sampling, population. The population is the group of variants you want to measure, which can be people, items, or any relevant measurable aspect.

The instructor began with an example of a coffee shop. What if you want to measure the coffee in different coffee shops to know the degree.

You can’t measure all the elements in one survey, for coffee cups you can’t measure every coffee cup that is served. So you take a sample and decide on a parameter by using the mean, which is called Mu, and Standard Deviation, which is Sigma. This is followed by the statistics of our sample, which is called the Latin X and SD for Standard Deviation.

How do you decide if your sample is sufficient or not? By using the confidence level, which is the amount of error allowed in A/B testing. Nowadays, there are tools on various websites that allow you to calculate your sample size and confidence level for free.

How do you know if your test is correct or random? By using the P-value, which is Statistical Significance. There is a misunderstanding between the P-value and the confidence level. The P-value is what you get after completing the research and getting the results to determine whether your research is correct or not, while the confidence level is what you get before starting the research to decide on your sample size.

One concept I didn’t understand about this topic is False Positive, so I searched Google for an answer, and here is what I found on Wikipedia that may be helpful in studying:

A false positive is an error in which a test result falsely indicates the presence of a condition such as a disease when the disease is not present, while a false negative is an opposite error in which the test result falsely does not indicate the presence of a condition when it is present.

Statistics Traps

There are four statistical traps you can fall into:

1- Regression to the mean and sampling error 2- Too many variations ​3- Click-through rates and conversion rates ​4- Frequentist vs. Bayesian testing methods.

1) Regression to the mean and sampling error.
The first stat trap you can fall into is regression to the mean and sampling error, which means stopping too early. When you select two variables to test, the variables fluctuate during the measurement, so at a certain point you will see a significant decrease in one variable and a decrease in another, and that is when you stop the test. To ensure that you do not fall into the regression to the mean and the sampling trap, make sure that you provide sufficient test duration and do not stop the test too early, but wait until the end and check the results.

You can find test duration calculators to help you determine the duration and avoid the first statistics trap.

2) Too many variants
The maximum variant you can test is three variants and no more. Don’t test too many variants assuming that it will give you a larger scope or a broader idea. Be sure to limit the variants in an investigation.

3) Click-through rates and conversion rates.
It is imperative to select and prioritize a Key Performance Indicator before you start a test. An increase in people adding items to the searching cart does not mean they would actually buy them.

There are two types of conversions, macro conversions, and micro conversions. Macro conversions include conversions, orders, sales, profit, and returns. Micro conversions include clicks, visits, views, scrolls, and bounces.
There are simple metrics like clicks and KPIs that are much more complicated, such as “ Add to Cart” and “The Higher the Uncertainty

4) Frequentist vs. Bayesian testing methods.
The last statistics trap is Frequentist vs. Bayesian test procedures, what is the difference between the Frequentist test and the Bayesian test? The difference is that in Bayesian, the hypothesis is assigned to a probability, whereas in Frequentist, the test is conducted without assigning a hypothesis to a probability.

An example to illustrate the difference between the two types of tests: imagine you have a 2d image of a dog that is black, but you can see a small snout that is white. From the Bayesian point of view, the dog is black, but from the frequentist point of view, the dog may be black and white.

In summary, Growth Marketing is all about tests and hypotheses, but there are fatal mistakes you can fall into. Decide on sufficient test duration, use the right sample size, and don’t stop the test prematurely. Most importantly, don’t be biased in your research.