A b test - Revision history

Vasodzh at 18:57, 28 January 2022

2022-01-28T18:57:01Z

2022-01-28T18:47:04Z

2022-01-28T18:43:33Z

2022-01-28T18:42:59Z

2022-01-28T18:40:48Z

2022-01-28T18:32:34Z

2022-01-28T18:32:14Z

2022-01-28T18:31:25Z

2022-01-28T18:31:11Z

2022-01-28T18:30:24Z

@@ Line 125: / Line 125: @@
 In case you run a test for Facebook, monthly users of the feature can be far more then 10k.
-)	Formulas can be used only if you are satisfied with 20% chance of not detection. Otherwise, you should either be a good mathematician to be able to theoretically create a formula to get a formula for a sample size to get, say, 95% chance of detecting a difference. OR you can just use my calculator.
+)	Formulas can be used only if you are satisfied with 20% chance of not detection. Otherwise, you should either be a good mathematician to be able to theoretically create a formula for a sample size to achieve, say, 95% chance of detecting a difference. OR you can just use my calculator, which is easier.
 '''Which I think is really something noone did before'''

@@ Line 125: / Line 125: @@
 In case you run a test for Facebook, monthly users of the feature can be far more then 10k.
-)	Formulas can be used only if you are satisfied with 20% chance of not detection. Otherwise, you should either be a good mathematician to be able to theoretically create a formula to get a formula for a sample size to get, say, 95% chance of detecting a difference. OR you can just use my calculator
+)	Formulas can be used only if you are satisfied with 20% chance of not detection. Otherwise, you should either be a good mathematician to be able to theoretically create a formula to get a formula for a sample size to get, say, 95% chance of detecting a difference. OR you can just use my calculator.
+'''Which I think is really something noone did before'''
 =Code=
 Code, explanatory letter and examples of generated CSVs
 [[File:Archive_sim.zip]]

@@ Line 27: / Line 27: @@
 Historical information on how users performed in the past for the old feature.
 Expectation of the metric
-Variation of the metric
+* Expectation of the metric
 How many users access the feature monthly

@@ Line 1: / Line 1: @@
 '''Name:''' Sample size calculation for A/B test<br>
-'''Author:''' Vaso Dzhinchvelashvili
+'''Author:''' Vaso Dzhinchvelashvili <br>
 '''Method:''' Monte Carlo<br>
 '''Tool:''' Python<br><br>

← Older revision		Revision as of 18:40, 28 January 2022
Line 125:		Line 125:

	3) Formulas can be used only if you are satisfied with 20% chance of not detection. Otherwise, you should either be a good mathematician to be able to theoretically create a formula to get a formula for a sample size to get, say, 95% chance of detecting a difference. OR you can just use my calculator		3) Formulas can be used only if you are satisfied with 20% chance of not detection. Otherwise, you should either be a good mathematician to be able to theoretically create a formula to get a formula for a sample size to get, say, 95% chance of detecting a difference. OR you can just use my calculator
		+
		+
		+	=Code=
		+	Code, explanatory letter and examples of generated CSVs
		+	[[File:Archive_sim.zip]]

@@ Line 96: / Line 96: @@
 What else is exported:
 Delta – expected difference. ‘what happens if the difference in metrics is 1%?’
 '''Experiment length months''' – ‘what happens if we wait 0.1 month (first line)?’

@@ Line 97: / Line 97: @@
 What else is exported:
 Delta – expected difference. ‘what happens if the difference in metrics is 1%?’
 Experiment length months – ‘what happens if we wait 0.1 month (first line)?’
-Sample size – these many users will we get if we wait 0.1 month with a monthly flow of 100 users   (note, users are divided equally into two groups, so if there is 100 monthly user flow, there will be 50-50 users in both groups).
+'''Experiment length months''' – ‘what happens if we wait 0.1 month (first line)?’
 Sample size power: how many users do we need according to pyhton function, if we need to detect 1% difference (and have some variation)
-Sample size Lehr’s rule: How many users do we need according to Lehr’s rule of thumb to detect 1% difference.
+'''Sample size''' – these many users will we get if we wait 0.1 month with a monthly flow of 100 users   (note, users are divided equally into two groups, so if there is 100 monthly user flow, there will be 50-50 users in both groups).
 Sample size formula: How many users do we need according to statistical formula from the article to detect 1% difference.
-Is random= 1 – is not random – what is a chance we will notice the difference in 0.1 month if the real difference in means is 1% (3.9% chance for the first line – really low)
+'''Sample size power''': how many users do we need according to pyhton function, if we need to detect 1% difference (and have some variation)

@@ Line 107: / Line 107: @@
 =Conclusions (business point of view)=
 )	All formulas give pretty much the same number of users. So any of them can be used.
 )	All formulas AND simulation display same number of users when it is 80% chance to detect a difference. That means, standard betta =20% is used in all the formulas.
 Here is how I know that:

@@ Line 80: / Line 80: @@
 For the line 153, 0.16% delta can be noticed in 1 month with 11% chance.
-[[File:Df_view.jpg|300px|]]
+[[File:Df_view.jpg|600px|]]
-[[File:Df_view2.jpg|300px|]]
+[[File:Df_view2.jpg|600px|]]
 % is low. Now analyst understands that either he should be expecting higher difference (which is hard to achieve by replacing 1 feature with another), or state to a product manager (owner of the feature) that there is no reason to conduct an experiment with existing limitations (monthly user flow, delta (expected difference)).
@@ Line 88: / Line 88: @@
 For me it is in a root folder:
-[[File:repo.jpg|400px|]]
+[[File:repo.jpg|600px|]]
 When downloading a file, the same table is visible, so it might be more convenient to navigate to the .csv file:
-[[File:csv_file.jpg|400px|]]
+[[File:csv_file.jpg|600px|]]
 What else is exported:
@@ Line 110: / Line 110: @@
 Here is how I know that:
 For the same parameters but monthly flow of 10 000 users (any huge number), the csv file generated the following statistics:
-  [[File:Resultofab.jpg|400px|]]
+  [[File:Resultofab.jpg|600px|]]