Mobile App A/B Testing

Mobile App A/B Testing gives you conversion insight for two or more variants of an element and helps you in taking a decision about which one is better or which variant has a higher conversion rate. For example if you have two different Registration UIs A and B in your app and you are not sure which one will bring in more registrations, you could use A/B Testing to figure this out. You can start with both these UI options and distribute these randomly to your app users. A/B Testing will objectively derive success rates for both the UI variants and you should be able to know which one has a higher conversion rate. If UI variant A has a higher conversion rate, you would like to make it a winner. Once A is made the winner, all the traffic will be directed to A and you will have the better performing element served in your app.

To start with A/B testing in your app with App42 platform, here are the steps

Login to AppHQ.

Click on Utility > A/B Test from AppHQ Menu and select your app

Click on Create Test button and Enter Test Name and other information. For example ‘Register Test’ is the Test Name for example explained here.

Once your test has been created you can create your test variants.

Click on the Add Variants button to add your variant name and % of traffic that should be served to this test variant. In this case let’s say, we have two registration variant A and B and each have traffic distribution of 50% each.

Click on save changes button.

Now click on Add Variable button to add profile information in a key-value format that you want to attach with this variant. For example if Register screen A has color red, you can save the key as color and value as red in the profile of variant A. Similarly you can add other profile information for both variants if you want.

Once you are done with creation of test and its variants, call the following methods from your app to execute and see the result.

Execute Test

To execute a given test, one has to call execute method inside ABTestService by passing the test name. This method will return an appropriate variant based on the distribution rule defined. For example, If A and B have 50 % distribution each, in the first call this will return variant A and its profile info, and in the second execution it will return variant B and its profile info. Based on the response received, you can display the UI you need and populate it based on its profile information as shown below in the snippet

• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 

String testName="<Enter TestName>"; 
abTestService.execute(testName, new App42CallBack() {
public void onSuccess(Object response) 
{
	ABTest abTest  = (ABTest)response;
	System.out.println("Test Name  is :  "+abTest.getName());
	System.out.println("Type is :" +abTest.getType());
	System.out.println("Is Active : "+abTest.isActive());
	for(int i=0;i<abTest.getVariantList().size();i++)
	{
		System.out.println("Variant Name : "+abTest.getVariantList().get(i).getName());
		System.out.println("get profile list : "+abTest.getVariantList().get(i).getProfileJSON());	
	}
}
public void onException(Exception ex) 
{
	System.out.println("Exception Message"+ex.getMessage());
}
});

Coming Soon 

public class Callback : App42Callback  
{  
String testName="<Enter TestName>"; 
abTestService.Execute(testName, this);
	void App42Callback.OnException(App42Exception exception)  
	{  
		Console.WriteLine("Exception Message" + exception);  
	}  
	void App42Callback.OnSuccess(Object response)  
	{  
		ABTest abTest = (ABTest) response;  
		Console.WriteLine("Test Name  is :  " + abTest.GetName());
		Console.WriteLine("Type is :" + abTest.GetType());
		Console.WriteLine("Is Active : " + abTest.IsActive());
		for (int i = 0; i < abTest.GetVariantList().Count; i++)
		{
			Console.WriteLine("Name  : " + abTest.GetVariantList()[i].GetName());
			Console.WriteLine("Get profile list : " + abTest.GetVariantList()[i].GetProfileJSON());
		}
	}  
}   

String testName="<Enter TestName>"; 
ABTest abTest=abTestService.execute(testName);
System.out.println("Test Name  is :  "+ abTest.getName());
System.out.println("Type is :" + abTest.getType());
System.out.println("Is Active : "+ abTest.isActive());
Vector variantList = abTest.getVariantList();
for(int i=0;i<variantList.size();i++)
{
	System.out.println("Name : " + ((ABTest.Variant)variantList.elementAt(i)).getName());
	System.out.println("get profile list : "+ ((ABTest.Variant)variantList.elementAt(i)).getProfileJSON());	
}  

var testName = "<Enter TestName>",
abTestObj;
abTestService.execute(testName,{  
	success: function(object) 
	{  
		var result = JSON.parse(object);  
		abTestObj = result.app42.response.abtest;
        console.log("Test Name : " + abTestObj.name);
		console.log("Variant Name is : " + abTestObj.variant.name);
		console.log("ProfileJSON is : " + JSON.stringify(abTestObj.variant.profile));
	},  
	error: function(error) 
	{ 
		console.log("Exception is : " + error)
	}  
});     

String testName="<Enter TestName>"; 
ABTest abTest=abTestService.execute(testName);
System.out.println("Test Name  is :  "+abTest.getName());
System.out.println("Type is :" +abTest.getType());
System.out.println("Is Active : "+abTest.isActive());
for(int i=0;i<abTest.getVariantList().size();i++)
{
	System.out.println("Variant Name : "+abTest.getVariantList().get(i).getName());
	System.out.println("Get profile list : "+abTest.getVariantList().get(i).getProfileJSON());	
}  

String testName="<Enter TestName>"; 
App42Log.SetDebug(true);        //Prints output in your editor console
abTestService.Execute(testName, new UnityCallBack());
public class UnityCallBack : App42CallBack
{
    public void OnSuccess(object response)
    {
       ABTest abTest = (ABTest)response;
       App42Log.Console("Test Name : " + abTest.GetName());
       for (int i=0; i < abTest.GetVariantList().Count; i++) 
		{
			 App42Log.Console("Variant Name : " + abTest.GetVariantList()[i].GetName());
			 App42Log.Console("ProfileJSON  : " + abTest.GetVariantList()[i].GetProfileJSON());
		}
	}

	public void OnException(Exception e)
	{
		App42Log.Console("Exception : " + e);
	}
}

String testName="<Enter TestName>"; 
ABTest abTest = abTestService.Execute(testName);
Console.WriteLine("Test Name  is :  " + abTest.GetName());
Console.WriteLine("Type is :" + abTest.GetType());
Console.WriteLine("Is Active : " + abTest.IsActive());
for(int i=0; i < abTest.getVariantList().Count; i++)
{
	Console.WriteLine("Variant Name : "+abTest.GetVariantList()[i].GetName());
	Console.WriteLine("Get profile list : "+abTest.GetVariantList()[i].GetProfileJSON());	
}  

Not Available

Not Available

var testName:String="<Enter TestName>";
abTestService.execute(testName,new callback());
class callback implements App42CallBack
{	
	public function onException(exception:App42Exception):void
	{
		trace("Exception Message " + exception);
	}
	
	public function onSuccess(response:Object):void
	{					
		if(response is ABTest)
		{		
			var abtest:ABTest = ABTest(response);
			trace("Test Name  is :  "+abtest.getName());
			trace("Type is :" +abtest.getType());
			trace("Is Active : "+abtest.IsActive());
			for(var i:int=0;i<abtest.getVariantList().length;i++)
			{
				trace("Variant Name : "+abtest.getVariantList()[i].getName());	
				var object:Object = com.adobe.serialization.json.JSON.decode(Variant(abtest.getVariantList()[i]).getProfileJSON());
				trace("get profile list : "+Util.toString(object));
			}						
		}   	
	}
}

Not Available

Tracking Conversion/Goal Reached

If the user takes desired action in the shown variant version, goalReached method has to be called to track the conversion. For example, if the user has done registration from variant A screen, it should consider as conversion and goalReached method needs to get called in this event by passing the variant name (A in this case). Snippet for the same is shown below.

• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 

String testName="<Enter TestName>";
String variantName="<Name Of the variant>";
abTestService.goalAchieved(testName,variantName,new App42CallBack() {
public void onSuccess(Object response)
{
	ABTest abTest  = (ABTest)response;
	System.out.println("Test Name  is :  "+abTest.getName());
	System.out.println("Type is :" +abTest.getType());
	System.out.println("Is Active : "+abTest.isActive());
	for(int i=0;i<abTest.getVariantList().size();i++)
	{
		System.out.println("Variant Name : "+abTest.getVariantList().get(i).getName());
		System.out.println("get profile list : "+abTest.getVariantList().get(i).getProfileJSON());	
	}
}
public void onException(Exception ex)
{
	System.out.println("Exception Message " + ex.getMessage());
}
});  

Coming Soon

public class Callback : App42Callback  
{  
String testName="<Enter TestName>"; 
String variantName="<Name Of the variant>";
abTestService.GoalAchieved(testName, variantName, this);
	void App42Callback.OnException(App42Exception exception)  
	{  
		Console.WriteLine("Exception Message" + exception.GetMessage());  
	}  
	void App42Callback.OnSuccess(Object response)  
	{  
		ABTest abTest = (ABTest) response;  
		Console.WriteLine("Test Name  is :  " + abTest.GetName());
		Console.WriteLine("Type is :" + abTest.GetType());
		Console.WriteLine("Is Active : " + abTest.IsActive());
		for (int i = 0; i < abTest.GetVariantList().Count; i++)
		{
			Console.WriteLine("Name  : " + abTest.GetVariantList()[i].GetName());
			Console.WriteLine("Get profile list : " + abTest.GetVariantList()[i].GetProfileJSON());
		}
	}  
} 

String testName="<Enter TestName>"; 
String variantName="<Name Of the variant>";
ABTest abTest = abTestService.goalAchieved(testName, variantName);
System.out.println("Test Name  is :  "+ abTest.getName());
System.out.println("Type is :" + abTest.getType());
System.out.println("Is Active : "+ abTest.isActive());
Vector variantList = abTest.getVariantList();
for(int i=0;i<variantList.size();i++)
{
	System.out.println("Name " + ((ABTest.Variant)variantList.elementAt(i)).getName());
	System.out.println("Get profile list : "+ ((ABTest.Variant)variantList.elementAt(i)).getProfileJSON());	
}      

var testName = "<Enter TestName>",
variantName="<Name Of the variant>",
abTestObj;
abTestService.goalAchieved(testName, variantName,{  
	success: function(object) 
	{  
		var result = JSON.parse(object);  
		abTestObj = result.app42.response.abtest;
        console.log("Test Name : " + abTestObj.name);
		console.log("Variant Name is : " + abTestObj.variant.name);
		console.log("ProfileJSON is : " + JSON.stringify(abTestObj.variant.profile));
	},  
	error: function(error) 
	{ 
		console.log("Exception is : " + error)
	}  
});     

String testName="<Enter TestName>";
String variantName="<Name Of the variant>";
ABTest abTest=abTestService.goalAchieved(testName, variantName);
System.out.println("Test Name  is :  "+abTest.getName());
System.out.println("Type is :" +abTest.getType());
System.out.println("Is Active : "+abTest.isActive());
for(int i=0;i<abTest.getVariantList().size();i++)
{
	System.out.println("Variant Name : "+abTest.getVariantList().get(i).getName());
	System.out.println("Get profile list : "+abTest.getVariantList().get(i).getProfileJSON());	
}  

String testName="<Enter TestName>"; 
String variantName="<Name Of the variant>";
App42Log.SetDebug(true);        //Prints output in your editor console
abTestService.GoalAchieved(testName, variantName, new UnityCallBack());
public class UnityCallBack : App42CallBack
{
    public void OnSuccess(object response)
    {
       ABTest abTest = (ABTest)response;
       App42Log.Console("Test Name : " + abTest.GetName());
       for (int i=0; i < abTest.GetVariantList().Count; i++) 
		{
			 App42Log.Console("Variant Name : " + abTest.GetVariantList()[i].GetName());
			 App42Log.Console("ProfileJSON  : " + abTest.GetVariantList()[i].GetProfileJSON());
		}
	}

	public void OnException(Exception e)
	{
		App42Log.Console("Exception : " + e);
	}
}

String testName="<Enter TestName>"; 
String variantName="<Name Of the variant>";
ABTest abTest=abTestService.GoalAchieved(testName, variantName);
Console.WriteLine("Test Name  is :  " + abTest.GetName());
Console.WriteLine("Type is :" + abTest.GetType());
Console.WriteLine("Is Active : " + abTest.IsActive());
for(int i=0; i < abTest.getVariantList().Count; i++)
{
	Console.WriteLine("Variant Name : "+abTest.GetVariantList()[i].GetName());
	Console.WriteLine("Get profile list : "+abTest.GetVariantList()[i].GetProfileJSON());	
}         

Not Available

Not Available

var testName:String="<Enter TestName>";
var variantName:String="<Name Of the variant>";
abTestService.goalAchieved(testName, variantName, new callback());
class callback implements App42CallBack
{	
	public function onException(exception:App42Exception):void
	{
		trace("Exception Message " + exception);
	}
	
	public function onSuccess(response:Object):void
	{					
		if(response is ABTest)
		{		
			var abtest:ABTest = ABTest(response);
			trace("Test Name  is :  "+abtest.getName());
			trace("Type is :" +abtest.getType());
			trace("Is Active : "+abtest.IsActive());
			for(var i:int=0;i<abtest.getVariantList().length;i++)
			{
				trace("Variant Name : "+abtest.getVariantList()[i].getName());	
				var object:Object = com.adobe.serialization.json.JSON.decode(Variant(abtest.getVariantList()[i]).getProfileJSON());
				trace("get profile list : "+Util.toString(object));
			}						
		}   	
	}
}

Not Available

View Report and Make Winner

You can analyze the conversion of each variant in the dashboard as shown below. This report will help you in declaring a winner among A and B. Variant which has the highest conversion ratio is the recommended winner. You can make a variant winner by selecting the radio button against that particular variant. Once a variant is selected as the winner, distribution for that variant would be 100% and all the requests will be served to that variant only. For example if A is a winner in this case, execute the method will always return variant A as execution result.

More Details?

Go To ABTest API Usage Guide