Web Site Measurement Hacks

Run Your Own Split-Path Tests

While there are a number of vendors providing split-path testing tools, relatively simple code will let you run your own tests.

While there are a handful of vendors that provide solutions and services for split-path testing—the practice of randomly showing visitors alternative pages or content to improve conversion—most charge between $25,000 and $100,000 annually for their services. However, if you have a reasonable amount of control over your web application platform, it’s not unreasonable to simply build the ability to distribute and track visitors right into your existing site.

Regardless of how you choose to implement split-path testing, the essence of a good strategy is as follows:

1. Check for test participation.

2. Assign the visitor to a test or control group.

3. Tag the visitor.

4. Redirect test subjects to the appropriate page.

5. Monitor for completion of goals.

Perhaps the most important is step 5: monitoring for the completion of goals. Keep in mind that split-path testing is a lightweight adaptation of the scientific method—you have a control group and one or more tests, and you’re looking for the differences between the two sets. In the case of a web site, the differences should be measured by an increase in leads collected, sales generated, pages viewed, etc., depending on your particular business objectives [Hack #38] .

The Code

The following code is written in VBScript for Microsoft’s Active Server Pages, although it could be quite easily adapted to PHP, Perl, or Java. You should save the following code as split-path_testing.inc and plan to include it in your header files (ideally, via a server-side include).

    <%
    '******************************************************************
    ' Define the tests to be performed.
    ' Tests are stored in an array. Be sure to dimension the
    ' array to the number of tests minus 1!@
    '*******************************************************************


   Dim Test(1) ' SET TO MAXIMUM NUMBER OF TESTS MINUS ONE
   Test(0) = "Home_Page_Test,index.asp,index_test.asp,12/1/2004,12/31/2004"
   Test(1) = "Buy_Now_Test,buy_now.asp,buy_now_test.asp,12/1/2004,12/11/2004"

   Dim SplitArray
   Dim TestName(20), TestFile(20), DefaultFile(20), TestStart(20), TestEnd(20)
   For i=0 to Ubound(Test)
   SplitArray = split(Test(i), ",")

       TestName(i) = SplitArray(0)
       DefaultFile(i) = SplitArray(1)
       TestFile(i) = SplitArray(2)
       TestStart(i) = SplitArray(3)
       TestEnd(i) = SplitArray(4)

	  Next
 
  ' Define a function called RandomNumber that will be used
  ' to assign visitors to the test or control group
  Function RandomNumber(intHighestNumber)
  Randomize
  RandomNumber = Int(Rnd * intHighestNumber) + 1
  End Function

 ' The bulk of the code first sets a few variables including the current date
 and the name of the || script currently being loaded in the visitor's browser
 (ScriptName):

 Response.Buffer = True
 ScriptName = Request.ServerVariables("SCRIPT_NAME")
 ThisDate = FormatDateTime(now(), vbShortDate)

 ' The rest of the code simply iterates through the tests already loaded into
 arrays to see if the page is one included in a test, if the visitor is already
 in a test group and if not, which
 ' test group they should be assigned to. As soon as the test assignment is
 made, the visitor is
 ' then redirected (if they've been determined to be part of a test) or nothing
 happens and the
 ' rest of the page is loaded

   For i = 0 to Ubound(Test) if InStr(ScriptName, DefaultFile(i)) then 
       if (CDate(ThisDate)  >= CDate(TestStart(i)) AND CDate(ThisDate) =< 
    CDate(TestEnd(i))) then
      if IsDate(Re quest.Cookies("TestCookie")(TestName(i))) then 
	Response.Redirect("./" & TestFile(i)  & "?TestGroup=" & TestName(i))
 elseif Request.Cookies("TestCookie")(TestName(i)) <> "CONTROL" then 
   Response.Cooki es("TestCookie").Domain = "www.webanalyticsdemystified.com"  
   Response.Cookies("TestC ookie").Expires = now() + 365 
' Here is where we assign the visitor to the test or contr ol group. 
' Note the TestGroup=[TestName] in the redirect—the critical piece to 
'  allow us to tag the visitor:
 
  if RandomNumber(100) > 50 then
      Response.Cookies("TestCookie")(TestName(i)) = ThisDate 
      Response.Redirect("./" & TestFile(i) & "?TestGroup=" & TestName(i))
  else 
      Response.Cookies("TestCookie")(TestName(i)) = "CONTROL"
end if end if
' If the end date for the test has expired, clean out the cookie so that
the test name can be reused if necessary:
 elseif CDate(ThisDate) > CDate(TestEnd(i)) then
  Response.Cookies("TestCookie")(TestName(i)) = ""

  end if
end if
next
%>

Check for Test Participation: Pages and People

You need a way to keep track of whether the viewed page is part of a test or not. To do, use a simple array to keep track of the name of the test (for your measurement application, a nice, readable name), the default and test filenames (physical script names), and the start and end dates for the test. Each element in the array holds information about an individual test.

   <%
   Dim Test(1) ' SET TO MAXIMUM NUMBER OF TESTS MINUS ONE
   Test(0) = "Home_Page_Test,index.asp,index_test.asp,12/1/2004,12/31/2004"
   Test(1) = "Buy_Now_Test,buy_now.asp,buy_now_test.asp,12/1/2004,12/11/2004"

For the most part, all you really need to know is that the format for the test definitions is important: you have to have the name of the test, followed by a comma, followed by the default filename (e.g., the control group), followed by a comma, the name of the test file, comma, the start date in MM/DD/ YYYY format, comma, and then the end date in MM/DD/YYYY format.

You also want to set a cookie in the visitor’s browser that lets you track his participation in your tests. The code to do this is very simple:

   Response.Cookies("TestCookie")(TestName(i)) = ThisDate

You create a cookie called TestCookie with a name/value pair where the name of the test (TestName(i)) equals today’s date (ThisDate). Then you’ll be able to check the cookie for a valid date to see if the visitor is participating in the test.

  if IsDate(Request.Cookies("TestCookie")(TestName(i))) then

Assign to a Test or Control Group

Assuming the visitor is not already participating in a test, the next step is to randomly assign him to the test group (so that he will see the test pages) or the control group (so that he will see the control pages). While there are many ways to do this, I like to use the simple “heads you’re in, tails you’re out” strategy:

  If RandomNumber(100) > 50 then

Basically, using a random-number generating function in VBScript, you generate a number between 1 and 100; 1 through 50 are in the control group and 51 to 100 are in the test group. Visitors are assigned to test or control groups on every tested page, in order to preserve random distribution throughout your site.

Tag the Visitor

If a visitor is going to be part of a test, you need to let your measurement application know this. The easiest way to do this is to modify the URL making the request for the test file so that either your logfile can be mined for the presence of a name/value pair like ?TestGroup=[TestName] or you can use this information to load a variable for your JavaScript page tag, e.g.:

  var _abTestGroup="TestName";

We do this by appending to the end of the test filename, contained in the TestFile(i) array variable.

 Response.Redirect("./" & TestFile(i) & "?TestGroup=" & TestName(i))

Warning

Note that if your test filename already has parameters in the query string, you’ll need to convert the “?” to an “&” for this redirection to work properly.

Keep in mind that the method you use to identify the test visitor will be defined based on which data source you’re using [Hack #3] and specific to the application you’re using.

Redirect Test Subjects to the Appropriate Page

As long as the TestFile(i) variable is properly set, the visitor is going to be redirected along to the test page and assigned membership to the appropriate group. One thing you want to keep track of is whether the distribution of visitors is roughly 50/50, based on our simple test assignment strategy. Monitor the traffic to both the control and test pages to make sure they’re receiving roughly the same number of page views; if they’re not, something may have gone wrong with the code.

Monitor for Completion of Goals

This is the single more important thing you’ll do with this code, though it has little or nothing to do with the code itself. You need to make sure that you’re using the ?TestGroup=TestName in the query string to let your measurement application know that this visitor has to be tracked as a separate visitor segment [Hack #48] or member of an A/B test. Again, how you make this assignment really depends on which data source and type of application you’re using, but a phone call to your vendor should yield a pretty simple explanation about how to do this.

Ideally, if you’re using a moderately powerful measurement application, you’ll then be able to see whether members of the test group are completing goals more frequently than those in the control group. This is important since it’s the only reason you would use the following code.

Running the Hack

The best way to run this hack is to make sure that split-path_testing.inc is included in your common header file using a server-side include.

  <!--#include file="split-path_testing.inc"-->

If you don’t have a common header file, I recommend you create one rather than adding this code to every page on your site. This is the most efficient way to make global changes to the array that defines the tests, including ending all of the tests if you need to.

Don’t forget, the most important thing I’m not explicitly showing you in the code is making sure that your measurement system knows about the tests. Getting test subjects into appropriate visitor segments and tracking those segments through the completion of goals is far and away the most important piece in split-path testing. Be sure and consult with your application vendor when you’re setting this up to capture as much useful information as possible.

Hacking the Hack

When you get really good at split-path testing, keep in mind that the code is already set up to let you test multiple pages at once. Because you’re building an array of tests, you can add as many as you’d like and really work hard to optimize your web site. The most important thing to keep in mind is that because arrays start at 0 rather than 1, you need to redimension the Test array to one less than the total number of tests you’re running or you’ll get a nasty ASP error.

For example, if you had 100 different tests, you would redimension the Test array to 99, and then add additional elements for each new test:

   Dim Test(99) ' SET TO MAXIMUM NUMBER OF TESTS MINUS ONE
   Test(0) = "Home_Page_Test,index.asp,index_test.asp,12/1/2004,12/31/2004"
   Test(1) = "Buy_Now_Test,buy_now.asp,buy_now_test.asp,12/1/2004,12/11/2004"
   Test(2) = "Splash_Test,splash.asp,splash_test.asp,12/1/2004,12/11/2004"
   …
   Test(99) = "Recommendation_Test,rec.asp,rec_test.asp,12/1/2004,12/11/2004"

Finally, know that there is such a thing as “test overkill” and if too many visitors pass through too many tests, you won’t know for sure which tests were working. I recommend running only three or four tests at a time, trying to isolate them in the site if possible. This strategy allows you to quickly test your ideas while not creating too much data to be analyzed reasonably.