Testing To Make IM Fast and Simple

I’ve pointed out that testing is a time-honored direct marketing technique that has taken on greater capabilities on the Internet. Several important items have drifted into my inbox over the past couple of weeks and I’d like to bring them together. Marketers need to be testing and there are an increasing number of services to help them deal with the complexities of doing reliable tests.

Many of you are familiar with the work of Avinash Kaushik, his blog and his book on Internet metrics. Recently he was interviewed by Brian Eisenberg of FutureNow and Grok.com, who has just written a book on testing. It’s a 30-minute interview (access from this page) but worth playing in the background while you read your mail.

Let me give you a brief overview. Kaushik emphasizes how fast the web moves and how flexible marketers have to be to keep up with it. The weeks and months it took to read the old direct mail tests just don’t cut it on the web. What the marketer needs to be able to do is:

• Encourage a flow of ideas and validate the good ones
• Lower the risk of making marketing decisions
• Do it in a way that’s fast and simple.

Testing is the key first step. Success comes from superb execution.

My last post on testing generated two comments that contribute to the discussion. They are from two marketing services firms that appear to serve different market segments with testing products. Both commenters basically agree that Google Website Optimizer may be a good way to get started in testing, but it doesn’t produce results that have a high level of statistical confidence.

Widemile is a service that targets agencies needing testing services for their clients. A lot of firms/a lot of agencies don’t have the in-house expertise. They need services and Widemile concentrates on the optimization of landing pages. They have a brief demo; check it out. Billy Shih has an informative blog that focuses on testing.


SiteSpec appears to serve clients directly with testing services that include A/B splits and multivariate tests. It has an interesting platform, well explained in a demo. There’s also a video on their home page.

Testing is important, but most of us need help. That includes knowing the right questions to ask and the kind of results to demand. Here are some good resources to help you upgrade your own knowledge about this key avenue to Internet marketing success.

Testing for Marketing ROI

Today seems to be testing day in marketing newsletters. I read an article in today’s SearchBuzz newsletter about multivariate tests and thought I should check out the tools. When I wrote about testing earlier I emphasized A/B splits (bivariate) because I think that’s an easier way to start. While I was working on that, along came this week’s chart from Marketing Sherpa that does a nice job of summarizing some of the strategy issues. But do you need a big budget to test?

In the earlier post I mentioned Offermatica (recently acquired by Omniture) as a testing tool. I really like their demos of both A/B splits and multivariate tests. They’re just a bit harder to find now. You have to look on Omniture’s product tours page and select Test & Target. But this is a hosted service, not DIY.

But the tools I wanted to take a look at are Google Website Optimizer and Widemile's Page Optimizer. Google’s is free. Google, of course, has a video.

Widemile’s business model is to provide technology to agencies and marketing services firms so they can offer testing services to their clients. They have a simple summary that describes the efficiency of A/B split testing against multivariate testing. You can test many variables at once when you use experimental designs (what we refer to as multivariate testing). That seems like stating the obvious. However, what may not be so obvious is that multivariate tests are much harder to “read.” The real question is, “Are these small differences statistically significant?” As the conceptual examples in my earlier post showed, that takes you right back to your statistics courses. And if you’re like me, you absorbed the fundamentals represented in the text box that starts “Choose the variables to be tested.” But you may share my reluctance to base serious marketing decisions on my personal statistical skills and my concern that the automated tools won't produce the right answers. Automated testing products need to take into account necessary sample sizes for statistical significance. They don’t want to burden users with that type of statistical detail (yes, I understand, it’s a bit scary), so you need to ask questions about statistical significance and confidence levels, even if it means you have to drag out an old stats textbook.

Since Google’s tool is DIY—and they don’t particularly want to talk about the significance issue either—I wondered how it was dealt with, if at all. They have a good blog that is well written and may help with some of your concerns. One of the entries I found was for a sample size estimation tool from one of their partners, a site called WebShare. If you followed my advice in the earlier post and read Paul Berger’s chapter on testing in our (free download) direct marketing text, you’ll find it follows the basic steps, although there are still issues. Specifically, it’s a one-tailed test; see the Improvement call-out on the tool page. It’s only looking for better results; not whether the results are either better or worse. That’s probably realistic for most marketing tests. They give you a nice comparison chart—various % improvement, various significance levels. Try it for yourself!

Most of us have a lot of questions about what will work best in our marketing programs. A/B splits will answer a few of these questions for us. Multivariate tests will answer many more questions in the same time period. Whether you are using marketing services of one kind or another, or whether you’re going to try it DIY, do it with care. I am reminded of the Mark Twain quote:

“It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so.”

Don’t accept insignificant results. And remember the old direct marketing rule to keep testing. If it’s an important marketing decision one test, especially one with a small sample size, is just not enough.

Social Landing Pages?

I noticed this term a few days ago in Ion Interactive’s newsletter and thought it was interesting. Landing pages are an important—and too often ignored—part of the conversion process. Improving them is a subject that should always be of interest. So, as usual, I set out to explore.

The first thing I found out is that the term doesn’t seem to be in use on sites like Grokdot.com, which specializes in conversion issues, or Omniture, which touts optimization of various aspects of emarketing. What I did find was a number of services that help marketers test their landing pages. Testing is always good, whether the pages are social or not.

So I went back to Ion to find their definition. Content from their newsletter (July 08) answered that:

Subscriber question: What is a social landing page?
Answer: Social landing pages are landing pages that include social networking features. They bring together marketer-managed marketing and social interactivity.

In the news release for their LiveBall product, Ion’s president Scott Brinker adds, “A landing page can serve as a ‘connector’, bringing together people with a specialized, shared interest.” The press release goes on to describe 7 benefits of adding social features to a landing page that include recipient interaction, brand development and lower bounce rate.

These are interesting, but I wanted some examples. Their LiveBall product demo is interesting, although it stopped working about half way through the second step. It makes it clear, though, that the Ion product is marketer-controlled, and I’m always a fan of DYI.

But I’m still looking for examples, so I did something I don’t usually do. I clicked on some PPC ads. (I’m such a considerate customer that I always use the free search listing link if it’s available.) I’ve used the Vertical Response product and like the company, so I looked to see what they had and found a good landing page. After clicking on several banners on several portal sites I found an HSBC ad on the NYT site that had a good landing page.

The search alone is instructive. A lot of companies are paying for advertising and not using landing pages associated with their ads. Some of the ads took me to product pages, which were at least relevant. Others just took me to a home page. Shameful!

Take a look at what I did find. I’m showing the landing page and the home page. Both landing pages have URLs that prove my point, but I’m showing the home page anyway to make the point that the landing pages take visitors more directly into relevant product information than do home pages.

The Vertical Response page is long, so I cut in in half and put it back together so you could see it all. It’s not particularly creative, but it gets their Create > Send > Track message across, has some promotional content, presents a compelling offer (500 free emails), and lets the viewer sign up directly on the landing page. Maybe not “creative,” but it really does the job. You could get to sign up from the home page, but you’d have to click through to the email page, where you’d find much of the same information, but an offer for only your first 25 emails (small list!). Vertical Response clearly has a landing page that’s meant to maximize the conversions from its PPC advertising.

HSBC’s term life insurance landing page (if you could see the URL, you’d see they call it the “simplestart” page) is even more interactive. It allows you to select profile characteristics and request a quote, again, right from the landing page. HSBC is a more complex enterprise and it’s home page represents that. It’s insurance main page is remarkably unattractive, but it allows you to get quotes for a variety of insurance, including term, which was featured when I wrote this post, presumably to integrate with the current banner advertising campaign.

Neither of these had “social” components in the social networking sense, although other components are possible. Ion’s blogs, for example, cover issues like use of widgets in online advertising and news feeds on landing pages.

So, while I’d call these landing pages “interactive,” instead of “social,” and there’s a large frontier to be explored either way. As you explore, don’t forget to test every step of the way. There’s so much a marketer can do; the question should always be “what should I do in order to be successful?” Test and find out!

Testing Critical to Internet Marketing Success

Last week I wrote about diy display advertising and pointed out that all interactive advertising and promotion offers marketers a splendid opportunity to test various approaches. That can include many marketing issues from competing email subject lines to competing messages or creative in a display ad to alternative landing pages and more.

Testing is a fundamental technique of direct marketing, but mass media marketers rarely had the opportunity to test. They have been slow to acquire the skill as some of their marketing has moved onto the web. Marketing programs can be much improved by skilled testing; it’s a skill that needs to be acquired.

In fact, many marketers are foggy about the difference between marketing research and testing. Both have their place in a well-run marketing program. However, in an interactive environment, I argue strongly that testing should come first. There are several reasons:
•Testing answers the question, “what works?” and gives a strong basis for good marketing decisions.
•Testing is faster.
•Testing is cheaper.

Testing, however, cannot answer the “why” questions--why customers think and do what they do. That’s where marketing research comes in. It’s also the reason that I argue to test first and find out what works, then ask “why” if and only if you need the answer to improve decision making.

Testing also requires a certain degree of technical skill to ensure the test is run properly and read correctly. In the earlier post I recommended Paul Berger’s chapter on testing in our direct marketing book. Paul is a superb statistician and his approach is based on classical hypothesis testing. You may find the outline familiar from your own statistics course. I developed a companion concept for my Internet marketing text that emphasis the managerial aspects of testing. The truth is that both perspectives are key to conducting valid tests.

Let me give just one example—the free membership trial from the Marketing Sherpa site. The test was conducted by Marketing Experiments, and you can access commentary and the full presentation via their blog. This test is a simple A/B split in which existing sign-up page A is tested against new sign-up page B. In this case A is the sign-up page currently in use, so it becomes the control (baseline) and the question is, “Does B work better than A?” A control (no treatment in experimentation parlance) is not strictly required in an A/B split. Then the question is, “Does A or B work better?”

In this case the control included a fairly generic paragraph about the benefits of membership and requested a minimal amount of information for signup. The treatment page included a feature matrix that detailed the content available free to newsletter subscribers and additional content available to members of Marketing Sherpa. The information required was the same; the difference was in the amount of information given to the potential registrant.

The test was run over a 30-day period and the metric used was conversion rate. More information did work, in spite of the fact that it led to a longer, more complex page. It improved conversion by more than half a percent, which—given generally low conversion rates—is huge.

What Marketing Sherpa has done with it is interesting. Instead of simply using the features listing, it has a page for each feature as part of the process. That would be deadly if you had to go all the way through in order to sign up for the free trial membership. The sign-up box is on each page, so the minute the visitor is convinced, she can enter her information. If not yet convinced, she goes on to the next page. Clever!

I’d encourage you to view you can access Dr. McGlaughlin’s entire presentation. It gives two other good examples. You can also find testing articles on GrokDotCom and examples of various types of testing on Omniture’s Offermatica site.

Whether the marketer uses marketing support services or chooses to develop testing expertise in house, it is an essential skill for the Internet age.