Online Testing: The Engine Behind Integration

AUDIENCE DEVELOPMENT

By Karlene Lukovitz

Monday, March 2, 2009

 

 

Even as the push to create integrated business models revolutionizes the publishing industry, some core dynamics don’t change. One of those is the fundamental importance of consumer marketing testing.

 

Testing is, of course, the foundation of all successful direct marketing, including traditional circulation marketing, but it’s become higher-profile with the pressing need to maximize online-generated revenue and audience reach.

Obviously, achieving a more profitable print circulation source mix is one core objective behind the emphasis on online testing. In rough terms, online-generated print subs can be five to 10 times more profitable than their direct mail-generated counterparts, according to audience developers.

 

The increasingly critical audience demographics factor is also behind the growing support of the testing needed to maximize online consumer touchpoints and revenue. Forming relationships with younger consumers online is not only key to developing a pool of future readers for print magazines, but building the audiences needed to compete for online and integrated program media advertising spending.

 

Every touchpoint, from e-newsletters to social network presence, has its own dynamics, and each requires testing not only to nail down the essentials but to optimize results by continually honing the marketing elements—offers, messaging, creative and navigation—or the enews or other content being offered. At the same time, the interfaces among online touchpoints, as well as among offline and online touchpoints, require testing to determine how channels combine to influence consumer behavior and how these synergies can be leveraged.

 

Roll these together, and it’s not hard to understand what’s driving the furious pace of online testing at some leading publishers.

Pushing the Online Envelope at Hearst

Hearst Magazines is among the most aggressive publishers on the online testing front, and the push is paying off.

EVP/general manager John Loughlin recently reported that online channels generated more than 2.2 million print subscriptions last year—making it Hearst’s top source—and confirmed that the costs are one-quarter to one-tenth of those for direct mail-generated subs. Hearst is “finding a huge new universe of consumers willing to buy magazines and techniques that are much more cost-effective,” Loughlin summed up. Moreover, the average age of online-generated subscribers is 8 to 10 years younger than the existing subscriber base.

 

When it comes to testing, Hearst’s mantra is that every day is a new opportunity to improve online results, according to Chris Wilkes, VP of the marketing and audience development group. “It’s all about keeping the marketing fresh,” says Wilkes. “We’re constantly testing different prices, marketing messages and creative, tracking the results and making changes accordingly on a daily basis.”

 

Whereas direct mail testing results in control packages that will generally continue to yield fairly predictable response and volumes for some time, that’s not the case online, Wilkes points out. “Online lifts can sometimes be several times the magnitude of those seen in direct mail, but they tend to hold for only a short period of time,” he explains. “Sometimes they’ll hold for a week or two, but not infrequently, the response differential is already starting to erode by the time the test is over.”

 

In lieu of the consistency or predictability of direct mail, Hearst’s online strategy focuses on relentless optimization: Continual testing and changes that result in frequent lifts, many of which may be relatively small, but add up to substantial gains over time.

 

Testing efforts extend across all of Hearst’s online channels. In terms of subscription volumes generated, these break down to about one-third each from Hearst’s branded magazine Web sites; internal marketing assets including email lists and an expanding network of affinity, non-magazine branded sites; and external digital marketing and partnerships.

While subscriptions generated through Hearst sites yield the most profitable subs on a cost-per-order basis, external partnerships represent the greatest volume growth potential, according to Wilkes. In this realm, partner testing comes into play up front, as a means of determining whether the potential partner’s channel can generate Hearst’s required volume and profitability thresholds. Once a partnership is formed, testing to optimize the offer and creative are conducted.

 

A critical key to Hearst’s growing success online is its investment over the past year in building out a relational database that incorporates the contact, transaction and other data for every individual with which Hearst has any touchpoint. The database includes all consumer-supplied postal and email addresses, including those provided off-line and online for magazine subscription orders (including expire data) and those provided through registrations (including enewsletters and sweepstakes registrations). Email addresses, driven increasingly by magazine-site traffic originating through the network of non-magazine sites, were at more than 25 million as of 2008.

 

The database is tied directly into Hearst’s custom content and promotion management system and the CDS fulfillment system to enable what Wilkes terms “identified marketing,” including auto-prompted offers that reflect the consumer’s specific scenario. For example, when an existing subscriber who’s previously provided an email address visits a Hearst Web site, the system uses custom targeting criteria to generate an appropriate offer, such as an incentive to pay now online for someone who’s in the billing cycle, or a renewal offer for someone close to expire. If it’s a renewal offer, and the subscriber is also being marketed offline, he or she will see the same offer received in the mail. Email marketing messages can also be tailored to reflect the individual’s touchpoints with the brand and transactional history.

The 10-member centralized digital team led by Wilkes and the traditional consumer marketing teams work collaboratively to share testing ideas and learning from offline and online tests that can be cross-leveraged.

 

Increasingly, offline marketing efforts include incentives, such as premiums or sweeps, to go online to place a subscription order. Such efforts yielded more than 150,000 paid subscriptions last year, and this year’s plan calls for increasing the volume by at least 20 percent. And among many other benefits, the database will facilitate the ability to conduct various types of offline-to-online testing efforts, including more testing of combining email and direct mail marketing—such as sending an email alert about an offer that will be arriving in the mail.

 

Online testing is by no means a large publisher-only game. Publishers of all sizes are taking advantage of its lower costs and speed to yield meaningful incremental bottom-line gains.

 

Here’s a small sampling of the types of subscription testing being conducted by various publishers:

 

*Web site testing: Key testing areas include increasing the number of subscription order ads and opportunities on the home page and pages throughout the site (testing results help consumer marketers justify getting more online “real estate”), as well as testing offer pricing/terms, premiums/incentives, ad creative, landing or order pages, and “rescue pages”—messages generated when a prospect is about to abandon the order process.

 

In addition to testing formats ranging from buttons to banners to full-blown ads, one big focus within creative testing is incorporating more interactive elements within sub promotions. Bonnier Corp.’s Popular Science has been testing including a “flipbook”—a digital issue that can be flipped through to “see what’s inside an issue” of Pop Sci—within a prominent, embedded subscription order form on its site’s home page. Tests of a wide variety of interactive ad formats such as scroll-overs and roll-overs are planned for various Bonnier titles in the year ahead, according to marketing manager Cliff Sabbag.

 

One great advantage of online testing is that, assuming that coding has been done correctly, behavior can be tracked to determine the strengths and weaknesses of each element in the conversion process. Monitoring click-throughs versus conversion rates enables marketers to see, for example, that a sub ad is generating high click-throughs, but low conversion rates—in which case, the problem is likely the offer being shown on the landing page or the functionality of the landing page, calling for more testing in this area, notes Michelle Murphy, senior marketing manager for American Express Publishing.

 

The ability to renew on site, as well as order gifts, are standard, and the same core types of testing go into enhancing click-throughs and conversion rates.

 

*Email: Acquisition testing spans pricing/offers, incentives/premiums, creative, and determining optimum frequencies or number of efforts.

 

On the frequency front, “There’s no one answer to this question, and I think that most publishers are in the process of figuring out what works for their audiences,” notes Jessica Morin, also a senior marketing manager at American Express Publishing. Testing frequency and managing opt-out levels is one key focus for the company within the email arena this year, she reports.

 

Testing at Playboy Enterprises has shown that re-emailing to prospects that opened an email but did not respond significantly increases conversions, according to SVP, circulation Phyllis Rotunno.

 

Email gift series are an important testing focus, as are e-renewal and e-billing series.

 

As publishers build the number of names for which they have both email and postal addresses (not only through online registrations for content or incentives, but offering incentives on direct mail pieces to provide email addresses as part of sub orders), using testing to coordinate email and direct mail efforts to increase overall response is becoming an increasingly promising area.

 

One major publisher saw significant lifts in tests in which prospects received an email alert about a subscription offer coming in the mail, then a follow-up booster email timed to arrive after the package (“By now, you should have received your special offer…”). Using email acknowledgements of subscriptions generated through direct mail (where email addresses are on file) or offline-to-online helps boost pay-up and provides opportunities for testing cross-sells and upsells.

 

Renewals are a prime opportunity to use testing to maximize the synergy between email and direct mail. Rodale, for example, uses e-renewal efforts as complements to print efforts, targeting/timing them to enhance response where less profitable direct mail efforts occur in a series, according to senior consumer marketing manager Kimberly Draves. Rodale uses the email or online “space” to promote additional points of value that are harder to market profitably in print, she explained at this year’s DMA Circulation Day. Metrics used for tracking success include email address volume and penetration, online transaction growth, and email delivery, open, click-on-open and response benchmarks.

 

*Search: Optimizing search to drive Web traffic and clicks into subscription marketing forms is an entire testing world unto itself—in fact, two worlds: Organic and pay-per-click optimization. And as Keith Kochberg of online marketing agency iMarketing Ltd. points out, many publishers are finding it highly effective to funnel a portion of search-generated traffic to tests of everything from landing pages to premiums.

 

Testing to optimize the cost-effectiveness of pay-per-click is increasingly important as publishers move into this arena to a much greater degree. As with direct mail, pay-per-click search costs are controllable, since the marketer determines how much he or she is willing to spend per click. On the other hand, competitive bidding means that the cost for a given keyword can vary day by day. “Success builds on itself, because the greater your conversion rate, the more you can afford to pay for that click, and the better your ad positioning becomes,” points out Kochberg.

 

A downside of search engine marketing from a testing standpoint is that the universe is obviously limited to the number of people who conduct searches on a given set of keywords and click through to a magazine’s site (presumably optimized by keyword testing). Because traffic volumes vary significantly by magazine, one of the fundamental do’s in SEM testing is to end the test based on having achieved a meaningful number of impressions, clicks or conversions rather than running a test for a predetermined timeframe, according to Kochberg.

 

True A/B or split testing—in which a new element is tested against the control (or another competing offer/creative presentation) in a test design that enables statistical significance or reliability of the results differential at a high confidence level—is standard procedure in direct mail.

 

In fact, employing test designs that enable statistically valid, reliable testing of multiple variables (A/B/C/D, etc.), sometimes called “grid testing,” is also standard procedure in the direct mail world.

 

 

In online testing, things are not so clear-cut. In theory, A/B testing should be a cinch online, with marketers simply instructing the programs or systems to generate or rotate the two variations being tested and automatically shut off the test when the needed number of impressions has been reached for each.

 

In practice, in many companies, a number of realities come into play that tend to limit the amount of true A/B testing that’s possible, at least at this stage. The pace and volume of online testing (in part reflecting the fleeting nature of the lifts generated in many cases), the need to prioritize staff and budget resources (within online and/or offline sources), and systems limitations can all make the time and work involved in setting up and analyzing such tests impractical on an every-test basis.

 

Instead, many audience developers, even at sizable companies, report that A/B or classic test designs are used, at least somewhat selectively, for testing areas or elements that have potential for generating substantial lifts and/or are likely to generate learning that contributes to core knowledge or can be leveraged across sources or brands.

One marketer, for instance, cites the example of an A/B premium test that resulted in including the winning premium in many efforts within the marketing mix. Another reports that within the email testing area, A/B is used mainly for testing subject lines—an element that has historically been shown to drive significant differences in response.

 

However, Bill Baird of Baird Direct Marketing, an online subscription marketing consultancy, reports that his site tracking across the industry has been indicating an upswing in A/B tests.

 

Publishers who want to do A/B also have the option of using outside agencies that offer expertise and turnkey programs and systems solutions.

 

On the next testing level is multivariate testing, which uses statistical modeling/regression techniques to enable simultaneous valid testing of multiple variables. This technique provides a way to accelerate learning. Sources agree that using an experienced multivariate testing service provider is nearly always the best way to go in setting up and conducting such tests.

 

Baird reports that site scans confirm that a number of larger publishers are using multivariate testing, but mainly on a selective or limited time basis. “I saw a definite shift among larger publishers toward multivariate testing last year, but they now seem to be reducing their usage or taking a break,” Baird says. “The general pattern over time seems to be to use this type of testing for a big project, and return to it again some time later, presumably for another special or large project.”

 

The reasons for this pattern? Although multivariate testing is clearly valuable—case studies of huge improvements in response and profitability resulting from use of this technique are routinely reported in online marketing sites—it can be labor-intensive. Such testing requires creation of multiple creative executions, and tends to produce results that require further multivariate testing, notes Baird. According to some audience developers interviewed, the results of such testing have not justified the expense, in their experience.

 

Given limited budgets and resources, Baird says the selective approach—narrowing down testing ideas to perhaps a handful that have the greatest potential for impact or bottom-line improvement and conducting A/B or limited multivariate testing in just those areas—is a sensible compromise.

 

How are tests being conducted in cases where neither A/B nor multivariate designs are being used?

 

In those cases, Hearst uses “a rolling approach of tracking overall response and order volumes on a daily basis very closely,” explains Chris Wilkes, VP of Hearst’s marketing and audience development group. “This can include a mix of promotions running concurrently that all have slight differences. Based on the daily trends, we’ll make changes, both minor and major, as frequently as needed to keep performance high. We establish baseline metrics for response—as a percentage of unique visitors or page views—and/or baselines for order volumes, using an average of historical volumes.”

 

Audience developers at other companies also report that it’s common to use known metrics based on historical performance, such as the average number of click-throughs and conversions generated by a certain number of impressions, and run each of two different ads for the same period of time.