#23 DOWN-‘N’-DIRTY, QUICK-‘N’-CHEAP: Bob Hacker’s Revolutionary New Testing Technique

http://dennyhatch.blogspot.com/2018/09/23-down-n-dirty-quick-n-cheap-bob.html 

 

Issue #23 — Tuesday, September 11, 2018

Posted by Denny Hatch

DOWN-‘N’-DIRTY, QUICK-‘N’-CHEAP:

Bob Hacker’s Revolutionary New Testing Technique

By Robert C. Hacker

"A good plan violently executed now is better

than a perfect plan executed next week."

                                   —Gen. George S. Patton Jr.

You have a control package—one that has been profitable for months or years.  Results from that package are in serious decline. Rising costs per order are unsustainable.

You need a replacement—fast!

If you cannot come up with one or more new controls—quickly—you will be out of business and frantically updating your résumé.

“There are two rules and two rules only for successful direct marketing. Rule #1: Test everything. Rule #2: See Rule #1.”

—Malcolm Decker

Ed Mayer’s Corollary to Mal Decker’s Rules: “Don’t test whispers.”

For example, DO NOT TEST $69.95 vs. $69.99.

Test serious differences: $49 vs. $69 vs. $89.

Traditional testing methodology tells us:

“The Holy Grail of direct marketing is the single variable test.”

—Don Nicholas

This means:

1) Only test one thing at a time.

E.g., Price, Offer, envelope teaser, or Letter copy (with identical price and offer on all tests).

2) Always test head-to-head.

Identical mail packages with just one element changed per panel.

3) Always test statistically valid samples.

This works fine when you mail in high volume and can afford to allocate 5,000 to 10,000—or more—names to each test cell.

For example, a mailer dropping 1,000,000 direct mail packages at a time (costing 70¢ each or $700,000 total) can afford to test 20 packages of 10,000 each and still mail the control package to 80% of the file.

But what if your entire mailing universe is constrained to 25,000 to 50,000 names? This is a situation many business-to-business—or geographically constrained—direct marketers find themselves in.

An old rule of direct mail testing dictated:

The minimum for an individual test cell is 5,000 mail pieces. The reason: back then the average response was 2%. Two percent of 5,000 is 100 inquiries or orders. This is the minimum for a statistically valid sample to be used in follow-up efforts. This technique is:

Expensive: A series of 5,000-piece mailing tests are expensive ($1 per piece or more in these minuscule quantities).

Time Consuming: Writing, designing, printing and mailing takes three weeks. Bulk snail mail takes at least an additional week-and-a-half to be delivered. Plus an additional two weeks to receive all orders. Even if many responses come immediately via 800 telephone and email, you’ve blown two months.  Turnaround time for getting confirming tests printed and mailed is two weeks with another week-and-a-half for all replies to be in. That’s two months minimum to see if you still have a business!

You may remember the famous scene from the movie. [In the winter of 1944] the Germans launched Operation Wacht am Rhein, better known as the Battle of the Bulge. Elements of the United States First Army are pinned down in Bastogne. British Field Marshal Bernard Montgomery will need weeks before his troops will be able to relieve the beleaguered Americans.

                 George C. Scott—second from left in the film, Patton—spars with 

                 dubious allied officers over his promise to save save Bastogne 

                 with two divisions in 48 hours.

Allied supreme commander General Dwight Eisenhower asks Patton, “How long will it take you to get Third Army moving north?”

“I can attack with two divisions in 48 hours,” Patton replies, to a round of snickering from the other generals present.

Patton’s boss, General Omar Bradley, is not amused. “Ike wants a realistic estimate, George. You’re in the middle of a fight now. It’s over a hundred miles to Bastogne.”

In the next scene, troops are marching north.

—SIDEWISE THINKING:

   Project Management and Business Creativity from a Sidewise Perspective

  

Biographer Martin Blumenson called this the penultimate moment of Patton’s career.

The Penultimate Rule Breaker: Multivariate Tests

With 50,000 names to mail, conventional test methodology would limit the number of test cells to 5-10, even if you tested the entire file.  Often, that’s just not enough testing, particularly if there is no control package and there is little or no time to find one.

Or what if you have to create a new control and you’ve only got one chance to do it? This is often the case for direct marketing agencies, vendors and freelancers.  They get one shot. And month later they’re either heroes or zeroes.

When either of these situations presents itself, I always run multivariate tests.

Often the cell sizes are not statistically valid. 

But, it’s still the best way I know to find a winner fast.

Let me give you an example.  

One of our prospective clients gave us the following opportunity one September.  Their current direct mail cost-per-sale was about $700.  The target was $95 and no agency or vendor could hit it. If she couldn’t hit the $95 target, her department was going away – and her with it.

If we could test in October and find a control package that hit their cost-per-sale target, they would allocate $2,500,000 to a rollout in January.  Due to a very tight time constraint, we had to find a winner on the first drop.  We recommended a multivariate test, as it was the only sure way to find a winner in time.

 In this case, the test matrix contained:

• 32 different copy/offer/package combinations

• 288 discrete test cells distributed over about 200,000 pieces of mail.

Did it work?  Yup, we hit the target on the first test. 

We never stopped testing like this for them and drove the cost-per-sale even lower.  A few years later we were mailing 3,000,000 pieces of mail per-day for this client.  It took six control packages to get through the universe and we found them all with multivariate testing.

                       

How to Do a Multivariate Test

The objective of a multivariate test is to develop a control package as fast as possible.  With a properly run test you can usually find a winner -- or several winners -- in the first test, almost always by the second test event.  Here’s how it works.

Phase 1

The objective of Phase 1 is to determine the key success drivers as soon as possible.  Always test big ideas, not whispers to . . .

• Discover which lists will work.

• Determine which offers will work.

• Find a winning copy platform or structure.

• Identify a winning package format.

To show you how a multivariate test might work, let’s look at a simple 25,000-piece test.

(This would be a very typical B2B test scenario.) The key recommendations are to test…

• Three different offers

• Two package formats, Package X and Package Y

• Five lists of 5,000 each.

Based on the above test recommendations, the test matrix would look like this:

In this matrix, there are 30 test cells, each cell with 833 names.  The results from each cell are not statistically valid.  To generate a statistically valid sample, this test would have to be about 150,000 pieces, which is three times larger than the entire mailing universe of 50,000.

However, multivariate test results are still highly indicative. We let the market tell us what they will respond to instead of sitting around the conference table guessing.  We move from the world of personal opinion to professional judgment ASAP!  You will see strong trends and “hot zones” that tell you what to do next.  Let’s assume that the test generated the following response rates:

In this example, it is safe to conclude the following, assuming we need a response rate of 3% or more to hit the economic target.  

• Package X is a clear winner over Package Y.

• Package Y was pretty good.  Can it be tweaked to pull up performance and then put it in rotation with Package X to slow down list fatigue? Typically, we’d do that after Package X in in rollout.

• Offer B worked best with Lists 1, 2 and 3.

• Offer C worked best with List 4.

• Nothing worked with List 5.

Phase 2

In Phase 2 we check-test the results of Phase 1, this time using cells that are large enough to generate a statistically valid sample size. The matrix might look like this, assuming we found anew list (List 6) that matches the profile of List 4.

 

With a response rate of 3% or higher and test cell counts of 5,000, the results from this test would be statistically projectable.  And we’ve been able to find a winning control package with only 50,000 names instead of the 150,000 that would have been required for a test of this magnitude with traditional testing methodology.

In this world, speed is a strategy.  When you are in trouble, traditional test methodologies are too slow.  If you don’t find the answer, somebody else will.

                       

So, when you have to find a winner fast, use multivariate testing.  If you have small list universes, use multivariate testing.  The key to making a multivariate test work is to only test big ideas.  By big ideas we mean testing things that are very obvious to the recipient. 

If you look at a package for three seconds and can’t see the big idea, you don’t have one.  Typically, big ideas are:

• Offers

• Copy platforms

• Major headlines, Johnson boxes, eyebrows and callouts

• Package design

• Package formats

• Outgoing envelope formats

• Outgoing envelope teaser copy

Many direct marketers today test whispers. It’s a waste of time and money. Don’t do it.

###

Word count: 1523