This post is also recorded as a podcast here. On the podcast, I’m telling the story, not just reading the post.
Much of a startup is about learning lessons and finding ways to not repeat your mistakes. This campaign described below provided a lot of lessons and might help to point the business in a new direction both short and long term.
For background, @Mssg is a platform that powers messaging campaigns over Facebook Messenger. We don’t call them bots. @Mssg scripts are comparable to a landing page, but on Messenger. The platform powers an automated conversation that is designed to collect data and drive action. This particular campaign was created for a non-profit organization to drive donations. It took place in Feb-March 2017, which was an incredibly good time for progressive organizations to drive donations.
The objective is to use Facebook Ads to start Messenger conversations that collect information (email, billing address, donation amount) and then drive the donation on the client’s donation web page. This donation happens inside Messenger, but their donation page. There are 2 KPIs for a campaign like this. Cost (ad cost) to acquire a donor and ROI — total donation amounts compared to the amount spent on the ads. I don’t throw around the word amazing lightly. From one aspect, these results were amazing. From another, not so much.
We launched on a Wednesday and hundreds of conversations were started over the first few days. The client, an agency, was running the Facebook Ads, and the @Mssg platform is powering the conversation that starts after the user engages. The agency took the organization’s best performing ads and simply duplicated them for our test campaign. The organization was also currently running these same ads separate from our test.
It’s important to understand that @Mssg sees the results once the conversation starts, and the agency/client sees the ad campaign that is starting the conversation.
Once the conversation starts the results were incredible. The first question in the conversation asked for the user’s email address, the second message asked for address, the third question asked for a donation amount. The fourth interaction was a message plus a button that opened the donation page inside Messenger.
On Friday afternoon we reviewed the numbers. The campaign was earning a 54% conversion rate for email address once the conversation started. That’s pretty amazing for digital marketing. Messaging campaign generally see high conversion rates. 54% is good, but not outstanding. The campaign also saw a 54% conversion rate for billing address, the second question. So there was zero falloff between the first and second questions. This is incredibly sticky. I’ve launched thousands of campaigns and never seen a 100% conversion from one step to the next!
The next stat is mind-blowing. 52% of the conversations replies with a non-zero donation amount when the campaign asked. This just doesn’t happen. An analogy would be if 96% of a website’s registrants put an item in their shopping cart. Finally, @Mssg tracks how many of the conversations actually open the donation page inside Facebook Messenger. 48% of the conversations opened the donation page.
All of this is promising no matter what, but it would be a downer if visitors weren’t converting once the page is opened in Messenger. Friday afternoon there was a checkin call with the agency. The donation page was converting at 87%. Wow! These people give us the information in the conversation and we fill that into the form on the page. When the user arrives all they need to do is enter their credit card and submit. This leads to an 87% conversion rate!!
Holy Shit. If we multiply all of these numbers together, once the conversation starts on Messenger, 41.76% of the conversations result in a financial transaction. That’s the amazing part. The Friday review call with the agency ends with us scheduling a call for Monday to discuss bulk pricing so they can quickly take @Mssg to every client.
That’s a great way to start the weekend. I wonder where you can buy a boat around here.
The call starts on Monday and something weird has happened. My contact at the agency asks, “Do people not make donations over the weekend? The numbers have really changed.” It turns out that over the weekend the cost to acquire a donor has shot up to $108 dollars. We’re trying to understand what is causing this.
We hadn’t discussed a target cost per donor the week before, but the client organization needed to be in the $5–$6 range. This is incredibly low, but again, it’s February 2017.
After a few minutes the agency discovered that over the weekend, the cost per click on the Facebook Ad rose to $8 per click. It’s fairly impossible to get donations at $6 if the initial ad-click is costing more than that. After a little more digging it appears that this test campaign was competing directly with the organization’s main Facebook Ad campaigns. We were bidding against ourselves and this was driving up the cost per click for the test audience.
It’s worth noting that the campaign was paying $8/click and converting donors at $108 per, so essentially the ad was running a 7.5% conversion rate for a transaction.
At this point it didn’t matter how impressive the conversation stats were. The ROI wasn’t there and it’s not in my customer’s (the agency’s) best interest to explain the nuance of the CPC driving the horrible results on an otherwise promising campaign. It all wrapped up with a dud. The current customer didn’t want to continue at the cost per donation from the test, and it was going to be hard for the agency to recommend messaging to future campaigns.
It was a bit of a roller coaster ride, but let’s focus on what can be learned.
- Ads are a fickle business. It’s important to get more details at the beginning of a test/pilot. What are the KPI’s we are trying to hit. What numbers are driving that?
- Always build a control group in the Ad Set. The focus was to test the results when we send a clicker into Messenger rather than to a mobile web page. In order to isolate this variable we need to have a control ad in the same set — assuming that the click would also have been $8 to send a clicker to the mobile page, messaging would have beat it.
- The biggest takeaway is that results are incredible once a conversation starts, so the goal is to start more conversations. Facebook Ads are a really good way to start conversations (possibly the best way), but they’re not without problems. Campaigns should diversify the source of conversations to get consistent traffic in case one source spikes or dips.
All in all this campaign example is a positive. In the few months since the campaign launched @Mssg has digested these lessons and we’ll have more overwhelmingly positive results to write about soon.