If you’ve been just about anywhere on the internet recently, you’ve heard about ChatGPT. You might even be groaning as you see this, “another article about ChatGPT?” But trust us, you’ll want to stick around for this one, because we put it to the test: emails written by ChatGPT vs. our own sales cadence. Who came out on top in the end? And what does it say about AI’s role in the future of our work?
Are robots officially coming for our jobs? Before you get too scared about a potential robot apocalypse, let’s talk about it.
An email battle of wits
As the hype for ChatGPT has grown, sales reps have begun to wonder, “can this save time and help my emails stand out?” A tool that writes killer emails for you is almost too good to be true for salespeople struggling to perfect their messaging on top of an overloaded schedule.
We wanted to figure out if ChatGPT was worth the fuss and could be helpful for sales reps. So we planned a test of intellect: an email written by our very own team vs. a sales email written by ChatGPT. The battleground? Our own .
The rules of the game
Based on our experience with previous AI language generators, we believed humans would probably win this battle, but we weren’t gonna call it without data to back it up.
For our experiment we chose an email aimed at business development leaders that we’d used before. It had a steady but slow reply rate of 3% that we wanted to beef up. Then we asked ChatGPT to write us an email for the same purpose.
We were impressed by the first email it generated, though it was longer than best practices suggest. But that didn’t keep us from testing it – this is all for science, after all.
However, we also wanted to give ChatGPT an honest shot and use it like someone in the real world would. Most people would continue to refine the result until it was perfect. So we added one more touchpoint and asked ChatGPT to write a shorter version.
That gave us three emails: one already used by our sales team, one generated using a single prompt in ChatGPT, and one from ChatGPT with extra refining.
Over the course of a week, we sent the three emails split among 100 recipients who fit the prospecting profile. To test the effectiveness of the messaging, we looked at click and reply rates to measure engagement.
So who won?
At first, we weren’t too impressed with ChatGPT’s performance. A peek in the middle of the experiment showed the human-written email was in the lead by almost twice as much. But by the end of the week there had been an upset.
Who came out on top?
Believe it or not, the highest engagement belonged to ChatGPT. But with a caveat – the winner was the second, shorter ChatGPT email that took extra – and human led- steps to perfect. In fact, the single touch ChatGPT-generated email performed the worst of the three.
Here are the numbers:
Ultimately, salesperson + AI turned out to be a winning combo. Like so many other things in life, the right answer is not either/or; it’s both.
Let’s talk about these results
ChatGPT isn’t a magic wand you can wave and *poof* – there’s the perfect pitch. It’s a tool, and like any tool, you need to know how to use it to get the kind of results you want.
With ChatGPT, it’s all about the prompt. The more information you give it, the better the results. Email marketer Chase Dimond shares tons of complex prompts that include details like “use the ‘Attention-Interest-Desire-Action’ framework.”
Are those prompts going to generate a fantastic email? Probably, yeah. But they’re not exactly simple. It’s less involved than poring over every word for hours to get things just right, but it still requires a decent amount of legwork.
Anything ChatGPT produces ultimately still comes from our own brains. A human has to write the prompt in the first place – and write it well, if good results are to be expected.
You need to have the knowledge of what makes a powerful message in order to guide the tool toward making one for you. Otherwise, it might produce something bland and personality-less. Not to mention something that can at times be factually incorrect. It’s a timesaver and template generator, but it’s not a replacement for the communication skills that sales or marketing demand.
It’s also important to note that like anything that goes viral, ChatGPT has experienced some growing pains. You may visit the site to see that it’s at capacity or experiencing delays due to “exceptionally high demand.”
All of that is expected from a growing tool, but it also means that if you rely on ChatGPT to spit out a masterpiece as soon as you ask, it might not be available the precise moment you need it. Plus, they’re coming out with a paid version soon; the ChatGPT you know and love right now may not look the same a few months down the line. So you shouldn’t develop total reliance on AI writing – you still need to know how to craft these messages on your own.
(This abomination, created using Dall-E, is all thanks to the beautiful human mind and what it can prompt AI to do.)
What does this mean for the future of sales and marketing?
It’s clear that AI continues to get more and more effective, not to mention omnipresent. ChatGPT isn’t the first or only AI language tool– it’s just (currently) the most popular. Denying its power is denying the future.
We don’t want to cover our ears and shout “la la la, nothing is going to change” while AI gets more sophisticated every day. AI is here to stay. Not to mention, neither sales nor marketing benefit from refusing to advance with and embrace technology.
Our experiment shows the power of AI. But it also serves as a reminder that no matter how polished AI gets, there’s a certain something about human connection that it simply can’t replace. And with how important personalization is to sales and marketing, you can’t leave that out. As marketing expert Ann Handley says, “You can write faster drafts, but you can’t shortcut relationships.”
While in the end one ChatGPT email emerged victorious in this battle of man vs. machine, it needed a little extra prompting to get there. So maybe the reality isn’t quite like a blockbuster movie where a ragtag band of humans has to fight against technologically superior robots. It’s more like a feel-good film where heroes from different backgrounds overcome their differences to work together (and maybe make a robot friend along the way). The conclusion of this tale isn’t one where humans or AI need to beat the other. It’s more about what we can accomplish with our powers combined.
Key Takeaways:
- In an experiment of a traditional email pitted against AI-written emails, the ChatGPT email created with detailed prompts and an extra human touch proved the most effective.
- ChatGPT and other AI writing software can be useful tools to create templates for emails or other sales and marketing content, but truly shine with human guidance.
- This human guidance is required to give writing produced by ChatGPT a “human touch” and personality.
See how Lusha can speed up your workflow
