Humans Vs. Generative AI: A Head-To-Head Ad Copy Content Test [Data Study]

Generative AI is already making a significant impact on advertising, creating pressure on advertisers to reassess their strategies and explore alternative ways to reach target audiences.

One prominent debate within this space centers around the effectiveness of human copywriting versus AI-generated ad copy.

Both sides claim the crown for crafting compelling messaging that drives clicks and conversions. However, the question remains: What truly resonates with audiences in the digital age?

The Click-Through Conundrum: A Deeper Dive Into The AI Debate

Click-through rates (CTR) and cost per click (CPC) are crucial metrics for gauging ad performance.

They reveal which messages entice users to engage with the advertised product or service and at what cost.

Our recent study at Hop Skip Media delves deeper into these metrics to shed light on the power of human copywriting in advertising.

Methodology: Decoding The Data With Carefully Defined Parameters

For our study, we utilized Copy AI, a popular AI-powered copy-generation platform. We chose Copy AI due to its widespread use and positive user reviews within the advertising industry.

The ad’s target audience was business owners and marketing managers looking for pay-per-click advertising services. The focus product/service advertised was our services at Hop Skip Media.

We designed a controlled test with one responsive search ad (RSA) sample size per copywriter (human and AI). This format allowed for a comprehensive content comparison across various headlines and descriptions.

Each ad contained 15 headlines and four descriptions, following the best practices for the RSA structure recommended by Google Ads. The ads were then placed on Google search via Google Ads for a duration of eight weeks. We allocated a total budget of $500*.

* We acknowledge that these two factors were limitations in our study and have plans to redo this study in 2024 with a larger budget and longer timeline.

Results: Human Copywriting Beats AI-Generated Ads – Exploring The Numbers In Detail

The results clearly show that human-written ads significantly outperformed AI-generated ones. Let’s delve deeper into the key metrics:

The human-written ads achieved 45.41% more impressions and 60% more clicks, resulting in a significantly higher CTR of 1.33%.

This suggests that the human-written ad copy resonated more effectively with the target audience, prompting them to engage with the ad at a higher rate.

Furthermore, the human-written ads boasted a lower average cost per click, indicating potentially greater efficiency in acquiring clicks.

These findings highlight the human copywriters’ effectiveness in crafting compelling ad copy that drives clicks and potentially leads to conversions.

Analysis: Why Humans Outperform AI – Unpacking The Reasons Behind The Results

Several factors likely contribute to the observed differences in performance between human-written and AI-generated copy:

1. Understanding The Audience Beyond Demographics

Humans possess innate skills in understanding the target audience’s emotions, needs, and desires.

They can tap into cultural nuances and tailor messaging to resonate on multiple levels, creating a deeper connection with potential customers.

While continuously evolving, AI still struggles with this level of nuanced understanding.

2. The Power Of Creativity And Emotional Appeal

Humans possess greater creativity and can craft messages that evoke emotional responses, humor, or a sense of urgency – all of which can significantly improve ad effectiveness.

AI, while adept at generating various text formats, often struggles with the subtle nuances of language and cultural references that resonate with audiences on an emotional level.

This can lead to generic ad copy lacking the emotional appeal necessary for click-throughs.

3. Training Data Quality And The Evolving Landscape Of AI

The effectiveness of AI-generated content hinges largely on the quality of the training data it is fed.

Limited data or biases within the data can lead to subpar outputs that miss the mark. Additionally, the capabilities of AI tools themselves are still evolving.

While advancements are promising, there’s room for improvement in their ability to replicate human copywriters’ creativity and emotional intelligence.

However, it’s important to acknowledge that AI tools can be valuable assets when used strategically.

By providing them with high-quality training data, in-depth information about the target audience and product/service, and careful human review and editing, they can offer valuable support to human copywriters.

We should also point out that Google’s Gemini (formerly Bard) and other large language models (LLMs) pointing to Google’s platform data were not available at the time of the study, which could be a limiter to Copy AI’s performance with this experiment.

4. The Importance Of Context And Adaptability

Humans excel at adapting their messaging to specific contexts and platforms.

They can tailor ad copy for different demographics and online behavior patterns, and even adjust based on real-time campaign performance data.

While AI is improving in its ability to adapt, it still struggles to replicate the level of flexibility and context awareness that human copywriters possess.

Optimizing The AI Advantage: Tips For Success And Responsible Use

While our study highlights the limitations of AI in crafting truly impactful ad copy, these tools still offer benefits when used strategically as long as you take the following into account to get the most out of them:

1. High-Quality Training Data

Ensure Copy AI or the AI tool of your choice is trained on a comprehensive and diverse dataset of high-quality ad copy relevant to your industry and target audience.

This will feed the AI with the necessary information to generate more relevant and impactful outputs.

2. Detailed Audience And Product/Service Information

Provide the AI tool with as much specific information as possible about your target audience, product/service, and marketing objectives. The more information you provide, the better the AI can tailor its suggestions to your specific needs.

3. Careful Review And Manual Editing

Never rely solely on AI-generated copy. Always review the outputs critically, edit for clarity and emotional appeal, and ensure alignment with your brand voice and messaging.

Additionally, use AI-generated copy as inspiration and a starting point, but don’t hesitate to inject your creativity and expertise to elevate the ad copy truly.

While Google recently introduced generative AI for Performance Max campaigns in the United States, it’s important to note that the rollout is currently limited and will reach other countries later in 2024.

Google rolls out features with a phased approach, allowing them to gather data and refine the feature before making it broadly available.

Also, it’s important to be mindful that Google will likely prioritize and showcase the content it generates through AI during the initial rollout.

This could potentially impact the initial performance data you see for your campaigns, as Google’s AI-generated content might receive more prominent placement compared to the content you’ve manually created.

Because of this, monitoring campaign performance over time is crucial, allowing Google’s AI to learn and adapt while observing how your own manually created content performs.

By keeping these considerations in mind, you can better understand the true potential of generative AI in your Performance Max campaigns.

Ethical Considerations: A Note of Responsibility In The Age Of AI-Generated Content

As with any technology, ethical considerations are paramount when utilizing AI-generated ad copy. Using this technology responsibly is crucial to ensure:

1. Accuracy And Transparency

Always double-check the accuracy of AI-generated information. AI is capable of creating highly convincing fraudulent articles that resemble genuine scientific studies.

Additionally, be transparent with consumers about using AI in your ad creation process if applicable.

2. Avoiding Deception And Manipulation

AI-generated content should never be used to deceive or manipulate consumers. The MIT Technology Review discovered that users are 3% less likely to spot false information in an AI-generated tweet than humans.

Although the credibility gap is small, it is concerning as that gap is likely to grow significantly. Ensure the messaging is clear and truthful and avoids any misleading claims.

3. Respecting User Privacy

Be mindful of user privacy concerns when utilizing AI for ad creation. Ensure any data used for training the AI is collected and used ethically, and in accordance with all relevant data privacy regulations.

Conclusion: The Human Touch Endures – A Look Toward The Future Of Advertising

Though narrow, this study clarifies the broader value of human copywriters in advertising.

Their ability to understand audiences on a deeper level, craft emotionally engaging narratives, and adapt to specific contexts remain unmatched by AI.

While AI continues evolving and offers valuable support when used strategically, the human touch will likely remain crucial in crafting ad copy that carries cultural and sensory nuance, leading to better interactions.

As the field progresses, further research and exploration will be key to understanding how AI can best complement and enhance human creativity in advertising.

Additionally, responsible and ethical use of AI-generated content should remain a top priority as this technology continues to play a growing role in advertising.

More resources: 

Featured Image: Rob Hyrons/Shutterstock