Editors note: We are republishing an article by WNIP that looks into the relationship between headline experimentation and higher engagement. This piece was originally published on What’s New in Publishing.

Headline experimentation has become an increasingly popular tool among publishers. Editorial teams often test several title variations for the same article, in order to find out which one performs the best.

Testing identifies not only the headlines that are being clicked on, but also the ones that lead to engagement with the content. This is particularly significant for the global publishers that need to ensure they’re allocating time to optimizations with the highest reader impact.

Some of the largest publishers in the world deem it essential to homepage optimization — we’ve seen websites running upwards of 1,000 tests per week.
Josh Schwartz, Chief Technology Officer at Chartbeat

Headline testing drives value in two ways. From a quantitative perspective, it directly lifts engagement for each story that’s tested. Informed by that data, content teams can then learn how to write more engaging headlines.

Chartbeat, the content intelligence platform for publishers, decided to determine whether the value of headline testing declines over time.

One might hypothesize that, after a few months of running headline tests, editors gain enough intuition to know which headlines will perform best. That, in turn, would make continuing to test become less important and headline testing a depreciating asset.
Headline experimentation: The data on its lasting impact | Chartbeat

The research suggests that not only does the value of headline experimentation increase over time, but that it continues to steadily drive engagement.

To investigate whether the value of headline testing truly depreciates, researchers looked at the dataset of over one million tests to see whether the lift of tests changes over time.

They found that lift is fairly constant over time. When a site first starts using headline testing, about 65% of tests have a non-original winner, and an average test has a roughly 45% lift.

headline experimentation over time rate of non-original headlines

“That’s what we’d expect after introducing a new culture of experimentation around headlines,” says Josh Schwartz, Chartbeat’s Chief Technology Officer. “Yet, two years later, those rates are the same, as we see in the graph below.”

Given that, the evidence is clear that testing continues to drive engagement. Yes, you can say editors are getting better at writing original headlines over time, but they’re also getting better at writing alternate headlines!
Headline experimentation: The data on its lasting impact | Chartbeat

The researchers also found that the most successful teams were those that build testing into their culture, with Slack rooms and team meetings dedicated to brainstorming headline ideas for important stories.

On average, the number of people running headline tests for a given site grows by about 44% during their first year of testing and another 49% their second year.

The data shows that the average publisher drives more than 5x the engagement with headline testing after two years when compared to the time they began.

headline experimentation over time average users running tests

Our analysis has shown that yes, there are concrete benefits to experimenting with headlines.
Josh Schwartz, Chief Technology Officer at Chartbeat

Key takeaways from the research:

1. Headline testing continues to drive engagement. Not only are editors are getting better at writing original headlines over time, their ability to create engaging variants are improving as well.

2. Consistency is key. The value of headline testing doesn’t wane with an increase in tests, it’s actually driving more engagement over time.

3. There’s a link between a culture of headline testing and their sustained engagement. The more tests are encouraged, analyzed, and iterated upon by teams, the stronger the results are over time.

More From The Fix: Media businesses face an engagement erosion crisis. So what?

Photo by Kelly Sikkema on Unsplash

headline testing

Latest in headline testing

“The evidence is clear”: Headline testing continues to drive higher engagement for publishers