Internet company Cloudflare is developing a means for the media to charge AI companies crawling the web for their content. Photo: Cloudflare blog
At the dawn of the internet, mainstream commercial media faced two choices: protect the copyright on its content and put up paywalls - or get with the programme and put its journalism on the World Wide Web for free.
Egged on by tech commentators projecting big audience growth and new rivers of digital ad money, most picked path two.
We all know what happened next.
The audiences arrived but the accompanying ad money, for the most part, did not.
The vast bulk of the revenue was hoovered up by tech platforms like the dominant search engine Google and the social media behemoth Facebook. And they thrived in part thanks to the content served to them by the legacy media.
In recent years, there's been some readjustment.
News organisations like Stuff, once leery of paywalls, have embraced them for some mastheads.
They've also reshaped their pitches around quality and trust, and tried to shore up their eroding ad revenue with memberships and premium subscriptions.
For the past year it's also been producing the 6 pm TV news for Three - and its digital wing has partnered 50/50 with Trade Me.
But just as some media are finding a new footing, another seachange is on the horizon.
AI is infusing itself into our largest and most powerful information distribution systems.
Its advocates say that's helping make data more accessible to us than ever. But in part, it's doing that by using web crawlers to train its LLMs (Large Language Models) on the work of journalists.
Photo: 123RF
AI crawlers 'scrape' data en masse and store it in ever-growing knowledge bases. That's much more extractive than the 'fetchers' used by Google to index data on the web.
That content is used to, among other things, create Google's AI Overviews that now appear often at the top of the page when you search for something.
That creates a dilemma for creators, who have reportedly seen their online search referrals slump 34.5 percent since Overviews appeared.
So - fewer website clicks and less traffic for media companies, and more people potentially getting information from unscrupulously-generated AI summaries.
All this is happening while other efforts to make tech companies pay for news - such as the Fair Digital News Bargaining Bill - flounder.
No wonder some media figures seem fed up, with Stuff chief executive Sinead Boucher among those warning AI could be apocalyptic for our legacy media.
But she and other like-minded media figures have a new ally.
Pay the toll
The internet infrastructure provider Cloudflare - which claims to run access to about a fifth of the Internet - has announced it will block AI crawlers from accessing online content without permission or compensation, by default.
Its new system allows website owners to essentially put up paywalls for AI, blocking it from training on their content until it either states a purpose the provider is comfortable with - or coughs up some cash under the terms of an agreed licence.
RNZ's Corey Fuimaono has been looking into this - and he says Cloudflare's move could be a step toward media companies finally reaping the rewards they hoped for more than 25 years ago - and actually making some real money off the internet.
Corey Fuimaono Photo: RNZ / Cole Eastham-Farrelly
"It's the future of media that we are talking about in terms of its sustainability," he told Mediawatch.
"If they don't hop on board with AI companies, there's a huge risk that their readership, their listenership, their viewership continues to dwindle as people just go to ChatGPT to ask queries, as people would go onto Google to do a bit of searching."
Fuimaono says finding a way to integrate information with AI will help the media stay relevant.
Despite that, he doesn't believe deals with AI companies will solve all the media's issues with dwindling ad revenue.
"It's a piece of the puzzle. And it'll be interesting to see over the next few months what that fair price is."
But will AI companies actually come to the table?
The downside risk is the possibility AI companies will hit the barriers put up by Cloudflare and other content delivery network (CDN) companies - and simply refuse to pay to get around them.
AI crawlers would be forced to train on the often lower-quality information they can access for free.
It would also increase the chance of AI training on other AI-generated content, creating a kind of ouroboros effect.
That would threaten not just media revenue, but the veracity of the information people find online.
Kelly Shortridge is a security vice-president at Fastly, another of the big CDNs which is also used by New Zealand companies online.
"It's a tricky discussion because on the one hand you do have larger and maybe more business-friendly companies that want to negotiate agreements with media companies or e-commerce companies," Shortridge told Fuimaono.
"They're looking to operate above-board business and they have a reputation they want to protect. I think they're more willing than even six months ago to consider solutions where they're able to verify they are legitimate and not trying to surreptitiously scrape content."
"That said, we have an excellent security research team and customer security operations - and they've seen quite a few instances of other kinds of AI companies or even individuals who are trying to impersonate those bigger, more legitimate companies. It's a little bit of a mess."
"Let's say everybody just decides to block all AI bots. There are some companies that are big enough that that will meaningfully hurt their business and they don't want that. That is why you're increasingly seeing them start to actually invest in alternate solutions [that could] help ... mostly media and e-commerce. It's evolving rapidly."
"You either have to move to paywall everyone, which isn't in everyone's best interests outside of the media space - or you have to figure out a way to better manage and control bots, whether those are crawlers or fetchers."
"It's always exciting when you see more of the market trying to figure out solutions that help the overall community. I think the more options that media and other companies have at their disposal, the better."
Fuimaono thinks bigger AI companies will seek to do deals with media, rather than take on the reputational risks and potential regulatory risks that come from refusing to pay.
"There's a lot at stake. And I think, as much as rock and roll as this whole ride has been so far, I think these AI companies will come to the table," he told Mediawatch.
You can hear Corey's full interview with Kelly Shortridge, VP of Security at Fastly, here: Extra Techverse: A tariff for AI crawlers?
Sign up for Ngā Pitopito Kōrero, a daily newsletter curated by our editors and delivered straight to your inbox every weekday.
