4 min read

Meta pauses November 20 business terms amid website-crawling dispute

The proposed rollout is on hold as a UK competition investigation proceeds. For creators, public portfolios, promotional pages and private content raise different access questions.

By Mia Esperansa, Editorial at Fanlock
Editorial illustration of a pause symbol above a scanner and a creator’s portfolio website
On this page

The November 20 rollout is on hold

Meta has paused the business and advertising terms scheduled for November 20, 2026, according to Search Engine Journal’s November 10 report, which cites a Meta Business Help Center notice. Existing terms remain in force. The report said Meta had not set a replacement date.

The proposed Commercial Terms had raised concerns about permission to crawl websites associated with businesses using Meta’s services. SEJ reports that the current advertising agreement already includes crawling permissions when an advertiser places an order. The pause therefore leaves existing permissions in place.

That distinction matters for creators who promote work through Facebook or Instagram while hosting a portfolio, shop or membership site elsewhere. A platform’s terms, a website’s crawler preferences and the controls protecting private files each need attention.

Meta lists different crawlers for different jobs

Meta’s crawler documentation separates link previews, AI search, advertising and model training. A request from one crawler should not automatically be treated as evidence of another purpose.

CrawlerPurpose described by Meta
FacebookExternalHitCollects the title, description and thumbnail for links shared on Meta’s apps.
Meta-WebIndexerSupports search results, citations and links in Meta AI.
Meta-ExternalAdsSupports advertising and other business products.
Meta-ExternalAgentIncludes foundation-model training and content indexing.
Meta-ExternalFetcherFetches links at a user’s request and supports agentic AI functions.

Meta documents robots.txt preferences for its crawlers, with exceptions: ExternalFetcher may bypass those rules for user-requested fetches, and FacebookExternalHit may do so for security checks. Meta says cached robots.txt changes can take up to 24 hours to apply.

A creator website needs an access policy of its own

For a creator, a useful review starts with the material itself. Promotional thumbnails may be intended for broad sharing. A full-resolution shoot, unreleased video or subscriber download may have a much narrower audience. Write down which pages and files belong in each group before changing crawler settings.

Fanlock’s practical recommendation: ask whoever manages your website to review crawler rules by purpose, then test the features you rely on. Check that a shared promotional link still produces the intended preview. Record which settings changed and why, so a later troubleshooting fix does not quietly undo your choices.

Private material needs access controls. Google’s robots.txt guidance explains that robots.txt cannot enforce crawler behavior and recommends password protection for private files. Ask your developer to test the actual image or video URL while signed out, as well as the page containing it. A restricted page should not leave its underlying file openly accessible.

These checks apply to a website you control. If your paid content lives on another platform, ask that platform about its access controls and crawler policy; settings on your own domain do not change the platform’s systems.

The UK investigation remains open

The UK Competition and Markets Authority opened an investigation on October 30 into whether Meta’s commercial terms could amount to an abuse of a dominant position. Its case page covers existing terms and revisions, with initial information gathering scheduled through January 2027. That timetable may change.

The regulator has made no finding that Meta broke competition law. Its investigation and the reported pause are separate developments; neither establishes that a particular creator’s website was crawled or that their work was used to train a model.

The next useful evidence will be a new terms notice from Meta and updates to the CMA’s case record. Until then, creators can document their website settings and check who can actually reach their files without waiting for the policy dispute to be resolved.

Sources and reporting notes

Prepared November 15, 2026. Pause reported by SEJ; the underlying Facebook notice required login. Meta’s crawler documentation, the CMA record and Google’s guidance were checked directly. Featured image: AI-generated for Fanlock.

About Mia Esperansa

Editorial at Fanlock

Mia Esperansa is part of Fanlock’s editorial team.

Published by Fanlock. Editorial standards and corrections