Cloudflare’s crawler now returns a 400 when robots.txt says no, and MiniMax H3 halves on Vercel until 13 September

Cloudflare changed the /crawl endpoint today so that it reads the target’s robots.txt and, where the answer is no, refuses. The changelog entry is dated 31 August 2026 [1]. There is a new parameter, contentUse. It takes reference or full, least permissive first, and it defaults to full. If the site you are pointing at sets a use level in its robots.txt that is stricter than the one you declared, the request comes back as a 400 rather than as content [1].

A robots.txt line is now an API error

The Robots Exclusion Protocol has always been voluntary. A crawler read the file, and the crawler decided, which meant the decision lived in code you owned. This one lives on Cloudflare’s side of the API. Nothing you run today stops working, because the default is the permissive value, so the first you will hear of it is a 400 in a nightly job pointed at a domain nobody audited.

The check worth doing this week is small. Take the domains at the top of your fetch list, read what each one now declares under the Content Signals standard [3], and set contentUse per call rather than leaving the default in place. Asking for reference where reference is all the pipeline needs costs nothing and turns a hard 400 into a crawl that finishes.

Half price on MiniMax H3, until 13 September

The other weekend item is money. Vercel cut MiniMax H3 and H3 Max by 50 per cent on AI Gateway on 30 August 2026, at every duration and in every aspect ratio the models support, running through 13 September [2]. The model IDs are unchanged, minimax/minimax-h3 and minimax/minimax-h3-max, so requests you already send collect the discount without a deploy [2]. Vercel prints no per-model rate on the AI Gateway pricing page, which sends you to the model list in the dashboard for the figure you will actually be billed [4].

Fourteen days will not carry a production workload onto a new video model, but it comfortably covers the evaluation most teams keep deferring, on the same model ID and at half the bill, with nothing to migrate afterwards and nothing to roll back.

Nothing at all from Visa, Mastercard or Stripe

I ran the sweep this morning, Monday 31 August, across agentic commerce announcements, AI payments at Visa, Mastercard and Stripe, stablecoin regulation and the protocol trackers. No payments item dated 29, 30 or 31 August cleared verification. The freshest verified items in the window are the ones earlier editions already carried: Dunamu and Visa on Friday, and Treasury’s GENIUS Act section 3 rulemaking, still open for comment.

Neither of these two changes needs anything from you to take effect. The discount lands on requests you already send, and the 400 waits inside a job you already scheduled. So how many of the domains at the top of your crawl list have you read the robots.txt of this year?