Back to news
Sunday report

Sunday Signal Report: November 22, 2026

Counting only independent runs cut the share of solved AI benchmarks from 63% to 24%. The biggest districts found their real AI policy was a search default. The week's question: who checked, and could we run the check ourselves?

Listen to this week's episode

Listen on Buzzsprout

Rob here

Who checked?

The lines I quote in the video come from chapter 5 of Ethan Mollick, Co-Existence: The Next Phase of AI (Portfolio, Oct. 20, 2026).

When an editor turned referees' objections into code and ran them against the data, they overturned the headline result in 77 of 156 agent-written studies. That is the week in one number. Ethan Mollick's SatBench found that AI labs' favorite benchmarks look 63% solved if you count launch posts and 24% if you count only independent runs. OpenAI's new flagship ranked second on an independent index and was called "essentially unusable" by testers who handed it real work. Two serious studies of phone pouches landed a week apart in time and miles apart in tone. Daisy Christodoulou did what every board should do with an AI-first school: she named the result that would change her mind. And the agent tools that release every few days shipped fixes for approval rules that were set but not enforced. None of this argues for slowing down. It argues for asking one question before the demo ends: who checked, and could we run the check ourselves?

Two-panel card titled Who checked? Left panel, AI benchmarks from SatBench, data as of November 14: share of the 173 most-cited benchmarks counted as solved is 63% using all reported scores, including lab launch posts, and 24% using independent runs only; three in four first crossings past 85% came from the lab's own launch post. Right panel, agent-written research from an alphaXiv preprint: when referee objections were turned into code, 77 of 156 major objections, 49.4%, overturned the headline result, and 24 of 120 minor objections flipped the verdict; text-only AI referees flagged 10 of 16 versions that had already been corrected. A note asks who checked a claim, and whether you could run the check yourself.
AI and learning

Ask for the testable claim

What changed: on Nov. 16 Daisy Christodoulou, author of Teachers vs Tech, published a long review of Alpha School, the private network where students learn academic content through apps for about two hours a day. She credits its use of Math Academy, which is built on science-of-learning principles, and its hiring of strong educators. Her worry is the heavy reliance on screens: she argues screens suit rapid retrieval practice better than deep reading, and that 25-year-olds can learn from screens in ways 8-year-olds can't. Then she does the useful part. She sets two bars that would change her mind: an AI maths tutor for upper secondary at least as effective as the Google and Eedi tutor, or 90% of 100,000 or more students passing a phonics screening check at age 7. She points to England's phonics reforms as the human-led benchmark: the pass rate rose from 58% in 2012 to 82% in 2019 and 89% by the end of Year 2 in 2026, across about 600,000 pupils a year with very little classroom technology. The same week supplied two warnings about unchecked claims. Ethan Mollick's SatBench found 109 of the 173 benchmarks AI labs cite most are counted as solved, but three in four first crossings came from the lab's own launch post, and counting only independent runs the figure falls from 63% to 24%. And in an alphaXiv preprint describing 936 AI agents across 35 human-run labs producing 936 studies, a guest editor turned 156 major referee objections into executable checks, and 77 of them, 49.4%, overturned the headline conclusion. Text-only AI referees flagged 10 of 16 versions that had already been fixed. Why it matters now: districts are being pitched AI-first models, AI tutors and AI-built assessments faster than anyone can evaluate them. The evaluation habit is the scarce skill. Rob's take: I don't need a vendor to promise transformation. I need them to name the result that would prove them wrong and who will measure it. That is also the real AI skill for adults. A pre-registered NBER study of 249 leaders found performance leading AI agents correlated 0.81 with impact leading human teams, and the good leaders of both asked more questions. Prompt tricks expire with each model. Defining the job, the audience and what good looks like does not. Concrete implication for a district leader: before the next AI pilot starts, write down in one sentence the result that would make you stop, the person outside the vendor who will measure it, and the date you will look. Then make your AI professional learning about delegation: scoping a task, reviewing the output and asking the next question.

Card titled Ask for the testable claim. Left panel lists Daisy Christodoulou's two bars for an AI-first school model: an AI maths tutor for upper secondary at least as effective as the Google and Eedi tutor, and 90% of 100,000 or more students passing a phonics screening check at age 7, plus her caution that screens suit rapid retrieval practice better than deep reading. Right panel is a bar chart of England's phonics check pass rate: 58% in 2012, 82% in 2019 and 89% by the end of Year 2 in 2026, across about 600,000 pupils a year in 16,000 schools with very little classroom technology. A note below says that in a pre-registered study of 249 leaders, performance leading AI agents correlated 0.81 with impact leading human teams, and good leaders of both asked more questions.
Sources: Daisy Christodoulou, No More Marking; NBER Working Paper 33662; Harvard Business School AI Institute.
AI governance

The default is the decision

What changed: Google's position that schools get Search with its AI features or not at all moved from a policy dispute to an exit plan. On Nov. 18 Los Angeles Unified said it had disabled AI Overviews for students and was evaluating whether Google Search supports learning; its chief information officer told the board's AI committee the district could change its default search engine "literally overnight," according to The New York Times as reported by NBC Los Angeles. The same day Google's vice president of education, Chris Phillips, told the Times that "Search and AI are deeply integrated" and that schools turning off Google's AI features "will not have access to Search." Google told The Verge that next year the Gemini app, Gemini Notebook and Search will be off by default until schools enable them, and that it is piloting Guided Learning in Search this month. On Nov. 19 New York City Chancellor Kamar Samuels called it an ultimatum: "we are not buying into that binary choice." Microsoft Bing and DuckDuckGo, which let admins turn off AI answers, are the named alternatives, and no decision has been made. On Nov. 20 Mother Jones reported that hundreds of Google workers had petitioned leadership, through the Alphabet Workers Union, to let schools keep Search without AI. Sen. Ed Markey's four questions to Google are due Nov. 30. Why it matters now: for most students, the AI policy that actually runs is not the one in the board binder. It is the search box on the device. Rob's take: I like that districts are finding their leverage, and I'd be careful about what we call a win. Switching search engines is a settings change. Teaching students to judge a summary, check a source and notice when a machine is guessing is the part no default will do for us. A district that swaps providers and stops there has moved the problem, not solved it. Concrete implication for a district leader: write down, today, what students actually get for each surface: search, AI summaries in search, AI mode, chat apps and AI inside the learning platform. Note which ones your filter and admin console can separate. Then decide what you would do if the vendor's answer on Nov. 30 is no, before someone asks you at a board meeting.

Timeline card titled The default is the decision. November 9: Senator Markey asks Google four questions about letting schools turn off AI without losing Search, with answers due November 30. November 18: LAUSD says it has disabled AI Overviews for students, and its CIO said the district could change its default search engine literally overnight, according to The New York Times. November 18: Google says Search and AI are deeply integrated; next year the Gemini app, Gemini Notebook and Search will be off by default until schools enable them, and Google is piloting Guided Learning in Search this month. November 19: NYC Chancellor Kamar Samuels says the city is not buying into that binary choice, and Bing and DuckDuckGo are named as alternatives with no decision yet. November 20: hundreds of Google workers petition leadership through the Alphabet Workers Union. A note says changing the default search engine is a settings change, and changing what students learn to do with a search box is not.
Sources: Office of Sen. Markey; NBC Los Angeles; The Verge; ABC7 New York; Mother Jones.
Devices and screens

Two big studies, one small effect

What changed: on Nov. 17 Harvard's Center for Education Policy Research, Dartmouth and NWEA released a student-level analysis of about 3.3 million students in 4,600 schools from 2021-22 through 2025-26. Strict cellphone bans using lockable pouches raised annual achievement growth in math and reading by about 0.01 standard deviations a year, which the authors equate to one to two weeks of added instruction, or about half the yearly loss seen before the pandemic. They also report the bans seemed to reduce the share of students with large reading losses, and found similar-sized effects for state bell-to-bell bans in 2025-26, with a warning that those state comparisons may be confounded. NPR's coverage adds the context that matters: an April study by Brian Jacob, Tom Dee and colleagues found "close to zero" overall effect, with high school math gains offset by middle school losses. The April study is peer reviewed and the new one is not yet. Dee calls the early results "largely disappointing"; co-author Tom Kane argues that at roughly $25 per student, pouches are still a cheap way to buy instructional time. In California, the first semester under the Phone-Free Schools Act looks like a patchwork. EdSource interviewed more than 40 people in seven districts and read the 15 largest districts' policies: some lock phones away all day, some allow them at lunch and passing periods, some leave enforcement to each teacher. Educators report more conversation and fewer fights, and researchers say it is too early to credit the bans for academic gains. Why it matters now: phone rules are becoming law faster than the evidence is settling, and both sides will quote a study this winter. Rob's take: restriction and instruction are separate questions. A pouch buys a small amount of attention back. It does not decide what happens with that attention. I'd rather a district be honest that the effect is small and cheap than oversell a lockbox as a learning strategy. Concrete implication for a district leader: if you adopt or tighten a phone rule, pair it with one measure you will actually track, such as office referrals, chronic absence or reading growth, a baseline from before the change, and a date to report back. Communicate the expected effect as small, because the evidence says it is.

Two-panel card titled Two big studies, one small effect. Left panel, Harvard CEPR, Dartmouth and NWEA, November 17: one to two weeks of added learning a year from strict locked-pouch bans, about 0.01 standard deviations a year in math and reading; about 3.3 million students in 4,600 schools from 2021-22 to 2025-26; gains did not fade through three years, the authors say; not yet peer reviewed; pouches cost roughly 25 dollars per student, per co-author Tom Kane. Right panel, the April study by Jacob, Dee and colleagues: close to zero overall effect on test scores, peer reviewed; gains in high school math offset by losses in middle school; Stanford's Tom Dee calls the early results largely disappointing. A note says EdSource talked with more than 40 people in seven California districts and read the 15 largest districts' policies, enforcement ranges from locked pouches all day to rules left to each teacher, and researchers say it is too early to credit the bans for gains.
Sources: Harvard Center for Education Policy Research; NPR; EdSource.
Cybersecurity

The AI plumbing is the target

What changed: the attack surface this week was the layer that connects AI to everything else. On Nov. 17 two poisoned versions of Microsoft's Agent Framework, 1.26.1 and 1.26.2, were published to PyPI with a credential stealer that ran during normal Python startup even when the package was never imported; the toolkit sees about 200,000 downloads a month. On Nov. 18 CISA added CVE-2026-40217 in LiteLLM, a widely used gateway for routing traffic to AI providers, to its Known Exploited Vulnerabilities catalog with a Dec. 2 deadline. LiteLLM's advisory says the sandbox escape gives code execution in the proxy process, which runs as root in the default Docker image, and that reaching it requires a proxy-admin credential; the fix is 1.83.11. On Nov. 16 Google's Threat Intelligence Group reported that, for the first time, it saw attackers use AI frameworks to fully automate finding and exploiting vulnerabilities with no human in the loop, comparing code before and after patches and firing exploits on their own. Zero-days exploited hit 22 in August, against a 2025 average of 8 a month. And on Nov. 19 Google and Mandiant detailed how a suspected China-linked group exploited VMware vCenter CVE-2026-123890 starting in early September, before any patch existed, after first breaking into a remote support appliance; it could then forge sign-in tokens to bypass MFA. CISA set a Nov. 23 federal deadline. Why it matters now: districts are standing up AI gateways, agent toolkits and integrations quickly, often outside the asset inventory, while attackers are shortening the time from patch to exploit. Rob's take: an AI gateway holds the provider keys and sees every prompt, which can include student data. That makes it infrastructure, not an app, and it deserves the same patch clock as a firewall. The uncomfortable lesson from vCenter is that keeping something off the internet is not enough when the attacker comes in through the remote support tool you trust. Concrete implication for a district leader: this week, ask your team and every AI vendor three things in writing. Which AI gateway or proxy sits in front of our data, and what version is it on? Do any district or vendor environments install agent toolkits from public package registries, and are versions pinned? Is vCenter patched, and was it checked for compromise rather than only updated?

Timeline card titled The AI plumbing is the target. Early September: a China-linked group exploited VMware vCenter CVE-2026-123890 before any patch existed, after first breaking into a remote support appliance; Broadcom patched it November 19 and CISA set a November 23 deadline. November 16: Google Threat Intelligence says for the first time it saw attackers use AI frameworks to find and exploit flaws with no human in the loop; zero-days exploited were 22 in August against a 2025 average of 8 a month. November 17: Microsoft Agent Framework versions 1.26.1 and 1.26.2 on PyPI carried a credential stealer that ran at Python startup even when the package was never imported, for a toolkit with about 200,000 downloads a month. November 18: CISA lists LiteLLM CVE-2026-40217 as exploited, due December 2; the proxy runs as root in its default Docker image and the fix is 1.83.11. A note says an AI gateway holds the provider keys and sees every prompt, so inventory it like a firewall, not like an app.
Sources: Google Threat Intelligence Group and Mandiant; CISA KEV catalog; BleepingComputer; BerriAI LiteLLM.
Frontier and open-weight AI

Cheaper at the meter, unproven at the desk

What changed: OpenAI released GPT-6.2 Astra and GPT-6.2 Sol on Nov. 17. Astra costs $8 per million input tokens and $40 per million output tokens, 20% below GPT-6 Astra, and both models are available to Edu users in ChatGPT Work and Codex as well as the API. OpenAI rates both at the Critical threshold for cybersecurity under its Preparedness Framework and launched them with tighter cyber safeguards. Its headline benchmark comparison carries its own footnote: the rival's leaderboard results exclude nine tasks that used a fallback model. The independent picture is split. Artificial Analysis ranks Astra second on its Intelligence Index and puts Sol at 57, on par with GPT-6 Astra at about 15% of the cost per task. Early testers who handed the models real work were harsher: VentureBeat reports Matt Shumer calling them "essentially unusable," and Dan Shipper finding Astra often stopped early on end-to-end builds. In open weights, Tencent's 770-billion-parameter Hy4 is downloadable on Hugging Face under the new Tencent Hy License, which is derived from Apache 2.0 but is not Apache 2.0: companies offering the model as a service with more than $50 million in revenue or 100 million monthly users need a separate written license, while internal use is exempt. Microsoft announced MAI Licensed Weights on Nov. 17, letting organizations deploy and customize Microsoft models in their own Azure tenant, with customer data used for training only with express agreement. Availability is "coming soon through our Frontier Company engagements." Why it matters now: education pricing and Edu availability make a new default model likely in contract talks, and the governance questions are now about where weights live and who controls upgrades. Rob's take: a price cut is not a productivity gain until the model finishes the work your staff actually hands it. A benchmark rank is a lab's homework. Downloadable is not the same as supported, and announced is not the same as shipped. Concrete implication for a district leader: before changing a default model, run ten of your own real tasks through the old and new options, count how often a person had to send the work back, and price that, not the token meter. For any open-weight or tenant-hosted option, get the license and the upgrade terms in writing.

Table card titled Cheaper at the meter, unproven at the desk, with three columns: release, what shipped, and the condition. GPT-6.2 Astra, OpenAI, November 17: 8 dollars input and 40 dollars output per million tokens, 20% below GPT-6 Astra, available to Edu users, number 2 on the Artificial Analysis Intelligence Index; condition: early testers called it essentially unusable for delegated work and one said it often stopped early on end-to-end builds. GPT-6.2 Sol, OpenAI, November 17: 2 dollars input and 10 dollars output, unchanged, scoring 57 on the index, on par with GPT-6 Astra at about 15% of the cost per task; condition: an independent ranking, not a classroom result. Hy4 weights, Tencent, November 20: 770 billion parameter model downloadable on Hugging Face; condition: not Apache 2.0, model-as-a-service providers over 50 million dollars in revenue or 100 million monthly users need a separate written license, and internal use is exempt. MAI Licensed Weights, Microsoft, November 17: Microsoft models deployed and customized inside your own Azure tenant, trained on your data only with express agreement; condition: coming soon through Frontier Company engagements, announced and not shipped.
Sources: OpenAI; Artificial Analysis; VentureBeat; Tencent (Hugging Face); Microsoft AI.
Agentic AI checkpoint

Policy set is not policy enforced

What changed: release status from the canonical GitHub releases, in UTC. OpenClaw shipped four stable builds in four days. 2026.11.4 (Nov. 16) requires approval for delegated persistent changes when authority is restricted or unverified, adds a standing approval setting, and says plainly that Full Access or standing approval can still authorize changes that enable host code execution even when direct execution tools are denied. 2026.11.5 (Nov. 17) changes the plugin SDK contract with no compatibility window, which breaks some custom plugins. 2026.10.33 (Nov. 18) is a gateway-only extended-stable build for the long-term line. 2026.11.6 (Nov. 19) fixes the Gateway allowing some shell commands to run without the approval the policy required, fixes automatic command review being silently disabled when an unchanged exec policy was reapplied, and fixes browser snapshots exposing filled passwords, one-time codes and card fields to the model. 2026.11.5-beta.4 (Nov. 16) and 2026.11.10-beta.1 (Nov. 19) are prereleases, not production builds. Hermes Agent shipped stable v0.23.2 on Nov. 16, which fixes a High-severity advisory (CVE-2026-130164), and stable v0.23.3 on Nov. 20, which turns proactive tool-result pruning on by default, sets prompt-cache lifetime to automatic and archives the tool results that compaction drops; full curated notes for both wait for v0.24.0. The daily canary builds are prereleases. On the research side, a Nov. 13 arXiv paper on agent harnesses found that a setup tuned on one model scored 68.41 when moved to another, below the plain starting setup's 73.89. Keeping a record of each fix's reason, prediction and result beat carrying over the bare setup, and in coding it improved 9 of 10 paired runs by 2.36 points on average. Why it matters now: two of this week's stable releases fixed approval rules that were configured but not enforced. Rob's take: a guardrail with no written reason is the first thing someone deletes during the next model switch, and a policy you set but never tested is a hope. The asset is not the model. It is the permissions, checks and review steps around it, plus the notes on why each one exists. Concrete implication for a district leader: for any agent tool in a pilot, keep a one-page decision log for each guardrail (what failure it targets, when it was added, whether it worked) and test one approval rule after every update by trying an action it should block. Keep prerelease builds in the sandbox.

Release status card titled Policy set is not policy enforced, covering November 15 to 22, 2026, from canonical GitHub releases. OpenClaw 2026.11.6, stable, November 19: fixes the Gateway letting some shell commands run without the approval policy required, and automatic command review being silently turned off when an unchanged policy was reapplied. OpenClaw 2026.11.4 and 11.5, stable, November 16 and 17: approval required for delegated changes when authority is unverified; Full Access or standing approval can still authorize changes that enable code execution; 11.5 breaks some custom plugins. OpenClaw 2026.10.33, extended stable, November 18: gateway-only long-term line with security and recovery fixes. OpenClaw 2026.11.10-beta.1, prerelease, November 19: test build only. Hermes Agent v0.23.2 and v0.23.3, stable, November 16 and 20: v0.23.2 fixes a High-severity advisory, CVE-2026-130164, and v0.23.3 turns tool-result pruning on by default, with curated notes waiting for v0.24.0 and daily canaries as prereleases. A note on harness research: an agent setup tuned on one model scored 68.41 when moved to another, below the plain starting setup's 73.89, and keeping a record of why each fix was made beat the bare setup in 9 of 10 paired coding runs.
Sources: OpenClaw release notes; Hermes Agent GitHub releases; Yu et al., arXiv:2611.18856.