Sunday Signal Report: November 22, 2026
Counting only independent runs cut the share of solved AI benchmarks from 63% to 24%. The biggest districts found their real AI policy was a search default. The week's question: who checked, and could we run the check ourselves?
Listen to this week's episode
Who checked?
The lines I quote in the video come from chapter 5 of Ethan Mollick, Co-Existence: The Next Phase of AI (Portfolio, Oct. 20, 2026).
When an editor turned referees' objections into code and ran them against the data, they overturned the headline result in 77 of 156 agent-written studies. That is the week in one number. Ethan Mollick's SatBench found that AI labs' favorite benchmarks look 63% solved if you count launch posts and 24% if you count only independent runs. OpenAI's new flagship ranked second on an independent index and was called "essentially unusable" by testers who handed it real work. Two serious studies of phone pouches landed a week apart in time and miles apart in tone. Daisy Christodoulou did what every board should do with an AI-first school: she named the result that would change her mind. And the agent tools that release every few days shipped fixes for approval rules that were set but not enforced. None of this argues for slowing down. It argues for asking one question before the demo ends: who checked, and could we run the check ourselves?

Ask for the testable claim
What changed: on Nov. 16 Daisy Christodoulou, author of Teachers vs Tech, published a long review of Alpha School, the private network where students learn academic content through apps for about two hours a day. She credits its use of Math Academy, which is built on science-of-learning principles, and its hiring of strong educators. Her worry is the heavy reliance on screens: she argues screens suit rapid retrieval practice better than deep reading, and that 25-year-olds can learn from screens in ways 8-year-olds can't. Then she does the useful part. She sets two bars that would change her mind: an AI maths tutor for upper secondary at least as effective as the Google and Eedi tutor, or 90% of 100,000 or more students passing a phonics screening check at age 7. She points to England's phonics reforms as the human-led benchmark: the pass rate rose from 58% in 2012 to 82% in 2019 and 89% by the end of Year 2 in 2026, across about 600,000 pupils a year with very little classroom technology. The same week supplied two warnings about unchecked claims. Ethan Mollick's SatBench found 109 of the 173 benchmarks AI labs cite most are counted as solved, but three in four first crossings came from the lab's own launch post, and counting only independent runs the figure falls from 63% to 24%. And in an alphaXiv preprint describing 936 AI agents across 35 human-run labs producing 936 studies, a guest editor turned 156 major referee objections into executable checks, and 77 of them, 49.4%, overturned the headline conclusion. Text-only AI referees flagged 10 of 16 versions that had already been fixed. Why it matters now: districts are being pitched AI-first models, AI tutors and AI-built assessments faster than anyone can evaluate them. The evaluation habit is the scarce skill. Rob's take: I don't need a vendor to promise transformation. I need them to name the result that would prove them wrong and who will measure it. That is also the real AI skill for adults. A pre-registered NBER study of 249 leaders found performance leading AI agents correlated 0.81 with impact leading human teams, and the good leaders of both asked more questions. Prompt tricks expire with each model. Defining the job, the audience and what good looks like does not. Concrete implication for a district leader: before the next AI pilot starts, write down in one sentence the result that would make you stop, the person outside the vendor who will measure it, and the date you will look. Then make your AI professional learning about delegation: scoping a task, reviewing the output and asking the next question.

- The great Alpha debateDaisy Christodoulou, No More Marking
- SatBench: a benchmark of AI benchmarksEthan Mollick, GitHub
- A Society of AI Scientists (preprint)alphaXiv
- Measuring Human Leadership Skills with AI AgentsNational Bureau of Economic Research, Working Paper 33662 (2025)
- Think You're a Good Leader? Try Leading an AI TeamHarvard Business School AI Institute
The default is the decision
What changed: Google's position that schools get Search with its AI features or not at all moved from a policy dispute to an exit plan. On Nov. 18 Los Angeles Unified said it had disabled AI Overviews for students and was evaluating whether Google Search supports learning; its chief information officer told the board's AI committee the district could change its default search engine "literally overnight," according to The New York Times as reported by NBC Los Angeles. The same day Google's vice president of education, Chris Phillips, told the Times that "Search and AI are deeply integrated" and that schools turning off Google's AI features "will not have access to Search." Google told The Verge that next year the Gemini app, Gemini Notebook and Search will be off by default until schools enable them, and that it is piloting Guided Learning in Search this month. On Nov. 19 New York City Chancellor Kamar Samuels called it an ultimatum: "we are not buying into that binary choice." Microsoft Bing and DuckDuckGo, which let admins turn off AI answers, are the named alternatives, and no decision has been made. On Nov. 20 Mother Jones reported that hundreds of Google workers had petitioned leadership, through the Alphabet Workers Union, to let schools keep Search without AI. Sen. Ed Markey's four questions to Google are due Nov. 30. Why it matters now: for most students, the AI policy that actually runs is not the one in the board binder. It is the search box on the device. Rob's take: I like that districts are finding their leverage, and I'd be careful about what we call a win. Switching search engines is a settings change. Teaching students to judge a summary, check a source and notice when a machine is guessing is the part no default will do for us. A district that swaps providers and stops there has moved the problem, not solved it. Concrete implication for a district leader: write down, today, what students actually get for each surface: search, AI summaries in search, AI mode, chat apps and AI inside the learning platform. Note which ones your filter and admin console can separate. Then decide what you would do if the vendor's answer on Nov. 30 is no, before someone asks you at a board meeting.

- NYC schools weigh alternatives to Google search amid AI disputeABC7 New York
- LAUSD disables Google AI Overviews, reviews Google SearchNBC Los Angeles (City News Service)
- The nation's largest school districts consider ditching Google SearchThe Verge
- Google workers petition leadership over school AI and Search policyMother Jones
- Markey demands Google allow schools to disable AI services without blocking Search accessOffice of Sen. Edward J. Markey
Two big studies, one small effect
What changed: on Nov. 17 Harvard's Center for Education Policy Research, Dartmouth and NWEA released a student-level analysis of about 3.3 million students in 4,600 schools from 2021-22 through 2025-26. Strict cellphone bans using lockable pouches raised annual achievement growth in math and reading by about 0.01 standard deviations a year, which the authors equate to one to two weeks of added instruction, or about half the yearly loss seen before the pandemic. They also report the bans seemed to reduce the share of students with large reading losses, and found similar-sized effects for state bell-to-bell bans in 2025-26, with a warning that those state comparisons may be confounded. NPR's coverage adds the context that matters: an April study by Brian Jacob, Tom Dee and colleagues found "close to zero" overall effect, with high school math gains offset by middle school losses. The April study is peer reviewed and the new one is not yet. Dee calls the early results "largely disappointing"; co-author Tom Kane argues that at roughly $25 per student, pouches are still a cheap way to buy instructional time. In California, the first semester under the Phone-Free Schools Act looks like a patchwork. EdSource interviewed more than 40 people in seven districts and read the 15 largest districts' policies: some lock phones away all day, some allow them at lunch and passing periods, some leave enforcement to each teacher. Educators report more conversation and fewer fights, and researchers say it is too early to credit the bans for academic gains. Why it matters now: phone rules are becoming law faster than the evidence is settling, and both sides will quote a study this winter. Rob's take: restriction and instruction are separate questions. A pouch buys a small amount of attention back. It does not decide what happens with that attention. I'd rather a district be honest that the effect is small and cheap than oversell a lockbox as a learning strategy. Concrete implication for a district leader: if you adopt or tighten a phone rule, pair it with one measure you will actually track, such as office referrals, chronic absence or reading growth, a baseline from before the change, and a date to report back. Communicate the expected effect as small, because the evidence says it is.

The AI plumbing is the target
What changed: the attack surface this week was the layer that connects AI to everything else. On Nov. 17 two poisoned versions of Microsoft's Agent Framework, 1.26.1 and 1.26.2, were published to PyPI with a credential stealer that ran during normal Python startup even when the package was never imported; the toolkit sees about 200,000 downloads a month. On Nov. 18 CISA added CVE-2026-40217 in LiteLLM, a widely used gateway for routing traffic to AI providers, to its Known Exploited Vulnerabilities catalog with a Dec. 2 deadline. LiteLLM's advisory says the sandbox escape gives code execution in the proxy process, which runs as root in the default Docker image, and that reaching it requires a proxy-admin credential; the fix is 1.83.11. On Nov. 16 Google's Threat Intelligence Group reported that, for the first time, it saw attackers use AI frameworks to fully automate finding and exploiting vulnerabilities with no human in the loop, comparing code before and after patches and firing exploits on their own. Zero-days exploited hit 22 in August, against a 2025 average of 8 a month. And on Nov. 19 Google and Mandiant detailed how a suspected China-linked group exploited VMware vCenter CVE-2026-123890 starting in early September, before any patch existed, after first breaking into a remote support appliance; it could then forge sign-in tokens to bypass MFA. CISA set a Nov. 23 federal deadline. Why it matters now: districts are standing up AI gateways, agent toolkits and integrations quickly, often outside the asset inventory, while attackers are shortening the time from patch to exploit. Rob's take: an AI gateway holds the provider keys and sees every prompt, which can include student data. That makes it infrastructure, not an app, and it deserves the same patch clock as a firewall. The uncomfortable lesson from vCenter is that keeping something off the internet is not enough when the attacker comes in through the remote support tool you trust. Concrete implication for a district leader: this week, ask your team and every AI vendor three things in writing. Which AI gateway or proxy sits in front of our data, and what version is it on? Do any district or vendor environments install agent toolkits from public package registries, and are versions pinned? Is vCenter patched, and was it checked for compromise rather than only updated?

- GTIG AI Threat Tracker: Agentic Threat Activity Crosses a Significant ThresholdGoogle Threat Intelligence Group
- Microsoft's Agent Framework PyPI package compromised to steal credentialsBleepingComputer
- Sandbox escape in custom-code guardrail (CVE-2026-40217)BerriAI LiteLLM, GitHub advisory
- ARMORY CASTLE Exploits vSphereGoogle Threat Intelligence Group and Mandiant
- Known Exploited Vulnerabilities Catalog (JSON feed)CISA
Cheaper at the meter, unproven at the desk
What changed: OpenAI released GPT-6.2 Astra and GPT-6.2 Sol on Nov. 17. Astra costs $8 per million input tokens and $40 per million output tokens, 20% below GPT-6 Astra, and both models are available to Edu users in ChatGPT Work and Codex as well as the API. OpenAI rates both at the Critical threshold for cybersecurity under its Preparedness Framework and launched them with tighter cyber safeguards. Its headline benchmark comparison carries its own footnote: the rival's leaderboard results exclude nine tasks that used a fallback model. The independent picture is split. Artificial Analysis ranks Astra second on its Intelligence Index and puts Sol at 57, on par with GPT-6 Astra at about 15% of the cost per task. Early testers who handed the models real work were harsher: VentureBeat reports Matt Shumer calling them "essentially unusable," and Dan Shipper finding Astra often stopped early on end-to-end builds. In open weights, Tencent's 770-billion-parameter Hy4 is downloadable on Hugging Face under the new Tencent Hy License, which is derived from Apache 2.0 but is not Apache 2.0: companies offering the model as a service with more than $50 million in revenue or 100 million monthly users need a separate written license, while internal use is exempt. Microsoft announced MAI Licensed Weights on Nov. 17, letting organizations deploy and customize Microsoft models in their own Azure tenant, with customer data used for training only with express agreement. Availability is "coming soon through our Frontier Company engagements." Why it matters now: education pricing and Edu availability make a new default model likely in contract talks, and the governance questions are now about where weights live and who controls upgrades. Rob's take: a price cut is not a productivity gain until the model finishes the work your staff actually hands it. A benchmark rank is a lab's homework. Downloadable is not the same as supported, and announced is not the same as shipped. Concrete implication for a district leader: before changing a default model, run ten of your own real tasks through the old and new options, count how often a person had to send the work back, and price that, not the token meter. For any open-weight or tenant-hosted option, get the license and the upgrade terms in writing.

- Introducing GPT-6.2 Astra and GPT-6.2 SolOpenAI
- OpenAI has the #2 model in the Intelligence Index with the release of GPT-6.2 Astra and GPT-6.2 SolArtificial Analysis
- 'Essentially unusable': OpenAI launches GPT-6.2 Astra and Sol to negative reception from AI power usersVentureBeat
- tencent/Hy4 model card and Tencent Hy LicenseTencent, Hugging Face
- Introducing MAI Licensed WeightsMicrosoft AI
Policy set is not policy enforced
What changed: release status from the canonical GitHub releases, in UTC. OpenClaw shipped four stable builds in four days. 2026.11.4 (Nov. 16) requires approval for delegated persistent changes when authority is restricted or unverified, adds a standing approval setting, and says plainly that Full Access or standing approval can still authorize changes that enable host code execution even when direct execution tools are denied. 2026.11.5 (Nov. 17) changes the plugin SDK contract with no compatibility window, which breaks some custom plugins. 2026.10.33 (Nov. 18) is a gateway-only extended-stable build for the long-term line. 2026.11.6 (Nov. 19) fixes the Gateway allowing some shell commands to run without the approval the policy required, fixes automatic command review being silently disabled when an unchanged exec policy was reapplied, and fixes browser snapshots exposing filled passwords, one-time codes and card fields to the model. 2026.11.5-beta.4 (Nov. 16) and 2026.11.10-beta.1 (Nov. 19) are prereleases, not production builds. Hermes Agent shipped stable v0.23.2 on Nov. 16, which fixes a High-severity advisory (CVE-2026-130164), and stable v0.23.3 on Nov. 20, which turns proactive tool-result pruning on by default, sets prompt-cache lifetime to automatic and archives the tool results that compaction drops; full curated notes for both wait for v0.24.0. The daily canary builds are prereleases. On the research side, a Nov. 13 arXiv paper on agent harnesses found that a setup tuned on one model scored 68.41 when moved to another, below the plain starting setup's 73.89. Keeping a record of each fix's reason, prediction and result beat carrying over the bare setup, and in coding it improved 9 of 10 paired runs by 2.36 points on average. Why it matters now: two of this week's stable releases fixed approval rules that were configured but not enforced. Rob's take: a guardrail with no written reason is the first thing someone deletes during the next model switch, and a policy you set but never tested is a hope. The asset is not the model. It is the permissions, checks and review steps around it, plus the notes on why each one exists. Concrete implication for a district leader: for any agent tool in a pilot, keep a one-page decision log for each guardrail (what failure it targets, when it was added, whether it worked) and test one approval rule after every update by trying an action it should block. Keep prerelease builds in the sandbox.

- OpenClaw 2026.11.6 release notesOpenClaw
- OpenClaw 2026.11.4 release notesOpenClaw
- Hermes Agent v0.23.3Nous Research, GitHub
- Hermes Agent v0.23.2Nous Research, GitHub
- Self-Improvement Memory Enables Continual Recursive Self-ImprovementarXiv