Skip to main content

AI’s growing role in business is exposing challenges that new models alone can’t solve. On the latest episode of This Week in AI, we examined restrictions on autonomous agents, NVIDIA’s leadership crisis, and research into making AI systems more reliable.

Agentic commerce is running into resistance

Anthropic is turning Claude into a broader workplace platform. Alongside its upgraded Fable 5.2 and Mythos 5.2 models, the company introduced Claude Meetings and Claude Sheets, Whiteboards, and Data. Anthropic also expanded its enterprise offerings with self-hosted agent infrastructure. Companies can connect agents to internal systems while keeping credentials and session histories on their own infrastructure, even though the models still run through an external provider.

OpenAI is bringing more of the shopping process into ChatGPT. Its new business profiles let companies manage how customers discover them, while business agents can answer questions and recommend products. The company also introduced ChatGPT Wallet, which allows agents to make purchases without accessing customers’ actual payment card details. Users still need to review and approve these transactions before payment. Bringing business discovery and payments into ChatGPT puts OpenAI in more direct competition with search engines and online marketplaces.

Airlines, banks, and retailers, however, are already restricting outside agents. United Airlines is blocking unauthorized agents from booking flights, while financial institutions and retailers, including Amazon and eBay, have imposed their own limits. Their concerns include privacy, security, and unintended purchases. An agent may be technically capable of completing a transaction but prohibited from accessing the service. Developers will need to account for those limits, while businesses that allow such transactions must establish who bears responsibility when an agent makes an unintended purchase.

NVIDIA faces a test of leadership and governance

NVIDIA founder Jensen Huang resigned as CEO following reports of personal relationships that prompted an investigation into potential national security concerns. US authorities have been investigating whether a Chinese businesswoman involved in the controversy had intelligence ties to Beijing. Those allegations remain unproven. NVIDIA’s board launched its own investigation, and longtime CFO Colette Kress has stepped in as interim CEO while the company searches for a permanent successor.

NVIDIA’s influence over the AI chip market makes its leadership transition especially consequential for customers and investors. Huang led the company for 33 years and helped establish its dominant position in AI infrastructure. Now Kress and the board must maintain customer confidence while navigating the investigation. We also discussed NVIDIA’s investments in other AI companies and the circular financing arrangements that have raised concerns about how the industry measures growth. Any changes to those investments could affect companies that depend on NVIDIA for both computing infrastructure and capital.

NVIDIA’s leadership transition also raises questions about succession at founder-led technology companies. After three decades with Huang at the helm, the board must show it can provide independent oversight and maintain the company’s direction through a period of uncertainty. How NVIDIA manages that transition could offer lessons for other AI companies whose identities remain closely tied to their founders.

Researchers are probing why AI systems fail

New research found that giving AI agents memory of previous tasks can make them perform worse than without memory when they learn the wrong lesson or apply past experiences out of context. The proposed WikiTrace system links memories to their original tasks and evidence, allowing agents to check what they’ve learned before acting.

Other researchers are investigating how to detect uncertainty in language models through internal signals, hidden states, token probabilities, and whether the model gives conflicting answers to the same question. This matters because AI can sound confident even when it is wrong. If we can spot when a model is unsure, we could prompt it to check its answer or acknowledge its uncertainty, helping people avoid relying on incorrect information when making decisions. Meanwhile, MIT researchers explored weight-space programming, using language models to inspect and edit other neural networks’ weights. In early experiments, the approach corrected a simulated driving problem and improved another model’s math performance.

Benchmark scores alone don’t explain why agents misuse memories, sound confident even when they are wrong, or fail at specific tasks. Research into these problems could give developers better ways to diagnose and correct mistakes as AI systems take on more responsibility.

What’s next

Next week, we’ll follow up on NVIDIA’s leadership transition and Anthropic’s anticipated IPO, along with other developments across the AI industry.

We’ll be back on Monday, November 23, for another episode covering the latest developments in AI research and applications. You can catch previous episodes on YouTube, Spotify, Apple, or wherever you get your podcasts.

Post topics: AI Agents•AI Governance•This Week in AI