Controversial AI Explainability Research Draws Attention from Sam Altman
In recent months, OpenAI has been in negotiations with AI startup Goodfire regarding a potential acquisition or collaboration (my colleagues and I reported this morning that OpenAI had considered acquiring the startup, which analyzes how AI "thinks," for $5 billion or more). During the negotiations, OpenAI CEO Sam Altman showed particular interest in the research of Nick Kamarata, who previously worked at OpenAI and is now at Goodfire. His recent insights on how artificial intelligence thinks have sparked significant reactions within the industry.
Kamarata published a paper last month and released a follow-up report last week, igniting heated discussions among AI researchers. Some praised the findings as a potential breakthrough in understanding AI models, while others dismissed it as mere hype. This polarized response indicates that certain areas of AI research remain immature, with researchers unable to reach a consensus on what constitutes high-quality research. However, even without validated technology, related startups can still achieve valuations in the billions.
Goodfire's core research focus is mechanistic interpretability, commonly referred to as "mech interp" in the industry. The goal of this field is to understand the internal workings of AI models, rather than just observing their external behaviors. AI models consist of trillions of values, functioning like a "black box" that cannot be directly read like traditional software code. This makes it difficult to predict how AI will behave in new scenarios and complicates error tracing when mistakes occur.
Kamarata's paper is titled "Neural Networks Are Readable," and it argues that the artificial neurons within AI models (mathematical functions built on vast amounts of data) are actually more interpretable than researchers previously believed. Following this reasoning, startups like Goodfire, which aim to decode AI's internal logic, may have a more optimistic future.
For example, as early as 2017, mechanistic interpretability researchers discovered a unique neuron in an image AI model: it would activate regardless of whether the input was a picture of a car or a cat. Many neurons exhibit this phenomenon, processing multiple seemingly unrelated tasks, which has puzzled researchers. In fact, in 2023, while still at OpenAI, Kamarata attempted to analyze the neurons of GPT-2 using GPT-4, only to find that at most 1.7% of the neurons could be interpreted.
However, Kamarata believes that advancements in model training techniques and interpretative methods have changed the landscape for the better. Taking Alibaba's Tongyi Qianwen Qwen3-32B as an example, about 90% of its neurons serve a single function and are thus interpretable. Upon re-examining the classic neuron that responds to both cats and cars, he found that the defining feature activating this neuron is the similar V-shaped outline of cat faces and car fronts. {#quote_BABA}
"In my view, the story of interpretability is largely about how what we observe under the microscope is far better than we anticipated," he wrote. "It's just that the microscope itself hasn't been calibrated properly, and the adjustments needed are often very subtle."
Several leading researchers in the field of mechanistic interpretability disagree with Kamarata's conclusions. They acknowledge the value of the research direction but argue that the paper exaggerates the significance of the findings.
Neil Nanda, head of mechanistic interpretability at Google DeepMind, stated on social media: "This work is interesting, but I don't think it brings a significant update to the overall picture of interpretability." Nanda wrote that the paper overestimates our current understanding of neurons; even if all neurons could be understood, many unresolved questions about the model's functioning would remain. {#quote_GOOG}
OpenAI researcher Leo Gao also participated in the discussion, publishing a rebuttal titled "Neural Networks Are Not Readable by Default." He argued: "The paper is far from sufficient to prove that we have understood the vast majority of neuron behaviors; these results do not indicate that the interpretability problem has been solved, nor do they suggest that this work is much simpler than previously thought."
Liv Gordon, who previously worked at Goodfire and is now at Anthropic, replicated some of Kamarata's experiments and obtained different results. She also pointed out that not all cat faces are V-shaped, and that this neuron does not respond to many V-shaped objects (such as bird beaks).
In the face of mixed reviews from various quarters, Kamarata remains optimistic. He posted on social media about AI interpretability: "I believe this problem can be solved. Whether existing simple solutions work, existing approaches are generally feasible but need new tools, or we require disruptive new ideas, we will eventually find the answer.
This content is provided for general informational purposes only and doesn't constitute financial, investment, legal, or tax advice. Any events, rewards, online promotions, or related information mentioned herein should not be considered a recommendation, solicitation, or invitation to purchase, sell, trade, or otherwise deal in any crypto assets. Crypto assets are highly volatile and may result in loss. The availability of WEEX services, products, and related events may vary by region. You are responsible for ensuring that your participation is in accordance with applicable local laws and regulations.
You may also like

Why Is Bitcoin Falling? BTC Near $64K as Anthropic IPO Looms

Does an AI Trading Tool Need Permission to Place Orders on WEEX?

Tether-backed Twenty One Capital buys $10M in Bitcoin as stock slides

The Next Hyperliquid Will Emerge from These 6 Companies

A $25 million MEV ruling exposes a dangerous blind spot in Wall Street’s tokenization push

A Super Empire on a Fragile Foundation: How Jensen Huang Has Controlled NVIDIA for 33 Years

Binance’s He Yi explains AI Pro transfers after hack-day withdrawal claims

Q3 Institutional Holdings Observation: Large-Cap Tech Underweight Narrowing, Funds Concentrating on AI Infrastructure, Software Stocks Neglected

Microsoft Ignite Conference Kicks Off Today! Focus on Copilot Monetization and AI Computing Returns, Can the Enterprise AI Flywheel Deliver?

How to Reassess Core Assets with 'Revenue + Narrative' After Market Crash?

Circle Expands Disparte and Walia Roles as Stablecoin Operations Scale

US Deploys Third Aircraft Carrier to the Middle East as Iran Intensifies Threats Amid Tanker Attack in the Strait of Hormuz

Wall Street Sees 'Gazumping' Phenomenon as Hedge Funds Compete for Top Talent
Shiba Inu (SHIB) Price Prediction 2027: What Would It Take to Reach $0.00001?
See our Shiba Inu price prediction for 2027, including the $0.000006 base estimate, the market cap needed for $0.00001, and how much burns could help.

Huya's Q3 2026 Financial Report: Operating Profit Turns Positive for the First Time in Nearly Four Years, Live Streaming and Game-Related Services Continue to Improve Efficiency

Aave Labs Reports: A 97-Person Team Supporting What Kind of DeFi Business?

Prediction: Anthropic Will Have a Successful IPO but Then the Stock Will Trade Like SpaceX

Intesa Sanpaolo raises XRP and Solana ETF holdings, cuts Bitcoin and Ethereum

Why Are Tokenized Stocks Seeing Strong Buying Pressure While Prices Keep Falling?

Report: Israel Uploads Millions of Photos to Amazon's Facial Recognition Platform to Track Palestinians

WEEX P2P now supports CAD —Merchant Recruitment Now Open

Cardano Foundation seeks 25M ADA for 2027 events despite $1.2M funding gap
SpaceX (SPCX) Stock Price Prediction 2027: How Much Can Starlink Growth Offset AI Spending?
SpaceX stock price prediction for 2027: See the $118–$148 base case, how much Starlink could offset AI spending, and what it would take to reach $200.

Chevron Expands in the Gulf of Mexico with Plans to Acquire Majority Stake in Oil Field for Over $5 Billion
Anthropic IPO Price Prediction: Where Could ANT Open After Pricing?
Anthropic IPO price prediction: See ANT’s $80–$87 base-case opening range, how the final offer price changes it, and what could push shares above $90.

IOSF: The Cost Paradox of Video AI - Why Is Generating Cheaper but Final Cuts Still Expensive?

Will the $50 billion request materialize? The U.S. transportation system urgently needs funding for upgrades, but Congress has become a "lame duck"

Japan Proposes Establishing an 'Asian Version of the International Energy Agency' to Enhance Southeast Asia's Oil Storage Capacity in Response to Energy Crises

Federal Reserve Warns Korean Investors About Leveraged ETF Risks

Mercado Bitcoin Withdrawal Update: Pix Restart and Crypto Suspension
Mercado Bitcoin set November 17 for Pix deposits and withdrawals in Brazilian reais. Read the withdrawal rules and the background to the crypto suspension.