SemiAnalysis Latest Interview: Storage Has Room to Double, Be Cautious with CPO in the Short to Medium Term, CPU is Just a Supporting Role

Source: Wall Street Watch

Every layer of AI infrastructure is under pressure, with opportunities and misjudgments coexisting.

Dylan Patel, founder of SemiAnalysis, recently participated in a podcast interview, systematically outlining the core dynamics and investment logic of the current AI infrastructure stack.

His insights cover model economics, memory supercycles, CPU repricing, the timeline risks of CPO, and structural opportunities in data center energy supply.

In response to the market's general skepticism about AI return on investment (ROI), Dylan revealed that Anthropic achieved positive free cash flow in the second quarter of this year, with annual recurring revenue exceeding $50 billion and a gross margin of over 70%. On the enterprise side, the productivity leap brought by the latest AI models far exceeds the increase in computing costs, prompting companies to cut other software expenses to maintain explosive growth in AI budgets.

In terms of hardware evolution, the shift towards inference models is reshaping market demand.

Dylan emphasized that storage is facing a structural shortage lasting several years, with still 2 to 3 times upside potential; meanwhile, although agents and reinforcement learning have driven up CPU demand, the seller's market has priced it too high. The growth of CPUs mainly comes from historical "catch-up," and their absolute value in AI servers is still far less than that of GPUs.

Dylan believes that the much-anticipated mass deployment of co-packaged optics (CPO) has been clearly postponed to the end of 2028 to 2029, unexpectedly extending the dividend period for copper cable connectors. The constraints on power grid transmission and distribution are forcing data centers to turn to "behind-the-meter power" (self-built power), creating a vast investment opportunity in industrial energy and power conversion supply chains beyond traditional chip investments.

Anthropic Leads the Way, AI Demand Narrative Begins to Materialize

In response to market skepticism about AI companies' ROI, Dylan Patel provided specific data.

"Anthropic achieved positive free cash flow in the second quarter, profitable in April, profitable in May, and June looks to be the same," he stated. Anthropic's annual recurring revenue has exceeded $50 billion, with a gross margin of over 70%. OpenAI's revenue has also rapidly increased with the rising adoption of Codex.

SemiAnalysis's own spending trajectory corroborates this trend. Last November, the company's 90-person team had an annual AI expenditure of less than $100,000; by the end of January this year, due to the large-scale rollout of Claude Code, this figure soared to $4 million annually; it has now reached $11 million, peaking at an annualized rate of $14 million. "Employee labor costs plus AI costs have already exceeded one-third, and by the end of the year, it is likely to reach half."

He also pointed out that newer, stronger models are not necessarily more expensive in practical use. An old model might require 100,000 tokens and 10 interactions to complete a task, while a new model might only need 25,000 tokens and 1 interaction. "Every time the model upgrades from 4.6 Opus to 4.7 Opus, our spending will first drop for a week and then soar again—because everyone sees that things that couldn't be done before can now be done."

He believes this is also one of the core reasons why Anthropic has an advantage in the competition with OpenAI: higher token efficiency and lower overall costs for users.

Memory: Structural Shortage, Not Ordinary Cycle

Among all hardware categories, Dylan Patel's judgment on memory is the most resolute.

"This is not a short-term shortage; it is a structural shortage that will last for years." He pointed out that memory capacity is only growing by 20% to 30% per year, while demand from the AI side is doubling and doubling again, and the gap between the two will continue to widen.

The core logic driving this judgment comes from the impact of inference models on KV caching. The context length of traditional conversational inference is measured in thousands of tokens, consuming limited KV cache; however, with the emergence of inference models represented by o1, the context length has exploded, and KV cache has sharply expanded, making memory the most directly benefited category. SemiAnalysis will release a report in December 2024 specifically pointing out this trend.

The rigid constraints on the supply side will force the downstream market to reallocate limited memory resources. He predicts that consumer electronics with low price elasticity will be the first to feel the pressure—mid-range and low-end smartphone manufacturers have seen shipments drop by 40%, and prices for iPhones and MacBooks will rise next year. "Memory prices will continue to rise, and consumer electronics will be compressed to a new level until AI gets the memory it needs, which will be truly sufficient."

He added that even if a cyclical downturn arrives, "long-term growth is undoubtedly certain from trough to trough."

CPU: Limited Catch-Up Market, Don't Overextend

The CPU has emerged as a new protagonist in this year's AI infrastructure narrative, but Dylan Patel holds a clear warning stance on this.

The logic behind the recovery of CPU demand is clear: reinforcement learning requires a large number of CPUs to run environmental validations (code unit testing, simulation operations, etc.); agent inference requires models to frequently call tools and interact with the real world, which heavily relies on CPU computing power.

At the same time, in recent years, a large number of AI chips have been shipped, but the accompanying CPUs are severely insufficient, and we are currently in a concentrated catch-up phase, with ARM, Intel, and AMD all benefiting, and NVIDIA's Vera CPU also providing a revenue guidance of $20 billion.

"But I want to give an important warning: there is a lot of catch-up effect here." He stated that once the historical backlog is caught up, only incremental demand will remain, and demand will return to normal. In absolute terms, Blackwell's single unit is about $50,000, while a CPU is about $5,000; even if the proportion of CPU increases, the dollar amount is still far less than AI acceleration chips.

"Memory and AI acceleration chips are the main components; CPUs are being revalued after being underestimated, and they are now priced more reasonably, but they will not grow indefinitely faster than AI chips."

Optical Interconnect: Long-Term Optimism, Caution with CPO in the Short to Medium Term

Networking and optical interconnects are another area of heightened market sentiment, but Dylan Patel takes a cautious stance on the deployment rhythm of CPO (co-packaged optics).

"I predict that CPO will not achieve large-scale mass production until the end of 2028 to 2029." He pointed out that the current manufacturing yield, chip design, and supply chain maturity have not yet reached the standards for large-scale deployment, and NVIDIA's Rubin and its subsequent architecture Feynman will still use a full copper solution, meaning CPO on the GPU side will need to wait for several generations of chip iterations.

He revealed that SemiAnalysis just released a report to institutional subscription clients last week, indicating that in the medium term, they are more optimistic about copper cables and non-CPO optical solutions, holding a cautious attitude towards CPO. Changes in the design of some downstream chips (such as the removal of the 800V design in Rubin Ultra) have further delayed the CPO deployment timeline. Companies like Amphenol, which produce copper cable connectors, will benefit more than expected as a result.

"CPO will happen in the long term, and copper cables will eventually be replaced, but the timeline has been delayed, and in the short to medium term, there are still significant opportunities for copper cables."

Power: Self-Built Power Will Become Mainstream, Diverse Innovation Paths

The power supply for data centers is becoming the hardest physical constraint for AI growth.

According to Dylan Patel's predictions, the new electricity consumption for data centers will be 20 GW this year, 30 GW next year, and 50 GW the year after, with near-explosive growth.

He breaks down the energy issue into three dimensions: transmission, generation, and conversion. Transmission is the most challenging aspect to overcome, involving regulatory policies, local power company monopolies, and cost-sharing mechanisms, which are difficult to change in the short term. Generation and conversion, however, present broad opportunities.

He predicts that in the coming years, half of the new electricity consumption for data centers will come from "behind-the-meter power," meaning self-built power by enterprises, rather than relying on the public grid.

The current mainstream solution is combined cycle gas turbine (CCGT) units from manufacturers like GE Vernova, Mitsubishi, and Siemens; at the same time, non-traditional solutions such as reciprocating engines, industrial gas turbines, and even retrofitted engines from ships, trains, and trucks are emerging. "It sounds rough, but it works, and it is already being used."

In the longer term, he predicts that within about two years, the comprehensive cost of solar energy combined with storage will be lower than that of gas power generation; further out, there are concepts of space data centers—deploying computing chips in orbit, where solar panels do not need to penetrate the atmosphere, achieving much higher energy density than on the ground and eliminating the need for storage.

The conversion side is also full of investment opportunities, from IGBT, silicon carbide to gallium nitride MOSFETs, as well as solid-state transformers, UPS, and supercapacitors, the entire voltage conversion chain is rapidly evolving.

Currently, SemiAnalysis's largest research department is no longer semiconductors, but a team internally referred to as "DEI" (Data Center, Energy, and Industry), tracking the deployment dynamics of every data center and power plant globally.

Disclaimer: This content is provided for general branding and informational purposes only and doesn't constitute financial, investment, legal, or tax advice. Any events, rewards, online events, or related information mentioned herein should not be considered a recommendation, solicitation, or invitation to purchase, sell, trade, or otherwise deal in any crypto assets or to use any services. Crypto assets are highly volatile and may result in loss. WEEX services and online events may not be available in all regions and are subject to applicable laws, regulations, and eligibility requirements. You are responsible for ensuring that your use of WEEX services complies with local laws and for carefully assessing the risks before participating in any crypto-related activities.