AI Compute Architecture and Evolution Trends

Abstract—The focus of AI development has shifted from academic research to practical applications. However, AI development faces numerous challenges at various levels. This article will attempt to analyze the opportunities and challenges of AI from several different perspectives using a structured approach. This article proposes a seven-layer model for AI compute architecture, including Physical Layer, Link Layer, Neural Network Layer, Context Layer, Agent Layer, Orchestrator Layer, and Application Layer, from bottom to top. It also explains how AI computing has evolved into this 7-layer architecture through the three-stage evolution on large-scale language models (LLMs). For each layer, we describe the development trajectory and key technologies. In Layers 1 and 2 we discuss AI computing issues and the impact of Scale-Up and Scale-Out strategies on computing architecture. In Layer 3 we explore two different development paths for LLMs. In Layer 4 we discuss the impact of contextual memory on LLMs and compares it to traditional processor memory. In Layers 5 to 7 we discuss the trends of AI agents and explore the issues in evolution from a single AI agent to an AI-based ecosystem, and their impact on the AI industry.

Introduction. HE focus of AI development has shifted from academic research to practical applications. Current AI development began with the 2012 Alexnet project [1], which demonstrated the potential of neural networks. Following the release of the Transformer architecture [2] in 2017 and the discovery of scaling laws [3], the number of AI model parameters and computational requirements increased dramatically, sparking a race to develop large language models (LLMs). By 2022, ChatGPT attracted widespread public interest, and led to emergence of various on generative AI. Then, AI computing has further expanded into the fields of Agentic AI and Physical AI. The rapid development of AI has the potential to boost productivity. If an AI-based ecosystem is successfully established, it will significantly increase global productivity, with an impact comparable to previous industrial revolutions. However, AI development faces numerous challenges, including scaling computing power, energy efficiency, neural network models training, AI agents, physical AI, AI-based ecosystem, and business models.

Discussion / Conclusion. In this article, we analyze AI compute architecture using Seven-layer model. The key points we observed during the analysis are: • The computing power required for AI training has increased 100 million-fold over the past decade. Scale- Up chip computing power is insufficient to meet this demand. The Scale-Out strategy is necessary, connecting many chips to provide computing power. This has driven enormous demand for advanced semiconductors. • The computing power required for AI inference is likely to be far greater than that required for training. As test-time computing becomes mainstream, computing power for inference will increase rapidly. Furthermore, future users of AI inference will include not only humans but also AI agents and robots, who will also require extensive AI inference. Therefore, it is foreseeable that AI inference will require extremely large computing demands in the future.