79 large-scale models with more than 1 billion parameters have been released in China, concentrated in Beijing and Guangdong

Source: The Paper

Reporter Zhang Jing

According to incomplete statistics, at present, 79 large-scale models with a scale of more than 1 billion parameters have been released in China, and 14 provinces/regions are carrying out research and development of large-scale models, mainly concentrated in Beijing and Guangdong, of which 38 large-scale models in Beijing and 20 in Guangdong large model. Among the large models in China, more than half of the large models have been open sourced, and universities/scientific research institutions are the main force of open source.

·Beijing, Jiangsu, Guangdong, Shanghai, etc. are areas with relatively large large-scale model talents, providing key intellectual support for large-scale model research and development. However, the total amount of talents is still scarce. Large-scale models have a high threshold and require high-quality AI talents. At present, the number of large-scale model talents in various places is insufficient.

Image source: Generated by Unbounded AI tool

There are 79 large-scale models with more than 1 billion parameters in China, mainly concentrated in Beijing and Guangdong.

On May 28, at the Artificial Intelligence Large Model Development Forum, one of the parallel forums of the 2023 Zhongguancun Forum, Zhao Zhiyun, director of the China Institute of Scientific and Technological Information, released the "Research Report on China's Artificial Intelligence Large Model Map". At present, China's scale of more than 1 billion parameters 79 large-scale models have been released, mainly in Beijing and Guangdong, including 38 large-scale models in Beijing and 20 large-scale models in Guangdong. Among the large models in China, more than half of the large models have been open sourced, and universities/scientific research institutions are the main force of open source.

The Transformer network structure released by Google in 2017 is the source technology for the development of large models. Since then, large model technology has achieved iconic technological breakthroughs in natural language understanding, computer vision, and intelligent speech. Generalization ability and other aspects have achieved leapfrog development. ChatGPT has further stimulated the explosive emergence of large language models, and has also attracted a large number of R&D teams to invest in the development of more large models such as general vision and multimodality. Zhao Zhiyun said that in just over five years, the large-scale model technology has formed a huge technology group, and derived a large-scale model family covering various parameter scales, various technical architectures, various modes, and various scenarios.

Judging from the development trend of global large-scale models, organizations such as Google and OpenAI in the United States continue to lead the forefront of large-scale model technology, and more and more R&D teams in Europe, Russia, Israel, etc. are also investing in the research and development of large-scale models. Judging from the distribution of large-scale models released around the world, China and the United States lead by a large margin, accounting for more than 80% of the global total. The United States has always ranked the highest in the world in terms of the number of large-scale models. Simultaneous growth trend.

In the development trend of large models in China, Zhao Zhiyun said that according to incomplete statistics, 79 large models with a scale of more than 1 billion parameters have been released in China so far, and 14 provinces/regions are carrying out research and development of large models, mainly in Beijing and Guangdong, including 38 large models in Beijing and 20 large models in Guangdong. In terms of model domain distribution, natural language processing is still the most active focus area for large-scale model research and development, followed by multimodal domains, and there are still fewer large-scale models in the fields of computer vision and intelligent voice. In terms of the distribution of research and development subjects, different innovation subjects such as universities, scientific research institutions, and enterprises are all participating in the research and development of large models, and the joint research and development between academia and industry is still insufficient. "Large models have been developing rapidly since the beginning of this year, but we have also observed a trend of shrinking cooperation, and we need to pay attention next."

At the same time, the computing power-model matching degree is high, and the public computing power is developing rapidly. Beijing, Guangdong, Zhejiang, Shanghai and other places have the largest number of large models, and these four places are also the regions with the highest number of artificial intelligence server purchases in the past three years. Localities are also supplementing the rapidly growing demand for artificial intelligence computing power by providing public computing power, and providing more computing power support for large-scale model research and development.

Talent is also an important pillar of artificial intelligence. Beijing, Jiangsu, Guangdong, Shanghai, etc. are regions with a relatively large number of large-scale model talents, providing key intellectual support for large-scale model research and development. However, the total amount of talents is still scarce. Large-scale models have a high threshold and require high-quality AI talents. At present, the number of large-scale model talents in various places is insufficient.

"Through the release of large model papers, we can see that the academic influence of China's large model is gradually forming. From the perspective of regional influence, Beijing, Guangdong, and Shanghai are the highest in the country in terms of the number of papers and the number of paper citations. "Zhao Zhiyun said that the CogView model paper jointly developed by Tsinghua University, Ali and Baidu has the highest number of citations. However, compared with the academic influence of foreign leading large models, there is still a big gap.

From the perspective of open source innovation ecology, more than half of the large models in China have been open sourced, and Beijing, Guangdong, and Shanghai are among the top three in the country in terms of the number and influence of open source. Universities/scientific research institutions are the main forces of open source. "ChatGLM-6B of Tsinghua University, MOSS of Fudan University, and Baidu's Wenxin series of large-scale models have the highest influence on open source." Zhao Zhiyun said.

The industrial application of Chinese large-scale models develops along two paths. The first is the continuous expansion of the application field of general-purpose large-scale models. The large-scale model is developing rapidly to create a cross-industry general artificial intelligence capability platform, and its application industry is accelerating its penetration from office, life, and entertainment to medical care, industry, and education. The second is the continuous deepening of professional large-scale models in vertical fields. A group of professional large-scale models for vertical fields such as biomedicine and remote sensing meteorology give full play to their deep advantages in the field and provide high-quality professional solutions for specific business scenarios.

Advocate to strengthen the overall planning of resources and research and development forces, and promote the orderly development of large models.

In view of the lack of development of large models, Zhao Zhiyun put forward 4 suggestions and prospects:

One is to strengthen the overall planning of resources and research and development forces to promote the orderly development of large models. By strengthening the overall planning of computing resources such as intelligent computing centers, supercomputing centers, and cloud computing centers, formulate public data sharing catalogs and sharing rules, and promote the orderly opening of data classification and classification.

The second is to accelerate basic research and technological innovation, and enhance academic and open source influence. Large-scale model technology is still in the early stages of development, and there is huge potential for basic theory and technological innovation. Through miniaturization techniques such as distillation and quantification, the model is "slimmed down", providing technical support for the miniaturization and green development of large models. Further strengthening industry-university-research cooperation and encouraging the open source of large models will also accelerate the technological progress of large models.

The third is to strengthen the leading role of the scene in the development of large models, and create a benchmark project for large models. Based on industry-specific training data sets, build professional large models in the fields of finance, medical care, and electric power, and achieve high-quality application breakthroughs in specific business scenarios. We also hope to reversely promote the iterative upgrade of large model technology through application scenarios and application data.

The fourth is to strengthen international cooperation and actively participate in global artificial intelligence governance. Jointly promote the governance of large models with a responsible attitude, and hope that the governance principles and ethical norms of artificial intelligence can further take root in the entire chain of large models. At the same time, strengthen global cooperation on artificial intelligence governance on the basis of increasing consensus.