Frost & Sullivan in collaboration with LeadLeo releases the '2024 China GenAI Technology Stack Market Report'

The design of the generative AI technology stack plays a crucial role in assisting developers in improving the development efficiency of generative AI applications, providing comprehensive support at the system level for generative AI tasks, and exploring solutions to new demand challenges.

For developers, the GenAI technology stack can improve application development efficiency, provide more intuitive editing and debugging tools, and allow for direct monitoring of all aspects of the AI cycle.

For generative AI tasks, the GenAI technology stack provides system-level full support. Firstly, in response to the challenges of big data and large models, the technology stack needs to support larger models and more modal inputs, offering more powerful and scalable computing capabilities. Secondly, in the context of cloud and cluster scenarios, the technology stack needs to consider and support the expansion and deployment of AI tasks to support distributed computing and elastic computing, allowing users to use resources as needed.

Finally, the expansion of new demands also brings many challenges. Firstly, in response to the growing demand for multi-organization, multi-user shared cluster resources, providing a fair, stable, and efficient environment is the primary consideration for the technology stack. Secondly, in the face of fragmented edge hardware and software stacks, how to enable models to be deployed across different software and hardware platforms through a single training is also an issue that needs to be solved. Lastly, the important information within models can lead to security issues with weights; for enterprise-level or public cloud environments, the technology stack needs to provide higher levels of security and privacy protection measures.

Building a complete end-to-end GenAI application involves complex and rich module components and processes, ranging from user interaction to the output of actionable application results. It involves key modules such as AI model preparation, tuning, service, and governance. During the process of supporting GenAI application construction, users should select and combine these module components based on factors such as project, business line, and organizational maturity, and not every application involves all component modules.

RAG——Build generative AI applications that understand business betterIn the context of the emergence of specialized models in various industries, the limitations of large language models in specific domains due to missing enterprise knowledge are becoming apparent. Therefore, RAG (Reactive Adaptive Generation) based on external databases that can timely call up the latest data and domain-specific knowledge for supplementation is crucial. In addition, the integration of authoritative external data information and control over data sources can improve the accuracy and interpretability of model responses. The separation of data from the model itself also provides better data privacy management.

MAS——Provides more efficient complex task solutionsMulti-agent systems are an important branch of distributed AI. They decompose 'large and complex' tasks into multiple individually serving agents that communicate with each other, share information and resources, and collaborate to achieve a common overall goal. In complex tasks, multi-agent systems compensate for the single perspective difficulty of individual agents and possess core advantages in terms of robustness and fault tolerance, flexibility and scalability.

Prompt Engineering --Guide for more accurate model outputThe prompt engineering design optimizes the prompts to guide the model to generate responses that meet expectations. The quality of prompts directly affects the context and output, and it has advantages such as efficiency improvement, low training cost, and flexibility. Therefore, it is particularly crucial for models. However, overly specific prompts may also limit the model's creativity, so users need to make trade-offs based on the balance point.

Guardrail --Build safer applicationsThe importance of guardrails cannot be underestimated in the dynamic development of AI technology. Guardrails play a core role in preventing the abuse of AI technology, ensuring content fairness, maintaining public trust, and complying with regulatory standards. Guardrails are a fortress against potential risks associated with AI technology deployment and an indispensable part in building secure and reliable applications.

API ServiceBoost smoother application integrationThe large model API service provides a powerful, flexible, and easily integrable method to assist users in quickly integrating AI functions seamlessly into programs and services, thereby reducing development barriers and improving development efficiency. The core advantages of the API service are reflected in promoting software integration, achieving data sharing, enhancing scalability, and strengthening system security.

MLOps——Achieve faster and more reliable application releasesMLOps integrates machine learning application development, system deployment, and operations. By automating and standardizing across the entire machine learning lifecycle, it enables faster product development and market launch, more efficient teamwork, and continuous model improvement and optimization. It is an important component that assists enterprise users in efficiently deploying models into production environments.

For users in different fields, the construction of high-quality models, optimization for security and compliance, reduction in inference costs, unlocking of data value, and the realization of product applicationization are all major considerations in building GenAI applications.

Construction of high-quality modelsLarge models are at the core of generative AI applications. For users in different fields, the adaptability to specific scenarios and the controllability of generated content quality are the main considerations for model quality. Model adaptability provides users with more targeted decision-making suggestions in professional fields, enhancing the flexibility of solutions in related areas. Controllability improves users' efficiency in generating domain-specific content. For model adaptability in different business scenarios, users can optimize the model-specific path by considering factors such as computing resource costs, effect presentation, and data volume levels. Common model-specific methods include direct tuning at the application layer, leveraging external knowledge bases, parameter fine-tuning, and continuing training. In addition, model selection and evaluation are common ways to improve the controllability of model-generated content quality. Model selection is the core of understanding problems, selecting the optimal model for business scenarios based on principles such as minimum validation error. Model evaluation quantifies and optimizes model performance through a series of evaluation metrics such as accuracy, determining the model's prediction accuracy and practicality.

Security and Compliance OptimizationThe process of building generative AI applications involves security and compliance issues such as data and models, posing potential risks to users such as privacy leaks and incorrect decisions. Security and compliance optimization are crucial in aspects such as user rights and privacy protection, market order, and corporate responsibility maintenance. Therefore, security and compliance are one of the main considerations in building generative AI applications. Building a complete security defense chain from application construction to business deployment is the basic framework for optimizing security and compliance. Based on three levels: known risk attacks, model security, and application deployment architecture, targeted defense detection mechanisms can be designed respectively. The platform can provide users with a full-process security management tool to achieve comprehensive security assurance.

Reduced inference costsModel inference is a crucial link in connecting terminal scenario requirements. With the increasingly diverse business scenarios integrated into generative AI, inference has great potential for development in the future market. However, at present, the main part of AI expenditure is still concentrated on inference. Therefore, reducing inference costs is an important factor in boosting the development of AI technology and enhancing users' accessibility to generative AI. Model inference optimization technology can be carried out comprehensively at the data layer, model layer, and system layer. Among them, model layer optimization measures are widely used. They include a series of technologies such as knowledge distillation, model pruning, and model quantization, which reduce the model size and computational difficulty while ensuring model performance.

Release of data valueThe development of large models has brought new demands for data architecture. In the implementation of generative AI applications within user enterprises, data collaboration between enterprises and models, as well as the release of data value, have become important considerations in the process of 'data-driven decision-making'. The unification of data assets, the operation of automated tools, and the computing capabilities of universities all affect the effectiveness of data value release. The release of data value from generative AI applications can form a complete data optimization loop through three layers: computing power, models, and decision-making. Among them, the model optimization and decision optimization paths are relatively lightweight and flexible. At the model level, users mainly combine specific data with large models through methods such as RAG to generate unique value; decision optimization can be achieved through 'AI+BI' tools to provide intelligent decision support.

Realization of product applicationThe application of generative AI products involves steps such as setting product application goals, selecting generative AI tools, integrating product processes, and collecting user experience feedback. Among these, generative interface design and interaction methods with users are the most direct factors affecting product application feedback and iteration. The application of generative AI products is a crucial process in transitioning generative AI technology from experimental theory to practical application. The application of generative AI products can be optimized based on the most direct user interaction experience. Platforms can innovate in user interface design, such as CUI dialogic user interfaces that integrate LUI and GUI interactions. By using more intuitive and natural language interactions, the difficulty of user operations can be reduced, providing users with more intelligent and convenient interaction methods, thereby optimizing the user experience of the product.

Under the new technology and service models, the GenAI technology stack will develop in directions such as modularization and standardization, platformization and simplification, and decentralization.

Modularization and standardizationCurrently, there are challenges such as fragmentation and incompatibility among generative AI technology stack tools and systems. Modularization and standardization will become important development directions in the future. By dividing the technology stack into a series of modules and standardized interfaces, it becomes easier to replace components and upgrade them, thereby improving the flexibility and scalability of the system.

Platformization and SimplificationThe generative AI technology stack, based on model development, scenario adaptation, inference deployment, and product application, will further improve the application development ecosystem and form a one-stop development service platform. In addition to platformization, there is also simplification of the platform. By providing user-friendly tools and open APIs, the development threshold for users is reduced, and a complete development solution with strong operability is provided.

DecentralizationCurrently, AI systems still rely heavily on centralized architectures for efficient management and control, but this has come with risks such as data privacy. AI decentralization aims to distribute the development, deployment, and control of AI across multiple entities or users, in order to improve transparency and reduce abuse risks. It is a major trend in future development.

Business Model Reconstruction under MaaSThe MaaS model, which aims to help users lower the threshold for model development through a series of tools and services, will become the mainstream business model for large models in the future. It will have a cascading impact on generative AI applications in terms of commercial ecosystems and product implementation, thereby reconstructing the generative AI business model in reverse.

The Transformation of Information Goods under GenAIAs an underlying innovation technology, GenAI has had a comprehensive and revolutionary impact on all aspects of information products. Moreover, GenAI holds great potential for creating new types of information products, which also boosts the possibility of a new era of products and the emergence of super products.