The global enterprise AI market is expanding rapidly, and more and more businesses are exploring AI’s potential to drive innovation and efficiency. The AI market is expected to reach an estimated $1,811.8 billion by 2030, and Gartner predicts that over 80% of enterprises will adopt generative AI models or APIs within the following year. However, only a tiny percentage of AI applications make it into production.
So what is the problem?
Significant challenges exist when you try to take your experimental AI system into production, specifically when it involves generative AI. As companies work towards integrating AI into their operations, it is critical to clearly understand the potential challenges and necessary strategies needed to successfully architect AI-powered APIs and applications.
AI Development: Value Proposition
The strategic value of AI lies in its potential to enhance operational efficiency, streamline processes, and improve the overall user experience. By automating repetitive tasks and freeing up resources, focus on higher-value activities.
However, to unlock the full potential of AI, its integration must be aligned with an organization’s core objectives, and this is no easy task.
Challenges in AI Development
Organizations have traditionally invested significant time in collecting data, training models, and testing them, making the development of AI applications lengthy. The advent of pre-trained models has significantly accelerated this process. By utilizing pre-trained models and integrating them with data, tools, and APIs, it is now possible to prototype AI systems more quickly.
Despite this advancement, developers face challenges building AI systems and transitioning applications from prototype to production. As a result, the percentage of AI applications that successfully reach production remains very low. Addressing these challenges can save time and effort while ensuring the successful deployment of scalable AI applications that provide long-term value.
Let us examine some specific challenges related to transitioning a Generative AI prototype to production and explore ways to address them effectively.
AI Model Development Challenges
Evaluation
Ensuring that your AI system achieves an acceptable level of accuracy is essential for its success. However, achieving the desired level of accuracy can be challenging, particularly for complex use cases, and often requires substantial effort.
Selecting the right combination of models is critical to achieving the desired accuracy. Factors such as model size (e.g., number of parameters), architecture, training data, and training techniques influence accuracy. Accuracy can also be enhanced by fine-tuning models and integrating external data sources to incorporate domain-specific knowledge.
The prompts used in your AI system play a key role in shaping its behavior. Prompt engineering provides established guidelines and best practices for optimizing outcomes. While techniques such as zero-shot and few-shot prompting are effective for straightforward tasks, advanced approaches like Chain-of-Thought (CoT), Tree-of-Thought (ToT), and ReAct (Reason and Act) are better suited for handling complex scenarios, as they enable structured reasoning and decision-making.
Evaluating AI models requires more than measuring the accuracy of the final output; it also involves examining the quality of intermediate steps. Overlooking these steps can lead to logical errors, inefficiencies, or other issues in the reasoning process. A thorough evaluation should address edge cases, fairness across different groups, robustness to adversarial inputs, and the validity and consistency of intermediate steps.
Building a practical AI system requires experimenting with different models, optimizing prompts, integrating private data, and fine-tuning models as needed. Evaluation should go beyond assessing the final outputs and include examining intermediate steps to ensure consistency, validity, and reliability.
Latency
Latency is a critical factor in system design, as it directly affects user experience. In Generative AI applications, high latency from slow models can frustrate users and degrade the overall experience. This challenge is amplified in agentic workflows, where AI systems must interact with models multiple times, resulting in high latency.
While faster models can alleviate this issue, they often involve trade-offs with accuracy, requiring careful consideration to find the right balance. Techniques not requiring changing models, such as caching frequently used data, can reduce the number of calls to models and lower latency. Additionally, enhancing the user interface (UI) can help mitigate the impact of latency on user experience. For example, partial results can be provided incrementally as the AI processes data, offering real-time feedback, reducing perceived wait times, and keeping users engaged.
Cost
High costs are a standard challenge organizations face when building AI systems. Pre-trained models are often accessed through APIs provided by companies such as Azure, OpenAI, or AWS, with pricing based on token usage — a token being a unit of text the model processes. Highly accurate models tend to be more expensive, leading to higher costs.
In some use cases, highly accurate models may not be necessary. For these scenarios, costs can be optimized using smaller, cheaper models that still meet the required accuracy. Another option is hosting models on your own, which can be expensive for larger models but may result in cost savings with smaller models if they are sufficient to achieve the desired accuracy. Furthermore, caching can reduce the number of calls to models, lowering token usage and overall costs.
Data-Related Challenges
Data-related challenges are significant when training or fine-tuning models and building Retrieval-Augmented Generation (RAG) systems. These challenges encompass several key areas: compliance, privacy, and data quality. If not addressed carefully, they can lead to complications that hinder effective model development.
Addressing these challenges requires careful planning and execution. Ensuring compliance involves understanding and adhering to relevant regulations to manage data responsibly. Privacy concerns can be mitigated by removing sensitive data while retaining the usability of datasets, possibly with the help of automated tools. Data quality issues can be resolved through thorough data cleaning and preprocessing; automating these workflows helps reduce errors and rework while ensuring datasets are suitable for model training. Systematically managing these aspects can make model development more effective and reliable.
Integration Complexity and Scaling AI
Unlike in the past, when AI systems had one or very few components (e.g., models explicitly trained for a single task), modern AI systems often include several components (such as agents, vector databases, etc.) that must interact with non-AI components to deliver the desired experience. The architectural complexity of modern AI systems can be significant and requires substantial effort to develop, build, and operate in a scalable manner.
Therefore, it is essential to carefully design and architect your AI system according to best architectural design practices. This includes applying principles such as microservice design and API design best practices. APIs play a critical role in your AI system, serving as the components’ interfaces.
Rather than building everything from scratch, software engineering platforms that provide abstractions and capabilities for architecting, building, and running systems can significantly save time and reduce costs.
AI Traffic Management
Modern AI systems rely on external models accessed through APIs. Managing access to these external AI services is vital for the functionality of these applications. Essential aspects of this management include authentication, throttling (based on costs and token limits), monitoring (such as tracking token usage), routing requests to the appropriate models, safeguarding the models, and protecting user data through methods like detecting and removing identifiable information (PII) personally.
Building capabilities into your AI system is challenging and requires significant effort to implement effectively. A practical approach is leveraging API management solutions specifically designed to handle AI traffic, often called AI gateways. These AI gateways provide the necessary capabilities to manage, secure, and optimize access to external AI traffic, ensuring seamless integration and effective operation of modern AI applications.
Continuous Monitoring, Accuracy Evaluation, and Improvements
It is important to establish strategies for continuously monitoring and measuring AI systems’ performance to identify areas for improvement. These improvements made to the system are iterative, guided by insights gained from monitoring and feedback, ensuring the AI system remains reliable and effective over time.
To achieve this, create an evaluation dataset and select relevant performance metrics—Automate monitoring and evaluation pipelines to maintain efficiency and consistency. Regularly reevaluate performance after system changes to prevent degradation, as even minor prompt adjustments can significantly impact accuracy.
Collecting user feedback is vital to enhance AI systems, as it plays a key role in the continuous improvement cycle, helping the system adapt to meet user needs. However, ensuring that feedback collection complies with privacy regulations and safeguarding sensitive user data while improving accuracy is equally important. Together, these components form a robust and effective strategy for continuously evaluating and enhancing system accuracy.
The Future of Architecting in the Enterprise
Successful generative AI development starts with a purposeful beginning. It is imperative to have a clear understanding of the problem and the value that is delivered by the desired application. Thorough planning and a user-centric design approach that prioritizes functionality and user experience are also necessary.
As technology continues to evolve, AI applications must also evolve to stay aligned with emerging trends and meet the changing needs of users.
AI development is a dynamic ongoing process that demands iterative learning, adaptation, and a commitment to innovation to stay competitive and deliver impactful and scalable solutions. Ultimately, it’s about defining what you want to build and understanding the value it brings to your users and your business.
The post Top Strategies for Building Scalable and Secure AI Applications appeared first on The New Stack.
AI development is a dynamic ongoing process that demands iterative learning, adaptation, and a commitment to innovation.