A roadmap for scaling AI agents in the modern enterprise

In collaboration with:

Johnny Chivers, EMEA AWS Technical Director
Aidan Caffrey, Distinguished Architect and UK FS Data and AI Engineering Lead

Maintaining a coherent system state 

As AI agents multiply within a system, managing a single, reliable source of truth becomes difficult. Traditional, synchronous service-to-service integrations often add latency and complexity, creating performance bottlenecks that lead to partial outages or slowdowns. For enterprises dealing with financial transactions or regulated data (e.g., healthcare), consistent state management and minimal downtime are non-negotiable.

Overcoming scalability bottlenecks 

Many AI-driven systems struggle to scale during peak demand. Direct service integrations can lock systems into rigid architectural constraints, limiting the ability to independently scale components. In addition, handling large volumes of event processing while preserving data consistency often proves a bottleneck. The result is a system that cannot seamlessly accommodate traffic spikes or seasonal surges.

Regulatory compliance and observability 

In highly regulated sectors, every transaction must be auditable. AI-powered decisions require traceability and explainability. Maintaining observability—to detect model drift, potential biases, or anomalies—often adds extra overhead to the system, calling for robust logging and easily auditable event trails.

To address these challenges, we propose an event-driven process that couples:

• Apache Kafka for asynchronous messaging and event logging.

• BPMN-based orchestration (e.g., Camunda 8 powered by Zeebe) for managing long-running workflows and decision trees.

Instead of direct calls between AI services, each event is published to Kafka, allowing agents and services to operate asynchronously. Key advantages:

• Seamless scalability 
  Individual components (e.g., speech-to-text or fraud detection services) scale independently based on message throughput. If a sudden spike occurs (peak trading day, e-commerce flash sale), only the relevant components need to scale.

• Resilience and fault tolerance 
  With Kafka’s event log, the system can replay events after downtime or disruptions. This is vital in regulated areas like finance and healthcare, where data loss is unacceptable.

• Clear separation of concerns 
  BPMN-based orchestration handles the business process flow, while AI agents focus on analysis and decision-making. This modular design reduces complexity and improves maintainability.

1. A long-running business process is modeled in BPMN (for example, loan origination or complex voice-based customer support).

2. Each process step is mapped to events flowing through Kafka topics.

3. AI agents subscribe to relevant Kafka topics, process the events (e.g., run a machine learning model), and publish results back to Kafka.

4. The BPMN engine listens for specific events to move the workflow to the next stage—fully decoupled from the AI services themselves.

By using this loose coupling, enterprises can adapt individual AI tasks, business logic, or compliance steps without affecting the entire system.

1. Conversational AI at scale 
   A conversational AI application uses Azure Speech Services for real-time voice interactions. Audio data streams into Kafka, which triggers separate services for speech-to-text conversion, context-aware response generation, and text-to-speech transformation. Each result is published as an event, so microservices can react asynchronously. By decoupling each step, the system scales horizontally to thousands of concurrent sessions while preserving low latency.

2. Direct voice-to-voice AI models 
   In advanced setups, AI models process voice inputs directly and generate synthetic voice outputs in real-time—bypassing text intermediaries. For example, frameworks like Google’s WaveNet, OpenAI’s Whisper, or Meta’s SeamlessM4T handle audio in and audio out.

   • Event-driven flow: Kafka topics manage each stage (voice input processing, intent recognition, response generation, voice synthesis) asynchronously.
   • Scaling and latency: Edge computing, GPU acceleration, and WebRTC-based streaming can further reduce network delays.
   • Context management: Redis or similar in-memory stores provide a “session memory,” so the AI agent can recall user context, enabling human-like back-and-forth conversations. 
     This design accommodates thousands of concurrent conversations with ultra-low latency—ideal for customer support, voice-based digital assistants, and hands-free IoT applications.

3. Regulatory compliance in banking 
   A global bank applies the same architectural principles for Basel III/IV reporting. Trading systems publish large volumes of data to Kafka in diverse formats. A BPMN workflow orchestrator (Camunda 8) coordinates AI agents that standardise and validate data. Redis ensures idempotent processing, preventing duplicate handling of events. Meanwhile, container orchestration (Kubernetes) auto-scales infrastructure based on trading volumes, ensuring cost optimisation. The entire pipeline is fully auditable—critical for meeting regulatory requirements.