Reference architecture for an enterprise support chatbot with RAG

The support problem

Customer support systems usually combine a natural-language question with conversation history, customer identity, and operational data such as an order or payment record. A retrieval-augmented generation (RAG) system can search approved documentation and return an answer grounded in the passages it retrieved. It still needs a path to a human when evidence is missing, conflicting, or outside the system’s authority.

This article presents a reference architecture. It describes components and controls, while leaving performance and business outcomes to a measured deployment.

Design goals

The system should answer supported questions, show the source passages used, protect customer data, and hand off cases that require an authenticated operation or human judgment. Performance and availability are requirements to measure against a stated workload, not fixed properties of an architecture.

The main components are:

• a client that sends a question and conversation context
• an application service that authenticates the caller and manages the session
• a retriever that applies document permissions before returning passages
• a generator that answers from the supplied evidence and cites it
• an escalation path for uncertain or operationally sensitive requests
• ingestion and monitoring jobs that keep the index current and observable

High-level request flow

For each request, the application can use this sequence:

1. Authenticate the caller and apply tenant and role filters.
2. Normalize the question with the permitted conversation context.
3. Retrieve candidate passages from the approved index.
4. Ask the generator to answer only from those passages, cite them, and abstain when support is insufficient.
5. Run policy checks on the answer and route an uncertain or restricted case to a human.
6. Record the request, retrieved source identifiers, decision, and latency under the application’s retention policy.

Retrieval, citations, access filtering, abstention, and human handoff are mechanisms for reducing unsupported answers. They do not ensure that every answer is correct.

Component boundaries

Client and application service

A web or mobile client sends a question over an authenticated HTTPS or WebSocket connection. The application service owns session state, request validation, rate limits, and authorization. A load balancer can distribute requests across stateless service instances. Session state belongs in a shared store when requests can reach different instances; a local container filesystem is not a shared session store.

The service should separate read-only informational answers from actions such as cancelling an order or changing a payment method. An action needs its own authenticated API, confirmation policy, and audit record. A model response is not an authorization decision.

Knowledge ingestion

Source documents, approved ticket excerpts, and metadata can be stored as S3 objects. An ingestion worker reads new object versions, extracts text, splits it into passages, computes embeddings, and writes index records with a stable source key, version, tenant, and page or section metadata.

The ingestion worker should be idempotent. Write the new index version, validate its document count and metadata, and then publish an index version for readers. S3 and OpenSearch do not provide one atomic transaction across both systems, so a failed write must be retried or reconciled with an explicit status record.

Storage choices

S3 stores objects. It is a durable object store for source documents, exported datasets, and model artifacts. It does not itself perform nearest-neighbour retrieval.

OpenSearch stores searchable index records. An index can contain passage text, metadata, and vectors when the selected OpenSearch mapping and engine support vector search. The index points back to the source object or source version; it does not make the S3 object and the index one transaction.

FAISS is a vector-search library and local index format. A FAISS index is a separate option from an OpenSearch vector index. It is useful inside a process or behind a service that owns its files. It does not provide OpenSearch’s distributed indexing, access control, or durability features by itself.

Choose one primary retrieval path for a deployment, or document why two are needed. If a fast cache is used, store the cache key, source version, permission scope, and expiry beside the cached answer so a stale or unauthorized answer cannot be reused.

LlamaIndex and the model service

LlamaIndex can compose document ingestion, retrievers, query engines, and response synthesis. It is an application framework; it does not turn a model into a trusted support operator. Keep the retriever and response policy behind the application’s authorization boundary.

SageMaker real-time endpoints manage the deployment of a model container behind an endpoint. SageMaker can provision instances, route requests, and apply configured scaling and monitoring policies. A SageMaker-managed endpoint is a managed serving path, even when the model weights are open source. The endpoint’s model, instance type, concurrency, and autoscaling policy must be measured for the target workload.

A genuinely self-hosted alternative runs a model server such as vLLM or Text Generation Inference on infrastructure the team operates, for example an EC2 instance or an ECS or Kubernetes service. That option gives more control over the server and network, while making capacity, patching, scaling, and incident response the team’s responsibility.

The model layer can therefore be described as one of these two paths:

1. an application service calls a SageMaker-managed endpoint, or
2. an application service calls a model server operated by the team.

The architecture should not describe both as the same deployment.

Monitoring

CloudWatch can collect application and endpoint metrics. X-Ray or another tracing system can connect request spans when its instrumentation is configured. Useful fields include retrieval latency, generation latency, token counts, model errors, abstentions, handoffs, source versions, and policy decisions. Avoid logging raw customer text unless the retention and access policy explicitly permits it.

A readable query pipeline

The same pipeline in two stages makes the decision boundaries explicit.

Retrieval stage

1. Authenticate the request and derive its tenant and document scope.
2. Search the selected index with the question and permitted context.
3. Filter results that fail authorization or freshness checks.
4. Return passages with source identifiers, version, page or section, and retrieval score.

Answer stage

1. Pass the question and authorized passages to the generator.
2. Instruct it to cite the passages and say when evidence is insufficient.
3. Check the output for policy violations, unsupported claims, and action requests.
4. Return the answer with citations, or create a human handoff with the relevant context.

This split keeps document access and model generation separate. It also gives an evaluator a clear boundary for retrieval recall, citation coverage, answer correctness, and handoff quality.

Performance and capacity planning

Measure the service under a named workload. Record the model and version, prompt length, retrieved passage count, hardware, concurrency, cache state, and latency percentile. Report component boundaries so an endpoint latency is not confused with end-to-end user latency.

Useful controls include:

• bounded retrieval and generation tokens
• request timeouts and retries with idempotency keys
• autoscaling based on queue depth and measured utilization
• a cache keyed by authorized scope and source version
• a fallback model or human handoff when the primary path is unavailable
• load tests that include long conversations and peak concurrency

These controls help an operator measure and manage latency, cost, and availability. They do not establish a universal response-time or uptime guarantee.

Evaluation and safety checks

Create a reviewed set of support questions with expected sources, acceptable answers, and handoff labels. Evaluate retrieval separately from generation:

• retrieval recall and source-version freshness
• citation correctness and coverage
• answer correctness and refusal when evidence is missing
• access-control leakage tests across tenants and roles
• action authorization and confirmation tests
• human-handoff precision and time to resolution

For production monitoring, sample answers for review under the data policy, track changes after document updates, and keep infrastructure failures separate from model failures. A failed endpoint, missing index, or timeout is an operational error. A fluent answer that contradicts its sources is a model or retrieval error.

Reference deployment checklist

Before exposing the assistant to customers, verify that:

• source objects and index records have stable version identifiers
• index writes are retryable and reconciliation handles partial failure
• retrieval enforces tenant and role permissions
• the generator cites passages and can abstain
• operational actions use authenticated APIs and confirmation rules
• the serving path is documented as SageMaker-managed or team-operated
• logs and traces follow the retention policy
• latency, capacity, cost, and handoff metrics have an observed workload

Conclusion

A support RAG system is a set of boundaries: authenticated requests, permission-filtered retrieval, evidence-aware generation, and a human path for uncertainty and operations. AWS services and LlamaIndex can implement those boundaries in several ways. The architecture becomes a production design only after its workload, evidence quality, access controls, and failure handling have been measured.