You cannot train yesterday’s shipment status into tomorrow’s answer.
Use retrieval for changing policy and product knowledge, live tools for customer-specific order state, and fine-tuning where supported for a repeated behavioral problem demonstrated by evaluation. These approaches can coexist. Do not use fine-tuning as a substitute for current facts or treat a retrieved document as permission to access an order.
What changed—and what the sources establish
OpenAI’s retrieval guide describes searching indexed files and filtering by attributes. That provides tools for scoped knowledge access, but the application still needs an appropriate data and permission design. OpenAI retrieval guide.
OpenAI’s model-optimization guide combines evaluation, prompting and fine-tuning where supported. Fine-tuning is an intervention to evaluate against a diagnosed behavior gap, not an automatic requirement for every support assistant. OpenAI model optimization.
LangChain’s retrieval documentation separates indexing and query-time access from generation. Diagnose retrieval failures and generation failures separately rather than trying to solve both by changing the model. LangChain retrieval architecture.
Choose the approach that fits the constraint
Classify the question by where its correct answer lives. A store’s latest order status is not the same kind of knowledge as its published return policy.
| Situation | Start here | Tradeoff / acceptance check |
|---|---|---|
| Changing policy or product knowledge | Permission-aware retrieval | Keep effective dates, source owners and citations |
| Current customer order or payment state | Authenticated live commerce tool | Check ownership and avoid stale document snapshots |
| Repeated format, tone or classification error | Prompt/evaluation, then supported fine-tuning | Use representative labeled examples and regression checks |
| Refund or cancellation request | Policy plus authorized action adapter | Retrieved text cannot grant write permission |
Build the knowledge boundary before choosing the retrieval stack
Inventory sources and their owners: public policies, product manuals, internal procedures and merchant-specific exceptions. Record effective dates and scope. Distinguish text that customers may see from internal instructions or restricted customer records. A useful index should preserve document identity, relevant headings and the conditions of each rule. An isolated sentence about returns can be misleading without its excluded products or regional terms.
Apply tenant and source permissions before generation, not only in the final answer. A model should never receive another store’s private document and be asked politely not to use it. Use metadata filters and authorized indexes as part of the retrieval boundary. Retire obsolete policies or keep explicit version selection; otherwise the assistant can cite a real document that no longer governs the customer’s situation.
Diagnose whether the error is knowledge, retrieval or behavior
Create support cases with known evidence and acceptable answers. If the right document is missing, repair ingestion. If it exists but is not retrieved, inspect query construction, filters, chunk boundaries and ranking. If the evidence arrives but the answer ignores an exception, improve the prompt and evaluate the model’s use of context. Fine-tuning a tone problem will not fix a missing policy file.
Consider fine-tuning only after a repeated behavior remains despite a good task specification and evidence path. Examples might include a stable classification taxonomy or a required response structure. Verify that the intended model supports the chosen training method. Keep held-out and adversarial cases, and evaluate refusals and evidence handling as well as the target behavior. Do not train confidential customer history into a broadly accessible assistant.
Connect current data and actions through narrow tools
For “where is my order,” retrieve order state from the authorized commerce system using the authenticated customer context. A knowledge article can explain statuses, but it should not supply the customer’s current tracking event. Keep dynamic facts such as inventory, price and payment state separate from documents that change less frequently. Tell the user when the live source is unavailable rather than filling the gap with a plausible status.
For a refund, combine policy evidence with deterministic eligibility checks and a separately authorized command. The model’s summary can prepare the case, but permissions and money belong in the execution layer. Return source references and action state so support can continue after escalation. Knowledge accuracy and business-action correctness need separate acceptance measures; a well-cited answer can still perform the wrong action.
Worked example: illustrative assumptions, not a reported result
Illustrative support case: a customer bought shoes 18 days ago and asks for a return. Retrieval finds the current policy and its category exception. A live tool verifies ownership, delivery date and existing return state. Approved rules determine eligibility. The assistant then explains the decision and, if allowed, prepares a return command rather than inventing approval from a generic policy paragraph.
For an invented evaluation set of 100 cases, suppose 20 fail because the right evidence never reaches the model and ten fail because the response ignores an exception. Prioritize the 20 retrieval failures and diagnose the ten behavior failures separately. These numbers are explanatory assumptions, not a measured RAG or fine-tuning benchmark. One aggregate accuracy score would obscure the different repairs.
Failure modes to catch before scaling
- Stale source truth: indexed content has no effective-date discipline. Use owners, version selection and ingestion checks.
- Permission afterthought: restricted documents reach the model before filtering. Enforce retrieval access at the source boundary.
- Training as a cure-all: fine-tuning is used to compensate for missing or changing facts. Fix the knowledge and tool path first.
A combined architecture is often sensible, but each layer adds maintenance. Start with the smallest design that meets acceptance and add training or retrieval complexity only for a demonstrated gap.
A concrete next step
Label twenty representative support cases by policy knowledge, live state, behavior and action. Trace their failures to the responsible layer, then pilot one repair with the same held-out acceptance cases before buying a more complex stack.
| Sources | Owner, scope, effective date and permitted audience |
|---|---|
| Retrieval | Relevant evidence arrives with correct access filtering |
| Generation | Answer uses conditions and cites its evidence |
| Actions | Separate authorization, execution and read-back status |
Track evidence availability, retrieval success, answer correctness, escalation and completed permitted actions separately. Preserve failed cases and source versions so a new model or index change can be evaluated without losing the reason for the original decision.
Practical questions
Should changing policies be handled with fine-tuning?
Use a maintained retrieval or rule source for changing policy facts. Fine-tuning may improve a repeated behavior, but it is not a reliable mechanism for keeping every policy update current.
Can RAG answer current order questions?
A policy index can explain order terminology, but customer-specific current state should come from an authenticated live tool. Verify ownership and report source unavailability honestly.
When is fine-tuning worth evaluating?
Consider it for a persistent, measurable behavior gap after improving task instructions and evidence. Confirm model support, use representative labels and keep held-out regression cases.
Does a cited answer prove the action is safe?
No. Citations establish evidence for the explanation. Authorization, eligibility, command execution and resulting business state require separate checks.
Primary sources checked 11 October 2026
- OpenAI retrieval guide — Retrieval searches indexed files and supports attribute filtering of search results.
- OpenAI model optimization — Model optimization combines evaluations, prompting and, where supported, fine-tuning.
- LangChain retrieval architecture — Retrieval separates knowledge indexing and query-time access from model generation.
The calculations in this guide are illustrative planning examples. Product documentation establishes supported features and conditions; it does not establish a measured return for your organization. Recheck account availability and release notes when implementing the recommendation.
Continue with Production retrieval controls and Commerce action workflow.
Research Path