Put expert feedback to work in your agent workflow

Developer integration brief · 19 October 2026

An Expert Review gives you a specialist's assessment of how your agent performs for its intended users. The feedback loop makes that assessment useful beyond a single review: findings become tests, tests guide changes, and the specialist assesses the next version against the original evidence.

Start by using the report in your normal development process. Add regression testing when you want repeatable checks. Add an automated improvement loop when you want a coding agent or prompt optimizer to propose and evaluate changes. You keep your runtime, repository, model provider and release process.

1. Choose how much to automate

Integration level	What you get from it	What you implement
Review only	Clear findings, supporting observations and a specialist's explanation of what should change	No integration. Make changes in your existing workflow and provide the next version for follow-up.
Review + regression tests	Repeatable checks based on approved expectations, with each failure linked to its source finding	Import the review package; map scenarios and criteria to your test runner; run your agent and retain results in CI.
Review + automated improvement	Candidate changes tested against expert feedback before you consider a release	Add an isolated coding agent or prompt optimizer, a bounded experiment controller and a comparison report for specialist follow-up.

Automation is optional. An Expert Review remains useful if your team makes every change by hand. A test suite remains useful if you never run an optimizer.

2. What IXO provides; what stays with you

The integration starts at a completed, endorsed review. IXO handles the specialist review process and delivery of the evidence package. Your workflow consumes that package and exercises your own agent.

IXO provides	Your team provides
A named specialist, recorded scope and basis for their relevant qualifications	The intended users, tasks, domain context and criteria that matter to your product
Findings and supporting observations tied to the reviewed agent version	A versioned agent and a test environment with representative data and resettable tools
A readable report and structured export, including specialist-approved expectations	An adapter that maps those expectations to your own executable tests
Review provenance, endorsement, integrity references, permitted uses and current review status	Import checks using the supplied verification mechanism, plus storage and processor controls in your environment
Specialist follow-up on the candidate behavior within the agreed review scope	Before-and-after results, remaining failures and access to the candidate version
A new review record linked to the earlier review	Decisions about patches, merge approval, deployment and rollback

You do not need to reproduce IXO's review service or build a credential-verification platform. Configure your importer against the IXO connection and verification details supplied for your integration. Keep your repository and production credentials in your own environment; agree any staging access the specialist needs for the review.

3. Prepare your agent for review

Provide a short review brief: who the agent serves, which tasks it should perform, what a successful outcome looks like, and which actions it must never take without approval. Include the domain policies and context needed to judge its behavior.

Identify the version under review. Record the code revision, prompt and tool configuration, model/provider settings, dependency versions and relevant knowledge snapshot. For a hosted agent, provide a version-specific staging endpoint. If the service changes underneath a run, treat it as a different version.

Use synthetic or permitted customer examples. Replace live payment, messaging and other side-effecting tools with test equivalents. Give the specialist enough access to observe the actual user journey and relevant tool activity. Private model chain-of-thought is not required.

For automated testing, your harness needs three capabilities:

Reset: establish the scenario's starting state without touching production data.
Run: invoke the specified agent version with the scenario input and its normal prompt, retrieval and tool flow.
Capture: return the user-visible response, structured outputs, ordered tool events, execution status, cost and timing. Record the executed version from the runner, independently of the agent's answer.

If an action cannot be observed, report the relevant check as unassessed. A convincing final answer does not establish that the agent used its tools correctly.

4. Receive and accept a review package

A review package contains the human-readable assessment and the evidence your workflow needs to reuse it. The review package schema and sample package show the data relationships. Use the synthetic sample to understand the format and test local parsing.

Package information	How your workflow uses it
Package ID, schema version, parent review and reviewed agent version	Track revisions and associate the feedback with the correct project and baseline
Specialist identity, scope and endorsement	Attribute the findings and determine which expectations the specialist approved
Scenario input, context, preconditions and observed trace	Reproduce the behavior in a controlled environment
Finding ID, severity, evidence location and explanation	Diagnose the failure and retain the reason for each test
Criterion ID, expected behavior, scoring mode and approval reference	Build assertions or a rubric without changing the specialist's meaning
Artifact hashes, review receipt and endorsement binding	Check that the imported evidence matches the endorsed review
Permitted uses, processors, retention and review status	Decide whether the package can be used in this workflow now

Begin with an authenticated package download and a manually triggered import. Automatic delivery can trigger the same import job using the connection documented at onboarding; the rest of your workflow can stay unchanged. A notification triggers retrieval, rather than serving as proof of authenticity.

Before allowing a package into CI or an optimizer, your importer must:

Validate the supported schema version and match the order, project and agent against your configured integration context.
Check the receipt, endorsement, artifact integrity and current review status using the supplied verification mechanism. Do not trust a verified field in uploaded content.
Confirm that test derivation, the selected optimizer and its model processors are permitted. Permission to use feedback does not automatically permit model-weight training or public sharing.
Store the original package unchanged and record the verification result. Derived cases retain the package digest and source finding/criterion IDs.
Deduplicate by project, package ID and digest. Re-importing identical bytes is safe; different bytes under the same immutable ID require investigation.

Verified provenance establishes who endorsed which evidence for which version. It does not make a specialist infallible or certify every future agent release. A completed review that finds serious problems is valid input for improvement. Unendorsed or disputed expectations require clarification before they become trusted test labels.

5. Turn a finding into a test in your harness

Consider a returns agent. The customer paid US$80 for a discounted item. The agent attempts a US$100 refund before the customer confirms. The specialist identifies two failures: the amount and the order of actions.

Approved expectation	Your test implementation
Propose the eligible paid amount: US$80	Assert `currency == "USD"` and the structured refund amount equals the fixture's eligible amount.
Wait for explicit customer confirmation	Assert that no refund-execution tool event occurs before the confirmation event.
Explain the amount and next action clearly	Apply the approved rubric to the response; route ambiguous results for specialist review.

Map scenario inputs into your existing fixture format. Map deterministic criteria into exact assertions over outputs and tool events. Map judgment criteria into a fixed rubric; a calibrated model judge can screen these during development. Keep expected answers and scoring instructions out of the agent's task input.

Your adapter should preserve a traceable chain:

review package → finding → approved criterion → test case → run result

The specialist approves the meaning of success. Your team implements the executable assertion. If that translation changes the meaning or introduces a new expectation, return it for approval before updating the benchmark. Generated variants should be labeled as synthetic and kept distinct from human-reviewed observations.

Run the reviewed baseline first. Confirm that the test reproduces the reported failure, or document why it cannot. Then run your existing successful scenarios as regression checks. Freeze the cases, scorer version and thresholds before comparing candidate changes.

6. Add the optional automated loop

Your coding agent or prompt optimizer becomes a proposer within your development workflow. It receives the approved feedback, visible development cases and a narrowly defined change request. Your evaluation runner decides whether the resulting behavior is better.

IXO Expert Review
    ↓
Your importer → approved cases in your test harness
    ↓
Baseline → propose a change → evaluate → keep or reject
                             ↑                │
                             └── repeat ──────┘
    ↓
Selected candidate + fresh evaluation + comparison report
    ↓
Specialist follow-up → your release approval
    ↓
Resolved findings become regression tests; new findings start the next cycle

Wire this into a manually dispatched CI job first. Once reliable, trigger it after accepting a new review package. A normal pull request can run the same regression checks without invoking the optimizer.

The following workflow pseudocode shows how to connect the steps in your integration:

review = import_and_verify(package, configured_review_context)
cases = map_approved_criteria(review, your_test_harness)
baseline = evaluate(pinned_agent, cases, regression_suite)

with bounded_experiment(policy) as experiment:
    candidate = experiment.search(
        baseline=baseline,
        propose=your_coding_agent_or_optimizer,
        evaluate=your_fixed_evaluator,
        feedback=review.permitted_feedback,
    )

if candidate is not None:
    final_results = evaluate_fresh_cases(candidate, pinned_agent)
    save_follow_up_packet(review, baseline, candidate, final_results)
else:
    save_unresolved_findings(review, experiment.results)
# Specialist follow-up and your normal release approval happen next.

Implement the controller so that it enforces these boundaries outside the coding agent:

Control	Practical implementation
Change scope	Start with one class of change: prompt, retrieval configuration or code/tool flow. Allowlist writable paths; reject other patches.
Fixed evaluation	Keep scorers, approved expectations and final evaluation data outside the optimizer's writable workspace.
Isolation	Use temporary workspaces and test tools; provide only approved model/network access and no production credentials. Treat transcript instructions as data.
Budget	Limit candidate count, elapsed time, model tokens, tool calls and retries. Reserve evaluation cost before dispatch, including the final comparison.
Selection	Reject new critical failures. Keep experimental repairs separate from release eligibility; all critical release checks must pass.
Stop conditions	Stop at the budget, repeated infrastructure errors, no useful progress, or a question that needs specialist judgment.

The example policy illustrates a US$100 machine budget, 20 candidate proposals and a two-hour limit. Set your own thresholds from measured costs and product needs. This file is configuration for your controller; it does not enforce limits by itself. Inference and compute are separate from the review fee.

7. Use your existing tools, or start with these

You can implement the loop with your current test runner, coding assistant and CI. No agent-runtime migration or model-weight training is required.

Tool	Where it fits	Your integration work
Inspect AI	Runs evaluation datasets, agent solvers and scorers, with retained logs	Map approved cases into samples, invoke your real agent through a solver or compatible bridge, and implement output/tool-event scorers.
OpenHands Software Agent SDK	Proposes code changes in a local or isolated workspace	Supply the scoped task and permitted files; submit patches to your fixed evaluator and controller.
GEPA	Optimizes prompts and other text components from evaluation feedback	Define the candidate components and an evaluator returning scores plus permitted diagnostic feedback.
Autoresearch pattern	Constrains edits, measures an experiment, retains or rejects it, and repeats	Apply that experiment pattern to your own agent evaluator and task-success criteria.

A practical starting combination is your agent + Inspect AI + OpenHands + a small controller in your CI environment. For prompt-only changes, GEPA can be the proposer instead. Pin and test the versions you use. These frameworks supply building blocks; your adapter connects them to the IXO review package and your agent.

8. Return evidence for specialist follow-up

Compare the selected candidate and baseline on fresh scenarios outside the optimizer's view, as well as the agreed regression suite. Repeatedly used selection cases are part of optimization; reserve independent cases for the final comparison. Keep related variants in the same dataset split.

Repeat unstable cases and include the run counts and variation. A small set of successful tests supports a scoped review; it does not establish universal reliability. Failed infrastructure runs are incomplete results, not evidence that a candidate passed or failed the task.

Provide the specialist with:

The original review, source finding IDs, candidate version and a concise change summary.
Paired baseline/candidate responses and tool traces, with evidence locations.
Per-criterion outcomes, scorer and benchmark versions, run counts, cost and latency.
Remaining failures, unassessed criteria, model-judge disagreements and fresh-case results.
Access to reproduce the candidate behavior in the agreed test environment.

The specialist records which findings are resolved, partly resolved or unresolved, and identifies new issues. Their follow-up applies to the identified candidate version. Your team decides whether to merge and release through its normal approval process.

After release, preserve the approved resolved cases as regression tests where permitted. Submit new or ambiguous behaviors for review. Each cycle adds evidence and sharper expectations; it may also reveal that a candidate should be rejected. Improvement is measured, not assumed.

9. Keep the integration reliable

Situation	Your workflow should do this
Unsupported schema, invalid endorsement or mismatched version/hash	Reject the import before any content reaches the optimizer; retain the reason for investigation.
Missing permission, expired rights or a withdrawn review	Pause new use, apply agreed retention rules and identify affected cases and pending candidates.
Duplicate delivery or interrupted job	Resume using the package digest and experiment ID; reconcile already-used budget before retrying.
Conflicting findings or a disputed expectation	Preserve the evidence and request clarification; do not silently choose a new label.
Changed rubric, model configuration or environment	Create a new benchmark or run version and rerun the baseline before comparison.
No qualifying candidate	Return the measured results and unresolved findings; keep the current release unchanged.

Store private evidence with project-scoped access. Send only permitted content to approved model processors. Keep originals, derived tests, logs and follow-up reports linked through their retention lifecycle. A public, redacted review receipt is not an optimization dataset.

10. Your first integration is complete when…

One approved specialist finding becomes a repeatable test in your environment. You can run the pinned baseline and a changed candidate, compare their behavior, and return evidence the specialist can inspect. Add multi-candidate search after that path works.

Before enabling unattended experiments, demonstrate that your workflow rejects an altered package, enforces permissions and budgets, prevents evaluator changes, keeps final cases private and cannot bypass release approval.

Start with an Expert Review and choose your integration level. The current pilot covers one agent version, up to five agreed scenarios and one follow-up within scope. Automation setup, additional test authoring and further review cycles are agreed separately. The developer kit contains the review package format, synthetic example, validation script and templates for your side of the integration.