Failure Modes of Agentic Systems in Scientific Workflows

TL;DR

• The most dangerous failure is often plausible, not obvious. A scientific agent can execute every tool successfully and still produce the wrong conclusion because an early assumption, sample link, unit, or transformation was wrong.
• Failures propagate across layers. Data, model reasoning, tools, memory, orchestration, security, and human review each create distinct risks; a final-answer score cannot reveal where the chain broke.
• More steps do not automatically mean more reliability. Long trajectories create more opportunities for compounding error, circular confirmation, repeated actions, and budget exhaustion.
• Detection must use independent evidence. Schema validation, physical or chemical invariants, duplicate checks, alternative hypotheses, tool-result reconciliation, and expert review are stronger than asking the same model whether it made a mistake.
• A safe agent needs bounded authority and a valid way to abstain. Least-privilege tools, action budgets, hard stops, reversible writes, and explicit escalation turn uncertainty into a controlled outcome.

Scientific work is vulnerable to silent agent failure. A file with the wrong sample or unit may still parse, and a candidate structure may explain most signals while violating one decisive constraint. Passed forward without challenge, such errors can produce an internally coherent but scientifically wrong report.

This guide separates failure modes by source, shows how they combine, and maps each one to detection and containment controls. It concerns software agents that coordinate data and analytical tools; physical instrument control requires additional safety engineering and validation.

Why agent failures become cascades

Unlike a fixed script, an agent can choose tools, update a plan, retain memory, and decide when the task is complete. This flexibility introduces feedback: an interpretation changes the next action, whose output changes the next interpretation.

Consider a spectrum assigned the wrong solvent. The shift reference may be adjusted incorrectly; peak annotations may then be misclassified; a structure-search tool may receive distorted constraints; and the agent may treat the resulting poor matches as evidence that the compound is novel. No individual program has necessarily crashed. The error is semantic and becomes larger as the trajectory continues.

A scientific result must therefore include its path. The NIST scientific-workflow programme describes capturing tools, parameters, inputs, assumptions, and provenance needed to reproduce and validate an experiment. An agent should record enough context to reconstruct its result.

The 2026 SciAgentArena preprint reports uneven performance across roughly 200 interactive scientific tasks: agents did better on well-specified analysis workflows and struggled more with open-ended exploration. It is one benchmark, not a universal measure, but it supports a practical principle: ambiguity should increase supervision.

A failure taxonomy for scientific agents

Separating overlapping categories makes incident review and control design more precise.

“Hallucination” is too broad to distinguish a model-generated claim from a parser error, stale memory, or misleading tool result. Controls depend on knowing which boundary failed.

Failure mode 1: bad inputs that look valid

Scientific garbage is often well formed. A CSV can satisfy its schema while a concentration uses the wrong unit; a readable spectrum can carry the wrong sample identifier.

Input validation therefore needs three layers:

1. Syntactic validation: Can the object be parsed, and are required fields present?
2. Semantic validation: Are units, dimensions, ranges, identifiers, and relationships meaningful for this task?
3. Scientific validation: Do independent observations and domain invariants agree with the declared experiment?

Provenance is part of validation. The NIST Research Data Framework highlights persistent instrument identifiers, metadata, standards, and data history. Every derived object should retain links to its source, transformation, parameters, and software version.

Quarantine missing or contradictory inputs, never overwrite the acquisition, and distinguish “unknown” from a default. Silently replacing a missing field without recording the assumption fabricates evidence.

Failure mode 2: correct tools used incorrectly

Tool access gives an agent leverage. It also creates failure modes that a standalone model does not have: selecting an inappropriate method, sending the wrong object, setting an invalid threshold, applying results outside the tool’s domain, or writing to an unintended record.

A tool contract should state:

• accepted input types, units, required metadata, and validity range;
• output schema, uncertainty, version, and known failure signals;
• side effects, permissions, reversibility, and approval requirements;
• how success is verified independently of an execution-status code.

Domain automation does not remove these requirements. For example, Bruker describes Advanced Chemical Profiling 2.0 as an NMR workflow spanning queued samples, instrument-control software, analysis, and reporting. That scope illustrates the number of boundaries an agentic layer may encounter; it does not establish that a separate agent can safely operate them without explicit interfaces and validation.

External content adds a security dimension. The AgentDojo paper demonstrates that instructions embedded in tool-returned data can redirect an LLM agent. A laboratory note, database field, or imported report must not become an authorised command. Separate data from instructions and grant only the actions required for the current task.

Failure mode 3: unsupported hypotheses and circular confirmation

A scientific agent may generate a plausible hypothesis early and then interpret later observations in its favour. It selects confirmatory tools, discounts conflicts as noise, or stops as soon as one candidate crosses an internal threshold. This is automation-assisted confirmation bias.

The control is not simply “reason longer.” Require three answers:

• What evidence supports the current hypothesis?
• What observed evidence does it fail to explain?
• What alternative hypothesis would also explain the observations, and which test would distinguish them?

Candidate generation and candidate evaluation should be separable where practical. Independent algorithms, held-out measurements, orthogonal techniques, or a blinded expert can challenge the favoured result. Repeating the same prompt with the same evidence may create variation, but it is not independent verification.

The NIST Generative AI Profile treats confabulation as a risk and recommends empirical testing, appropriate human review, provenance, and post-deployment monitoring. Scientific monitoring should include unsupported assertions, constraint violations, sensitivity to reasonable input changes, and missed abstentions.

An apparently successful completion can be more dangerous than a visible crash. A 2026 case study of astrophysical workflows, “Plausible but Wrong”, reports syntactically valid computations that produced plausible but inaccurate results and physically inconsistent outputs without self-diagnosis. The domain differs from analytical chemistry and the study covers only eighteen tasks, so its numerical results should not be generalised. Its operational lesson transfers: scientific validity needs external checks.

Failure mode 4: loops, overthinking, and premature stopping

Agents can fail by doing too much or too little. In a loop, the system repeats searches, reprocesses the same data, alternates between equivalent hypotheses, or calls tools without increasing information. At the other extreme, it may mistake a plausible candidate or successful tool response for task completion.

Different tool calls can still be semantically redundant. Track unresolved constraints, candidate-set change, new evidence, and distance to acceptance criteria. If successive actions make no measurable progress, stop and escalate.

Useful control limits include:

• maximum steps, time, cost, and external queries;
• per-tool retry limits and explicit error classes;
• duplicate-action detection using normalised inputs and state;
• a requirement to predict the expected information gain before a costly action;
• completion criteria based on evidence, not the agent’s verbal confidence;
• a safe terminal state for inconclusive, invalid, and out-of-scope cases.

Budget exhaustion is a workflow status, not evidence that the leading hypothesis is correct. Preserve what is known, unresolved, and needed next.

Worked incident: a plausible result built on the wrong record

Imagine an agent asked to assess whether a newly acquired spectrum supports an expected material identity. The file parses, preprocessing succeeds, and a comparison tool returns a high similarity to the reference. The agent writes “identity supported” into a draft report.

Later, a scientist notices that a weak diagnostic region is absent. The incident trace reveals that the acquisition filename was copied from the previous sample. The agent joined records by filename rather than immutable sample ID, then retrieved the previous sample’s reference spectrum. Every downstream tool operated as designed.

The root cause is an unsafe identity join, weak negative-evidence checks, and concealed provenance. The fixes are stable identifiers, testable joins, visible conflicts, and a hard block when identity checks fail.

After containment, replay the case and neighbouring cases, add the incident to a regression set, and check earlier results produced by the affected versions.

A practical detection-and-containment architecture

No single guardrail covers the full trajectory. Defence in depth should place controls before, during, and after action:

Match controls to the decision. A research assistant proposing hypotheses can tolerate different uncertainty and permissions from a system influencing batch release. Consequential decisions need stronger evidence, separation of duties, and independent review.

Lessons from building agentic NMR workflows at Rombo AI

NMR structure interpretation makes cascading failures easy to see. Reference errors alter shifts; peak-picking choices alter constraints; overlap makes correlations ambiguous; and one unsupported connection can change a candidate structure. The useful role for an agent is therefore not an opaque jump from spectrum to answer. It is to coordinate bounded tools while preserving the evidence chain, rejected hypotheses, unresolved conflicts, and chemist review.

Rombo AI applies this principle to specialist NMR workflows: reusable spectral representations and analytical tools can support individual steps, while orchestration determines what to do next and when to abstain. Spectra AI and Spectra API remain a distinct product line for compound identification and structure elucidation; they are not interchangeable with Rombo AI’s broader platform for material and mixture analysis.

This article does not claim autonomous instrument operation, universal file compatibility, specific integrations, or validated performance for every laboratory context. Those boundaries must be documented and tested for the intended use.

Designing a governed agentic workflow for scientific data? Contact Rombo AI to discuss evidence requirements, failure tests, and review boundaries for your application.

Frequently asked questions

What is the most dangerous failure mode for a scientific AI agent?

A plausible but invalid result is often more dangerous than an explicit error because it can pass into reports or decisions without triggering investigation. Independent scientific checks, provenance, and contradiction reporting help expose silent failure.

Can chain-of-thought or longer reasoning prevent agent failures?

Not by itself. More reasoning steps can help plan a task, but they also create more opportunities for error propagation and loops. Reliability comes from grounded observations, independent validation, bounded tools, progress checks, and defined stopping rules.

How do you detect an agent loop?

Track whether successive actions add evidence, resolve constraints, change the candidate set, or reduce uncertainty. Repeated or semantically equivalent actions without measurable progress should trigger a stop, not unlimited retries.

Should a scientific agent be allowed to write results automatically?

Authority should match consequence. Drafting a traceable result can be acceptable, while changing a system of record, controlling an instrument, or approving a release may require separate permissions and human approval. Writes should be scoped, logged, and reversible where possible.

What belongs in a scientific-agent incident report?

Record the objective, inputs and provenance, full trajectory, tool and model versions, parameters, permissions, observations, warnings, human actions, affected outputs, root cause, containment, and regression test. Avoid reducing the incident to a generic label such as hallucination.