AI Agents and Agentic Systems

Enterprise interest has moved past “a model that answers questions” to “a system that does the work”. That step changes the engineering problem: an agent does not just produce text, it takes actions with real consequences — it writes to systems, spends money, and talks to people.

This handbook is a field reference for building those systems under control. Thirty-seven short sections cover the anatomy of an agent, the harness around it, tool risk and permissions, memory, multi-agent patterns, and the evaluation and failure modes that decide whether any of it is safe to run in production. Each section closes with a plain-language summary, and the last two are a misconception table and a glossary you can hand to a non-specialist.

1. From Tool Use to Agency

A fixed tool-use pipeline executes a predefined sequence. An agentic system allows the model to choose the next action repeatedly based on the current goal and observations.

Agency is a system property created by the model, tools, state manager, policies and runtime together.

In plain languageThe LLM becomes agentic only when surrounding software lets it continue deciding and acting.

2. What an Agent Is

An AI agent is a software system that receives a goal, maintains state, selects actions, uses tools, observes results and decides whether to continue.

The model is usually the decision component, while the harness performs execution and control.

In plain languageAn agent is not merely a chatbot. It is a chatbot connected to an action loop.

3. Workflows Versus Agents

A workflow has steps and branches defined mainly by developers. An agent dynamically determines its process and tool usage.

Workflows are more predictable. Agents are more flexible but harder to test and control.

In plain languageUse a workflow when the path is known. Use an agent when the path genuinely depends on what is discovered.

4. The Agent Control Loop

A common loop is: interpret goal, plan, select action, execute, observe, update state and decide whether to stop.

Some agents plan explicitly; others choose one next action at a time.

In plain languageThe agent repeatedly asks what to do next, what happened and whether it is finished.

5. The Agent Harness

The harness is the operational software around the LLM. It manages tools, state, retries, permissions, budgets, logging and stopping conditions.

LangGraph models agents as stateful graph execution with model, tool and middleware nodes.

In plain languageThe harness turns model suggestions into controlled software actions.

6. Goals and Instructions

A goal defines the desired outcome. Instructions define constraints, available resources, quality requirements and prohibited behaviour.

Goals should be testable and bounded. Open-ended instructions create uncontrolled loops.

In plain languageTell the agent what finished looks like and what it must never do.

7. Planning

Planning decomposes a goal into steps or subgoals. Plans may be created once, revised after observations or generated incrementally.

Planning adds token use and can create false confidence when the environment changes.

In plain languageA plan is a working hypothesis, not proof that the task will succeed.

8. Reasoning and Observable State

Operational systems should rely on observable actions, state changes and outputs rather than hidden reasoning.

Plans, tool calls, results and decisions can be logged without exposing private chain-of-thought.

In plain languageTrust the trace of what the agent did, not an unverifiable story about how it thought.

9. Tools and Actions

Tools may search, read files, query databases, send messages, update records, execute code or control applications.

Every tool needs a strict schema, validation, timeout and error contract.

In plain languageTools are the agent’s hands. Their permissions determine what damage or value it can create.

10. Tool Risk Tiers

Read-only tools are generally lower risk. Reversible writes require stronger controls. Irreversible, financial, legal or external communication actions require explicit approval.

Tool policy should consider data sensitivity, side effects and blast radius.

In plain languageA search query and a bank transfer must never share the same approval policy.

11. Observations and State Updates

A tool result becomes an observation. The harness records it in state and decides what portion returns to the model.

Large observations may require summarisation, filtering or structured extraction.

In plain languageThe agent learns what happened because the application feeds the result back into the next step.

12. Termination Conditions

An agent must stop after success, failure, user cancellation, maximum steps, time limit, cost limit or repeated lack of progress.

Stopping only when the model says it is finished is unsafe.

In plain languageThe harness needs a circuit breaker independent of the model.

13. Error Handling and Recovery

Tools can time out, return malformed data or partially complete actions. Recovery policies include retry, fallback, compensation, escalation and safe failure.

Retries should be bounded and use backoff where appropriate.

In plain languageA reliable agent expects failure and has a defined next move.

14. Human-in-the-Loop

Human approval can be required before sensitive tools, final publication or irreversible actions.

The system should present the proposed action, evidence and consequences before execution.

In plain languageThe agent prepares the action; the person authorises it.

15. Memory in Agentic Systems

Working state tracks the current task. Conversation history stores interaction. Long-term memory stores selected persistent facts. External knowledge is retrieved through RAG.

Memory writes and retrieval require relevance, privacy and expiry policies.

In plain languageAgent memory is managed storage, not permanent awareness.

16. RAG Inside an Agent

An agent may decide when to retrieve, reformulate queries and compare sources. This adds flexibility beyond a fixed, pre-defined RAG pipeline.

Retrieval permissions and citation requirements remain enforced by the harness.

In plain languageThe agent can choose when to consult the library, but it cannot bypass library rules.

17. Code Execution and Sandboxes

Code execution enables calculations, file transformation and testing but is high risk.

It should run in an isolated sandbox with limited filesystem, network, CPU, memory and execution time.

In plain languageGive the agent a disposable workshop, not unrestricted access to the whole computer.

18. Browser and Computer Use

Browser tools manipulate webpages; computer-use tools can interact with graphical interfaces. These environments are variable and exposed to malicious content.

Actions such as login, purchase, deletion or publishing require policy gates.

In plain languageVisual automation is powerful but less predictable than a stable API.

19. Single-Agent Patterns

ReAct alternates action decisions and observations. Plan-and-execute separates planning from execution. Reflection adds a critique or revision stage.

Each additional loop increases cost and possible failure paths.

In plain languageChoose the simplest pattern that solves the task.

20. Multi-Agent Systems

Multi-agent systems divide work among specialised agents. They may communicate through messages, shared state or a supervisor.

AutoGen supports event-driven agents, messages and local or distributed runtimes.

In plain languageA multi-agent system is a team of specialised software roles, not automatically a smarter model.

21. Supervisor Pattern

A supervisor assigns tasks to worker agents, receives results and decides the next delegation.

The supervisor can become a bottleneck or single point of failure.

In plain languageOne coordinator manages several specialists.

22. Handoffs and Swarms

In a handoff pattern, one agent transfers control and context to another. Swarm-style systems use decentralised handoffs between specialised agents.

Handoffs need clear ownership and termination rules.

In plain languageThe current specialist decides which specialist should take over next.

23. Debate and Critic Patterns

Debate asks multiple agents to propose or challenge answers. Critic patterns use a separate evaluator to review plans or outputs.

Agreement between agents does not guarantee truth because they may share the same model and errors.

In plain languageMore opinions can expose mistakes, but they are not independent evidence.

24. Communication and Shared State

Agents communicate through structured messages. Shared state should define ownership, versioning and conflict resolution.

AutoGen Core treats messages as the mechanism through which agents communicate.

In plain languageA team works only when information is passed clearly and consistently.

25. Determinism and Reproducibility

Agent outputs vary with model sampling, tool results, timing and external state.

Reproducibility requires fixed versions, recorded traces and mocked tools for tests.

In plain languageRecord the environment and every action if you want to understand or repeat a run.

26. Cost and Token Control

Agent loops can consume many model calls and large contexts. Budgets should limit steps, tokens, tool calls, elapsed time and expensive models.

Use smaller models for routing or classification where quality permits.

In plain languageEvery extra decision step has a cost, even when no useful progress occurs.

27. Latency and Concurrency

Sequential tool loops increase latency. Independent tasks can run concurrently, but parallel actions can create race conditions or duplicate side effects.

The runtime must coordinate state updates and cancellation.

In plain languageParallel workers are faster only when their work does not interfere.

28. Security and Least Privilege

Each tool should receive only the permissions required for its task. Credentials should be scoped, short-lived and stored outside prompts.

Separate read, write and administrative identities.

In plain languageGive the agent the smallest key that opens only the required door.

29. Prompt Injection and Tool Abuse

Webpages, emails and documents may contain malicious instructions. Retrieved content must be treated as data, not authority.

The model should not gain new permissions because untrusted content requested them.

In plain languageA webpage can lie to the agent. The harness must enforce the rules.

30. Auditability and Observability

An agent trace should record model calls, tool requests, results, state transitions, approvals, errors, cost and final outcome.

Sensitive prompt content should be redacted according to policy.

In plain languageA trustworthy agent leaves a clear operational trail.

31. Agent Evaluation

Evaluation should measure task success, tool correctness, evidence quality, step efficiency, cost, latency, safety and recovery.

Trace-level evaluation reveals failures hidden by a plausible final answer.

In plain languageJudge both the destination and the route taken to reach it.

32. Failure Modes

Common failures include looping, premature stopping, wrong tool selection, fabricated tool arguments, stale state, unsafe actions, excessive cost and coordination breakdown.

Fallback to a deterministic workflow or human operator should be available.

In plain languageAn agent can fail even when every individual component works correctly.

33. Framework Landscape

LangGraph models workflows and agents as stateful graphs. AutoGen provides event-driven single- and multi-agent runtimes. CrewAI provides agents, crews and flows.

Framework choice should follow control, observability and deployment needs.

In plain languageFrameworks supply scaffolding; you still design the building.

34. NVIDIA NeMo Agent Toolkit

NVIDIA’s NeMo Agent Toolkit is a framework-agnostic library for connecting enterprise agents to tools and data sources and for profiling workflows.

Its orchestration does not inherently require a GPU; model inference can run locally or remotely.

In plain languageThe toolkit coordinates and observes agent workflows; the LLM endpoint can run separately.

35. Practical Implementation Workflow

Start with a deterministic workflow. Add one model decision point, then one low-risk read-only tool. Introduce state, limits, tracing and evaluation before write tools.

Add human approval before external communication or irreversible actions. Multi-agent design should be the final step, not the first.

In plain languageEarn autonomy gradually through measured tests.

36. Common Misunderstandings

Misunderstanding	Correction
An agent is simply an LLM with tools	It also requires a control loop, state, policies and runtime.
More autonomy always improves results	It increases flexibility and failure surface.
Multi-agent means multiple independent opinions	Agents may share the same model and biases.
The model can enforce its own permissions	Permissions must be enforced by external software.
Human approval after execution is sufficient	Approval must precede sensitive actions.
A plausible final answer proves a good run	The action trace and evidence must also be evaluated.
Local agents are automatically secure	Local tools, credentials and network services still require controls.
Agent memory is equivalent to model training	Memory is external state retrieved during execution.

37. Glossary and Abbreviations

Term	Definition
Agent	System that selects and executes actions toward a goal.
Agent harness	Runtime controlling tools, state, policies and loops.
Autonomy	Degree to which a system chooses actions without human intervention.
Handoff	Transfer of control from one agent to another.
Human-in-the-loop	Required human review or approval within execution.
Least privilege	Granting only minimum required permissions.
Multi-agent system	Several interacting specialised agents.
Observation	Result returned after an action.
ReAct	Pattern alternating action decisions and observations.
Sandbox	Isolated environment for risky execution.
State	Stored information describing current execution.
Supervisor	Agent coordinating worker agents.
Termination condition	Rule that ends the loop.
Tool	External function available to an agent.
Trace	Recorded sequence of model, tool and state events.
Workflow	Developer-defined sequence of steps and branches.