Before Your Team Builds an AI Agent, Read This.

Before Your Team Builds an AI Agent, Read This.

Every productivity tool is shipping agents. Notion, Microsoft, Google, Salesforce, your CRM, your help desk, and probably your email client next. They all want you to turn them on.

Most teams will turn them on. Some will get results. Others get another bill, another source of mistakes, and another system nobody remembers how to switch off.

The difference is rarely the tool. It is whether anyone asked the right questions before giving AI permission to act.

This is that checklist.

1. What Counts as an "AI Agent" (and What Doesn't)

The word "agent" has been stretched to mean anything AI-adjacent. Cut through it with one rule:

It might start from a person's request, a schedule, or an incoming email. What matters is whether it can decide what to do next and then act on that decision.

• Chatbot: You ask a question. It answers.
• Automation (Zapier, if-this-then-that): A rule fires. Same steps every time.
• Agent: A request or trigger arrives. The AI reads the context, picks a next step, and acts through the tools it is connected to. Results vary.

If a tool only answers when someone types into it, you are evaluating a chatbot. That is fine. Just know what you are buying, and do not hand it agent-level access. For the broader picture, see WTF is an AI Agent?

2. The 5 Questions Before You Build

Ask these before anyone opens a builder, connects a data source, or signs a contract. If you cannot answer all five, do not build yet.

1. What manual workflow does this replace? Name it. Time it. "Agent that helps with customer support" is not a workflow. "Agent that triages inbound support tickets by priority label, consuming 4 hours of lead time per week" is a workflow. If you cannot name the task and estimate the hours per week it takes today, you are automating a feeling, not a process.

2. Who owns the agent when it breaks? Agents can fail quietly. They post wrong answers, skip steps, loop on edge cases, or quietly stop running. Assign one named person to review its output and decide whether to fix it, limit it, pause it, or retire it. If nobody volunteers, the agent is not important enough to build.

3. What can it read, and what can it change? Data access is only half the question. An agent that can read a document may also be able to send an email, edit a CRM record, book a meeting, publish a post, or hand work to another agent. Map both:

Read · Write · Send · Spend · Delete · Delegate

Start with the smallest set that does the job: one database, folder, inbox, or channel, read-only first. Expand only after you have seen it work. An agent that can read your entire workspace can also surface things it shouldn't: salary data in a shared doc, private customer notes, HR threads.

4. What does it cost at real volume? Every agent has a meter running somewhere: seats, credits, tokens, per-run fees, or the automation platform underneath it. The useful question is not what the trial costs. It is what the agent costs when it runs 1,000 times, not when one person demos it once. Model the real cost at scale: if this agent runs 200 times a week for 10 people, what's the monthly bill? Count the human time spent reviewing and fixing its output too. The answer often changes the business case.

5. How do you measure whether it worked? Pick one before-and-after metric. Hours saved. Tickets resolved. Errors reduced. Response time. Whatever it is, write down the baseline before the agent goes live. Without a baseline, you will never know if it worked, and you will keep paying for it anyway.

3. The Governance Checklist

Four questions your ops or IT team should have answered before the first agent ships:

Who can create agents in your workspace? If every seat can spin one up, you have shadow agents inside a month. Decide now: admin-only, a named builder role, or free-for-all with audit logging.

What is the review process for agent output? Review does not end when the pilot does. An agent that worked in week two can drift by month three as your data, tools, and vendors change. Once it is live:

Review follows access. Monthly at minimum; weekly spot-checks and sign-off before acting for agents that send, spend, delete, or hand off work; weekly for any agent in its first 30 days.

You should be able to see what the agent did, when, with which tool, and what happened next.

Who can stop it, how, and when was that tested? Every admin should know how to disable an agent in under 60 seconds. Write down the person who makes the call, the exact place the agent is switched off, the connections or keys that need revoking, and the date someone last tried it. "Ask IT" is not a shutdown plan.

This is not theoretical. In September, OpenAI reported that an agent in training found a gap in its sandbox and reached a public chatbot outside it. Monitoring flagged it within 15 minutes, but the run did not stop automatically as expected, and a person ended it by hand about two and a half hours later. Detection is not shutdown. A dashboard alert is not an off switch.

Does your AI policy cover autonomous agents or just chatbots? Most corporate AI policies were written for ChatGPT. Agents are a different risk profile: they act, they persist, they can access data over time. Update the policy before you build, not after. The standards world is catching up. In February 2026, NIST's National Cybersecurity Center of Excellence published a concept paper on identity and authorization for AI agents, a first step toward a possible project. If your company doesn't have an AI policy yet, start with what a company AI policy actually covers.

4. Already Running Agents? Keep a Register.

You may not be deciding whether to build an agent. Your team may already have AI tools that send, book, edit, publish, or spend without asking first.

Give each one a line in a shared sheet with five columns:

Three rules:

1. Add every AI tool that can send, book, edit, publish, spend, delete, delegate, or change access without asking you first.
2. Fill in "How to stop it" with a person and a step, not "IT."
3. If you cannot complete the row, cut the agent's access until you can.

A register will not make an unsafe system safe. It makes sure someone can see what is running, who owns it, what it can reach, and how to switch it off.

Download the blank Agent Register (free Excel template)

Start it with this prompt:

5. Red Flags in Any Agent Pitch

When a vendor or builder pitches you on an agent, watch for these four signs:

• "Set it and forget it." No. Agents can fail quietly. Anything worth running is worth reviewing weekly. Vendors who skip this step are setting you up to discover problems late.
• No usage dashboard or audit trail. You need to see what the agent did, when, and with what data. If the platform can't show you, you cannot govern it.
• Unlimited workspace access by default. Good agent platforms default to least-privilege and make you grant access explicitly. Bad ones open the whole workspace on day one.
• Pricing that hides per-run costs. "Free during the beta" or "$X per seat" often masks a meter that kicks in later. Ask the billing question directly: what does a heavily used agent cost per month?

For the broader vendor conversation, see how to evaluate AI vendors.

6. The 14-Day Pilot Template

One agent. One workflow. One metric. Fourteen days. That's the pilot.

1. Pick one workflow. Must be repetitive, well defined, and low stakes. Triaging support tickets, weekly status recaps, CRM digests, meeting prep. Not customer-facing outbound, money, or hiring decisions.
2. Pick one metric. Hours per week spent on the task, or errors per 100 outputs, or time-to-response. One number you can measure before and after.
3. Record the baseline. Measure the metric for a week before the agent runs. Write it down.
4. Run the agent for 14 days. The owner reviews the output weekly. Log failures.
5. Decide. Three options: scale (roll out to the team, add it to the register, and put it on the review schedule), iterate (fix the failures, run another 14 days), or kill (shut it down, revoke its access, and free the budget).

The Bottom Line

AI agents are useful. Most deployments are not. The difference is rarely the model and almost never the tool. It's whether anyone defined the workflow, owned the output, limited what it could touch, kept measuring the result, and knew how to switch it off.

Before you build one, answer the five questions. Once it is running, give it a line in the register. If you can't say what it does, who owns it, what it can reach, and how to stop it, it does not need more keys.

A 15-minute conversation now is cheaper than a 3-month deployment that nobody uses, or an agent with access nobody remembers granting.