An AI agent platform should finish a defined job inside a boundary you control. The evaluation must test the job and the boundary.
A polished demo proves that the happy path can work once. The twelve questions below test whether the system can carry routine work after the demo ends.
#1. What exact result does it own?
Start with a noun and a finish line. "Help with finance" is not a job. "Reconcile yesterday's payments and list every mismatch" is a job.
Ask the vendor to state what counts as complete, failed, and blocked. If the answer is always a chat response, you are evaluating an assistant.
#2. Can it reach every required system?
List each system in the workflow. Mark whether the product can read, change, and send through it. These are different abilities.
Also ask what happens when the app has no packaged integration. A platform can use an API, a standard tool server, or a browser. Each route has different limits. Alfera documents its routes on the integrations page and its browser on the cloud computer page.
#3. Are permissions enforced outside the model?
A sentence that says "never send" is an instruction. It is not an access control.
Ask the vendor to show the check that runs between the model's request and the external action. Then test a message that asks for a forbidden action. The tool must refuse it even when the model wants to continue.
Read the permission test we use and compare it with the product's permission controls.
#4. Can read, change, and send access differ?
"Connected" is too broad. A finance agent may need to read invoices without being able to edit the ledger. A support agent may draft email without sending it.
Require grants per agent and per connection. Test an allowed read and a blocked write in the same app.
#5. What requires a person's approval?
Ask whether approval rules can use the action, amount, system, or agent. Then ask who receives the request and what context they see.
The work must pause before the action. A warning after the action is not an approval control.
#6. What record does each run leave?
Require a record that answers these questions:
- Who started the work?
- Which sources did it read?
- Which tools did it call?
- What changed outside the platform?
- What did each step cost?
- What finished, failed, or remains blocked?
Ask the vendor to export that record. A conversation transcript alone does not show the external actions.
#7. What happens after a partial failure?
Use a workflow with three actions. Make the second action fail. Then retry.
The product must show which work already happened. It must avoid repeating an external action without evidence that repetition is safe. Ask what happens when the result of a request is unknown.
This is a better reliability test than running the happy path ten times.
#8. Can the work start without a person present?
Recurring work needs a trigger. Test the trigger you need: a set time, an app event, an incoming message, or an API call.
Then pause it, run it once outside the schedule, and inspect the result. Alfera supports these starts through scheduled jobs.
#9. How does company context stay scoped?
Ask what the agent can know before a question arrives. Then remove one source and repeat a question that only that source could answer.
The old answer must disappear. A retained summary can leak knowledge after the source is removed. Our memory guide explains the test in detail.
#10. Can you choose the model?
Different jobs need different cost, speed, and reasoning profiles. Ask whether you can select approved models, use your own provider contract, and restrict models by workspace.
Alfera lists its supported choices on the models page. The useful feature is the policy, not the size of the list.
#11. What does the full job cost?
Ask for the subscription, included usage, model charges, markup, user limits, and overage rules. Then price the same real workflow on each platform.
Do not compare one token price with one seat price. Compare the monthly cost of finished work, including the human time needed to check or repair it. Alfera's pricing page shows the plan and usage lines separately.
#12. What work should stay with a person?
A credible vendor names the boundary.
Keep the work with a person when the standard is unwritten. Do the same when the outcome depends on a relationship or new judgment. An agent can prepare evidence and drafts. It should not invent the standard.
#Run the same two-week test for every vendor
Choose one real job with a clear finish line. Give each platform the same source data, tools, permission limits, and difficult case.
Count five results:
- Finished without repair.
- Finished after a person intervened.
- Wrong but detected.
- Wrong and not detected.
- Blocked with a useful explanation.
The fourth result carries the most risk. The fifth can be healthy when the boundary worked as designed.
The first-month rollout guide gives a longer adoption plan. For a vendor comparison, two weeks on one job is enough to find the major limits.