AI Agent Evaluations
An agent evaluation is an evaluation completed by an AI agent instead of a staff member. Staff write the evaluation criteria in plain language; the agent reads an application, returns a score from 0 to 100 with a short explanation, and Full Fabric saves the result as a submitted evaluation. Agent evaluations sit alongside the evaluations done by staff and never replace them: they give reviewers a first reading, and the decision stays with staff.
Before You Start
Agent evaluations need all of the following:
| Requirement | Where |
|---|---|
| The AI module, with AI model access turned on | Settings → Automation & AI → Governance. See AI Console & Assistant. |
| The AI evaluations setting turned on for the institution | Institution settings. See Permissions and Settings. |
| An evaluation template set up as an agent evaluation, with its criteria | See below. |
| For automatic runs, the agent template selected on the application template | Application template settings. |
Note: When AI model access is off, no application data is sent to the model provider and agent evaluations do not run.
Setting Up an Agent Template
When you create an evaluation template (see Evaluation Templates), set Type of evaluation to MentorAI Agent instead of Manual Evaluation. The choice appears only while the AI evaluations setting is on. A template is either a team evaluation or an agent evaluation, never both.
Full Fabric then sets up two things for you:
- An agent with the same name as the template. Renaming the template renames the agent.
- A form with one section, Agent Evaluation, and two system fields: Agent Score (a number) and Explanation (a paragraph).
An agent template has a Criteria tab instead of the Schema and Automation tabs. Write the Agent Instructions there: the evaluation criteria the agent applies, in plain language (for example, the qualities you look for and how to weigh them). Test prompt runs the instructions on up to five submitted applications and shows the score and explanation for each, without saving an evaluation. Editing the instructions needs the permission to view, create and edit agent evaluation prompts. See Permissions and Settings.
To run the agent automatically, open the application template's Evaluations tab and, under Mentor AI, pick the template in the Agent field. The default, Don't use an agent to score applications, runs none.
What the Agent Reads
For each application, the agent receives:
- The application's name, its class or course, its status, and its last updated, submitted, unsubmitted and deadline dates.
- The answers in each of the application's form documents.
Blank values are left out. The agent is asked to score only from this data and the criteria in the prompt, and to report when there is not enough information to evaluate.
When Agent Evaluations Run
On Submission
When an applicant submits an application, and its application template has an agent evaluation selected, Full Fabric runs that agent in the background. The run starts only when every requirement above is met; otherwise the submission completes without one.
On Demand
Staff can also run an agent template on a selection of applications with Create evaluations in the applications list, the same bulk action used to start staff evaluations for many applications at once. Each selected application gets its own background run. See Starting and Assigning Evaluations.
Warning: Running an agent template on demand replaces the existing evaluation of the same template on each selected application. The previous score and explanation are deleted before the new run.
The Result
When the agent returns both a score and an explanation, Full Fabric creates an evaluation from the template, fills in Agent Score and Explanation, and submits it straight away. If the agent returns no score or no explanation (for example, because the application holds too little data), no evaluation is created.
Agent evaluations differ from staff evaluations in a few ways:
| Behaviour | Agent evaluation |
|---|---|
| State | Submitted as soon as it is created. |
| Assignment | Recorded against the person whose action started the run. |
| Score | Feeds the application's Agent score, not its Score. See Scoring and Decisions. |
| Bulk field updates | Not allowed. |
| Duplicating to another application | Not allowed. |
| Inbox cards | None are created for assignment or submission. |
Who Sees Agent Evaluations
Opening an agent evaluation needs the permission to view agent evaluation scores and outputs. Without it, staff cannot open agent evaluations, do not see them under Other evaluations on another evaluation, and do not get them in the application PDF.
Selecting an agent template on an application template needs its own permission. See Permissions and Settings.
Duplicating and Deleting Agent Templates
- Duplicating an agent template also duplicates its agent, with the same prompt, so the copy can be changed without affecting the original.
- Deleting an agent template deletes its agent and every evaluation made from it, and removes it from any application template that had it selected.