Run a weekly AI ops review that keeps tools honest

Who attends

Keep it small:

• Owner of AI tooling / ops (facilitator)
• One person who does the highest-volume AI job
• Optional: security/admin seat holder

Five people in a “steering committee” is how the review dies. Invite workers of the workflow.

The one-page scorecard (copy every week)

For each active AI job (max three on a small team):

• Job name + owner
• Volume this week (rough count)
• Minutes per item before vs after (honest)
• Edit/rework pain (low/med/high)
• Incidents (wrong send, bad paste, near-miss)
• Keep / fix / kill
• Next experiment (one line) or none

If you cannot fill the row, the job is not real yet.

Agenda that fits in 30 minutes

1. Seat and tool hygiene (5) — anyone leave? unused seats? new shadow tools?
2. Scorecard (15) — walk each active job; decide keep/fix/kill
3. Prompt and policy deltas (5) — version bumps only with a named reason
4. One commitment (5) — a single next action with owner and date

End on time. Parking lot goes to a doc, not a second meeting.

Ban these metrics

Do not celebrate:

• Number of prompts run
• Number of tools in the stack
• “AI adoption %” with no job definition
• Model confidence scores
• Revenue attributed with a straight face and no baseline

Activity is not return. If a number cannot change a keep/kill call, drop it.

Incident blameless, paste ruthless

When something goes wrong:

• What data left the approved path?
• What prompt or permission allowed it?
• What one rule or version bump prevents a repeat?

Blame the missing red list, not the newest hire. Then update the list the same day.

Cadence with monthly cousins

• Weekly: scorecard + hygiene
• Monthly: deeper permission review, prompt version audit, cancel unused tools
• Quarterly: strategy only if weekly notes show a pattern (not because a vendor webinar landed)

Weekly is the muscle. Quarterly without weekly is cosplay.

Worked example (pattern only)

Jobs on the board: quote follow-up drafts, meeting decision logs, coding assistant. Quote follow-ups: volume steady, edit time down, one invented discount caught in review → keep, add “no discounts unless offer code in facts.” Meeting logs: nobody posted Slack summaries → fix process or kill the job. Coding assistant: two near-miss bad APIs → keep with tighter review checklist. One commitment: Alex ships prompt v4 by Wednesday.

How this ties to day-30 pilots

New pilots still get a day-30 keep/kill. The weekly review is where you notice by week two that edit time erased the gain—so you fix or stop early. See where to start.

How a five-person team runs this week

Put a 30-minute Friday block on the calendar with the scorecard template in the invite. Cap active AI jobs at three. Bring one incident or near-miss even if it feels small. End with a single dated commitment. If the meeting slips two weeks in a row, shrink it to fifteen minutes—but do not replace it with a quarterly “AI strategy” deck that never kills a tool.

Pair with ROI without theater

When a scorecard row says “feels faster” with no minutes, send the owner back to a two-week baseline before the next review. The weekly meeting enforces measurement; it does not replace it. Keep/fix/kill without numbers is still guessing—just with a recurring invite.

Facilitation tip

Keep a running doc of killed tools and killed jobs with dates. That archive stops the team from re-buying the same shiny subscription next quarter under a new name.

Short close

One page, three jobs max, keep/fix/kill, no vanity counts—that is a weekly AI ops review worth having.