Can You Trust AI With That Task? Score It, Then Decide

Your calendar assistant nudges an internal meeting 15 minutes later. An AI tax module files the firm’s quarterly return with a digit off. Both lean on AI. One slip just shuffles a coffee break; the other invites the IRS and torches your reputation. So which jobs can roll on autopilot and which stay human-led?

I use a simple tool to help with that – the AI Risk Matrix. Easy enough to scribble on a napkin, strong enough to steer seven-figure calls (or in a company like Apple’s case, to guide fraud detection activities).

The Wrong Way

Some firms drop automated AI into every process: proposals, legal responses, vendor picks, you name it. Don’t Do It! When accuracy is shaky or the cost of failure is brutal, blind automation equals carnage, rework, lost deals, and clients who wonder if anyone is still driving.

Welcome to the AI Risk Matrix

To use the matrix we need to score the process we’re considering automating with AI.

First, how accurate do we believe the AI can get?

How accurate is the data that we’re feeding the AI system (this is a huge factor)? How accurate would a human be able to be with that same data? How well will an AI be able to interpret the data? Are we thinking of automating a fuzzy process or something very well defined and understood?

Based on that knowledge, score how accurate an AI can potentially be on this task:

1 = Foggy – might as well flip a coin (what an intern might do)

2 = Reasonably good – ok hit rate (entry level professional)

3 = Very Reliable – solid (reasonably trained professional)

4 = Almost never wrong – gold standard confidence (highly experienced expert – yes it can fly the plane)

Second, what is the cost of being wrong?

How badly do things go if the AI makes a mistake? Could this cost us money? A customer? Our reputation? The company?

Think through the worst case of a bad AI result and score the impact:

1 = Pocket change mistake – shrug it off

2 = Minor annoyance, quick fix – small speed-bump

3 = Serious rework, real money – real money hit

4 = Legal or brand-killer disaster – kiss clients goodbye

Why the Grid Works

1. No hiding in the middle. The four point scale forces a judgement call.
2. Risk meets reality. A typo never equals a bad merger clause.
3. Healthy debate. Scoring sparks talk about the actual task to be accomplished, data quality, failure impact, and fixes.

What to do With the Result?

Based on the quadrant you’re now ready to take action and approach automation with AI in a more structured way.

Assist – Human directed with lots of back and forth to get to the desired result because of the low accuracy makes this quadrant perfect to address with an assistant like ChatGPT.

Help Decide – AI is a talented researcher, bringing lots of appropriate information to a human to make the final decision on how to handle the situation. My team has a task agent that reviews legal documents posted in a Slack channel and provides feedback on the dozen key items we do or don’t want to see in an agreement to help the team quickly move forward. Another example is an AI tool that analyzes an EOS Level 10 meeting transcript to provide coaching and suggestions weekly.

Automate – Amazing scale opportunities for the company to eliminate the grunt work and let people focus on much higher value activities. Automate the heck out of this category. That doesn’t mean fire and forget as you still need to craft a feedback loop that identifies any exception cases for human input and handling. An example is a digital sales coach that triggers for each new sales call transcript, reviews the call against company standards for objection handling, positioning and flow, and then shares the scores and suggestions for potential improvement with the sales person and manager.

Human in the Loop – High risk and high reward opportunities (often those services we provide for clients) are handled with the AI doing a large chunk of the work and a human reviewing and tuning the output. For us at Lextech that means writing software and designing new systems. There are a significant number of tasks the AI can attack and then our engineers, architects and designers can validate that work and fine tune things for exceptionally productive results.

Where Should You Start?

If your team is just getting started with AI, the Assist and Help Decide categories are great places to build AI fluency and mental muscles in the organization and start to see real results. Your first AI initiative probably shouldn’t be in the more aggressive categories.

If you’re past ChatGPT and diving into the higher value opportunities, a solid understanding of the business value of each opportunity weighed against the risks becomes critical to prioritize and select quick wins and long term transformations.

1. Define repeatable process. What is the process we’re considering AI tackling? Are we agreed on the key steps to be taken?
2. Rate AI accuracy. What exactly does the real data look like and how good a job can AI do? Use real data, not gut feel.
3. Price the downside. Cash, client hours, fines – put a number on pain.
4. Identify the AI approach. Where does this task fall in the AI Risk Matrix and is that a good fit for where you’re at in your AI journey?

The Stakes Are High for Getting AI Right

Handing the wrong task to full autopilot can explode a client account before lunch. Ready to explain to the team that a chatbot lost a million-dollar retainer?

Stop guessing. Score the task, read the grid, and know exactly when AI can drive and when your people need both hands on the wheel.

Take Action

• Learn from top firm owners: Apply for membership in Collective 54 
• Stay ahead of industry trends: Subscribe to Collective 54 Insights 
• Get practical, real world AI insights: Connect with me on LinkedIn 

About the Author

Alex Bratton is the author Practical AI for Leaders and Billion Dollar Apps, an adjunct professor of computer science and the CEO & Chief Geek of Lextech, an Apple enterprise partner that applies cutting edge tech to drive employee efficiency for teams through tech powered experiences people love.

Alex is an applied technologist, bringing real business value from cutting edge tech like AI. Alex’s superpower is his keen sixth sense to look around the corner of what’s next with technology to empower people and help businesses thrive – at least that’s what AI says.