Imagine the following
3 doctors and each one is right 70% of the time.
You take the majority vote.
You’re now right 78% of the time, correct?
Wait ? 3 average opinions just beat every single one of them, clear!
That’s exactly the challenge hiding in every multi-agent setup I see.
In the next 5 minutes you’ll know:
Why 3 average opinions beat 1 good one, and the one condition that breaks it.
How to set up 3 agents that catch each other’s mistakes instead of copying them.
The 9 checks to run before you trust “the agents agree”.
And two things before we start:
Save the checklist at the end. You’ll need it on your next build.
Send this to the friend who just wired up 10 agents.
Vote
For the majority to be wrong, 2 of the 3 doctors must be wrong on the same case.
That’s much harder than 1 being wrong.
So the vote wins:
3 doctors: 78%
5 doctors: 84%
7 doctors: 87%
Mathematically for 3: 0.7³ + 3 × 0.7² × 0.3 = 0.784
What is described here has a name “ensembling”, and machine learning used it long before agents and LLM hype. Also you need to know that we had a concept called A random forest that turns hundreds of weak trees into one strong model, because each tree saw different data.
Catch
The setup only works if their mistakes land in different places.
Say all 3 doctors studied from the same textbook.
They’re wrong on the same cases, and for the same reasons (almost exactly) and then the vote isn’t reliable
Back to 70%.
The three of them became one doctor speaking three times and the catch here is:
Many weak voices beat one strong voice only when their mistakes don’t line up.
Your agents are the doctors
Run 3 agents with same model and same prompt, what do you expect?
They’ll agree a lot.
That’s not a sign they’re right but more of a sign they’re copies.
And because agreement looks like confidence, → we always say: avoid the “confidently wrong“ perception.
So, keep two numbers apart:
Agreement: how often they give the same answer.
Accuracy: how often that answer is right.
Most setups measure the first and assume the second.
And how-to make their mistakes different
Change at least 2 of these per agent:
Model → Different families fail differently.
Role → One hunts for mistakes, one defends and/or one judges.
Information → One sees support tickets, while one sees usage data.
Persona → A first-time user and a power user read the same screen differently for sure.
Now, if they agree, it means something.
That’s how synthetic users and ensembling been created in tools like fustat.ai: a panel of varied users tells you what a single prompt never will for your UI Designs.
Weigh the votes
Back to the doctors; If one is right 85% of the time and the others 70%, this vote should count more and they can’t be treated equally, unfair !
That’s called “weighted voting” and how to get the weights:
Take 10 to 20 cases where you already know the answer.
Run all 3 agents.
Write down how often they agreed, and how often the agreed answer was right.
High agreement + low accuracy = a very consistent mistake.
Love the disagreements
And when your agents disagree, it’s actually a good sign and please don’t be annoyed.
Because a split is the cheapest quality check you have and it shows you exactly where the system isn’t sure.
Before you close, hereby the checklist
As a Product builder, you don’t need to code for this
Think about your day; you already do it with people.
5 friends with the same background loving your idea tells you less than 3 very different customers agreeing.
Rule of thumb → Measure where they’re wrong and not just how much they agree.
PS: The Product Voyagers grows from your shares and if this saved you one bad build, send it to one PM building with agents.





Handy checklist to use, thank you.