"Your brand appeared" is not "your brand was recommended"

The cheap way to score an AI answer is to search the text for your brand name. If it is there, mark it cited. That is fast and free, and it is wrong often enough to matter. "Acme is not a good fit for this" contains your brand and is a negative. "Acme" inside "architecture" is not a mention at all. If your scoring cannot tell those apart, your citation rate is partly fiction. We built an optional semantic layer to fix it.

Where simple matching breaks

There are two failure modes. The first is the common word. If your brand is "Arc" or "Notion" or "Apple," a plain text search lights up on every incidental use of the word. We already tightened this: the probe now matches your brand on word boundaries, so "Arc" no longer triggers on "architecture." That removes the silliest false positives.

The second failure mode is harder: meaning. A wordlist cannot reliably read tone. "X is often recommended" is positive. "People used to recommend X, but not anymore" is negative, and a naive scorer counts the word "recommend" and calls it a win. Sarcasm, negation, and "mentioned but dismissed" all slip through.

The semantic judge

For when accuracy matters, the probe now has an optional scoring tier: a neutral language model reads the full answer and decides three things by meaning. Was the brand genuinely cited or recommended (not just name-dropped)? Was what the answer said about it accurate? And what was the sentiment toward it?

One design choice we care about: the judge is never from the same model family as the engine being judged. We do not grade Claude's answer with Claude, or ChatGPT's with ChatGPT. Judging your own family invites a quiet bias toward your own outputs, and the easiest way to avoid that critique is to not do it. We also report how often the cheap heuristic and the semantic judge agree, using a chance-corrected score (Cohen's kappa) so that "they both said not-cited" does not inflate the agreement.

Why we keep both

We did not throw the heuristic away. It is fast, free, and good enough for a quick read, so it stays the default. The semantic judge costs an extra model call per answer, so it is a deliberate, paid step for when you need the accurate number (a client report, a published case study). Two tiers, honest about the trade-off: speed by default, precision on demand.

What to do next

If you are scanning your AI visibility casually, the default heuristic is fine. When you are about to make a decision or show a number to someone else, run the semantic tier and check the agreement score. If the two methods disagree a lot, trust the judge and treat the heuristic number with caution. For why the raw rate is uncertain even before scoring choices, see One AI Probe Will Lie to You.

Key Takeaways

• Searching for your brand name overcounts: common-word names and negative mentions both fool it.
• The probe matches brands on word boundaries to kill the obvious false positives.
• An optional semantic tier judges, by meaning, whether you were genuinely cited, accurately, and with what sentiment.
• The judge is always a different model family from the engine under test, to avoid self-preference bias.
• Heuristic stays the fast default; semantic is the paid, accurate option for reports and decisions.