500 Prompts Probably Won’t Help You

M. Heouaichi

M. Heouaichi

When we started looking more closely at AI Visibility, one thing initially seemed pretty logical: The more prompts we monitor, the better our picture should become of how a brand appears in ChatGPT, Gemini, and other AI systems. If 100 prompts are good, 500 should be better. And with 2,000, you should basically know everything.

I now think that’s the wrong way to think about prompt tracking.

The problem, in my view, isn’t that having a lot of prompts is fundamentally bad. The problem is that you very quickly start confusing quantity with coverage of all possible prompts. 500 prompts can describe a surprisingly small part of the actual market if they’re all variations of the same question. At the same time, 100 well-chosen prompts can provide a fairly nuanced picture of when a brand is mentioned, which competitors appear, and which questions it doesn’t show up for at all.

For AI Visibility, I therefore think we need a different metric than “How many prompts are we tracking?”

The more important question is:

Which decision-making situations in our market do these prompts actually cover? I think this is where the art begins. For us, it’s not easy to simply work out which prompts are important.

A Prompt Is Not a Keyword With a Question Mark

Many prompt sets are still created using what is perhaps, without wanting to offend anyone here, a universally familiar SEO logic. You take a keyword list and turn “AI Visibility Tool” into “What is the best AI Visibility Tool?”. Then you turn that into “Which AI Visibility Tools are there?”, “Which tool is suitable for AI Visibility?” and “What are the best tools for AI Visibility?” and so on and so forth. I think you get what I’m getting at.

Congratulations. We’ve now built four prompts. But, but, but, in terms of content, we probably still only have one prompt.

For classic rank tracking, it can make sense to look at different search terms individually because each query has its own demand, SERP, and ranking dynamics. With prompt tracking, I’m initially interested in something else: Which information or decision-making situation are we simulating right now?

Let’s take an example. A company wants to understand how visible its brand is in AI Search. Around this topic, you could easily generate hundreds of different phrasings. But the truly different questions don’t begin with the phrasing, they begin with the intent.

Someone might first want to understand what AI Visibility actually means. Another person already knows what the problem is and is looking for tools to measure it. Someone else is comparing several providers. An existing user might want to know why a competitor is recommended more often for certain topics. And a marketing team is trying to find out which measures can improve this visibility.

Five similar terms. Five completely different situations. That’s the level on which I would build prompt tracking.

Many similar prompts compared with a smaller, more diverse prompt set.

Start With Topics, Not Prompts

If I had to build a prompt set for a brand from scratch today, I therefore wouldn’t start with the question of how we can generate 500 questions as quickly as possible. I would first define the topics in which that brand actually wants to be relevant.

Let’s take a simplified SaaS example. We might find that six topic clusters are particularly important to the business:

  • Understand the problem and category

  • Discover solutions and products

  • Compare providers

  • Select a product for specific use cases

  • Prepare a purchase decision

  • Optimize or replace an existing solution

Now, for a specific topic like “AI Visibility Monitoring,” we can develop different questions along these situations. This allows us to avoid measuring 30 variations of “best AI Visibility Tool” and then convincing ourselves that we’ve examined the market particularly thoroughly.

A reasonable set could, for example, include questions like:

  • What does AI Visibility mean for companies?

  • How can I measure whether my brand is mentioned in ChatGPT?

  • Which tools can monitor AI Visibility?

  • What should a good AI Visibility tool measure?

  • Which solution is suitable for a B2B company?

  • How can I compare my AI Visibility with competitors?

  • Why is my competitor recommended by AI assistants and my brand isn’t?

  • Can I measure AI Visibility with traditional SEO tools?

  • What data do I need to evaluate GEO measures?

  • How can I improve the visibility of my brand in AI answers?

  • What role do sources and mentions play in AI Visibility?

  • Is a separate AI Visibility tool worth it in addition to my existing SEO stack?

Now we only have twelve prompts. But each one has a job.

And that’s exactly the crucial difference.

12 to 15 Prompts per Cluster Are a Good Start, Not a Law of Nature

As a working heuristic, I like 12 to 15 prompts per important topic cluster. Not because some magical threshold is suddenly crossed at prompt number 16, but because the number forces you to make selection decisions.

With eight topic clusters, that puts you at roughly 96 to 120 prompts. With twelve clusters, it would be around 144 to 180. That’s large enough to cover different situations, intents, and phrasings, but still small enough that you understand what is actually being measured.

Within a cluster, however, I wouldn’t simply generate twelve prompts just like that. A useful framework should cover several dimensions:


Dimension

Example

Problem

How do I find out whether my brand is visible in AI Search?

Education

How does AI Visibility Tracking work?

Discovery

Which tools measure AI Visibility?

Comparison

Which AI Visibility tools are suitable for B2B SaaS?

Use Case

How do I compare my AI Visibility with competitors?

Alternative

Can I also measure AI Visibility with my SEO tool?

Branded

What is Palagus?

Branded Comparison

What alternatives are there to Palagus?

Evaluation

What should I look for in an AI Visibility tool?

Action

How can I improve my mentions in AI answers?

I would also pay attention to the mix of branded and non-branded prompts. Anyone who exclusively monitors questions containing their own brand name is, in the worst case, only measuring how well AI systems respond to questions in which we have already put the desired brand into their mouths. More interesting are often situations in which the model itself has to decide which brands to mention.

The Most Interesting Prompts Are Often the Ones Where You Don’t Appear

This is exactly where prompt tracking becomes strategically interesting. If a brand appears for “What is Palagus?”, that is fundamentally relevant. But it wouldn’t be particularly surprising.

Far more interesting is a commercial question where several competitors are mentioned or recommended and your own brand is missing.

Because suddenly we no longer have an abstract visibility number, but a concrete question: Why does the AI system consider other providers more relevant in exactly this decision-making situation?

Perhaps content for this use case is missing. Perhaps the brand barely appears in the sources that corresponding answers draw on. Perhaps the positioning on the company’s own website is too vague. Perhaps competitors have simply built a much clearer connection between brand, category, and problem. And sometimes the most honest explanation is also: The competitor is actually the more plausible answer to this specific question.

That is much more valuable than another number on a dashboard.

Prompt Tracking Should Be a Sample, Not a Phone Book

I would therefore think of prompt tracking more like a sample. We are not trying to write down every theoretically possible question that someone might ever ask an AI system. That would be hopeless anyway and not particularly useful, especially since you can’t represent all of humanity or the entire target audience.

We want to observe a representative set of business-relevant situations. Welcome to the world of statistics.

That also changes how I would add new prompts. I no longer ask: “Can we track this phrasing too?” Of course we can. The better question would be: “What new information do we get from it?”

If prompt 184 practically measures the same thing as prompts 30, 77, and 131, I would probably not include it. If, on the other hand, a new prompt represents a previously missing use case, a different persona, or a later stage of the purchase decision, it can be valuable even if it sounds very specific.

This leads to a somewhat counterintuitive consequence: A good prompt set should not simply grow. It should also be able to shrink.

A prompt set is regularly pruned like a plant.

After a Few Weeks, I Would Start Deleting

At the beginning, you can be more generous. If it’s not yet clear which prompts actually provide different insights, a slightly larger initial set is better than artificial precision.

Once enough measurements are available, however, I would regularly check which prompts are redundant. Two questions can look completely different linguistically and still practically always produce the same brands, sources, and patterns. In that case, we might not need both. However, this differs from case to case.

Conversely, two prompts that sound almost identical can produce surprisingly different results. A small addition such as company size, industry, region, or use case can potentially change the set of recommended providers. Those are exactly the kinds of differences we want to keep because they tell us something about how AI systems structure the market.

My ideal prompt set would not be a static benchmark that someone once put together in an Excel file and then left untouched for twelve months. It would be a living model of the market: New products emerge, competitors change their positioning, customers ask new questions, and some topics become less important. Well, the thing is alive after all. Prompt research therefore doesn’t stop once monitoring begins.

Actually, that’s when it really begins. I think the most important thing is to identify the right prompts in the first place. After all, everything that follows is built on exactly that.

The Relevant Number Is Not the Number of Your Prompts

Of course, at the end of the day, an AI Visibility system needs concrete prompts that are regularly executed and evaluated. But I would be cautious with any dashboard, any strategy, and even with ourselves, as soon as a large number of prompts is automatically understood as a mark of quality.

10,000 monitored prompts sound impressive. 10,000 bad prompts are still bad prompts. Well, you know the input-output scheme 😁.

The number I would be interested in is Coverage – I know, it’s a fancy term: Have we mapped the topics that are relevant to the business? Have we taken different phases of the customer journey into account? Do we have brand questions and generic questions? Are important use cases, personas, and competitive decisions included? And do we know, for each cluster, roughly what decision we can derive from the results?

If we can answer these questions, maybe 120 prompts are enough. If we can’t answer them, even 5,000 won’t help us.

And that’s important: More data is pretty easy to produce. A good measurement model is considerably harder. That’s true for many things and isn’t limited to AI Visibility.