Comment by nworley
19 days ago
I don’t think there’s a clean solution yet but I’m not convinced brute force prompt enumeration scales either, given how much randomness is baked in. I guess that’s why I’ve started thinking about this less as prompt tracking and more as signal aggregation over time. Looking at repeat fetches, recurring mentions, and which pages/models seem to converge on the same sources. It doesn’t tell you what the user asked, but it can hint at whether your product is becoming a defensible reference versus a lucky mention.
From someone who's built a tool in this space, curious if you’ve seen any patterns that cut through the noise? Or if entropy is just something we have to design around.
Disclaimer: I've built a tool in this space as well (llmsignal.app)
Signal aggregation is definitely the right mental model. We've found that tracking 'Share of Model' over time (e.g. how often a brand appears in the top 3 recommendations for a category query) is much more stable than individual prompt outputs, which can vary wildly due to temperature.
It's similar to share-of-voice in traditional PR. You can't control every mention, but you can track the aggregate trend of whether the model 'knows' you exist and considers you relevant.