Comment by qurio_dev
5 days ago
Great question. Testing multi-turn Socratic logic is much harder than testing standard RAG. We currently use a 'Shadow Evaluator'—a separate LLM instance that reviews session logs to flag cases where the tutor 'collapsed' and gave a direct answer.
The biggest learning so far: 'Instruction Drift' is real. You can't just give one long prompt. You have to break the reasoning into smaller 'Cognitive Process Capsules' (CPCs) to keep the model from losing the Socratic thread during long sessions.
Thanks for the insight. This resonates with the same explanation I tried to give around non determinism tasks that feed into each other [1]. If you don't break the cycle with determinism or, in your case, CPCs, you get to a stage where error is almost guaranteed.
Now you have the very useful and opinionated sochratic method. Are you playing around with teaching meta thinking as well? E.g. helping your daughter also approach other meta systems of interaction such as breaking down problems or systems thinking.
In any case, good luck with the endeavour and I love that's you're doing something that immediately solves your need because it's the most real UX ROI there is. I'll try to stay tuned.
Can I RSS subscribe to the signals? I don't see a feed or newsletter at a glance.
[1] https://alexhans.github.io/posts/series/evals/error-compound...
Thanks for the insight, alexhans. You hit the nail on the head regarding the deterministic cycle—CPCs were born out of that exact frustration.
To answer your question: Yes, Meta Thinking is the eventual north star. Right now, I’m focused on the "Architectural" logic of solving a single problem, but the goal is to help students recognize their own patterns across different domains.
I don't have an RSS feed or newsletter set up just yet, but given your interest, I’ll prioritize adding a simple "Signals" feed to the site so supporters can follow the technical journey. A question for you (or anyone else in the thread): As a solo dev with zero marketing budget and no sales background, I’m struggling with how to get this in front of the right parents without getting lost in the "AI hype" noise. Since I’m self-funding the API costs, I can't afford big ad spends.
If you have any advice on organic growth for a tool that's intentionally designed to be "harder" (more friction) than the competition, I’d love to hear it.
Thanks for the insights.
You don't change your priorities based on my opinion but adding an RSS should be quite easy as it's just an auto-generated XML.
I don't have any concrete answers but some things to consider:
- Are you really targeting unknown people who stumble onto your site? Why not literally try to see if the parents of your daughter's school would like it? What's a payment plan or type they'd be willing to consider to justify the product subscription or purchase. What's the real barrier at this stage? They might not use it even if free? (Discoverability) Or is it cost/the model for the perceived value?
> Since I’m self-funding the API costs, I can't afford big ad spends.
I'd be very careful with ad spends without talking to a trusted person who has gone through the journey and forced you to ask the right questions about expectations and the path to get there. Being very harsh on your pitches/answers might get you to different decisions.
One invariant in marketing is networking and audience. If you have it already, you can leverage it and they can help you expand. If you don't have it, how do you build it overtime without feeling unseen? I think that's a lot of market study at the personal level.
> If you have any advice on organic growth for a tool that's intentionally designed to be "harder" (more friction) than the competition, I’d love to hear it.
Who do you think would resonate with this and agree that your product meets that goal?