Comment by alexhans
5 days ago
I've naturally done this a lot and suggested that other people prompt this way. I can see how a "ready made" solution with this behaviour could be interesting.
The compliance parts are good to make clear considerating one segment of the user target audience.
May I ask what techniques do you use to test regressions or correct behaviour of your multi turn conversation in your product? What are the biggest lessons and learnings in that space?
Great question. Testing multi-turn Socratic logic is much harder than testing standard RAG. We currently use a 'Shadow Evaluator'—a separate LLM instance that reviews session logs to flag cases where the tutor 'collapsed' and gave a direct answer.
The biggest learning so far: 'Instruction Drift' is real. You can't just give one long prompt. You have to break the reasoning into smaller 'Cognitive Process Capsules' (CPCs) to keep the model from losing the Socratic thread during long sessions.
Thanks for the insight. This resonates with the same explanation I tried to give around non determinism tasks that feed into each other [1]. If you don't break the cycle with determinism or, in your case, CPCs, you get to a stage where error is almost guaranteed.
Now you have the very useful and opinionated sochratic method. Are you playing around with teaching meta thinking as well? E.g. helping your daughter also approach other meta systems of interaction such as breaking down problems or systems thinking.
In any case, good luck with the endeavour and I love that's you're doing something that immediately solves your need because it's the most real UX ROI there is. I'll try to stay tuned.
Can I RSS subscribe to the signals? I don't see a feed or newsletter at a glance.
[1] https://alexhans.github.io/posts/series/evals/error-compound...
Thanks for the insight, alexhans. You hit the nail on the head regarding the deterministic cycle—CPCs were born out of that exact frustration.
To answer your question: Yes, Meta Thinking is the eventual north star. Right now, I’m focused on the "Architectural" logic of solving a single problem, but the goal is to help students recognize their own patterns across different domains.
I don't have an RSS feed or newsletter set up just yet, but given your interest, I’ll prioritize adding a simple "Signals" feed to the site so supporters can follow the technical journey. A question for you (or anyone else in the thread): As a solo dev with zero marketing budget and no sales background, I’m struggling with how to get this in front of the right parents without getting lost in the "AI hype" noise. Since I’m self-funding the API costs, I can't afford big ad spends.
If you have any advice on organic growth for a tool that's intentionally designed to be "harder" (more friction) than the competition, I’d love to hear it.
1 reply →