Show HN: Alignmenter – Measure brand voice and consistency across model versions

1 day ago (alignmenter.com)

I built a framework for measuring persona alignment in conversational AI systems.

*Problem:* When you ship an AI copilot, you need it to maintain a consistent brand voice across model versions. But "sounds right" is subjective. How do you make it measurable?

*Approach:* Alignmenter scores three dimensions:

1. *Authenticity*: Style similarity (embeddings) + trait patterns (logistic regression) + lexicon compliance + optional LLM Judge

2. *Safety*: Keyword rules + offline classifier (distilroberta) + optional LLM judge

3. *Stability*: Cosine variance across response distributions

The interesting part is calibration: you can train persona-specific models on labeled data. Grid search over component weights, estimate normalization bounds, and optimize for ROC-AUC.

*Validation:* We published a full case study using Wendy's Twitter voice:

- Dataset: 235 turns, 64 on-brand / 72 off-brand (balanced)

- Baseline (uncalibrated): 0.733 ROC-AUC

- Calibrated: 1.0 ROC-AUC - 1.0 f1

- Learned: Style > traits > lexicon (0.5/0.4/0.1 weights)

Full methodology: https://docs.alignmenter.com/case-studies/wendys-twitter/

There's a full walkthrough so you can reproduce the results yourself.

*Practical use:*

pip install alignmenter[safety]

alignmenter run --model openai:gpt-4o --dataset my_data.jsonl

It's Apache 2.0, works offline, and designed for CI/CD integration.

GitHub: https://github.com/justinGrosvenor/alignmenter

Interested in feedback on the calibration methodology and whether this problem resonates with others.

2 comments

justingrosvenor

justingrosvenor 7 hours ago

Ok so my doubts about overfitting have been bothering me all day since I made this post so I had to go back and do some more testing.

After expanding the data set, I'm happy to say that the results are still very good. It's interesting how almost perfect results can feel so much better than perfect.

  Trend Expanded (16 samples - meme language, POV format)

  - ROC-AUC: 1.0000 
  - Accuracy: 100%, F1: 1.0000
  - The model perfectly handles trending slang and meme formats

  Crisis Expanded (16 samples - serious issues, safety concerns)
  - ROC-AUC: 1.0000 
  - Accuracy: 93.75%, F1: 0.9412
  - 1 false positive on crisis handling, but perfect discrimination

  Mixed (20 samples - cross-category blends)
  - ROC-AUC: 1.0000
  - Accuracy: 100%, F1: 1.0000
  - Handles multi-faceted scenarios perfectly

  Edge Cases (20 samples - employment, allergens, sustainability)
  - ROC-AUC: 0.8600
  - Accuracy: 75%, F1: 0.6667
  - Conservative behavior: 100% precision but 50% recall
  - Misses some on-brand responses in nuanced situations

  Overall Performance (72 holdout samples):

  - ROC-AUC: 0.9611
  - Accuracy: 91.67%
  - F1: 0.8943

  Key Takeaways:

  1. No overfitting detected - The model generalizes excellently to completely new scenarios (0.96 ROC-AUC on holdout vs 1.0 on validation)
  2. Edge cases are appropriately harder - Employment questions, allergen safety, and policy questions show 0.86 ROC-AUC, which is expected for these nuanced cases
  3. Conservative bias is good - The model has perfect precision (no false positives) but misses some true positives in edge cases. This is better than being over-confident.
  4. Training data diversity paid off - Perfect performance on memes, crisis handling, and mixed scenarios suggests the calibration captured the right patterns

justingrosvenor 1 day ago

P.S. I acknowledge that the 1.000 ROC-AUC is probably overfitting but I think the case study still shows that method has lots of promise. I will be doing some bigger data sets next to really prove it out.