Comment by erader
5 days ago
I've been working on something like this the last few months specifically around service quote analysis (repairs, construction, hvac, auto, etc.) and it's really cool. I think LLM analysis is the way to go because the amount of complexity is absolutely staggering - just to start the difference in quality and information available on a quote is drastically different between vendors within the SAME vertical. Then to do actual do analysis on local laws, the details of your property (not just photos/videos, but zoning and lot details), vendor analysis, etc.
On top of it all, the most important thing to consider is intent -> An emergency plumbing visit is often very different than a proactive upgrade.
edit: spelling
This is in line with my thinking, can you say more about how intent changes how you would use a system like this?
I had a really complex negotiation for car repairs (goodwill warranty, balancing a long list of repairs/recalls etc) which was pretty time sensitive. If I had already had my service record in a structured format along with the manufacturer's policies I feel like I could have responded with better preparation. Same for any other big maintenance items on the house, mortgage, insurance, etc.
And then there's the flip side--what do my policies and healthcare/loyalty plans cover that I'm not taking advantage of? What can be combined towards my goals etc.
For my initial system I'm not building full historical service history, insurance policies, etc. because it's a serious amount of scope on top of the core value prop, which is point-in-time "is this a good quote?". When I eventually do this, I'd need to do it proper with LLM + RAG, etc.
I do have the concept of an "asset" which could be a car, house, etc. and with enough basic info it's pretty easy for the LLM to cross reference common problems, or at least suggest questions that you should follow up on.
I'm leaving intent pretty free-form for now, the most friction I'm willing to add is 2 things:
- Basic enum preferences around budget and flexibility to help with prompting
- A claude code style "a few questions" follow up
Any additional form friction I think gets too complex.
It's funny, a lot of my research has been from subreddits for auto, homeownership, questions for people who work in trades, etc. Every time someone asks "is this quote fair", the response from the experts is almost always "But what do you want"
So in a time-sensitive repairs scenario, intent could "What get's my car safe to drive again for daily commute.... or for a long roadtrip". The output analysis could recommend which fixes are highest priority, where work could be split up, delayed etc.
how do you handle the LLM hallucinations in analysis? I like it for data extraction but i never trust it to analyze anything
First, I've spent a ton of time becoming opinionated about a normalized data model that supports the product experience I'm trying to build. This applies both to the extraction (line items, warranty sections, vendors, etc.) and the analysis portion. The latter is imperfect, but aligns philosophically with what I'm willing to stand behind. For example
- building outputs for price fairness (based on publicly available labor data)
- scope match (is vendor over/under scoping user's intent)
- risk (vendor risk, timeline, price variability, etc.)
- value (some combination of price, service, longevity, etc.)
I don't get much hallucinations in my testing, but overall it's pretty complex pipeline since it is broken down into so many steps.