Comment by qurren
1 day ago
> Home Assistant already knows which bins (trash cans, for the international readers) are due for collection. [...] What it can’t tell me is whether I’ve actually done anything about it.
My first gut reaction to this was: Have you tried using a vision-language model?
I used to have an elaborate Home Assistant setup with tons of zigbee sensors here and there telling me if front door is not closed properly, if the rice cooker is left on (via current monitoring), if the garage door is left open (with a magnetic contact sensor), if window is left open, if the soil is dry, etc. I had to keep changing batteries of all these things and manage flaky Zigbee connections in the past.
More recently, I've found the absolute most reliable, ~zero-maintainence solution to be an LLM. Home Assistant already has an LLM plugin, you can wire it up to a local LLM if you want, and you can just feed it an image and ask "Is this garage door open or closed" and I found that to be much better for my sanity than changing batteries every 6 months and dealing with flaky sensors sending me false alerts.
I even found it useful for "Did the cat vomit on the floor" (if so notify me and pause the cleaning robot schedule) and "Do any plants look like they need attention"
For TFA's use case, if you already have a security camera facing front, you can just feed it that image and ask "Are the garbage bins out"
Also, if you're worried about cameras sending home images to the cloud, there are non-cloud options. (I use Unifi cameras and Home Assistant gets a RTSP stream over LAN.)
Yah, this is how I've solved this in my Home Assistant setup. My doorbell camera can see where I keep my bins when they're not out. We have three types of bins- a trash bin, a recycle bin, and two yard waste bins. Recycling and yard waste are picked up on alternate weeks, and I often forget which bins need to go out.
I have an automation that will grab an image from that camera and send it to the Anthropic API with a prompt describing where the bins are, and the color of each bin. The prompt comes back with JSON indicating which bins it can see. If it can see e.g. the yard waste bin, and it's time to take the yard waste bin to the curb, it will send me a notification.
It also reminds me when it's time to return them to their "resting" position after pickup- unless I already have.
Home Assistant orchestrates this all. I also have it hooked up to a "Bindicator"- a white 3D printed bin with a color-changing LED inside, controlled by an ESP8266 using ESPHome. When it's time to move the bin it lights up the color of the bin that needs to be moved. If I haven't moved the bins back after a day it turns orange to remind me.
Claude configured the automations for me via MCP but it's not particularly complicated. It's pretty reliable, though sometimes just because of where the doorbell camera is it can't clearly see the bins. If it can't make a ruling on the bin location it just falls back to notifying me so I will sometimes get a reminder I don't need.
I found all of this device, software, and battery management exhausting to the point where it was harder than just remembering if I closed the door or noticed that the cat puked on the carpet overnight. I let it all go and just lived like a normal person and everything got easier.
- I have to change batteries of zigbee devices maybe every three years or so
- how often do you have to change your cameras' batteries, or are you accepting ugly wiring for powering from mains?
- how many cameras do you have installed to cover your home?
- IP cameras are flaky, too
- constantly sending images of one's private space to a cloud LLM is Orwellian
- are you polling your LLM minutewise "Did the cat vomit on the floor"?
> how often do you have to change your cameras' batteries, or are you accepting ugly wiring for powering from mains
Meh (a) I want security and cat-monitoring cameras anyway (b) I'm willing to take the effort to install cameras neatly since they're capable of serving as 100 different sensors
> constantly sending images of one's private space to a cloud LLM is Orwellian
I agree. I have a local server with a GPU. The security side of things records to a local NVR. Nothing goes to the cloud.
> IP cameras are flaky, too
haven't had nearly as many problems here
> are you polling your LLM minutewise "Did the cat vomit on the floor"?
Kind of, but it only re-polls if the camera detected motion, and then every 15 minutes as a catch-all.
agreed it's Orwellian. fortunately, open-weight VLMs you can run on a Mac mini or a strix Halo are more than adequate for this task.
PoE cameras solve all of the cameras problems. Just drag a cable, it's one time bother and if you're clever it won't be any uglier than camera itself
Kinda liking this idea :)
I’ve not played with vision models much. I did some experiments with OpenCV and Processing many years ago, but times have changed.
I think you just wrote me next weekend project ;)
Yeah, when it comes to sensors wired power (or solar power I guess) is absolute minimum, it's extra effort that's always pays off.
My first-gut reaction was:
why are so few hacking their cognitive abilities (basic short-term memory)?
Honestly if I could figure out how to fix my problems in this area it would be worth my time, do you have any suggestions?
I don't have a problem with short or long term memory, in certain contexts. I can hold entire problem definitions and codebases in my head while working, but I often set down my keys on the way to pick up my water bottle and get to my car only to have no idea where they are. I can remember details of specifications, language features, video game mechanics, etc. for years, but often can't remember someone's name.
There's a whole cottage industry of social media telling me this is ADHD but I don't know if that's real or just for clicks, and if I can improve these issues.
That just sounds like short vs long term memory
> Honestly if I could figure out how to fix my problems in this area it would be worth my time, do you have any suggestions?
Many, having unfortunately been all the way at the worst-version of that – and figured out the way back.
No guarantees I can help, obviously, but this rhymes with the types of intermittency I dealt with (if you start going into rooms and forgetting what you went in for, that really sucks and is not “normal” - despite being common.)
Give a way to reach you? If you’re motivated, you can probably optimize.
[dead]