Comment by TeMPOraL
16 hours ago
Now this is how[0] we get some of the most magical Star Trek technology that eludes us to this day, such as automatic doors. Because if you notice, they work much, much better than real-life ones, because they seem to be doing something like this:
if(within 10 meters of door then) {
if(Jev(
[A] Intends to go through, expects doors to open
[B] Approaches with no intent to pass
[C] Passing by, loiters, or otherwise
[D] Other
) == most definitely A) {
// open doors, +/- identity/security/interlocks check
} else {
// ignore
}
}
Keywords: ambient awareness, understanding of intent.
Most interactive tech on Star Trek is like this - from phasers to consoles to communicators to voice interactions with the ship's computer. The computer seems to be aware of the user and surrounding, and actively infers intent from context, to DWIM ("do what I mean") and when they mean it, instead of doing dumb things[1] on simple triggers.
--
[0] - The direction, not final implementation - surely we can work out how to do it more efficiently than wrapping around final stage of LLM. But the point is, multimodal.
[1] - Obviously it's a fictional show, but in this, both Watsonian and Doylist explanations align near-perfectly: this is/portrays advanced technology, that Just Works and doesn't do stupid shit. Same intent recognition algorithm is there - fictionally in the computer, in reality in the minds of on-set technicians.
To do this you need video/motion understanding, the intent cannot be judged from still images or state descriptions.
We’ve had the tool to do this since mid 2025, V-JEPA2 [1], Yann Lecun’s last work at Meta.
It runs at several FPS on a macbook and can even be trained locally. Chaining it with Jev for decision-making would probably work great!
[1] https://ai.meta.com/research/vjepa/
The technique i mentioned in the blogpost works with video too. I tested succesfully with Qwen/Qwen3-VL-8B-Instruct. Effecient caching is a little trickier though, but very doable.
"doing dumb things" and "stupid shit" is an odd choice to describe tools that only trigger on explicit activation. Is a windshield being lowered by a switch being held a "dumb thing"?
Dumb things start to happen when you try to build Star Trek interfaces. When you build DWIM interfaces in real life, they are annoying and trigger unwanted and the implementation is without exception, by necessity, a growing ball of spaghetti.
> Dumb things start to happen when you try to build Star Trek interfaces. When you build DWIM interfaces in real life, they are annoying and trigger unwanted and the implementation is without exception, by necessity, a growing ball of spaghetti.
This is what I'm talking about.
"Growing ball of spaghetti" happens because system cannot recognize intent. That problem, itself, was something most engineering teams still seem to fail to recognize.
Automated doors are my favorite example, because the "simple solution" is ubiquitous and wrong and we got used to it, and complex solution is usually leading people the wrong path. In short:
Current doors: if(user triggers proximity detector) { open(); }
Failed attempt at DWIM: if(user triggers proximity detector && this && not that && except when ...) { open(); }
Star Trek: if(user intends to walk through the door) { open(); }
LLMs are the first tool we have that allow us to infer user intent directly, and use that as an input.
And recognizing intent itself cannot be done with a single sensor. It requires both general understanding of how humans behave, and awareness of surrounding and subjects - their movements and behavior, as well as who/what they are, and what they are doing.
Just because you're using an LLM (even a big one) doesn't mean there won't be cases of ambiguous intent or random classification errors.
Automatic doors (as used in the real world) are pretty much always located in areas intended for actively moving foot traffic (not as interior doors for every room). So this feels like adding a lot of complexity and unwelcome probabilistic behavior to something that in most locations does the right thing 99% of the time already.
> Most interactive tech on Star Trek is like this - from phasers to consoles to communicators to voice interactions with the ship's computer.
Almost like the Star Trek mechanisms can infer perfect intent.
Like there’s a hidden script or something.
More seriously, I think there’s real value in an automatic door that behaves consistently rather than one that tries to infer messy human intent. Real life isn’t a TV show and there’s both ambiguity in how people behave and how they even intend to behave. It’s mostly not hard to understand how a proximity sensor door will function. Using a black-box classifier to improve that won’t necessarily make people like it more. And calling up to the cloud for every sensor event, ignoring privacy issues, adds weird latency and a huge failure mode during data center outages.
Spoken like a person who's never had to queue in a shop with an automatic door and then the queue reaches too close to the door and then you're the unfortunate person who keeps on accidentally opening the door while standing at the back of the queue and then everyone else in the queue glares at you.
So now you turn around to check if anyone else is coming in behind you and the door triggers. Or alternatively, you finally get sick of waiting and turn around but the door takes 5 seconds to realize you're not just doing aimless human fidgeting and actually want to leave.
Also .15% of the time the door just randomly opens or stays shut with no rhyme or reason. Bug previously filed with cloud door vendor but told it's within door SLA and sent previously signed copy of agreement that "all neural nets are a black box and inherently probabilistic so errors may occur at any time and not a product defect"
> It’s mostly not hard to understand how a proximity sensor door will function. Using a black-box classifier to improve that won’t necessarily make people like it more.
This is fundamentally the same problem as with self-driving cars. There's "dumb and predictable" and there's "DWIM" and the space in between is "mostly useless or dangerous". For DWIM interfaces, you have to reach a level of reliability[0] where you just know what the device will do and trust it to do it. Like, you walk at the doors "on autopilot" while distracted by phone or conversation, and the doors open in time to let you comfortably through.
Note that the currenetly used automated doors rarely meet even basic reliability standards for a dumb, predictable device. Notably:
1. Most don't open fast enough to let a person walk through them at normal pace, forcing people to slow down or stop;
2. Many don't react well, or at all, when you approach from a side.
That 2. is IMO exemplary of the problem: it's typically some tuning to prevent the doors from getting triggered by people passing by or leaving the store through exit doors at 90deg to entry doors - but there is no complete solution here and any attempt just makes things inconvenient for users, because it's trying to solve the wrong problem. It's solving for "open for people approaching, ignore people passing by", whereas the problem that needs to be solved is, "open only for people who clearly intend to go through".
> And calling up to the cloud for every sensor event, ignoring privacy issues, adds weird latency and a huge failure mode during data center outages.
You do not need to call up cloud for this.
In fact, this post inspired me to get the equivalent working off local model to finally get "computer" activation DWIM (no bullshit "hotword recognition", but continuous awareness of conversations, and picking up only on those clearly intended for the "home's computer" -- exactly like Star Trek does it). Turns out, it's totally doable on the humble RTX 4070 Ti (which is like TFA card but only half the memory bandwidth).
Hell, a doors PoC would be too (gonna try this later for triggering lamps via Home Assistant), but since determining intent here requires processing multiple frames, that's gonna be tricky to do with local model in near real-time. Tricky, but not impossible.
--
[0] - Possible interactive reliability, so you're able to negotiate your intended meaning on the spot, without even thinking about it, through a gesture, word, face movement, pose change, etc.
How does the model get context to decide ABCD ?
As described in the blog post – e.g. image via a camera feed.