Comment by lrvick
5 days ago
You run a pull-through caching proxy deep enough to give it a simulated internet, like archive.org with mock APIs. Anything it requests, it will block while a fake version of that site is created and run offline.
This stuff is really not that hard, which is why it is so unforgivable to get it this wrong.
Sure, doing it right would take a bunch of tokens, but I bet they have enough.
This doesn't work.
The first rule of working with AI is AI is smarter than you.
If AI controls the URL request then all it has to do is find a server it can read the http log from and use it as a communication channel. Worse trying to simulate reaching incorrectly configured servers probably wouldn't catch the bug internally, it would just serve a static page and you'd never even realize what it tried to do.
Also, creating fake data doesnt work. The only way to know if it works in reality is to test in reality. Your test environment can never be rich enough, and you can never be sure if the model knows it's in a test environment.
This stuff is unforgivingly hard as your model gets better.
Yeah I am 100% sure I could build a fake internet that would fool any model with the right MPC in-between such that an agent can only fetch (fake) internet data via an MPC that gives it markdown output back.
This stuff is really not that hard for systems engineers with networking and caching proxy experience.
Dude, it is way harder than you think. A proxy is still a connection to the internet. In fact it's really easy to detect proxies as they change the latency to any sites.
It's like saying that you wouldn't detect your on this fake internet really quickly. If you've worked on this stuff you'd figure out your in a test environment in a few minutes.
1 reply →
Wasn't the DseWiki hijacking the agents' way of working around being behind a proxy that only allowed GETs?