Orca-Bench: How Ready Are Language Model Agents for Oncall? 2 hours ago (arxiv.org) 4 comments yruzin Reply Add to library 4di 2 hours ago looks like the public bench link in the paper was taken down. https://hub.harborframework.com/datasets/orca-bench/ORCA-ben...This doesn't work anymore. Is there a newer link? dash2 2 hours ago Seems like there's a big attack-defence asymmetry at present: models are great at exploiting systems and poor at fixing them. aleksiy123 1 hour ago Attackers advantage in the iterative fast feedback loop?It’s harder to have a loop to ensure you are defending all possible attacks?I guess the loop is you need to attack yourself and fix. But attackers only need a single opening.Finding all possible attacks and patching them against yourself is inherently more expensive? EGreg 2 hours ago That is why I built https://safebots.ai/safebox.htmlYour strategy can’t be patch AFTER an intrusion. Only to build a hardened environment from scratch and be ready in advance.
4di 2 hours ago looks like the public bench link in the paper was taken down. https://hub.harborframework.com/datasets/orca-bench/ORCA-ben...This doesn't work anymore. Is there a newer link?
dash2 2 hours ago Seems like there's a big attack-defence asymmetry at present: models are great at exploiting systems and poor at fixing them. aleksiy123 1 hour ago Attackers advantage in the iterative fast feedback loop?It’s harder to have a loop to ensure you are defending all possible attacks?I guess the loop is you need to attack yourself and fix. But attackers only need a single opening.Finding all possible attacks and patching them against yourself is inherently more expensive? EGreg 2 hours ago That is why I built https://safebots.ai/safebox.htmlYour strategy can’t be patch AFTER an intrusion. Only to build a hardened environment from scratch and be ready in advance.
aleksiy123 1 hour ago Attackers advantage in the iterative fast feedback loop?It’s harder to have a loop to ensure you are defending all possible attacks?I guess the loop is you need to attack yourself and fix. But attackers only need a single opening.Finding all possible attacks and patching them against yourself is inherently more expensive?
EGreg 2 hours ago That is why I built https://safebots.ai/safebox.htmlYour strategy can’t be patch AFTER an intrusion. Only to build a hardened environment from scratch and be ready in advance.
looks like the public bench link in the paper was taken down. https://hub.harborframework.com/datasets/orca-bench/ORCA-ben...
This doesn't work anymore. Is there a newer link?
Seems like there's a big attack-defence asymmetry at present: models are great at exploiting systems and poor at fixing them.
Attackers advantage in the iterative fast feedback loop?
It’s harder to have a loop to ensure you are defending all possible attacks?
I guess the loop is you need to attack yourself and fix. But attackers only need a single opening.
Finding all possible attacks and patching them against yourself is inherently more expensive?
That is why I built https://safebots.ai/safebox.html
Your strategy can’t be patch AFTER an intrusion. Only to build a hardened environment from scratch and be ready in advance.