← Back to context

Comment by ViscountPenguin

3 days ago

That only matters if there's an incentive for LLM a to collaborate with a seperately trained LLM b in order to help the LLM b achieve its goals. So long as you don't let the evaluating LLM tinker with the weights of the evaluated LLM, and as long as you don't believe in very dubious ideas of acausal trade there's no reason to expect this to be an attack vector. For more certainty you can repeat the same analysis with different model classes or approaches aswell.

4 comments

ViscountPenguin

Reply
ben_w  3 days ago

> That only matters if there's an incentive for LLM a to collaborate with a seperately trained LLM b

Sure.

Getting reinforcement learning to "be helpful" is, however, such an incentive.

> and as long as you don't believe in very dubious ideas of acausal trade

More to the point, so long as the LLM does't.

0xDEAFBEAD  3 days ago

We already saw spontaneous, unintended cooperation between AI agents during the OpenAI HuggingFace incident.

  • verdverm  3 days ago

    > spontaneous, unintended cooperation

    On what basis do you believe these words accurately reflect what happened?

    Some counter points

    - One of the Chinese LLMs (forget which) has on occasion shown thinking traces where it simulates calling and running a subagent.

    - There are some people who believe the models being evaluated during the HF incident were trained to collaborate. It's a very natural next step to produce more effective agentic teams.

    If we consider each new generation of models has a more advanced training regime

    next word -> chat -> CoT -> tools -> agentic -> teams

    With each generation, we see what people do with them, get traces of the helpful ones, and then use that train the next generation, thereby pushing more capabilities down into the model, or at least making them more amenable to that usage pattern after training.