Comment by spwa4

What is the difference between loop engineering and hyperparameter search?

I mean, sure, LLMs add a new textual dimension to hyperparameter search. You used to write a program to write a program from some parameters (how many layers, hidden dimension, which inputs, how often to drop, what momentum, ... note: in practice limit the choices), and now you just tell an LLM to try them. Then run and collect results. People did this in 1990.

Hell, I've done "compile options search" with this principle. Just a big python array with options, run through them, benchmark. Then decide it's taking to long and find some way to "intelligently" run through them.

Boom, "loop engineering". I hate that everything keeps getting renamed. Give me back my HAAR cascades!