How did you make your own harness, I am curious to know more about the building process and please feel free to share your harness.
If someone were to not use your harness and rather use some stock harness though, what is the one that you would recommend? I am curious about that too.
I really don't mean to discourage anyone but I find that making your own harness is pretty complex process. We where very naive when we started, thinking this is just a loop, but then it turned out it has all kinds of nuances, edge-cases, things to be considered, etc.
As I am writing this I am fixing a bug in the harness that could cause infinite loops under some conditions.
For simple interactions looping over the the llm complete function is not really that difficult. Put some tools, write the loop and exit.
It starts getting more tricky when you need to detect cycling behaviour, when some tools might need to abort but do so gracefully, when you need to wait for something to complete while allowing other parts to continue, when you want to protect against too much usage after you hit certain thresholds, when you need to retry whatever, when you need to compact or truncate to maintain good context window and do so with the model in mind (kimi), when messages need to be structured in certain ways to handle various model capabilities, etc.
Pi.dev, Hermes and all the other I have seen do not do even half of the things I have enumerated above - not at least out of the box.
We have our own... so I was making the comment form our own experience working with the model.
It is a lot trickier to use kimi compared to sonnet - hence why it seems that sonnet is more powerful while I think it is down to the harness.
How did you make your own harness, I am curious to know more about the building process and please feel free to share your harness.
If someone were to not use your harness and rather use some stock harness though, what is the one that you would recommend? I am curious about that too.
I really don't mean to discourage anyone but I find that making your own harness is pretty complex process. We where very naive when we started, thinking this is just a loop, but then it turned out it has all kinds of nuances, edge-cases, things to be considered, etc.
As I am writing this I am fixing a bug in the harness that could cause infinite loops under some conditions.
For simple interactions looping over the the llm complete function is not really that difficult. Put some tools, write the loop and exit.
It starts getting more tricky when you need to detect cycling behaviour, when some tools might need to abort but do so gracefully, when you need to wait for something to complete while allowing other parts to continue, when you want to protect against too much usage after you hit certain thresholds, when you need to retry whatever, when you need to compact or truncate to maintain good context window and do so with the model in mind (kimi), when messages need to be structured in certain ways to handle various model capabilities, etc.
Pi.dev, Hermes and all the other I have seen do not do even half of the things I have enumerated above - not at least out of the box.
4 replies →