Comment by nodja
7 months ago
Back in the GPT3 days people said that prompt engineering was going to be dead due to prompt tuning. And here we are 2 major versions later and I've yet to see it in production. I thought it would be useful not only to prevent leaks like these, but they would also produce more reliable results no?
If you don't know what prompt tuning is, it's when you freeze the whole model except a certain amount of embeddings at the beginning of the prompt and train only those embeddings. It works like fine tuning but you can swap them in and out as they work just like normal text tokens, they just have vectors that don't map directly to discrete tokens. If you know what textual inversion is in image models it's the same concept.
I think prompt tuning might be worth doing for specific tasks in agentic workflows. For general prompts using words instead of fine tuned input vectors might be good enough. It also easier to update.
The fact that the model leaks some wordy prompt doesn't mean it's actual prompt aren't finetuned emeddings. It wouldn't have a way to leak those using just output tokens and since you start finetuning from a text prompt it would most likely return this text or something close.