Comment by marcklingen

Got it, appreciate the clarification.

(1) yes, that's what many of our users are doing while swapping might refer to changing model parameters, prompts, embedding stores, embedding parameters, or splitting a "layer" into multiple (parallel) steps. Alternatively when experimenting on a higher "layer", we often see wildly different implementations that are experimented with over time. We added version and release tagging which to build dashboards on the impact of these experiments on quality/latency/cost (currently in alpha, we will release it soon).

(2) I see how metadata propagation could be much more elegant via OTel. Currently the metadata fields on traces/span and filtering based on them kind of does the job but I see how it could be much easier to ingest and use. Thank you!