← Back to context

Comment by lostphilosopher

9 days ago

I’ve been looking for tooling that would evaluate my prompt and give feedback on how to improve. I can get somewhere with custom system prompts (“before responding ensure…”) but it seems like someone is probably already working on this? Ideally it would run outside the actual thread to keep context clean. There are some options popping up on Google but curious if anyone has a first anecdote to share?

2 comments

lostphilosopher

Reply
docjay  7 days ago

It really depends on how deep you want to go.

1. Just jazz up and expand on a simple prompt.

2. A full context deficiency analysis and multiple question interview system to bounds check and restructure your prompt into your ‘goal’.

3. Realizing that what looks like a good human prompt is not the same as what functions as a good ‘next token’ prompt.

If you just want #1:

import dspy

class EnhancePrompt(dspy.Signature):

    """Assemble the final enhanced prompt from all gathered context"""

    essential_context: str = dspy.InputField(desc="All essential context and requirements")

    original_request: str = dspy.InputField(desc="The user's original request")

    enhanced: str = dspy.OutputField(desc="Complete, detailed, unambiguous prompt. Omit politeness markers. You must limit all numbered lists to a maximum of 3 items.")

def enhance_prompt(prompt: str, temperature: float = 0.2) -> str:

    with dspy.context(lm=dspy.LM("_MODEL_", temperature=temperature)): return dspy.ChainOfThought(EnhancePrompt)(essential_context=f"Direct enhancement request: {prompt}", original_request=prompt).enhanced

res = enhance_prompt("Read bigfile.py and explain the do_math() function.")

print(res)

Read the file `bigfile.py` and provide a detailed explanation of the `do_math()` function. Your explanation should cover:

1. The function's purpose and what it accomplishes

2. The input parameters it accepts and the output/return value it produces

3. The step-by-step logic and algorithm used within the function

Include relevant code snippets when explaining key parts of the implementation.