Created a TypedPredictorSignature class that builds a signature from Pydantic models - this signature is optimized for use with TypedPredictor and TypedChainOfThought

[drawal1]

📝 Changes Description

This MR/PR contains the following changes:

Created a TypedPredictorSignature class with a single function called create that takes the pydantic classes that define input and output fields and builds a signature optimized to use with the dspy.TypedPredictor to extract structured information from the input.

The advantages of this implementation are:

1. It significantly reduces the chances of the dreaded "retries..." exception
2. If default value is specified, predictor will now return default value if the information cannot be extracted
3. For fields with constraints, an invalid value can be specified along with a validator function (mode="wrap"), and the predictor will return the invalid value if the extracted information does not satisfy the constraints
4. For fields defined as Optional, the predictor will return null if the information cannot be extracted
5. Field description and example values will be used to construct the signature for better prediction, if they are specified in the pydantic model
6. Prefix text can be specified optionally, and it will be used to construct the prompt
7. Using enum fields in pydantic models will no longer generate parse errors

Here is example usage:

    class CommandExtractionInput(BaseModel):
        command: str

    class OutputParamsSchema(BaseModel):
        @field_validator('name', mode='wrap')
        @staticmethod
        def validate_name(name, handler):
            try:
                return handler(name)
            except ValidationError:
                return 'INVALID_NAME'

        @field_validator('age1', 'age2', 'age3', 'age4', 'age5', 'age6', mode='wrap')
        @staticmethod
        def validate_age(age, handler):
            try:
                return handler(age)
            except ValidationError:
                return -8888

        @field_validator('email1', 'email2', mode='wrap')
        @staticmethod
        def validate_email(email, handler):
            try:
                return handler(email)
            except ValidationError:
                return 'INVALID_EMAIL'

        name: Annotated[str,
                        Field(default='NOT_FOUND', max_length=15,
                            title='Name', description='The name of the person',
                            examples=['John Doe', 'Jane Doe'], json_schema_extra={'invalid_value': 'INVALID_NAME'})
                    ]
        age1: Annotated[int, 
                       Field(gt=0, lt=150, default=-999, json_schema_extra={'invalid_value': '-8888'})]
        age2: Annotated[int, 
                       Field(gt=0, lt=150, json_schema_extra={'invalid_value': '-8888'})] = -999
        age3: Optional[Annotated[int, 
                       Field(gt=0, lt=150, json_schema_extra={'invalid_value': '-8888'})]]

        age4: Annotated[int, 
                       Field(gt=0, lt=150, default=-999, json_schema_extra={'invalid_value': '-8888'})]
        age5: Annotated[int, 
                       Field(gt=0, lt=150, json_schema_extra={'invalid_value': '-8888'})] = -999
        age6: Optional[Annotated[int, 
                       Field(gt=0, lt=150, json_schema_extra={'invalid_value': '-8888'})]]

        email1: Annotated[str, 
                         Field(default='NOT_FOUND', 
                            pattern=r'^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$',
                            json_schema_extra={'invalid_value': 'INVALID_EMAIL'})
                    ]
        email2: Annotated[str, 
                         Field(default='NOT_FOUND', 
                            pattern=r'^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$'),
                            json_schema_extra={'invalid_value': 'INVALID_EMAIL'}
                    ]

    dspy_signature_class = TypedPredictorSignature.create(
        CommandExtractionInput, OutputParamsSchema)

    lm = dspy.LM('openai/gpt-3.5-turbo')
    with dspy.context(lm=lm):
        extract_cmd_params = dspy.TypedChainOfThought(
            dspy_signature_class)

        input_for_parameter_extraction = CommandExtractionInput(
            # command = "A random command."
            # command = "My name is kjhd and I am 200 years old. My email is 9236"
            command = "Hello, my name is John Doe and I am 25 years old. My email is [email protected]."
        )
        prediction = extract_cmd_params(**input_for_parameter_extraction.model_dump())

        dspy.inspect_history(n=1)
        print(prediction)

...

✅ Contributor Checklist

• Pre-Commit checks are passing (locally and remotely)
• Title of your PR / MR corresponds to the required format
• [] Commit message follows required format {label}(dspy): {message}

⚠️ Warnings

Anything we should be aware of ?

[okhat]

Amazing. Having some discussions on Discord at https://discord.com/channels/1161519468141355160/1294140517764042794/1297707179054207028

[okhat]

Interesting to see special support for optional fields hmm, do we need optional fields? Or are Optional[.] types enough? I see that you have this for input fields here and for output fields below.

[drawal1]

Pydantic allows multiple ways to specify a field schema, including Optional property. It could be on the field spec inside the annotation or outside. IMO, we should not let users guess what pydantic field metadata we do and don't support. If we can support it, we should

[okhat]

interesting to see examples and metadata per field

[drawal1]

these are definitely valuable signals. We should leverage Pydantic field metadata instead of duplicating in DSPy signature fields

[okhat]

ah interesting concept here, requiring a default value for every required output field, why?

[drawal1]

Users may provide input without passing the value of a required parameter. So how to indicate that the required value was missing in the input, without throwing an exception? And without forcing the LM to hallucinate a required missing value?

Luckily, you can specify that Pydantic default value should not be subject to validation. This means that regardless of the type (str, int, float, ...), you can specify a default value of "NOT_FOUND" and this signature will correctly detect and return it without hallucinating.

No wrap validators necessary, which in any case don't distinguish between invalid and missing

[drawal1]

The unique nature of agentic applications is that we cannot assume user input is being validated in the UI for required input. Combine this with LM ability to hallucinate when forced to do so...

[okhat]

Interesting choice of words. "When extracting" and "return". Something to keep in mind.

[drawal1]

Seems to work. Any pitfalls here? Are you thinking if the field name was "extracted_name" or field.default was "return"? Not quite sure how to get around that

[okhat]

naive quotes will fail on complex values? same with the naive join below

[drawal1]

Only issue I have seen is the LM sometimes returning string values enclosed in quotes. For example, returning "'John Doe'", instead of "John Doe". But this is easy to strip. It correctly extracts numbers without putting them in quotes but I guess the prompt could be enhanced to detect if the example values should be quoted. Any other suggestions on how this could be improved?

Also, re. complex types - isn't that what I am testing with complex Input and Output BaseModels? So may be the code for pulling apart complex types and building a custom signature is unavoidable. Thoughts?

[okhat]

Very interesting, asking the model to signal bad/hard fields

[drawal1]

Detecting missing and invalid values is a "must-have" requirement for production-quality apps. Here, I am basically trying to avoid the multiple retries that the framework makes when the extracted value does not match the constraints specified in the Pydantic BaseModel.

The LM may be incorrectly extracting the parameter or the user has specified an invalid value. In both cases, the system should not hallucinate the closest value that matches the actual input. Rather, it should flag the invalid field and let the application handle the error with a proper error-correction workflow ("Provided value was invalid. It must be... Here are some examples... Did you mean...?")

[okhat]

why use Annotated instead of assignment = Field(...)?

[drawal1]

From Pydantic docs - "In case you use field constraints with compound types, an error can happen in some cases. To avoid potential issues, you can use Annotated:"

See https://docs.pydantic.dev/latest/concepts/fields/#validation-alias