Best way for generating functions directly from the LLM?

[kmheckel]

Hi, I was wondering if there is a best practice for generating python functions/Callables directly from LLMs as part of structured outputs? I tried adding Callables as part of a pydantic model but didn't have much luck so I've tried using a wrapper class which I registered using the guide (https://magentic.dev/examples/registering_custom_type/).

Is there a better way to go about this? I'd like to generate python tools dynamically to use as part of the workflow. I know there's issues with imports and scope and other things but wanted to ask.

[jackmpcollins]

@kmheckel One way to do something like this is using @e2b-dev and provide the model a tool to use to execute python code in an E2B sandbox. See the magentic cookbook here https://github.com/e2b-dev/e2b-cookbook/blob/1bd110b03748751bfeac1e8509cf91c037f59814/examples/magentic-code-interpreter/magentic_code_interpreter.ipynb

(not recommended) A DIY version of doing this more natively would be to create a prompt-function returning a str of the code, and exec it. You would need to include in your prompt all the symbols (variables, functions, etc.) it can use in its code, and probably also things like types, signatures, or generally how to use them.

---

I've been thinking about how higher-order functions might work in magentic. I think you would ideally be able to write prompt-functions with return types

• -> PromptFunction[Foo] (string prompt + list of functions/tools + ChatModel instance + parameters/signature + return type)
• -> Callable[[Foo, Bar], Baz] (executable code that can use available variables, functions etc.)

For prompt-functions returning PromptFunction, a nice approach would be to allow it to give the prompt-function being created access to any of the same functions that are provided as its functions argument. The other components of a PromptFunction should be more straightforward to (de)serialize.

Pydantic has an ImportString type which could additionally be used to allow the LLM specify a function. Though it still needs to be told somehow which ones are available. https://docs.pydantic.dev/2.8/api/types/#pydantic.types.ImportString

For prompt-functions returning Callable, this is much trickier and would require much more information/context as outlined above.

---

I'm very interested to hear how this goes for you!

[kmheckel]

Thanks for the timely reply @jackmpcollins !

The possibility of higher-order functionality is exciting and something I'd be really interested in, such as the ability to create a promptchain with relevant functions and execute it. It would be powerful too if the functions could be retrieved as well as added to/updated over time, enabling approaches such as Voyager where the system compositionally builds tools to work with an environment.

Returning a prompt function seems straightforward except for how to incorporate the added functions - everything else seems easy to read/write in the FunctionSchema.

My current solution involves using exec(), but context/function scope for it is tricky as you said since you want the function to be executed/defined in the scope of the callsite so that other local functions are visible. As for serialization, while I can use Python's inspect functionality to obtain the source of the function, this isn't always reliable either or necessary for all occaisions. For example when doing function calling, all that's needed is the function signature and docstring, but for functions that modify the code i.e. given this function, modify/fix it you'd want the source as well; I think this could just be handled by explicitly passing the source code in and returning a new Callable? ImportString looks like it could do the job, I'd just need to work out how to write generated code to a local editable module and then import it. Right now I'm using ChromaDB and storing the function definitions in the metadata and doing semantic search over the document strings but still lack a clean way to rebuild the functions from text.

I've looked at Trace (https://github.com/microsoft/trace) to see how they handle working with code since it has a similar idea of generating and executing code, there might be some pointers there on how to do things.

[jackmpcollins]

I would avoid trying to store the generated code as importable .py files and instead treat it as data. To solve getting the source code for a generated function you could just store this as an attribute on the function when you generate it. So you would pass around a more specialized version of function.

# pseudo
class Function(Callable):
	source_code: str

For functions defined in code you could just use inspect.getsource, either to add the source_code attribute when defining the function or to get the source when needed.

for functions that modify the code i.e. given this function, modify/fix it you'd want the source as well; I think this could just be handled by explicitly passing the source code in and returning a new Callable?

Yes. Then the prompt-function would be like below. You could even make Function show the source code when formatted (docs: https://magentic.dev/formatting/).

@prompt("Improve the performance of this Python function: {func.source_code}")
def improve_perf(func: Function) -> Function: ...

Can you share some code using your exec approach? A quick way to prototype the above would be to use functions/tools:

def create_function(code: str):
	eval(...)


@prompt(
    "Improve the performance of this Python function: {source_code}",
    functions=[create_function],
)
def improve_perf(source_code: str) -> FunctionCall: ...


func_call = improve_perf()
generated_function = func_call()

[kmheckel]

Sure thing, here it is. This mostly works but does run into the issue of str2func building the tool/function in an isolated scope, meaning that foo may try to call bar but when executed will hit a runtime error. I'll have to test what you've suggested in providing str2func to the prompt-function, this seems like it'll work to an extent; it probably makes more sense to have a multi-step workflow for generating prompt-functions etc. rather than trying to do everything in oneshot directly from the LLM.

import inspect
from magentic.chat_model.function_schema import FunctionSchema, register_function_schema
from typing import Any, Iterable, Union
from collections.abc import Callable
from functools import wraps
import json

def _str2func(func_str):

    # Step 1: get the function name from the string.
    # Create a local dictionary to store the function
    local_dict = {}
    # Execute the string in the local dictionary's scope
    exec(func_str, {}, local_dict)
    # Retrieve the function object from the local dictionary
    func_name = list(local_dict.keys())[0]

    func = local_dict[func_name]
    func.__src__ = func_str
    return func

# this can be improved. subclass Callable?
class Tool(Callable):
    def __init__(self, fun):
        scope = globals()
        self.fun = self._get_function(fun)
        wraps(self.fun)(self)
        if hasattr(self.fun, "__src__"):
            self.__src__ = self.fun.__src__
        else:
            self.__src__ = inspect.getsource(self.fun)

    def _get_function(self, fun: Union[str, Callable]) -> Callable:
        if isinstance(fun, str):
            return _str2func(fun)
        elif callable(fun):
            return fun
        else:
            raise TypeError(f"Expected a callable or a string, got {type(fun).__name__}")

    def __str__(self) -> str:
        return self.__src__

    def __format__(self, format_spec: str) -> str:
        return self.__src__

    def __repr__(self) -> str:
        return f"<Tool(name={self.__name__})>"

    def __call__(self, *args, **kwargs):
        return self.fun(*args, **kwargs)


@register_function_schema(Tool)
class ToolFunctionSchema(FunctionSchema[Tool]):

    @property
    def name(self) -> str:
        """The name of the function.

        Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
        """
        return "tool"

    @property
    def description(self) -> str | None:
        return "A single callable Python function."

    @property
    def parameters(self) -> dict[str, Any]:
        """The parameters the functions accepts as a JSON Schema object."""
        return {
            "properties": {
                "function_definition": {"type": "string"}
            },
            "required": ["function_definition"],
            "type": "object",
        }

    def parse_args(self, chunks: Iterable[str]) -> Tool:
        """Parse an iterable of string chunks into the function arguments."""
        args = json.loads("".join(chunks))
        return Tool(args["function_definition"])

    def serialize_args(self, func: Tool) -> dict:
        """Serialize an object into a JSON string of function arguments."""
        return {
            "function_definition": str(func)
        }