Cosette’s source

Setup

Exported source 
models = 'o1-preview', 'o1-mini', 'gpt-4o', 'gpt-4o-mini', 'gpt-4-turbo', 'gpt-4', 'gpt-4-32k', 'gpt-3.5-turbo', 'gpt-3.5-turbo-instruct'
Exported source 
text_only_models = 'o1-preview', 'o1-mini'
model = models[2]

For examples, we’ll use GPT-4o.

OpenAI SDK

cli = OpenAI().chat.completions
m = {'role': 'user', 'content': "I'm Jeremy"}
r = cli.create(messages=[m], model=model, max_completion_tokens=100)
r
ChatCompletion(id='chatcmpl-ALXzEpOZShv71v2TVAaoMv2dhKVTl', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Hello, Jeremy! How can I assist you today?', refusal=None, role='assistant', function_call=None, tool_calls=None))], created=1729698896, model='gpt-4o-2024-08-06', object='chat.completion', service_tier=None, system_fingerprint='fp_a7d06e42a7', usage=CompletionUsage(completion_tokens=11, prompt_tokens=9, total_tokens=20, completion_tokens_details=CompletionTokensDetails(audio_tokens=None, reasoning_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0)))

Formatting output

source

find_block

 find_block (r:collections.abc.Mapping)

Find the message in r.

Type Details
r Mapping The message to look in
Exported source 
def find_block(r:abc.Mapping, # The message to look in
              ):
    "Find the message in `r`."
    m = nested_idx(r, 'choices', 0)
    if not m: return m
    if hasattr(m, 'message'): return m.message
    return m.delta

source

contents

 contents (r)

Helper to get the contents from response r.

Exported source 
def contents(r):
    "Helper to get the contents from response `r`."
    blk = find_block(r)
    if not blk: return r
    if hasattr(blk, 'content'): return getattr(blk,'content')
    return blk
contents(r)
'Hello, Jeremy! How can I assist you today?'
Exported source 
@patch
def _repr_markdown_(self:ChatCompletion):
    det = '\n- '.join(f'{k}: {v}' for k,v in dict(self).items())
    res = contents(self)
    if not res: return f"- {det}"
    return f"""{contents(self)}

<details>

- {det}

</details>"""
r

Hello, Jeremy! How can I assist you today?

  • id: chatcmpl-ALXzEpOZShv71v2TVAaoMv2dhKVTl
  • choices: [Choice(finish_reason=‘stop’, index=0, logprobs=None, message=ChatCompletionMessage(content=‘Hello, Jeremy! How can I assist you today?’, refusal=None, role=‘assistant’, function_call=None, tool_calls=None))]
  • created: 1729698896
  • model: gpt-4o-2024-08-06
  • object: chat.completion
  • service_tier: None
  • system_fingerprint: fp_a7d06e42a7
  • usage: CompletionUsage(completion_tokens=11, prompt_tokens=9, total_tokens=20, completion_tokens_details=CompletionTokensDetails(audio_tokens=None, reasoning_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0))
r.usage
CompletionUsage(completion_tokens=11, prompt_tokens=9, total_tokens=20, completion_tokens_details=CompletionTokensDetails(audio_tokens=None, reasoning_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0))

source

usage

 usage (inp=0, out=0)

Slightly more concise version of CompletionUsage.

Type Default Details
inp int 0 Number of prompt tokens
out int 0 Number of completion tokens
Exported source 
def usage(inp=0, # Number of prompt tokens
          out=0  # Number of completion tokens
         ):
    "Slightly more concise version of `CompletionUsage`."
    return CompletionUsage(prompt_tokens=inp, completion_tokens=out, total_tokens=inp+out)
usage(5)
CompletionUsage(completion_tokens=0, prompt_tokens=5, total_tokens=5, completion_tokens_details=None, prompt_tokens_details=None)

source

CompletionUsage.__repr__

 CompletionUsage.__repr__ ()

Return repr(self).

Exported source 
@patch
def __repr__(self:CompletionUsage): return f'In: {self.prompt_tokens}; Out: {self.completion_tokens}; Total: {self.total_tokens}'
r.usage
In: 9; Out: 11; Total: 20

source

CompletionUsage.__add__

 CompletionUsage.__add__ (b)

Add together each of input_tokens and output_tokens

Exported source 
@patch
def __add__(self:CompletionUsage, b):
    "Add together each of `input_tokens` and `output_tokens`"
    return usage(self.prompt_tokens+b.prompt_tokens, self.completion_tokens+b.completion_tokens)
r.usage+r.usage
In: 18; Out: 22; Total: 40

source

wrap_latex

 wrap_latex (text, md=True)

Replace OpenAI LaTeX codes with markdown-compatible ones

Creating messages

Creating correctly formatted dicts from scratch every time isn’t very handy, so we’ll import a couple of helper functions from the msglm library.

Let’s use mk_msg to recreate our msg {'role': 'user', 'content': "I'm Jeremy"} from earlier.

prompt = "I'm Jeremy"
m = mk_msg(prompt)
r = cli.create(messages=[m], model=model, max_completion_tokens=100)
r

Hi Jeremy! How can I assist you today?

  • id: chatcmpl-ALXzFkDqG2syVjHdYTGU4bfriaDeX
  • choices: [Choice(finish_reason=‘stop’, index=0, logprobs=None, message=ChatCompletionMessage(content=‘Hi Jeremy! How can I assist you today?’, refusal=None, role=‘assistant’, function_call=None, tool_calls=None))]
  • created: 1729698897
  • model: gpt-4o-2024-08-06
  • object: chat.completion
  • service_tier: None
  • system_fingerprint: fp_a7d06e42a7
  • usage: CompletionUsage(completion_tokens=10, prompt_tokens=9, total_tokens=19, completion_tokens_details=CompletionTokensDetails(audio_tokens=None, reasoning_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0))

We can pass more than just text messages to OpenAI. As we’ll see later we can also pass images, SDK objects, etc. To handle these different data types we need to pass the type along with our content to OpenAI.

Here’s an example of a multimodal message containing text and images.

{
    'role': 'user', 
    'content': [
        {'type': 'text', 'text': 'What is in the image?'},
        {'type': 'image_url', 'image_url': {'url': f'data:{MEDIA_TYPE};base64,{IMG}'}}
    ]
}

mk_msg infers the type automatically and creates the appropriate data structure.

LLMs, don’t actually have state, but instead dialogs are created by passing back all previous prompts and responses every time. With OpenAI, they always alternate user and assistant. We’ll use mk_msgs from msglm to make it easier to build up these dialog lists.

msgs = mk_msgs([prompt, r, "I forgot my name. Can you remind me please?"]) 
msgs
[{'role': 'user', 'content': "I'm Jeremy"},
 ChatCompletionMessage(content='Hi Jeremy! How can I assist you today?', refusal=None, role='assistant', function_call=None, tool_calls=None),
 {'role': 'user', 'content': 'I forgot my name. Can you remind me please?'}]
cli.create(messages=msgs, model=model, max_completion_tokens=200)

It looks like you mentioned your name is Jeremy. How can I help you further?

  • id: chatcmpl-ALXzGveStdH76ojrSQjXb9fei0tp8
  • choices: [Choice(finish_reason=‘stop’, index=0, logprobs=None, message=ChatCompletionMessage(content=‘It looks like you mentioned your name is Jeremy. How can I help you further?’, refusal=None, role=‘assistant’, function_call=None, tool_calls=None))]
  • created: 1729698898
  • model: gpt-4o-2024-08-06
  • object: chat.completion
  • service_tier: None
  • system_fingerprint: fp_a7d06e42a7
  • usage: CompletionUsage(completion_tokens=17, prompt_tokens=38, total_tokens=55, completion_tokens_details=CompletionTokensDetails(audio_tokens=None, reasoning_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0))

Client

source

Client

 Client (model, cli=None)

Basic LLM messages client.

Exported source 
class Client:
    def __init__(self, model, cli=None):
        "Basic LLM messages client."
        self.model,self.use = model,usage(0,0)
        self.text_only = model in text_only_models
        self.c = (cli or OpenAI()).chat.completions
c = Client(model)
c.use
In: 0; Out: 0; Total: 0
Exported source 
@patch
def _r(self:Client, r:ChatCompletion):
    "Store the result of the message and accrue total usage."
    self.result = r
    if getattr(r,'usage',None): self.use += r.usage
    return r
c._r(r)
c.use
In: 9; Out: 10; Total: 19

source

get_stream

 get_stream (r)

source

Client.__call__

 Client.__call__ (msgs:list, sp:str='', maxtok=4096, stream:bool=False,
                  audio:Optional[ChatCompletionAudioParam]|NotGiven=NOT_GI
                  VEN,
                  frequency_penalty:Optional[float]|NotGiven=NOT_GIVEN, fu
                  nction_call:completion_create_params.FunctionCall|NotGiv
                  en=NOT_GIVEN, functions:Iterable[completion_create_param
                  s.Function]|NotGiven=NOT_GIVEN,
                  logit_bias:Optional[Dict[str,int]]|NotGiven=NOT_GIVEN,
                  logprobs:Optional[bool]|NotGiven=NOT_GIVEN,
                  max_completion_tokens:Optional[int]|NotGiven=NOT_GIVEN,
                  max_tokens:Optional[int]|NotGiven=NOT_GIVEN,
                  metadata:Optional[Dict[str,str]]|NotGiven=NOT_GIVEN, mod
                  alities:Optional[List[ChatCompletionModality]]|NotGiven=
                  NOT_GIVEN, n:Optional[int]|NotGiven=NOT_GIVEN,
                  parallel_tool_calls:bool|NotGiven=NOT_GIVEN,
                  presence_penalty:Optional[float]|NotGiven=NOT_GIVEN, res
                  ponse_format:completion_create_params.ResponseFormat|Not
                  Given=NOT_GIVEN, seed:Optional[int]|NotGiven=NOT_GIVEN, 
                  service_tier:"Optional[Literal['auto','default']]|NotGiv
                  en"=NOT_GIVEN,
                  stop:Union[Optional[str],List[str]]|NotGiven=NOT_GIVEN,
                  store:Optional[bool]|NotGiven=NOT_GIVEN, stream_options:
                  Optional[ChatCompletionStreamOptionsParam]|NotGiven=NOT_
                  GIVEN, temperature:Optional[float]|NotGiven=NOT_GIVEN, t
                  ool_choice:ChatCompletionToolChoiceOptionParam|NotGiven=
                  NOT_GIVEN, tools:Iterable[ChatCompletionToolParam]|NotGi
                  ven=NOT_GIVEN,
                  top_logprobs:Optional[int]|NotGiven=NOT_GIVEN,
                  top_p:Optional[float]|NotGiven=NOT_GIVEN,
                  user:str|NotGiven=NOT_GIVEN,
                  extra_headers:Headers|None=None,
                  extra_query:Query|None=None, extra_body:Body|None=None,
                  timeout:float|httpx.Timeout|None|NotGiven=NOT_GIVEN)

Make a call to LLM.

Type Default Details
msgs list List of messages in the dialog
sp str System prompt
maxtok int 4096 Maximum tokens
stream bool False Stream response?
audio Optional[ChatCompletionAudioParam] | NotGiven NOT_GIVEN
frequency_penalty Optional[float] | NotGiven NOT_GIVEN
function_call completion_create_params.FunctionCall | NotGiven NOT_GIVEN
functions Iterable[completion_create_params.Function] | NotGiven NOT_GIVEN
logit_bias Optional[Dict[str, int]] | NotGiven NOT_GIVEN
logprobs Optional[bool] | NotGiven NOT_GIVEN
max_completion_tokens Optional[int] | NotGiven NOT_GIVEN
max_tokens Optional[int] | NotGiven NOT_GIVEN
metadata Optional[Dict[str, str]] | NotGiven NOT_GIVEN
modalities Optional[List[ChatCompletionModality]] | NotGiven NOT_GIVEN
n Optional[int] | NotGiven NOT_GIVEN
parallel_tool_calls bool | NotGiven NOT_GIVEN
presence_penalty Optional[float] | NotGiven NOT_GIVEN
response_format completion_create_params.ResponseFormat | NotGiven NOT_GIVEN
seed Optional[int] | NotGiven NOT_GIVEN
service_tier Optional[Literal[‘auto’, ‘default’]] | NotGiven NOT_GIVEN
stop Union[Optional[str], List[str]] | NotGiven NOT_GIVEN
store Optional[bool] | NotGiven NOT_GIVEN
stream_options Optional[ChatCompletionStreamOptionsParam] | NotGiven NOT_GIVEN
temperature Optional[float] | NotGiven NOT_GIVEN
tool_choice ChatCompletionToolChoiceOptionParam | NotGiven NOT_GIVEN
tools Iterable[ChatCompletionToolParam] | NotGiven NOT_GIVEN
top_logprobs Optional[int] | NotGiven NOT_GIVEN
top_p Optional[float] | NotGiven NOT_GIVEN
user str | NotGiven NOT_GIVEN
extra_headers Headers | None None
extra_query Query | None None
extra_body Body | None None
timeout float | httpx.Timeout | None | NotGiven NOT_GIVEN
Exported source 
@patch
@delegates(Completions.create)
def __call__(self:Client,
             msgs:list, # List of messages in the dialog
             sp:str='', # System prompt
             maxtok=4096, # Maximum tokens
             stream:bool=False, # Stream response?
             **kwargs):
    "Make a call to LLM."
    assert not (self.text_only and bool(sp)), "System prompts are not supported by the current model type."
    assert not (self.text_only and stream), "Streaming is not supported by the current model type."
    if 'tools' in kwargs: assert not self.text_only, "Tool use is not supported by the current model type."
    if any(c['type'] == 'image_url' for msg in msgs if isinstance(msg, dict) and isinstance(msg.get('content'), list) for c in msg['content']): assert not self.text_only, "Images are not supported by the current model type."
    if stream: kwargs['stream_options'] = {"include_usage": True}
    if sp: msgs = [mk_msg(sp, 'system')] + list(msgs)
    r = self.c.create(
        model=self.model, messages=msgs, max_completion_tokens=maxtok, stream=stream, **kwargs)
    if not stream: return self._r(r)
    else: return get_stream(map(self._r, r))
msgs = [mk_msg('Hi')]
c(msgs)

Hello! How can I assist you today?

  • id: chatcmpl-ALXzIRMSnolsXgCkpNZhRkP6He04H
  • choices: [Choice(finish_reason=‘stop’, index=0, logprobs=None, message=ChatCompletionMessage(content=‘Hello! How can I assist you today?’, refusal=None, role=‘assistant’, function_call=None, tool_calls=None))]
  • created: 1729698900
  • model: gpt-4o-2024-08-06
  • object: chat.completion
  • service_tier: None
  • system_fingerprint: fp_a7d06e42a7
  • usage: CompletionUsage(completion_tokens=9, prompt_tokens=8, total_tokens=17, completion_tokens_details=CompletionTokensDetails(audio_tokens=None, reasoning_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0))
c.use
In: 17; Out: 19; Total: 36
for o in c(msgs, stream=True): print(o, end='')
Hello! How can I assist you today?
c.use
In: 25; Out: 28; Total: 53

Tool use

def sums(
    a:int,  # First thing to sum
    b:int # Second thing to sum
) -> int: # The sum of the inputs
    "Adds a + b."
    print(f"Finding the sum of {a} and {b}")
    return a + b

source

mk_openai_func

 mk_openai_func (f)

source

mk_tool_choice

 mk_tool_choice (f)
sysp = "You are a helpful assistant. When using tools, be sure to pass all required parameters, at minimum."
a,b = 604542,6458932
pr = f"What is {a}+{b}?"
tools=[mk_openai_func(sums)]
tool_choice=mk_tool_choice("sums")
msgs = [mk_msg(pr)]
r = c(msgs, sp=sysp, tools=tools)
r
  • id: chatcmpl-ALXzJN8S3M8xOpyLYt9JlP7swcae1
  • choices: [Choice(finish_reason=‘tool_calls’, index=0, logprobs=None, message=ChatCompletionMessage(content=None, refusal=None, role=‘assistant’, function_call=None, tool_calls=[ChatCompletionMessageToolCall(id=‘call_ED1LI54AaSkk0T6aDrX5foal’, function=Function(arguments=‘{“a”:604542,“b”:6458932}’, name=‘sums’), type=‘function’)]))]
  • created: 1729698901
  • model: gpt-4o-2024-08-06
  • object: chat.completion
  • service_tier: None
  • system_fingerprint: fp_a7d06e42a7
  • usage: CompletionUsage(completion_tokens=21, prompt_tokens=94, total_tokens=115, completion_tokens_details=CompletionTokensDetails(audio_tokens=None, reasoning_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0))
m = find_block(r)
m
ChatCompletionMessage(content=None, refusal=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_ED1LI54AaSkk0T6aDrX5foal', function=Function(arguments='{"a":604542,"b":6458932}', name='sums'), type='function')])
tc = m.tool_calls
tc
[ChatCompletionMessageToolCall(id='call_ED1LI54AaSkk0T6aDrX5foal', function=Function(arguments='{"a":604542,"b":6458932}', name='sums'), type='function')]
func = tc[0].function
func
Function(arguments='{"a":604542,"b":6458932}', name='sums')

source

call_func_openai

 call_func_openai
                   (func:openai.types.chat.chat_completion_message_tool_ca
                   ll.Function, ns:Optional[collections.abc.Mapping]=None)
Exported source 
def call_func_openai(func:types.chat.chat_completion_message_tool_call.Function, ns:Optional[abc.Mapping]=None):
    return call_func(func.name, ast.literal_eval(func.arguments), ns)
ns = mk_ns(sums)
res = call_func_openai(func, ns=ns)
res
Finding the sum of 604542 and 6458932
7063474

source

mk_toolres

 mk_toolres (r:collections.abc.Mapping,
             ns:Optional[collections.abc.Mapping]=None, obj:Optional=None)

Create a tool_result message from response r.

Type Default Details
r Mapping Tool use request response
ns Optional None Namespace to search for tools
obj Optional None Class to search for tools
Exported source 
def mk_toolres(
    r:abc.Mapping, # Tool use request response
    ns:Optional[abc.Mapping]=None, # Namespace to search for tools
    obj:Optional=None # Class to search for tools
    ):
    "Create a `tool_result` message from response `r`."
    r = mk_msg(r)
    tcs = getattr(r, 'tool_calls', [])
    res = [r]
    if ns is None: ns = globals()
    if obj is not None: ns = mk_ns(obj)
    for tc in (tcs or []):
        func = tc.function
        cts = str(call_func_openai(func, ns=ns))
        res.append(mk_msg(str(cts), 'tool', tool_call_id=tc.id, name=func.name))
    return res
tr = mk_toolres(r, ns=ns)
tr
Finding the sum of 604542 and 6458932
[ChatCompletionMessage(content=None, refusal=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_ED1LI54AaSkk0T6aDrX5foal', function=Function(arguments='{"a":604542,"b":6458932}', name='sums'), type='function')]),
 {'role': 'tool',
  'content': '7063474',
  'tool_call_id': 'call_ED1LI54AaSkk0T6aDrX5foal',
  'name': 'sums'}]
msgs += tr
res = c(msgs, sp=sysp, tools=tools)
res

The sum of 604,542 and 6,458,932 is 7,063,474.

  • id: chatcmpl-ALXzK678Vwp480g7tu1eT5wIURo8Q
  • choices: [Choice(finish_reason=‘stop’, index=0, logprobs=None, message=ChatCompletionMessage(content=‘The sum of 604,542 and 6,458,932 is 7,063,474.’, refusal=None, role=‘assistant’, function_call=None, tool_calls=None))]
  • created: 1729698902
  • model: gpt-4o-2024-08-06
  • object: chat.completion
  • service_tier: None
  • system_fingerprint: fp_45c6de4934
  • usage: CompletionUsage(completion_tokens=23, prompt_tokens=126, total_tokens=149, completion_tokens_details=CompletionTokensDetails(audio_tokens=None, reasoning_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0))
class Dummy:
    def sums(
        self,
        a:int,  # First thing to sum
        b:int=1 # Second thing to sum
    ) -> int: # The sum of the inputs
        "Adds a + b."
        print(f"Finding the sum of {a} and {b}")
        return a + b
tools = [mk_openai_func(Dummy.sums)]

o = Dummy()
msgs = mk_toolres("I'm Jeremy")
r = c(msgs, sp=sysp, tools=tools)
msgs += mk_toolres(r, obj=o)
res = c(msgs, sp=sysp, tools=tools)
res

Hello Jeremy! How can I assist you today?

  • id: chatcmpl-ALXzMvnxaEgzwABU1ojD5KD9KMvur
  • choices: [Choice(finish_reason=‘stop’, index=0, logprobs=None, message=ChatCompletionMessage(content=‘Hello Jeremy! How can I assist you today?’, refusal=None, role=‘assistant’, function_call=None, tool_calls=None))]
  • created: 1729698904
  • model: gpt-4o-2024-08-06
  • object: chat.completion
  • service_tier: None
  • system_fingerprint: fp_a7d06e42a7
  • usage: CompletionUsage(completion_tokens=11, prompt_tokens=106, total_tokens=117, completion_tokens_details=CompletionTokensDetails(audio_tokens=None, reasoning_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0))
msgs
[{'role': 'user', 'content': "I'm Jeremy"},
 ChatCompletionMessage(content='Hello Jeremy! How can I assist you today?', refusal=None, role='assistant', function_call=None, tool_calls=None)]
tools = [mk_openai_func(Dummy.sums)]

o = Dummy()
msgs = mk_toolres(pr)
r = c(msgs, sp=sysp, tools=tools)
msgs += mk_toolres(r, obj=o)
res = c(msgs, sp=sysp, tools=tools)
res
Finding the sum of 604542 and 6458932

The sum of 604542 and 6458932 is 7,063,474.

  • id: chatcmpl-ALXzNUuTFv6YooYC3BDKLUG8JW1E9
  • choices: [Choice(finish_reason=‘stop’, index=0, logprobs=None, message=ChatCompletionMessage(content=‘The sum of 604542 and 6458932 is 7,063,474.’, refusal=None, role=‘assistant’, function_call=None, tool_calls=None))]
  • created: 1729698905
  • model: gpt-4o-2024-08-06
  • object: chat.completion
  • service_tier: None
  • system_fingerprint: fp_a7d06e42a7
  • usage: CompletionUsage(completion_tokens=20, prompt_tokens=132, total_tokens=152, completion_tokens_details=CompletionTokensDetails(audio_tokens=None, reasoning_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0))

source

mock_tooluse

 mock_tooluse (name:str, res, **kwargs)
Type Details
name str The name of the called function
res The result of calling the function
kwargs
Exported source 
def _mock_id(): return 'call_' + ''.join(choices(ascii_letters+digits, k=24))

def mock_tooluse(name:str, # The name of the called function
                 res,  # The result of calling the function
                 **kwargs): # The arguments to the function
    ""
    id = _mock_id()
    func = dict(arguments=json.dumps(kwargs), name=name)
    tc = dict(id=id, function=func, type='function')
    req = dict(content=None, role='assistant', tool_calls=[tc])
    resp = mk_msg('' if res is None else str(res), 'tool', tool_call_id=id, name=name)
    return [req,resp]

This function mocks the messages needed to implement tool use, for situations where you want to insert tool use messages into a dialog without actually calling into the model.

tu = mock_tooluse(name='sums', res=7063474, a=604542, b=6458932)
r = c([mk_msg(pr)]+tu, tools=tools)
r

The sum of 604542 and 6458932 is 7063474.

  • id: chatcmpl-ALXzOlnmqz2JsWy3I803CW2jSTlQ0
  • choices: [Choice(finish_reason=‘stop’, index=0, logprobs=None, message=ChatCompletionMessage(content=‘The sum of 604542 and 6458932 is 7063474.’, refusal=None, role=‘assistant’, function_call=None, tool_calls=None))]
  • created: 1729698906
  • model: gpt-4o-2024-08-06
  • object: chat.completion
  • service_tier: None
  • system_fingerprint: fp_a7d06e42a7
  • usage: CompletionUsage(completion_tokens=18, prompt_tokens=111, total_tokens=129, completion_tokens_details=CompletionTokensDetails(audio_tokens=None, reasoning_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0))

Structured outputs

source

Client.structured

 Client.structured (msgs:list, tools:Optional[list]=None,
                    obj:Optional=None,
                    ns:Optional[collections.abc.Mapping]=None, sp:str='',
                    maxtok=4096, stream:bool=False, audio:Optional[ChatCom
                    pletionAudioParam]|NotGiven=NOT_GIVEN,
                    frequency_penalty:Optional[float]|NotGiven=NOT_GIVEN, 
                    function_call:completion_create_params.FunctionCall|No
                    tGiven=NOT_GIVEN, functions:Iterable[completion_create
                    _params.Function]|NotGiven=NOT_GIVEN,
                    logit_bias:Optional[Dict[str,int]]|NotGiven=NOT_GIVEN,
                    logprobs:Optional[bool]|NotGiven=NOT_GIVEN, max_comple
                    tion_tokens:Optional[int]|NotGiven=NOT_GIVEN,
                    max_tokens:Optional[int]|NotGiven=NOT_GIVEN,
                    metadata:Optional[Dict[str,str]]|NotGiven=NOT_GIVEN, m
                    odalities:Optional[List[ChatCompletionModality]]|NotGi
                    ven=NOT_GIVEN, n:Optional[int]|NotGiven=NOT_GIVEN,
                    parallel_tool_calls:bool|NotGiven=NOT_GIVEN,
                    presence_penalty:Optional[float]|NotGiven=NOT_GIVEN, r
                    esponse_format:completion_create_params.ResponseFormat
                    |NotGiven=NOT_GIVEN,
                    seed:Optional[int]|NotGiven=NOT_GIVEN, service_tier:"O
                    ptional[Literal['auto','default']]|NotGiven"=NOT_GIVEN
                    , stop:Union[Optional[str],List[str]]|NotGiven=NOT_GIV
                    EN, store:Optional[bool]|NotGiven=NOT_GIVEN, stream_op
                    tions:Optional[ChatCompletionStreamOptionsParam]|NotGi
                    ven=NOT_GIVEN,
                    temperature:Optional[float]|NotGiven=NOT_GIVEN, tool_c
                    hoice:ChatCompletionToolChoiceOptionParam|NotGiven=NOT
                    _GIVEN, top_logprobs:Optional[int]|NotGiven=NOT_GIVEN,
                    top_p:Optional[float]|NotGiven=NOT_GIVEN,
                    user:str|NotGiven=NOT_GIVEN,
                    extra_headers:Headers|None=None,
                    extra_query:Query|None=None,
                    extra_body:Body|None=None,
                    timeout:float|httpx.Timeout|None|NotGiven=NOT_GIVEN)

Return the value of all tool calls (generally used for structured outputs)

Type Default Details
msgs list Prompt
tools Optional None List of tools to make available to OpenAI model
obj Optional None Class to search for tools
ns Optional None Namespace to search for tools
sp str System prompt
maxtok int 4096 Maximum tokens
stream bool False Stream response?
audio Optional[ChatCompletionAudioParam] | NotGiven NOT_GIVEN
frequency_penalty Optional[float] | NotGiven NOT_GIVEN
function_call completion_create_params.FunctionCall | NotGiven NOT_GIVEN
functions Iterable[completion_create_params.Function] | NotGiven NOT_GIVEN
logit_bias Optional[Dict[str, int]] | NotGiven NOT_GIVEN
logprobs Optional[bool] | NotGiven NOT_GIVEN
max_completion_tokens Optional[int] | NotGiven NOT_GIVEN
max_tokens Optional[int] | NotGiven NOT_GIVEN
metadata Optional[Dict[str, str]] | NotGiven NOT_GIVEN
modalities Optional[List[ChatCompletionModality]] | NotGiven NOT_GIVEN
n Optional[int] | NotGiven NOT_GIVEN
parallel_tool_calls bool | NotGiven NOT_GIVEN
presence_penalty Optional[float] | NotGiven NOT_GIVEN
response_format completion_create_params.ResponseFormat | NotGiven NOT_GIVEN
seed Optional[int] | NotGiven NOT_GIVEN
service_tier Optional[Literal[‘auto’, ‘default’]] | NotGiven NOT_GIVEN
stop Union[Optional[str], List[str]] | NotGiven NOT_GIVEN
store Optional[bool] | NotGiven NOT_GIVEN
stream_options Optional[ChatCompletionStreamOptionsParam] | NotGiven NOT_GIVEN
temperature Optional[float] | NotGiven NOT_GIVEN
tool_choice ChatCompletionToolChoiceOptionParam | NotGiven NOT_GIVEN
top_logprobs Optional[int] | NotGiven NOT_GIVEN
top_p Optional[float] | NotGiven NOT_GIVEN
user str | NotGiven NOT_GIVEN
extra_headers Headers | None None
extra_query Query | None None
extra_body Body | None None
timeout float | httpx.Timeout | None | NotGiven NOT_GIVEN
Exported source 
@patch
@delegates(Client.__call__)
def structured(self:Client,
               msgs: list, # Prompt
               tools:Optional[list]=None, # List of tools to make available to OpenAI model
               obj:Optional=None, # Class to search for tools
               ns:Optional[abc.Mapping]=None, # Namespace to search for tools
               **kwargs):
    "Return the value of all tool calls (generally used for structured outputs)"
    tools = listify(tools)
    if ns is None: ns=mk_ns(*tools)
    tools = [mk_openai_func(o) for o in tools]
    if obj is not None: ns = mk_ns(obj)
    res = self(msgs, tools=tools, tool_choice='required', **kwargs)
    cts = getattr(res, 'choices', [])
    tcs = [call_func_openai(t.function, ns=ns) for o in cts for t in (o.message.tool_calls or [])]
    return tcs

OpenAI’s API doesn’t natively support response formats, so we introduce a structured method to handle tool calling for this purpose. In this setup, the tool’s result is sent directly to the user without being passed back to the model.

c.structured(mk_msgs(pr), tools=[sums])
Finding the sum of 604542 and 6458932
[7063474]

Chat

source

Chat

 Chat (model:Optional[str]=None, cli:Optional[__main__.Client]=None,
       sp='', tools:Optional[list]=None, tool_choice:Optional[str]=None)

OpenAI chat client.

Type Default Details
model Optional None Model to use (leave empty if passing cli)
cli Optional None Client to use (leave empty if passing model)
sp str Optional system prompt
tools Optional None List of tools to make available
tool_choice Optional None Forced tool choice
Exported source 
class Chat:
    def __init__(self,
                 model:Optional[str]=None, # Model to use (leave empty if passing `cli`)
                 cli:Optional[Client]=None, # Client to use (leave empty if passing `model`)
                 sp='', # Optional system prompt
                 tools:Optional[list]=None,  # List of tools to make available
                 tool_choice:Optional[str]=None): # Forced tool choice
        "OpenAI chat client."
        assert model or cli
        self.c = (cli or Client(model))
        self.h,self.sp,self.tools,self.tool_choice = [],sp,tools,tool_choice
    
    @property
    def use(self): return self.c.use
sp = "Never mention what tools you use."
chat = Chat(model, sp=sp)
chat.c.use, chat.h
(In: 0; Out: 0; Total: 0, [])

source

Chat.__call__

 Chat.__call__ (pr=None, stream:bool=False,
                audio:Optional[ChatCompletionAudioParam]|NotGiven=NOT_GIVE
                N, frequency_penalty:Optional[float]|NotGiven=NOT_GIVEN, f
                unction_call:completion_create_params.FunctionCall|NotGive
                n=NOT_GIVEN, functions:Iterable[completion_create_params.F
                unction]|NotGiven=NOT_GIVEN,
                logit_bias:Optional[Dict[str,int]]|NotGiven=NOT_GIVEN,
                logprobs:Optional[bool]|NotGiven=NOT_GIVEN,
                max_completion_tokens:Optional[int]|NotGiven=NOT_GIVEN,
                max_tokens:Optional[int]|NotGiven=NOT_GIVEN,
                metadata:Optional[Dict[str,str]]|NotGiven=NOT_GIVEN, modal
                ities:Optional[List[ChatCompletionModality]]|NotGiven=NOT_
                GIVEN, n:Optional[int]|NotGiven=NOT_GIVEN,
                parallel_tool_calls:bool|NotGiven=NOT_GIVEN,
                presence_penalty:Optional[float]|NotGiven=NOT_GIVEN, respo
                nse_format:completion_create_params.ResponseFormat|NotGive
                n=NOT_GIVEN, seed:Optional[int]|NotGiven=NOT_GIVEN, servic
                e_tier:"Optional[Literal['auto','default']]|NotGiven"=NOT_
                GIVEN,
                stop:Union[Optional[str],List[str]]|NotGiven=NOT_GIVEN,
                store:Optional[bool]|NotGiven=NOT_GIVEN, stream_options:Op
                tional[ChatCompletionStreamOptionsParam]|NotGiven=NOT_GIVE
                N, temperature:Optional[float]|NotGiven=NOT_GIVEN, tool_ch
                oice:ChatCompletionToolChoiceOptionParam|NotGiven=NOT_GIVE
                N, tools:Iterable[ChatCompletionToolParam]|NotGiven=NOT_GI
                VEN, top_logprobs:Optional[int]|NotGiven=NOT_GIVEN,
                top_p:Optional[float]|NotGiven=NOT_GIVEN,
                user:str|NotGiven=NOT_GIVEN,
                extra_headers:Headers|None=None,
                extra_query:Query|None=None, extra_body:Body|None=None,
                timeout:float|httpx.Timeout|None|NotGiven=NOT_GIVEN)

Add prompt pr to dialog and get a response

Type Default Details
pr NoneType None Prompt / message
stream bool False Stream response?
audio Optional[ChatCompletionAudioParam] | NotGiven NOT_GIVEN
frequency_penalty Optional[float] | NotGiven NOT_GIVEN
function_call completion_create_params.FunctionCall | NotGiven NOT_GIVEN
functions Iterable[completion_create_params.Function] | NotGiven NOT_GIVEN
logit_bias Optional[Dict[str, int]] | NotGiven NOT_GIVEN
logprobs Optional[bool] | NotGiven NOT_GIVEN
max_completion_tokens Optional[int] | NotGiven NOT_GIVEN
max_tokens Optional[int] | NotGiven NOT_GIVEN
metadata Optional[Dict[str, str]] | NotGiven NOT_GIVEN
modalities Optional[List[ChatCompletionModality]] | NotGiven NOT_GIVEN
n Optional[int] | NotGiven NOT_GIVEN
parallel_tool_calls bool | NotGiven NOT_GIVEN
presence_penalty Optional[float] | NotGiven NOT_GIVEN
response_format completion_create_params.ResponseFormat | NotGiven NOT_GIVEN
seed Optional[int] | NotGiven NOT_GIVEN
service_tier Optional[Literal[‘auto’, ‘default’]] | NotGiven NOT_GIVEN
stop Union[Optional[str], List[str]] | NotGiven NOT_GIVEN
store Optional[bool] | NotGiven NOT_GIVEN
stream_options Optional[ChatCompletionStreamOptionsParam] | NotGiven NOT_GIVEN
temperature Optional[float] | NotGiven NOT_GIVEN
tool_choice ChatCompletionToolChoiceOptionParam | NotGiven NOT_GIVEN
tools Iterable[ChatCompletionToolParam] | NotGiven NOT_GIVEN
top_logprobs Optional[int] | NotGiven NOT_GIVEN
top_p Optional[float] | NotGiven NOT_GIVEN
user str | NotGiven NOT_GIVEN
extra_headers Headers | None None
extra_query Query | None None
extra_body Body | None None
timeout float | httpx.Timeout | None | NotGiven NOT_GIVEN
Exported source 
@patch
@delegates(Completions.create)
def __call__(self:Chat,
             pr=None,  # Prompt / message
             stream:bool=False, # Stream response?
             **kwargs):
    "Add prompt `pr` to dialog and get a response"
    if isinstance(pr,str): pr = pr.strip()
    if pr: self.h.append(mk_msg(pr))
    if self.tools: kwargs['tools'] = [mk_openai_func(o) for o in self.tools]
    if self.tool_choice: kwargs['tool_choice'] = mk_tool_choice(self.tool_choice)
    res = self.c(self.h, sp=self.sp, stream=stream, **kwargs)
    self.h += mk_toolres(res, ns=self.tools)
    return res
chat("I'm Jeremy")
chat("What's my name?")

You mentioned that your name is Jeremy. How can I help you further?

  • id: chatcmpl-ALXvI7Tp8wLo1L4kt94VzHSffNy6y
  • choices: [Choice(finish_reason=‘stop’, index=0, logprobs=None, message=ChatCompletionMessage(content=‘You mentioned that your name is Jeremy. How can I help you further?’, refusal=None, role=‘assistant’, function_call=None, tool_calls=None))]
  • created: 1729698652
  • model: gpt-4o-2024-08-06
  • object: chat.completion
  • service_tier: None
  • system_fingerprint: fp_a7d06e42a7
  • usage: CompletionUsage(completion_tokens=15, prompt_tokens=43, total_tokens=58, completion_tokens_details=CompletionTokensDetails(audio_tokens=None, reasoning_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0))
chat = Chat(model, sp=sp)
for o in chat("I'm Jeremy", stream=True):
    o = contents(o)
    if o and isinstance(o, str): print(o, end='')
Hello, Jeremy! How can I assist you today?

Chat tool use

pr = f"What is {a}+{b}?"
pr
'What is 604542+6458932?'
chat = Chat(model, sp=sp, tools=[sums])
r = chat(pr)
r
Finding the sum of 604542 and 6458932
  • id: chatcmpl-ALXvKAJK3G7PzLoatbfcY3fE2z9Y6
  • choices: [Choice(finish_reason=‘tool_calls’, index=0, logprobs=None, message=ChatCompletionMessage(content=None, refusal=None, role=‘assistant’, function_call=None, tool_calls=[ChatCompletionMessageToolCall(id=‘call_0Iu9kqTCeGXNzm61r2yew7Dz’, function=Function(arguments=‘{“a”:604542,“b”:6458932}’, name=‘sums’), type=‘function’)]))]
  • created: 1729698654
  • model: gpt-4o-2024-08-06
  • object: chat.completion
  • service_tier: None
  • system_fingerprint: fp_45c6de4934
  • usage: CompletionUsage(completion_tokens=21, prompt_tokens=80, total_tokens=101, completion_tokens_details=CompletionTokensDetails(audio_tokens=None, reasoning_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0))
chat()

The sum of 604542 and 6458932 is 7063474.

  • id: chatcmpl-ALXvLXwVVockYTulBBNb4ZBrViss6
  • choices: [Choice(finish_reason=‘stop’, index=0, logprobs=None, message=ChatCompletionMessage(content=‘The sum of 604542 and 6458932 is 7063474.’, refusal=None, role=‘assistant’, function_call=None, tool_calls=None))]
  • created: 1729698655
  • model: gpt-4o-2024-08-06
  • object: chat.completion
  • service_tier: None
  • system_fingerprint: fp_72bbfa6014
  • usage: CompletionUsage(completion_tokens=18, prompt_tokens=112, total_tokens=130, completion_tokens_details=CompletionTokensDetails(audio_tokens=None, reasoning_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0))

Images

As everyone knows, when testing image APIs you have to use a cute puppy.

# Image is Cute_dog.jpg from Wikimedia
fn = Path('samples/puppy.jpg')
display.Image(filename=fn, width=200)

img = fn.read_bytes()

OpenAI expects an image message to have the following structure

{
  "type": "image_url",
  "image_url": {
    "url": f"data:{MEDIA_TYPE};base64,{IMG}"
  }
}

msglm automatically detects if a message is an image, encodes it, and generates the data structure above. All we need to do is a create a list containing our image and a query and then pass it to mk_msg.

Let’s try it out…

q = "In brief, what color flowers are in this image?"
msg = [mk_msg(img), mk_msg(q)]
c = Chat(model)
c([img, q])

The flowers in the image are purple.

  • id: chatcmpl-ALXvd9DSVpvfd9BjVttUQWhi1fyhP
  • choices: [Choice(finish_reason=‘stop’, index=0, logprobs=None, message=ChatCompletionMessage(content=‘The flowers in the image are purple.’, refusal=None, role=‘assistant’, function_call=None, tool_calls=None))]
  • created: 1729698673
  • model: gpt-4o-2024-08-06
  • object: chat.completion
  • service_tier: None
  • system_fingerprint: fp_72bbfa6014
  • usage: CompletionUsage(completion_tokens=8, prompt_tokens=273, total_tokens=281, completion_tokens_details=CompletionTokensDetails(audio_tokens=None, reasoning_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0))

Third Party Providers

Azure OpenAI Service

azure_endpoint = AzureOpenAI(
  azure_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT"), 
  api_key=os.getenv("AZURE_OPENAI_API_KEY"),  
  api_version="2024-08-01-preview"
)
client = Client(models_azure[0], azure_endpoint)
chat = Chat(cli=client)
chat("I'm Faisal")