Function Calling
Function Calling allows you to connect Flow to external tools and systems. This unlocks Flow's ability to act in the real-world and better serve the needs of your users.
This could involve needing real-time information such as opening/closing times or validation services for authentication or action APIs that control a fast food system while placing a drive-thru order.
Based on what the user says in the conversation, Flow will recognise the user's intentions and extract out the key information that your system needs to complete the function call.
For example, you may want Flow to add reminders in a user's calender:
{
"name": "add_reminder",
"description": "Use this to schedule reminders. Needs a confirmation.",
"parameters": {
"type": "object",
"properties": {
"date" : {
"type" : "string",
"description" : "The date for the reminder in dd/mm/yyyy format"
},
"time" : {
"type": "string",
"description" : "The time for the reminder in 24 hour hh:mm format"
},
"title" : {
"type": "string",
"description" : "The title for the reminder"
},
"project": {
"type": "string",
"description": "Which project the reminder is related to. If not provided, leave blank."
}
},
"required": ["project"]
},
},
Configuring Function Calling
Functions must be declared within a list of tools when your client sends the StartConversation message. Each function must be defined with the following:
name- The name of the function that should be called. This name is passed as a field in the ToolInvoke messagedescription- A natural language string that instructs the LLM about the condition in which the function must be called Each function can have multiple parameters, which are input parameters gathered from the conversation by the LLM. Each of them are defined by:parameters- An object containing thepropertiesof the function call which should be collected from the conversation. Each parameter is defined by:type- The type of input it is. e.g. String, Integer, Object, etc.description- This describes to the LLM how to detect and format the input parameter during conversation. e.g. The date to set the reminder for in dd/mm/yyyy format
required- (List) : Optional. The list of input parameters for the function which are required.
Quick Start
1import asyncio
2import io
3import sys
4import json
5
6import pyaudio
7
8from speechmatics_flow.client import WebsocketClient
9from speechmatics_flow.models import (
10 ConnectionSettings,
11 Interaction,
12 AudioSettings,
13 ConversationConfig,
14 ServerMessageType,
15 ClientMessageType
16)
17from speechmatics_flow.tool_function_param import ToolFunctionParam
18
19AUTH_TOKEN = "Place your auth token here"
20
21# Example configuration which could add a reminder to a calendar.
22reminder_config = ToolFunctionParam(
23 type="function",
24 function={
25 "name": "add_reminder",
26 "description": "Use this to schedule reminders. Needs a confirmation.",
27 "parameters": {
28 "type": "object",
29 "properties": {
30 "date" : {
31 "type" : "string",
32 "description" : "The date for the reminder in dd/mm/yyyy format"
33 },
34 "time" : {
35 "type": "string",
36 "description" : "The time for the reminder in 24 hour hh:mm format"
37 },
38 "title" : {
39 "type": "string",
40 "description" : "The title for the reminder"
41 },
42 "project": {
43 "type": "string",
44 "description": "Which project the reminder is related to. If not provided, leave blank."
45 }
46 },
47 "required": ["project"]
48 },
49 },
50 )
51
52
53
54# Callback for handling reminder ToolInvoke in your system.
55async def reminder_handler(msg: dict):
56 print("Attempting to add reminder")
57 print(msg)
58 response_message = {
59 "message": ClientMessageType.ToolResult,
60 "id": msg["id"],
61 "status": "ok", # Used to inform user the status of the function call. Could be "failed" or "rejected".
62 "content": "Added reminder successfully to calendar" # LLM response helper message
63 }
64
65 await client.websocket.send(json.dumps(response_message))
66
67# Create a websocket client
68client = WebsocketClient(
69 ConnectionSettings(
70 url="wss://flow.api.speechmatics.com/v1/flow",
71 auth_token=AUTH_TOKEN,
72 )
73)
74
75# Create a buffer to store binary messages sent from the server
76audio_buffer = io.BytesIO()
77
78
79# Create callback function which adds binary messages to audio buffer
80def binary_msg_handler(msg: bytes):
81 if isinstance(msg, (bytes, bytearray)):
82 audio_buffer.write(msg)
83
84
85# Register the callback which will be called
86# when the client receives an audio message from the server
87client.add_event_handler(ServerMessageType.AddAudio, binary_msg_handler)
88
89# Handling ToolInvoke message
90client.add_event_handler(ServerMessageType.ToolInvoke, reminder_handler)
91
92async def audio_playback(buffer):
93 """Read from buffer and play audio back to the user"""
94 p = pyaudio.PyAudio()
95 stream = p.open(format=pyaudio.paInt16, channels=1, rate=16000, output=True)
96 try:
97 while True:
98 # Get the current value from the buffer
99 audio_to_play = buffer.getvalue()
100 # Only proceed if there is audio data to play
101 if audio_to_play:
102 # Write the audio to the stream
103 stream.write(audio_to_play)
104 buffer.seek(0)
105 buffer.truncate(0)
106 # Pause briefly before checking the buffer again
107 await asyncio.sleep(0.05)
108 finally:
109 stream.close()
110 stream.stop_stream()
111 p.terminate()
112
113
114async def main():
115 print(f"Starting...")
116 tasks = [
117 # Use the websocket to connect to Flow Service and start a conversation
118 asyncio.create_task(
119 client.run(
120 interactions=[Interaction(sys.stdin.buffer)],
121 audio_settings=AudioSettings(),
122 conversation_config=ConversationConfig(),
123 tools=[reminder_config]
124 )
125 ),
126 # Run audio playback handler which streams audio from audio buffer
127 asyncio.create_task(audio_playback(audio_buffer)),
128 ]
129 await asyncio.gather(*tasks)
130
131asyncio.run(main())
132
133Considerations
- Holding message - Flow has the ability to play a holding message automatically when the function call is first triggered. Adding
holding_phrase: falseto the function definition will encourage the LLM to not say anything when calling the function. This can be of use when the reply will be instant and the verbalisation of the process could detract from the user experience. - Function
status- The client must inform the service of whether the function call succeeded or not. This allows the service to inform the user of the result. There is no automatic timeout on the Flow API. - Asynchronous - Function calling is fully asynchronous. Once the client is informed of the function call, the conversation will continue to progress until a function call status update is received from the client. This is to continue providing a natural conversational experience to the customer.
- Completion Message - Flow can play a message on completion of the function call. The Client can switch this off by passing
<NO_RESPONSE_REQUIRED/>in the content field of the ToolResult message.
info
Since LLMs are semantically instructed, complete, narrow, & unambiguous function calls with simple descriptions can create a reliable customer experience. Complex business logic should be handled within your client.