Agent STT Reference
GETwss://global.rt.speechmatics.com/v2/agent
The global endpoint automatically routes each connection to the nearest region. You can also pin to a specific region for data residency, for example wss://eu.rt.speechmatics.com/v2/agent. See Supported endpoints.
Session Flow
A basic Agent STT session will have the following message exchanges:
Three groups of received messages carry the conversation itself, and are the ones your agent logic acts on:
Pass AddSegment downstream to your LLM, and use AddPartialSegment only for display.
The remaining received messages manage the session rather than the conversation: RecognitionStarted, AudioAdded, SpeakersResult, EndOfTranscript, Info, Warning and Error. See Received messages.
Browser based transcription
When starting an Agent STT session in the browser, temporary keys should be used to avoid exposing your long-lived API key. Agent STT uses Realtime (type=rt) temporary keys.
To do so, you must provide the temporary key as a part of a query parameter. This is due to a browser limitation. For example:
wss://global.rt.speechmatics.com/v2/agent?jwt=<temporary-key>
Handshake responses
Successful response
101 Switching Protocols- Switch to WebSocket protocol
Here is an example for a successful WebSocket handshake:
GET /v2/agent HTTP/1.1
Host: eu.rt.speechmatics.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: ujRTbIaQsXO/0uCbjjkSZQ==
Sec-WebSocket-Version: 13
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Authorization: Bearer wmz9fkLJM6U5NdyaG3HLHybGZj65PXp
User-Agent: Python/3.12 websockets/15.0
A successful response should look like:
HTTP/1.1 101 Switching Protocols
Connection: upgrade
Upgrade: WebSocket
Sec-WebSocket-Accept: 87kiC/LI5WgXG52nSylnfXdz260=
Malformed request
A malformed handshake request will result in one of the following HTTP responses:
400 Bad Request401 Unauthorized- when the API key is not valid405 Method Not Allowed- when the request method is not GET
Client retry
Following a successful handshake and switch to the WebSocket protocol, the client could receive an immediate error message and WebSocket close handshake from the server. For the following errors only, we recommend adding a client retry interval of at least 5-10 seconds:
4005 quota_exceeded4013 job_error1011 internal_error
Message handling
Every message in both directions is a stringified JSON object, with one exception: audio chunks sent from the client to the server are binary messages, referred to throughout this reference as AddAudio.
Every JSON message includes:
message(string): The message type. All other fields depend on this value and are documented in Sent messages and Received messages.
Sent messages
The below messages are sent from the client to the server.
StartRecognition
Initiates a new recognition session.
audio_format required
Possible values: [pcm_f32le, pcm_s16le]
16000rawStartRecognitiontranscription_config objectrequired
additional_vocab object[]
List of custom words or phrases that should be recognized. Alternative pronunciations can be specified to aid recognition.
- MOD1
- MOD2
Possible values: non-empty, Value must match regular expression \S
Possible values: non-empty, Value must match regular expression \S
Possible values: >= 1
Specify whether speaker labels are added to the transcript.
The default is none.
- none: no diarization labels are added.
- speaker: speaker attribution is performed based on acoustic matching.
Possible values: [none, speaker]
Request a specialized model based on 'language' but optimized for a particular field, e.g. 'finance' or 'medical'.
Possible values: non-empty
Whether or not to split multi-sentence segments on sentence boundaries, so that each segment contains a single sentence.
The default is false.
falseWhether or not to send partials as well as finals.
falseLanguage model to process the audio input, normally specified as an ISO language code
Possible values: non-empty
The model to use for transcription. Agent STT is powered by linden-1.
Possible values: [linden-1]
Language locale to be used when generating the transcription output, normally specified as an ISO language code
speaker_diarization_config objectnull
If true, speaker identifiers will be returned at the end of transcript.
Specifies the maximum number of speakers allowed in a single audio stream. If not provided, the number of speakers is unrestricted.
Possible values: >= 2
If true, the diarization engine will prefer to stick with the currently active speaker if good enough, even if not the best. This is useful for cases where we can flip incorrectly between similar speakers during a single speaker section.
Controls the number of unique speakers detected; a higher value is likely to result in more unique speakers being identified in the output. You may want to increase this value if you are seeing fewer speaker labels than expected. The default is 0.5.
Possible values: >= 0 and <= 1
speakers object[]
Speaker label, which must not match the format used internally (e.g. S1, S2, etc).
Possible values: non-empty
Possible values: >= 1
transcript_filtering_config objectnull
Configuration for applying filtering to the transcription
If true, words that are identified as disfluencies will be removed from the transcript. If false, they are tagged in the transcript as 'disfluency'.
replacements object[]
The pattern to be replaced.
The word or phrase to replace the 'from' value.
turn_config object
How the end of a speaker turn is detected.
The default is vad.
- vad: turn ends are detected automatically from voice activity.
- external: turn ends are signalled by the client with the
ForceEndOfUtterancemessage.
Possible values: [vad, external]
vadAddAudio
A binary chunk of audio. The server confirms receipt by sending an AudioAdded message.
EndOfStream
Declares that the client has no more audio to send.
EndOfStreamSetRecognitionConfig
Allows the client to re-configure the recognition session.
The language field can be included in SetRecognitionConfig, but its value must match the language specified in the initial StartRecognition message.
Attempting to change the language or any other transcription configuration parameter not listed below will result in an error.
SetRecognitionConfigtranscription_config required
Whether or not to send partials as well as finals.
falseLanguage model to process the audio input, normally specified as an ISO language code
Possible values: non-empty
GetSpeakers
Requests any detected speaker identifiers to be returned.
GetSpeakersForceEndOfUtterance
Closes the current speaker turn. The transcript up to timestamp is flushed as an AddSegment, followed by EndOfTurn.
Supported only with turn_config.turn_detection_mode: external, where the client detects turn ends itself.
In the default vad mode the service closes turns from voice activity.
ForceEndOfUtteranceTimestamp of the audio data that corresponds to the force end of utterance request. It's the number of seconds since the beginning of the audio.
Possible values: >= 0
Received messages
RecognitionStarted
Server response to StartRecognition, acknowledging that a recognition session has started.
language_pack_info object
Properties of the language pack.
Whether or not language model adaptation has been applied to the language pack.
Whether or not ITN (inverse text normalization) is available for the language pack.
Full descriptive name of the language, e.g. 'Japanese'.
The character to use to separate words.
The direction that words in the language should be written and read in.
Possible values: [left-to-right, right-to-left]
RecognitionStartedAudioAdded
Server response to AddAudio, indicating that audio has been added successfully.
When clients send audio faster than real-time, the server may read data slower than it's sent. If binary AddAudio messages exceed the server's internal buffer, the server will process other WebSocket messages until buffer space is available. Clients receive AudioAdded responses only after binary data is read. This can fill TCP buffers, potentially causing WebSocket write failures and connection closure with prejudice. Clients can monitor the WebSocket's bufferedAmount attribute to prevent this.
AudioAddedPossible values: >= 0
AddPartialSegment
An interim, best-guess preview of the segment currently being built. Subject to change.
AddPartialSegmentmetadata objectrequired
Segment end, in seconds from session start.
Segment start, in seconds from session start.
segment objectrequired
Speaker label (e.g. S1), present only when diarization attributed one.
The rendered segment text.
AddSegment
A finalized, stable segment, sent when a segment boundary is reached. It will not change.
AddSegmentmetadata objectrequired
Segment end, in seconds from session start.
Segment start, in seconds from session start.
segment objectrequired
Speaker label (e.g. S1), present only when diarization attributed one.
The rendered segment text.
SpeechStarted
Sent when voice activity detection detects the onset of speech.
SpeechStartedmetadata objectrequired
Time speech began, in seconds from session start.
SpeechEnded
Sent when voice activity detection detects the end of speech.
SpeechEndedmetadata objectrequired
Time speech ended, in seconds from session start.
StartOfTurn
Sent when turn detection marks the start of a speaker turn.
StartOfTurnmetadata objectrequired
Time the turn began, in seconds from session start.
EndOfTurn
Sent when turn detection marks the end of a speaker turn. It is the last message of a turn.
EndOfTurnmetadata objectrequired
Time the turn ended, in seconds from session start.
EndOfTranscript
Server response to EndOfStream, sent after the server has finished sending all segment messages.
EndOfTranscriptError
Error messages sent from the server to the client. After any error, transcription is terminated and the connection is closed.
ErrorPossible values: [invalid_message, invalid_model, invalid_language, invalid_config, invalid_audio_type, not_authorised, not_allowed, job_error, protocol_error, start_recognition_timeout]
Warning
Warning messages sent from the server to the client.
Possible values: >= 0
WarningPossible values: [duration_limit_exceeded, idle_timeout, session_timeout, add_audio_after_eos, speaker_id, unsupported_language]
Info
Additional information sent from the server to the client.
InfoPossible values: [broadcast]
Possible values: [recognition_quality, concurrent_session_usage]
SpeakersResult
Server response to GetSpeakers, returning any detected speaker identifiers.
SpeakersResultspeakers object[]required
Speaker label.
Possible values: non-empty
Possible values: >= 1
Websocket errors
An in-band Error message can be followed by a WebSocket close message. The table below shows the possible WebSocket close codes and associated error types. The error types are provided in the payload of the close message.