# Agent STT Reference ``` GET wss://global.rt.speechmatics.com/v2/agent ``` The global endpoint automatically routes each connection to the nearest region. You can also pin to a specific region for data residency, for example `wss://eu.rt.speechmatics.com/v2/agent`. See [Supported endpoints](/get-started/authentication.md#supported-endpoints). ## Session Flow[​](#session-flow "Direct link to Session Flow") A basic Agent STT session will have the following message exchanges: Three groups of received messages carry the conversation itself, and are the ones your agent logic acts on: | Group | Messages | What they mean | | -------------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | Speech signals | `SpeechStarted`, `SpeechEnded` | Voice activity. Sent in `vad` mode only, and not turn boundaries — a mid-sentence pause ends speech without ending the turn. | | Segments | `AddPartialSegment`, `AddSegment` | The transcript. Each partial is a snapshot that replaces the last; `AddSegment` is final and will not change. | | Turn signals | `StartOfTurn`, `EndOfTurn` | Where a speaker's turn starts and ends. `EndOfTurn` is the turn's last message, and the cue for your agent to respond. | Pass `AddSegment` downstream to your LLM, and use `AddPartialSegment` only for display. The remaining received messages manage the session rather than the conversation: `RecognitionStarted`, `AudioAdded`, `SpeakersResult`, `EndOfTranscript`, `Info`, `Warning` and `Error`. See [Received messages](#received-messages). #### Browser based transcription[​](#browser-based-transcription "Direct link to Browser based transcription") When starting an Agent STT session **in the browser**, [temporary keys](/get-started/authentication.md#temporary-keys) should be used to avoid exposing your long-lived API key. Agent STT uses Realtime (`type=rt`) temporary keys. To do so, you must provide the temporary key as a part of a query parameter. This is due to a browser limitation. For example: ``` wss://global.rt.speechmatics.com/v2/agent?jwt= ``` ## Handshake responses[​](#handshake-responses "Direct link to Handshake responses") **Successful response** * `101 Switching Protocols` - Switch to WebSocket protocol Here is an example for a successful WebSocket handshake: ``` GET /v2/agent HTTP/1.1 Host: eu.rt.speechmatics.com Upgrade: websocket Connection: Upgrade Sec-WebSocket-Key: ujRTbIaQsXO/0uCbjjkSZQ== Sec-WebSocket-Version: 13 Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits Authorization: Bearer wmz9fkLJM6U5NdyaG3HLHybGZj65PXp User-Agent: Python/3.12 websockets/15.0 ``` A successful response should look like: ``` HTTP/1.1 101 Switching Protocols Connection: upgrade Upgrade: WebSocket Sec-WebSocket-Accept: 87kiC/LI5WgXG52nSylnfXdz260= ``` **Malformed request** A malformed handshake request will result in one of the following HTTP responses: * `400 Bad Request` * `401 Unauthorized` - when the API key is not valid * `405 Method Not Allowed` - when the request method is not GET **Client retry** Following a successful handshake and switch to the WebSocket protocol, the client could receive an immediate error message and WebSocket close handshake from the server. For the following errors only, we recommend adding a client retry interval of at least 5-10 seconds: * `4005 quota_exceeded` * `4013 job_error` * `1011 internal_error` ## Message handling[​](#message-handling "Direct link to Message handling") Every message in both directions is a stringified JSON object, with one exception: audio chunks sent from the client to the server are binary messages, referred to throughout this reference as `AddAudio`. Every JSON message includes: * `message` (string): The message type. All other fields depend on this value and are documented in [Sent messages](#sent-messages) and [Received messages](#received-messages). ## Sent messages[​](#sent-messages "Direct link to Sent messages") The below messages are sent from the client to the server. ### StartRecognition[​](#startrecognition "Direct link to startrecognition") Initiates a new recognition session. **audio\_format** required **encoding**required Possible values: \[`pcm_f32le`, `pcm_s16le`] **sample\_rate**required Possible values: \[`8000`, `16000`] **type**required Constant value: `raw` **message**required Constant value: `StartRecognition` **transcription\_config** objectrequired **additional\_vocab** object\[] List of custom words or phrases that should be recognized. Alternative pronunciations can be specified to aid recognition. * Array \[ oneOf * MOD1 * MOD2 ****string Possible values: `non-empty`, Value must match regular expression `\S` **content**stringrequired Possible values: `non-empty`, Value must match regular expression `\S` **sounds\_like**string\[] Possible values: `>= 1` * ] **diarization**string Specify whether speaker labels are added to the transcript. The default is `none`. * **none**: no diarization labels are added. * **speaker**: speaker attribution is performed based on acoustic matching. Possible values: \[`none`, `speaker`] **domain**string Request a specialized model based on 'language' but optimized for a particular field, e.g. 'finance' or 'medical'. Possible values: `non-empty` **emit\_sentences**boolean Whether or not to split multi-sentence segments on sentence boundaries, so that each segment contains a single sentence. The default is `false`. **Default value: **`false` **enable\_partials**boolean Whether or not to send partials as well as finals. **Default value: **`false` **language**stringrequired Language model to process the audio input, normally specified as an ISO language code Possible values: `non-empty` **model**stringrequired The model to use for transcription. Agent STT is powered by `linden-1`. Possible values: \[`linden-1`] **output\_locale**string Language locale to be used when generating the transcription output, normally specified as an ISO language code **speaker\_diarization\_config** objectnull **get\_speakers**boolean If true, speaker identifiers will be returned at the end of transcript. **max\_speakers**integer Specifies the maximum number of speakers allowed in a single audio stream. If not provided, the number of speakers is unrestricted. Possible values: `>= 2` **prefer\_current\_speaker**boolean If true, the diarization engine will prefer to stick with the currently active speaker if good enough, even if not the best. This is useful for cases where we can flip incorrectly between similar speakers during a single speaker section. **speaker\_sensitivity**number Controls the number of unique speakers detected; a higher value is likely to result in more unique speakers being identified in the output. You may want to increase this value if you are seeing fewer speaker labels than expected. The default is 0.5. Possible values: `>= 0` and `<= 1` **speakers** object\[] * Array \[ **label**stringrequired Speaker label, which must not match the format used internally (e.g. S1, S2, etc). Possible values: `non-empty` **speaker\_identifiers**bytes\[]required Possible values: `>= 1` * ] **transcript\_filtering\_config** objectnull Configuration for applying filtering to the transcription **remove\_disfluencies**boolean If true, words that are identified as disfluencies will be removed from the transcript. If false, they are tagged in the transcript as 'disfluency'. **replacements** object\[] * Array \[ **from**stringrequired The pattern to be replaced. **to**stringrequired The word or phrase to replace the 'from' value. * ] **turn\_config** object **turn\_detection\_mode**string How the end of a speaker turn is detected. The default is `vad`. * **vad**: turn ends are detected automatically from voice activity. * **external**: turn ends are signalled by the client with the `ForceEndOfUtterance` message. Possible values: \[`vad`, `external`] **Default value: **`vad` ### AddAudio[​](#addaudio "Direct link to addaudio") A binary chunk of audio. The server confirms receipt by sending an AudioAdded message. **string**binary ### EndOfStream[​](#endofstream "Direct link to endofstream") Declares that the client has no more audio to send. **last\_seq\_no**integerrequired **message**required Constant value: `EndOfStream` ### SetRecognitionConfig[​](#setrecognitionconfig "Direct link to setrecognitionconfig") Allows the client to re-configure the recognition session. The `language` field can be included in SetRecognitionConfig, but its value must match the language specified in the initial StartRecognition message. Attempting to change the language or any other transcription configuration parameter not listed below will result in an error. **message**required Constant value: `SetRecognitionConfig` **transcription\_config** required **enable\_partials**boolean Whether or not to send partials as well as finals. **Default value: **`false` **language**string Language model to process the audio input, normally specified as an ISO language code Possible values: `non-empty` ### GetSpeakers[​](#getspeakers "Direct link to getspeakers") Requests any detected speaker identifiers to be returned. **final**boolean **message**required Constant value: `GetSpeakers` ### ForceEndOfUtterance[​](#forceendofutterance "Direct link to forceendofutterance") Closes the current speaker turn. The transcript up to `timestamp` is flushed as an `AddSegment`, followed by `EndOfTurn`. Supported only with `turn_config.turn_detection_mode: external`, where the client detects turn ends itself. In the default `vad` mode the service closes turns from voice activity. **message**required Constant value: `ForceEndOfUtterance` **timestamp**floatrequired Timestamp of the audio data that corresponds to the force end of utterance request. It's the number of seconds since the beginning of the audio. Possible values: `>= 0` ## Received messages[​](#received-messages "Direct link to Received messages") ### RecognitionStarted[​](#recognitionstarted "Direct link to recognitionstarted") Server response to StartRecognition, acknowledging that a recognition session has started. **id**stringrequired **language\_pack\_info** object Properties of the language pack. **adapted**boolean Whether or not language model adaptation has been applied to the language pack. **itn**boolean Whether or not ITN (inverse text normalization) is available for the language pack. **language\_description**string Full descriptive name of the language, e.g. 'Japanese'. **word\_delimiter**stringrequired The character to use to separate words. **writing\_direction**string The direction that words in the language should be written and read in. Possible values: \[`left-to-right`, `right-to-left`] **message**required Constant value: `RecognitionStarted` **orchestrator\_version**string ### AudioAdded[​](#audioadded "Direct link to audioadded") Server response to AddAudio, indicating that audio has been added successfully. When clients send audio faster than real-time, the server may read data slower than it's sent. If binary `AddAudio` messages exceed the server's internal buffer, the server will process other WebSocket messages until buffer space is available. Clients receive `AudioAdded` responses only after binary data is read. This can fill TCP buffers, potentially causing WebSocket write failures and connection closure [with prejudice](https://websockets.spec.whatwg.org#the-closeevent-interface). Clients can monitor the WebSocket's [`bufferedAmount`](https://www.w3.org/TR/websockets#dom-websocket-bufferedamount) attribute to prevent this. **message**required Constant value: `AudioAdded` **seq\_no**integerrequired Possible values: `>= 0` ### AddPartialSegment[​](#addpartialsegment "Direct link to addpartialsegment") An interim, best-guess preview of the segment currently being built. Subject to change. **message**required Constant value: `AddPartialSegment` **metadata** objectrequired **end\_time**floatrequired Segment end, in seconds from session start. **start\_time**floatrequired Segment start, in seconds from session start. **segment** objectrequired **speaker**string Speaker label (e.g. `S1`), present only when diarization attributed one. **transcript**stringrequired The rendered segment text. ### AddSegment[​](#addsegment "Direct link to addsegment") A finalized, stable segment, sent when a segment boundary is reached. It will not change. **message**required Constant value: `AddSegment` **metadata** objectrequired **end\_time**floatrequired Segment end, in seconds from session start. **start\_time**floatrequired Segment start, in seconds from session start. **segment** objectrequired **speaker**string Speaker label (e.g. `S1`), present only when diarization attributed one. **transcript**stringrequired The rendered segment text. ### SpeechStarted[​](#speechstarted "Direct link to speechstarted") Sent when voice activity detection detects the onset of speech. **message**required Constant value: `SpeechStarted` **metadata** objectrequired **start\_time**floatrequired Time speech began, in seconds from session start. ### SpeechEnded[​](#speechended "Direct link to speechended") Sent when voice activity detection detects the end of speech. **message**required Constant value: `SpeechEnded` **metadata** objectrequired **end\_time**floatrequired Time speech ended, in seconds from session start. ### StartOfTurn[​](#startofturn "Direct link to startofturn") Sent when turn detection marks the start of a speaker turn. **message**required Constant value: `StartOfTurn` **metadata** objectrequired **start\_time**floatrequired Time the turn began, in seconds from session start. ### EndOfTurn[​](#endofturn "Direct link to endofturn") Sent when turn detection marks the end of a speaker turn. It is the last message of a turn. **message**required Constant value: `EndOfTurn` **metadata** objectrequired **end\_time**floatrequired Time the turn ended, in seconds from session start. ### EndOfTranscript[​](#endoftranscript "Direct link to endoftranscript") Server response to EndOfStream, sent after the server has finished sending all segment messages. **message**required Constant value: `EndOfTranscript` ### Error[​](#error "Direct link to error") Error messages sent from the server to the client. After any error, transcription is terminated and the connection is closed. **message**required Constant value: `Error` **reason**stringrequired **type**required Possible values: \[`invalid_message`, `invalid_model`, `invalid_language`, `invalid_config`, `invalid_audio_type`, `not_authorised`, `not_allowed`, `job_error`, `protocol_error`, `start_recognition_timeout`] ### Warning[​](#warning "Direct link to warning") Warning messages sent from the server to the client. **duration\_limit**number Possible values: `>= 0` **message**required Constant value: `Warning` **reason**stringrequired **type** Possible values: \[`duration_limit_exceeded`, `idle_timeout`, `session_timeout`, `add_audio_after_eos`, `speaker_id`, `unsupported_language`] ### Info[​](#info "Direct link to info") Additional information sent from the server to the client. **message** Constant value: `Info` **quality** Possible values: \[`broadcast`] **reason**string **type** Possible values: \[`recognition_quality`, `concurrent_session_usage`] ### SpeakersResult[​](#speakersresult "Direct link to speakersresult") Server response to GetSpeakers, returning any detected speaker identifiers. **message**required Constant value: `SpeakersResult` **speakers** object\[]required * Array \[ **label**stringrequired Speaker label. Possible values: `non-empty` **speaker\_identifiers**bytes\[]required Possible values: `>= 1` * ] ## Websocket errors[​](#websocket-errors "Direct link to Websocket errors") An in-band `Error` message can be followed by a WebSocket close message. The table below shows the possible WebSocket close codes and associated error types. The error types are provided in the payload of the close message. | WebSocket Close Code | WebSocket Close Payload | | -------------------- | ----------------------- | | 1003 | `protocol_error` | | 1008 | `policy_violation` | | 1011 | `internal_error` | | 4001 | `not_authorised` | | 4003 | `not_allowed` | | 4004 | `invalid_model` | | 4005 | `quota_exceeded` | | 4006 | `timelimit_exceeded` | | 4013 | `job_error` |