For AI agents: a documentation index is available at /llms.txt. Markdown versions of all pages can be requested by appending `.md` to the URL, or by setting the `Accept` header to `text/markdown`.
Skip to main content
DeploymentsContainer

GPU Speech to text container (Melia 1 and Oak 1)

Learn about the Speechmatics Melia 1 and Oak 1 GPU container system

Melia 1 and Oak 1 are multilingual, GPU-only models. Oak 1 is tuned for healthcare audio. Both models run on the same transcriber and inference server images. For an overview, see Melia 1 and Oak 1.

Prerequisites​

System requirements​

The system must have:

  • Nvidia GPU(s) with at least 16GB of GPU memory
  • Nvidia drivers (see below for supported versions)
  • CUDA compute capability of 7.5-12.1 inclusive, which corresponds to the Turing, Ampere, Lovelace, Hopper, Blackwell architectures. Cards with the Volta architecture or below are not able to run the models
  • 24 GB RAM
  • The nvidia-container-toolkit installed
  • Docker version > 19.03

See Performance and cost for more information on the performance and cost of the container.

Nvidia drivers​

  • The GPU Inference Container is based on CUDA 13.0.1, which requires NVIDIA Driver release 580 or later.

Driver installation can be validated by running nvidia-smi. This command should return the Nvidia driver version and show additional information about the GPU(s).

Cloud instances​

The GPU node can be provisioned in the cloud.

Running the image​

Currently, each GPU Inference Container can only run on a single GPU. If a system has more than one GPU, the device must be specified using CUDA_VISIBLE_DEVICES or selecting the device using the --gpus argument. See Nvidia/CUDA documentation for details.

Pull the inference server image as described in Accessing images, then run it:

docker run --rm -it \
-v $PWD/license.json:/license.json \
--gpus '"device=0"' \
-e CUDA_VISIBLE_DEVICES \
-p 8001:8001 \
speechmaticspublic.azurecr.io/sm-asr-inference-server:1.13.0

When the Container starts you should see output similar to this, indicating that the server has started and is ready to serve requests.

I0705 08:12:55.419608 1 server.cc:709]
+--------------+---------+--------+
| Model | Version | Status |
+--------------+---------+--------+
| aed | 1 | READY |
| body | 1 | READY |
| detokenizer | 1 | READY |
| dz | 1 | READY |
| ensemble | 1 | READY |
| preprocessor | 1 | READY |
+--------------+---------+--------+
...
I0705 08:12:55.515473 1 grpc_server.cc:2579] "Started GRPCInferenceService at 0.0.0.0:8001"
I0705 08:12:55.515940 1 http_server.cc:4961] "Started HTTPService at 0.0.0.0:8000"
I0705 08:12:55.598672 1 http_server.cc:400] "Started Metrics Service at 0.0.0.0:8002"

The inference server runs in one of two modes: Batch (the default) or Realtime. A server supports only one mode at a time.

The transcriber offloads inference to the server, so it does not require a GPU. It uses a single image for both modes, rather than the separate rt- and batch- images of the Standard and Enhanced GPU container. Set SM_ASR_MODE to batch or rt to select the mode.

Selecting a model​

Set model to melia-1 or oak-1 in the transcription config, and set language to multi:

{
"transcription_config": {
"model": "oak-1",
"language": "multi"
}
}

Batch inference​

Run the transcriber with SM_INFERENCE_ENDPOINT set to the gRPC endpoint of the inference server and SM_ASR_MODE set to batch:

docker run --rm -it \
-e SM_INFERENCE_ENDPOINT=<server>:<port> \
-e SM_ASR_MODE=batch \
-v $PWD/license.json:/license.json \
-v $PWD/example.wav:/input.audio \
speechmaticspublic.azurecr.io/sm-asr-transcriber:1.13.0

To accept multiple jobs over an HTTP API without restarting the container between jobs, run the transcriber as a batch persistent worker.

Realtime inference​

To run the inference server in Realtime mode, set SM_STREAMING=true:

docker run --rm -it \
-v $PWD/license.json:/license.json \
--gpus '"device=0"' \
-e CUDA_VISIBLE_DEVICES \
-e SM_STREAMING=true \
-p 8001:8001 \
speechmaticspublic.azurecr.io/sm-asr-inference-server:1.13.0

When the server is ready, the model table lists the streaming models:

I0911 11:04:02.321449 1 server.cc:709]
+--------------------+---------+--------+
| Model | Version | Status |
+--------------------+---------+--------+
| aed | 1 | READY |
| body | 1 | READY |
| preprocessor | 1 | READY |
| streaming | 1 | READY |
| streaming_ensemble | 1 | READY |
+--------------------+---------+--------+
...
I0911 11:04:02.449550 1 grpc_server.cc:2579] "Started GRPCInferenceService at 0.0.0.0:8001"
I0911 11:04:02.449744 1 http_server.cc:4961] "Started HTTPService at 0.0.0.0:8000"
I0911 11:04:02.491582 1 http_server.cc:400] "Started Metrics Service at 0.0.0.0:8002"

Run the transcriber with SM_INFERENCE_ENDPOINT set to the gRPC endpoint of the inference server and SM_ASR_MODE set to rt:

docker run --rm -it \
-e SM_INFERENCE_ENDPOINT=<server>:<port> \
-e SM_ASR_MODE=rt \
-v $PWD/license.json:/license.json \
-p 9000:9000 \
speechmaticspublic.azurecr.io/sm-asr-transcriber:1.13.0

Connect over WebSocket as described in Realtime transcription, and select the model in the StartRecognition message. For the Realtime features both models support, refer to Melia 1 and Oak 1 on Realtime.

Realtime inference requires container version 1.13.0 or later.

Multi-session containers​

By default, the Realtime transcriber accepts one WebSocket connection at a time. To accept more, set SM_MAX_CONCURRENT_CONNECTIONS to the maximum number of sessions. The /ready health check endpoint then returns true while a connection is free.

CPU usage scales with the number of active sessions. Most memory is shared between sessions.

Monitoring the server​

The inference server is based on Nvidia's Triton architecture and as such can be monitored using Triton's inbuilt Prometheus metrics, or the GRPC/HTTP APIs. To expose these, configure an external mapping for port 8002(Prometheus) or 8000(HTTP).

Docker compose example​

These Docker Compose files will create a Speechmatics Melia 1 and Oak 1 GPU Inference Server and connect an applicable transcription worker:

(assumes your license.json file is in the current working directory)

Realtime​

docker-compose.yml
version: "3.8"

networks:
transcriber:
driver: bridge

services:
triton:
image: speechmaticspublic.azurecr.io/sm-asr-inference-server:1.13.0
deploy:
resources:
reservations:
devices:
- driver: nvidia
### Limit to N GPUs
# count: 1
### Pick specific GPUs by device ID
# device_ids:
# - 0
# - 3
capabilities:
- gpu
container_name: triton
networks:
- transcriber
expose:
- 8000/tcp
- 8001/tcp
- 8002/tcp
environment:
- NVIDIA_DRIVER_CAPABILITIES=all
- NVIDIA_VISIBLE_DEVICES=all
- CUDA_VISIBLE_DEVICES=0
- SM_STREAMING=true
volumes:
- $PWD/license.json:/license.json:ro
transcriber:
image: speechmaticspublic.azurecr.io/sm-asr-transcriber:1.13.0
networks:
- transcriber
ports:
- 9000:9000
environment:
- SM_INFERENCE_ENDPOINT=triton:8001
- SM_MAX_CONCURRENT_CONNECTIONS=1
- SM_ASR_MODE=rt
volumes:
- $PWD/license.json:/license.json:ro

Persistant Batch Worker​

docker-compose.yml
version: "3.8"

networks:
transcriber:
driver: bridge

services:
triton:
image: speechmaticspublic.azurecr.io/sm-asr-inference-server:1.13.0
deploy:
resources:
reservations:
devices:
- driver: nvidia
### Limit to N GPUs
# count: 1
### Pick specific GPUs by device ID
# device_ids:
# - 0
# - 3
capabilities:
- gpu
container_name: triton
networks:
- transcriber
expose:
- 8000/tcp
- 8001/tcp
- 8002/tcp
environment:
- NVIDIA_DRIVER_CAPABILITIES=all
- NVIDIA_VISIBLE_DEVICES=all
- CUDA_VISIBLE_DEVICES=0
volumes:
- $PWD/license.json:/license.json:ro
transcriber:
image: speechmaticspublic.azurecr.io/sm-asr-transcriber:1.13.0
networks:
- transcriber
ports:
- 18000:18000
environment:
- SM_INFERENCE_ENDPOINT=triton:8001
- SM_ASR_MODE=batch
command: --run-mode http --parallel=10
volumes:
- $PWD/license.json:/license.json:ro