GPU Speech to text container (Melia 1 and Oak 1)
Melia 1 and Oak 1 are multilingual, GPU-only models. Oak 1 is tuned for healthcare audio. Both models run on the same transcriber and inference server images. For an overview, see Melia 1 and Oak 1.
Prerequisites
- A license file or a license token
- There is no specific license for the GPU Inference Container, it will run using an existing Speechmatics license for the Batch Container
- Access to our Docker repository
System requirements
The system must have:
- Nvidia GPU(s) with at least 16GB of GPU memory
- Nvidia drivers (see below for supported versions)
- CUDA compute capability of 7.5-12.1 inclusive, which corresponds to the Turing, Ampere, Lovelace, Hopper, Blackwell architectures. Cards with the Volta architecture or below are not able to run the models
- 24 GB RAM
- The nvidia-container-toolkit installed
- Docker version > 19.03
See Performance and cost for more information on the performance and cost of the container.
Nvidia drivers
- The GPU Inference Container is based on CUDA 13.0.1, which requires NVIDIA Driver release 580 or later.
Driver installation can be validated by running nvidia-smi. This command should return the Nvidia driver version and show additional information about the GPU(s).
Cloud instances
The GPU node can be provisioned in the cloud.
Running the image
Currently, each GPU Inference Container can only run on a single GPU.
If a system has more than one GPU, the device must be specified using CUDA_VISIBLE_DEVICES
or selecting the device using the --gpus argument. See Nvidia/CUDA documentation for details.
Pull the inference server image as described in Accessing images, then run it:
docker run --rm -it \
-v $PWD/license.json:/license.json \
--gpus '"device=0"' \
-e CUDA_VISIBLE_DEVICES \
-p 8001:8001 \
speechmaticspublic.azurecr.io/sm-asr-inference-server:1.13.0
When the Container starts you should see output similar to this, indicating that the server has started and is ready to serve requests.
I0705 08:12:55.419608 1 server.cc:709]
+--------------+---------+--------+
| Model | Version | Status |
+--------------+---------+--------+
| aed | 1 | READY |
| body | 1 | READY |
| detokenizer | 1 | READY |
| dz | 1 | READY |
| ensemble | 1 | READY |
| preprocessor | 1 | READY |
+--------------+---------+--------+
...
I0705 08:12:55.515473 1 grpc_server.cc:2579] "Started GRPCInferenceService at 0.0.0.0:8001"
I0705 08:12:55.515940 1 http_server.cc:4961] "Started HTTPService at 0.0.0.0:8000"
I0705 08:12:55.598672 1 http_server.cc:400] "Started Metrics Service at 0.0.0.0:8002"
The inference server runs in one of two modes: Batch (the default) or Realtime. A server supports only one mode at a time.
The transcriber offloads inference to the server, so it does not require a GPU. It uses a single image for both modes, rather than the separate rt- and batch- images of the Standard and Enhanced GPU container. Set SM_ASR_MODE to batch or rt to select the mode.
Selecting a model
Set model to melia-1 or oak-1 in the transcription config, and set language to multi:
{
"transcription_config": {
"model": "oak-1",
"language": "multi"
}
}
Batch inference
Run the transcriber with SM_INFERENCE_ENDPOINT set to the gRPC endpoint of the inference server and SM_ASR_MODE set to batch:
docker run --rm -it \
-e SM_INFERENCE_ENDPOINT=<server>:<port> \
-e SM_ASR_MODE=batch \
-v $PWD/license.json:/license.json \
-v $PWD/example.wav:/input.audio \
speechmaticspublic.azurecr.io/sm-asr-transcriber:1.13.0
To accept multiple jobs over an HTTP API without restarting the container between jobs, run the transcriber as a batch persistent worker.
Realtime inference
To run the inference server in Realtime mode, set SM_STREAMING=true:
docker run --rm -it \
-v $PWD/license.json:/license.json \
--gpus '"device=0"' \
-e CUDA_VISIBLE_DEVICES \
-e SM_STREAMING=true \
-p 8001:8001 \
speechmaticspublic.azurecr.io/sm-asr-inference-server:1.13.0
When the server is ready, the model table lists the streaming models:
I0911 11:04:02.321449 1 server.cc:709]
+--------------------+---------+--------+
| Model | Version | Status |
+--------------------+---------+--------+
| aed | 1 | READY |
| body | 1 | READY |
| preprocessor | 1 | READY |
| streaming | 1 | READY |
| streaming_ensemble | 1 | READY |
+--------------------+---------+--------+
...
I0911 11:04:02.449550 1 grpc_server.cc:2579] "Started GRPCInferenceService at 0.0.0.0:8001"
I0911 11:04:02.449744 1 http_server.cc:4961] "Started HTTPService at 0.0.0.0:8000"
I0911 11:04:02.491582 1 http_server.cc:400] "Started Metrics Service at 0.0.0.0:8002"
Run the transcriber with SM_INFERENCE_ENDPOINT set to the gRPC endpoint of the inference server and SM_ASR_MODE set to rt:
docker run --rm -it \
-e SM_INFERENCE_ENDPOINT=<server>:<port> \
-e SM_ASR_MODE=rt \
-v $PWD/license.json:/license.json \
-p 9000:9000 \
speechmaticspublic.azurecr.io/sm-asr-transcriber:1.13.0
Connect over WebSocket as described in Realtime transcription, and select the model in the StartRecognition message. For the Realtime features both models support, refer to Melia 1 and Oak 1 on Realtime.
Realtime inference requires container version 1.13.0 or later.
Multi-session containers
By default, the Realtime transcriber accepts one WebSocket connection at a time. To accept more, set SM_MAX_CONCURRENT_CONNECTIONS to the maximum number of sessions. The /ready health check endpoint then returns true while a connection is free.
CPU usage scales with the number of active sessions. Most memory is shared between sessions.
Monitoring the server
The inference server is based on Nvidia's Triton architecture and as such can be monitored using Triton's inbuilt Prometheus metrics, or the GRPC/HTTP APIs. To expose these, configure an external mapping for port 8002(Prometheus) or 8000(HTTP).
Docker compose example
These Docker Compose files will create a Speechmatics Melia 1 and Oak 1 GPU Inference Server and connect an applicable transcription worker:
(assumes your license.json file is in the current working directory)
Realtime
version: "3.8"
networks:
transcriber:
driver: bridge
services:
triton:
image: speechmaticspublic.azurecr.io/sm-asr-inference-server:1.13.0
deploy:
resources:
reservations:
devices:
- driver: nvidia
### Limit to N GPUs
# count: 1
### Pick specific GPUs by device ID
# device_ids:
# - 0
# - 3
capabilities:
- gpu
container_name: triton
networks:
- transcriber
expose:
- 8000/tcp
- 8001/tcp
- 8002/tcp
environment:
- NVIDIA_DRIVER_CAPABILITIES=all
- NVIDIA_VISIBLE_DEVICES=all
- CUDA_VISIBLE_DEVICES=0
- SM_STREAMING=true
volumes:
- $PWD/license.json:/license.json:ro
transcriber:
image: speechmaticspublic.azurecr.io/sm-asr-transcriber:1.13.0
networks:
- transcriber
ports:
- 9000:9000
environment:
- SM_INFERENCE_ENDPOINT=triton:8001
- SM_MAX_CONCURRENT_CONNECTIONS=1
- SM_ASR_MODE=rt
volumes:
- $PWD/license.json:/license.json:ro
Persistant Batch Worker
version: "3.8"
networks:
transcriber:
driver: bridge
services:
triton:
image: speechmaticspublic.azurecr.io/sm-asr-inference-server:1.13.0
deploy:
resources:
reservations:
devices:
- driver: nvidia
### Limit to N GPUs
# count: 1
### Pick specific GPUs by device ID
# device_ids:
# - 0
# - 3
capabilities:
- gpu
container_name: triton
networks:
- transcriber
expose:
- 8000/tcp
- 8001/tcp
- 8002/tcp
environment:
- NVIDIA_DRIVER_CAPABILITIES=all
- NVIDIA_VISIBLE_DEVICES=all
- CUDA_VISIBLE_DEVICES=0
volumes:
- $PWD/license.json:/license.json:ro
transcriber:
image: speechmaticspublic.azurecr.io/sm-asr-transcriber:1.13.0
networks:
- transcriber
ports:
- 18000:18000
environment:
- SM_INFERENCE_ENDPOINT=triton:8001
- SM_ASR_MODE=batch
command: --run-mode http --parallel=10
volumes:
- $PWD/license.json:/license.json:ro