# GPU Speech to text container (Melia 1)

Learn about the Speechmatics Melia 1 GPU container system

Melia 1 is a multilingual, GPU-only model. For an overview, see [Models](/speech-to-text/models.md#melia-1).

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

* [A license file or a license token](/deployments/container/licensing.md)
  * There is no specific license for the GPU Inference Container, it will run using an existing Speechmatics license for the Batch Container
* [Access to our Docker repository](/deployments/container/accessing-images.md)

## System requirements[​](#system-requirements "Direct link to System requirements")

The system must have:

* Nvidia GPU(s) with at least 16GB of GPU memory
* Nvidia drivers (see below for supported versions)
* CUDA [compute capability](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-capabilities) of 7.5-12.1 inclusive, which corresponds to the Turing, Ampere, Lovelace, Hopper, Blackwell architectures. Cards with the Volta architecture or below are not able to run the models
* 24 GB RAM
* The [nvidia-container-toolkit](https://github.com/NVIDIA/nvidia-docker) installed
* Docker version > 19.03

See [Performance and cost](/deployments/container/performance-and-cost.md) for more information on the performance and cost of the container.

### Nvidia drivers[​](#nvidia-drivers "Direct link to Nvidia drivers")

* The GPU Inference Container is based on CUDA 13.0.1, which requires NVIDIA Driver release 580 or later.

Driver installation can be validated by running `nvidia-smi`. This command should return the Nvidia driver version and show additional information about the GPU(s).

#### Cloud instances[​](#cloud-instances "Direct link to Cloud instances")

The GPU node can be provisioned in the cloud.

## Running the image[​](#running-the-image "Direct link to Running the image")

Currently, each GPU Inference Container can only run on a single GPU. If a system has more than one GPU, the device must be specified using `CUDA_VISIBLE_DEVICES` or selecting the device using the `--gpus` argument. See [Nvidia/CUDA documentation for details](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-environment-variables).

Pull the Melia 1 inference server image as described in [Accessing images](/deployments/container/accessing-images.md#melia-1-model), then run it:

```
docker run --rm -it \
  -v $PWD/license.json:/license.json \
  --gpus '"device=0"' \
  -e CUDA_VISIBLE_DEVICES \
  -p 8001:8001 \
  speechmaticspublic.azurecr.io/sm-asr-inference-server-melia-1:1.3.0
```

When the Container starts you should see output similar to this, indicating that the server has started and is ready to serve requests.

```
I0705 08:12:55.419608 1 server.cc:709]
+--------------+---------+--------+
| Model        | Version | Status |
+--------------+---------+--------+
| aed          | 1       | READY  |
| body         | 1       | READY  |
| detokenizer  | 1       | READY  |
| dz           | 1       | READY  |
| ensemble     | 1       | READY  |
| preprocessor | 1       | READY  |
+--------------+---------+--------+
...
I0705 08:12:55.515473 1 grpc_server.cc:2579] "Started GRPCInferenceService at 0.0.0.0:8001"
I0705 08:12:55.515940 1 http_server.cc:4961] "Started HTTPService at 0.0.0.0:8000"
I0705 08:12:55.598672 1 http_server.cc:400] "Started Metrics Service at 0.0.0.0:8002"
```

### Batch inference[​](#batch-inference "Direct link to Batch inference")

The Melia 1 inference server currently runs in *batch* mode only, processing whole files and returning the transcript at the end.

Once the inference server is running, run the Melia 1 transcriber with the `SM_INFERENCE_ENDPOINT` environment variable set to the gRPC endpoint of the inference server. The transcriber offloads inference to the server, so it does not require a GPU.

The transcriber is a single unified image for both batch and real-time, so the `SM_ASR_MODE` environment variable must also be set, to batch or rt.

```
docker run --rm -it -e SM_INFERENCE_ENDPOINT=<server>:<port> -e SM_ASR_MODE=batch -v $PWD/license.json:/license.json -v $PWD/example.wav:/input.audio speechmaticspublic.azurecr.io/sm-asr-transcriber-melia-1:1.3.0
```

To accept multiple jobs over an HTTP API without restarting the container between jobs, run the transcriber as a [batch persistent worker](/deployments/container/batch-persistent-worker.md).

### Monitoring the server[​](#monitoring-the-server "Direct link to Monitoring the server")

The inference server is based on [Nvidia's Triton architecture](https://developer.nvidia.com/nvidia-triton-inference-server) and as such can be monitored using Triton's inbuilt Prometheus metrics, or the GRPC/HTTP APIs. To expose these, configure an external mapping for port 8002(Prometheus) or 8000(HTTP).

## Docker compose example[​](#docker-compose-example "Direct link to Docker compose example")

This Docker Compose file will create a Speechmatics Melia 1 GPU Inference Server:

(assumes your `license.json` file is in the current working directory)

docker-compose.yml

```
version: "3.8"

networks:
  transcriber:
    driver: bridge

services:
  triton:
    image: speechmaticspublic.azurecr.io/sm-asr-inference-server-melia-1:1.3.0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              ### Limit to N GPUs
              # count: 1
              ### Pick specific GPUs by device ID
              # device_ids:
              #   - 0
              #   - 3
              capabilities:
                - gpu
    container_name: triton
    networks:
      - transcriber
    expose:
      - 8000/tcp
      - 8001/tcp
      - 8002/tcp
    environment:
      - NVIDIA_DRIVER_CAPABILITIES=all
      - NVIDIA_VISIBLE_DEVICES=all
      - CUDA_VISIBLE_DEVICES=0
    volumes:
      - $PWD/license.json:/license.json:ro
```