Skip to main content

Deploying LLM Inference

Targon supports GPU-backed LLM inference on dedicated rentals. This guide covers image selection, port configuration, and persistence patterns for production serving.

Overview​

A typical LLM rental deployment includes:

  • A GPU rental (h200-small or larger depending on model size).
  • A container image with your inference server (vLLM, SGLang, TensorRT-LLM, etc.).
  • A service port exposed for HTTP traffic (for example 8080).
  • A volume for model weights so downloads survive container restarts.

Choose GPU Resources​

Pick a tier based on model size and throughput:

  • h200-small: Small and medium models, single-GPU inference.
  • h200-medium: Larger models or higher concurrency.
  • h200-large / h200-xlarge: Multi-GPU workloads or very large models.

Run targon inventory --gpu to see what is available. See the Compute reference for the full list.

Create the Rental​

Dashboard​

  1. Configure credentials and open the dashboard.
  2. Create a Volume for model weights if the model is larger than a few gigabytes.
  3. Go to Rentals → Create Rental.
  4. Select a GPU configuration and either:
    • Use a public template, or
    • Choose Custom and specify your inference image (for example lmsysorg/sglang:latest or a private image you built).
  5. Set Commands and Arguments to start your inference server, or bake the startup command into the image.
  6. Expose the service port your server listens on (for example 8080).
  7. Mount the volume at your model cache path (for example /root or /data).
  8. Deploy and connect via SSH or the proxied URL shown on the workload.

See the Rentals guide and the DeepSeek deployment walkthrough for step-by-step examples.

API​

Register and deploy a rental workload with the Workloads API:

curl -X POST https://api.targon.com/tha/v2/workloads \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "vllm-inference",
    "image": "my-registry/vllm-server:latest",
    "resource_name": "h200-small",
    "type": "RENTAL",
    "ports": [
      {"port": 8080, "protocol": "TCP", "routing": "PROXIED"}
    ],
    "volumes": [
      {"uid": "vol-<UID>", "mount_path": "/data"}
    ]
  }'

Use targon workload get with the workload UID to inspect status and URLs.

Build a Custom Image​

Build a container image with a Dockerfile or your preferred tooling, push it to a registry, and reference the tag when creating your rental or workload. A typical inference image starts from a CUDA base, installs your server (vLLM, SGLang, etc.), and sets environment variables such as HF_HUB_ENABLE_HF_TRANSFER=1 for faster model downloads.

Production Tips​

  • Pin package and model versions to avoid breaking changes.
  • Store weights on a volume rather than in the container filesystem.
  • Pre-download models to the volume on first boot to speed up subsequent restarts.
  • Monitor GPU utilization and upgrade tiers only when needed.

For managed inference without operating rentals yourself, use Sybil.com, powered by Targon.