Feature

GPU autoscaling on RunPod and Vast.ai

Rent GPUs when the queue fills up, release them when it is quiet. Runqy's server watches each queue, rents machines from RunPod or Vast.ai, and destroys them once their workers are idle.

Rented machines boot the runqy worker image, register with your server, clone your task code and start taking jobs, like any machine you own. Mix them freely with the GPU under your desk.

How it works

  1. 1. Watch

    Every 30 seconds, the server reads each queue's pending jobs, latency and live workers.

  2. 2. Rent

    A trigger fires: the server rents a machine matching your GPU, disk and price cap.

  3. 3. Work

    The worker boots, registers, deploys your code from Git and takes jobs from the queue.

  4. 4. Release

    Once idle, the worker is drained and the machine destroyed. You stop paying.

One block in your queue config

Autoscaling is set per queue, next to its deployment. The same settings are editable in the dashboard.

queues:
  inference:
    deployment:
      git_url: "https://github.com/you/models.git"
      startup_cmd: "python main.py"
    autoscale:
      enabled: true
      provider: gpu-vast
      min_workers: 0
      max_workers: 4
      scale_up:
        - trigger: no_workers
        - trigger: queue_depth
          threshold: 50
        - trigger: latency
          max_latency: "5m"
      scale_down:
        - trigger: idle
          timeout: "10m"
      instance:
        gpu: "RTX 4090"
        disk_gb: 50
        max_price_per_hour: 0.60

Scale up

  • no_workers Jobs are pending and no worker serves the queue: start one machine.
  • queue_depth More than N jobs pending: add a machine.
  • latency The oldest pending job has waited too long: add a machine.
  • schedule Cron: pre-warm machines before your peak, for a set duration.

Scale down

  • idle A worker has not run a task for a while: release its machine.
  • schedule Cron: go back to the minimum, for example every night.

Providers

RunPod: a list of acceptable GPU types, rented in order of preference. Vast.ai: the cheapest verified offer under your price cap, with filters on country, bandwidth and CUDA version. A mock provider lets you test your triggers without renting anything.

Built not to waste your money

An autoscaler spends real money, so it errs on the side of not leaking machines and not killing work.

Busy workers are never destroyed

A worker running a task is spared, even by a schedule. On scale-down, the worker is drained first and its machine is destroyed once it reports drained.

No stacking

While a machine is booting, no other one is added for the same load. Failed rentals back off exponentially, up to 10 minutes.

No leaked machines

A machine without a ready worker after the boot timeout, dead at the provider, or silent for 5 minutes is destroyed.

Only its own machines

Runqy only reads and destroys instances it created. Hand-made machines on the same account are left alone.

No admin key on rented hosts

Rented machines receive a worker-scoped API key. Provider credentials are stored encrypted on your server.

Survives a server restart

Backoffs and Vast.ai hosts to avoid are stored in Redis, and one server runs the loop at a time.

What you need

  • A runqy server reachable from the internet, and a Redis protected with a password and TLS: rented machines connect to both.
  • A RunPod or Vast.ai account. Other providers are not supported yet.
  • Patience on the first boot: the worker image weighs several GB, and your models load on top.

Set it up

Register a provider, add an autoscale block, and watch instances in the dashboard or with runqy autoscale status.

Autoscaling guide Runqy vs Ray