Comparison

Runqy vs Ray

Ray is a distributed compute framework. Runqy is a task queue. Both can run Python inference on GPUs you own or rent, but they solve different problems.

Ray splits a computation across a cluster. Runqy takes independent jobs from your application and hands each one to a worker, wherever that worker runs. Runqy is our project; this page says when Ray is the better tool.

What you get out of the box

Features available without writing glue code or adding another service.

Out of the boxRayRunqy
What it is Distributed compute framework: tasks, actors, Data, Serve, Train Task queue: jobs in, workers out
Submitting work Python driver; Jobs API and Ray Serve over HTTP REST API, CLI or Python client, from any language
Durable job queue✗ tasks live in the running cluster✓ Redis, survives restarts
Keeps a model loaded between jobs✓ actors✓ @load
GPU-aware placement✓ num_gpus, fractional GPUs✗ one worker per GPU, routed by queue
One job across several GPUs or nodes✓ ✗
Network between machines Nodes reach the head and each other on a range of ports Workers only connect out to the server and Redis
Ships your code to workers✓ runtime_env per job✓ git clone + deps per queue
Autoscaling machines✓ AWS, GCP, Azure, Kubernetes…✓ RunPod, Vast.ai
Roll out new code to running workers~ new job, new runtime_env✓ code_update, no restart of the machine
Priorities✗ ✓ weighted or strict sub-queues
Retries✓ max_retries✓ per queue
Execution timeout✗ handle it in your code✓ per queue, process restarted
Dashboard✓ ✓
Alerts~ via Prometheus and Grafana✓ email
Secrets for workers✗ ✓ encrypted vaults
License Apache-2.0 MIT

✓ built in · ~ partial or limited · ✗ not built in (you add a separate tool or write it yourself)

Choose Ray when

  • One job needs several GPUs or several nodes (tensor parallelism, distributed training).
  • You want GPU-aware scheduling: Ray places each task where the GPUs are free.
  • You process datasets in parallel with Ray Data, or serve models online with dynamic batching through Ray Serve.
  • Your machines share a network, such as one cloud region or one Kubernetes cluster.

Choose Runqy when

  • Your application submits independent jobs over HTTP and reads the results later.
  • Workers are scattered: a GPU at home, a lab server, rented machines. They only need outbound access.
  • Rented GPUs come and go, and you want RunPod or Vast.ai machines started and stopped with the queue.
  • You want a durable queue with priorities, timeouts, alerts and secrets, without building it on top of a framework.

The same GPU worker, both ways

Ray actor

import ray

@ray.remote(num_gpus=1)
class Generator:
    def __init__(self):
        self.pipe = load_pipeline()

    def generate(self, prompt):
        return self.pipe(prompt)

gen = Generator.remote()
image = ray.get(gen.generate.remote("a fox"))

The driver holds the reference. If the cluster goes down, pending work is gone.

Runqy task

from runqy_python import load, task, run

@load
def setup():
    return {"pipe": load_pipeline()}

@task
def generate(payload, ctx):
    return save(ctx["pipe"](payload["prompt"]))

run()
# Enqueue from anywhere:
# POST /queue/add {"queue": "images", "data": {...}}

Jobs wait in Redis until a worker takes them, with retries and timeouts per queue.

Using both

They also combine. A Runqy worker can start a Ray job inside a task when one request needs a multi-GPU computation, while Runqy keeps the queue, the priorities and the machines.

Try Runqy

Open source and self-hosted. Redis, the server and a worker run with one Docker Compose file.

Quickstart GPU autoscaling