Feature

Dashboard and email alerts

See every queue, task and worker in one place, and get an email when something needs you: a queue with no worker, jobs piling up, a crashed GPU process, a machine that could not be rented.

Both ship with the server. No extra service to run, no separate monitoring stack to wire up.

The dashboard

Served by the server at /monitoring, behind a login. It reads live data from Redis.

Queues

Pending, active, retry and archived tasks per queue, and processed and failed counts over time.

Tasks

Inspect a task, its payload and result; retry or delete it.

Workers

Health, version, active tasks, last heartbeat, and the outcome of the last remote command.

Control

Drain, resume, update code or upgrade a worker from its page.

Autoscaling

Rented instances, their state and their cost, and the provider registry.

Alerts

Active alerts, sent notifications and email settings.

Email alerts

Global rules for every queue, with per-queue overrides to mute a queue or tighten a rule.

alerts:
  enabled: true
  provider: smtp          # or sendgrid
  from: "runqy <alerts@example.com>"
  recipients: [ops@example.com]
  smtp:
    host: smtp.example.com
    port: 587
    password: "vault://mail/smtp_password"
  rules:
    - type: no_workers
      for: 2m
    - type: latency
      max_latency: 10m
    - type: worker_unhealthy
    - type: autoscale_failure
  • no_workers Jobs are pending and no worker serves the queue.
  • latency The oldest pending job has waited longer than max_latency.
  • pending More than threshold jobs are waiting.
  • worker_unhealthy A worker's task process crashed or the worker is degraded.
  • autoscale_failure Renting a GPU machine failed: no offer, bad key, price too low.

Each rule can wait a for duration before firing. You get one digest email per check, a resolved email when it clears, and a reminder while it keeps firing. Paused queues are skipped.

From the terminal and Prometheus

runqy queue list
runqy worker list
runqy worker info 3fc7e7ee1e37:1
runqy alerts status
runqy alerts test

The CLI shows the same data, locally or against a remote server. If you already run Prometheus and Grafana, the server exposes queue and task metrics at /metrics.

Read the guides

Monitoring Email alerts