RabbinatixDocs
Sign inGet started

Run and observe

Report errors from your services

Let your running services report the errors a health check can't see, such as a failed payment, so the map and the Connections view show which systems are in trouble right now.

Open Systems in the app

Why use it

A health check only sees whether your service answers. It can't see a payment that failed or a queue that is backing up while the service still answers. Reported signals close that gap: your service tells Rabbinatix about those problems itself, as they happen.

Health checksReported signals
Who starts itRabbinatix calls your serviceYour service sends to Rabbinatix
WhenEvery few minutesAs soon as something goes wrong
What it findsDown, too slow, wrong answerFailures only the service knows about
What you seeStatus and response timeA message, a code and details
Works behind a firewallNo, Rabbinatix must reach the URLYes, the service only sends out

You don't have to use it. Without a reporting key nothing is sent, and health checks work the same. Start with health checks for every system, and add reporting where a hidden failure would hurt, such as payments or checkout. On the Health checks tab, the ? next to Reported by the system explains the same.

What reported signals are

Each signal has a level:

LevelMeaning
errorSomething failed, such as a payment that timed out
warningSomething is wrong but still working, such as a slow queue
infoFor the record only, such as a deployment. It never raises an alert
okIt works again. It resolves the open alerts

Repeated signals are grouped into one alert: the same level, code, environment and message count together, even when numbers or ids in the message change. An alert shows how many times it happened and when it was first and last seen.

Make a reporting key

  1. Open the system and go to Running → Health checks.
  2. Under Reported by the system, click Make a reporting key.
  3. Copy the key. It is shown only once. Put it in your service's configuration, for example as an environment variable.

A key can only send signals for its own system. Rotate key makes a new one and stops the old one at once. Revoke key stops it without a new one; the alerts stay. You need permission to edit systems to manage keys and alerts.

Send a signal

Send POST /signal on this site with the header Authorization: Bearer <your key> and a JSON body:

{"level": "error", "message": "Payment provider timed out", "code": "PAY_TIMEOUT", "environment": "production", "source": "checkout"}

Only level and message are required. You can also send details (an object of up to 4 KB, kept with the signal) and at (when it happened). Send a list to report up to 50 signals at once. The Health checks tab has ready-to-copy examples for curl, JavaScript or TypeScript, Python, C# and Java.

C#

A small helper for .NET 6 or later. Put the key in configuration rather than in code:

using System.Net.Http.Json;
using System.Text.Json;
using System.Text.Json.Serialization;

public static class Signals
{
    private static readonly HttpClient Http = new() { Timeout = TimeSpan.FromSeconds(3) };
    private static readonly JsonSerializerOptions Json = new() { DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull };

    public static async Task ReportAsync(string level, string message, string? code = null, string? source = null, object? details = null)
    {
        try
        {
            using var req = new HttpRequestMessage(HttpMethod.Post, "https://<this site>/signal")
            {
                Content = JsonContent.Create(new { level, message, code, source, details }, options: Json),
            };
            req.Headers.Add("Authorization", "Bearer " + Environment.GetEnvironmentVariable("RABBINATIX_SIGNAL_KEY"));
            await Http.SendAsync(req);
        }
        catch
        {
            // reporting is best effort
        }
    }
}

// in a catch block:
await Signals.ReportAsync("error", ex.Message, code: "PAY_TIMEOUT", source: "checkout");

Java

A small helper for Java 11 or later, with no libraries. It sends in the background, so reporting never slows your service:

public final class Signals {
    private static final HttpClient HTTP = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(3)).build();

    public static void report(String level, String message, String code, String source) {
        try {
            String body = "{\"level\":" + json(level) + ",\"message\":" + json(message)
                + (code != null ? ",\"code\":" + json(code) : "") + (source != null ? ",\"source\":" + json(source) : "") + "}";
            HttpRequest req = HttpRequest.newBuilder(URI.create("https://<this site>/signal"))
                .timeout(Duration.ofSeconds(3))
                .header("Authorization", "Bearer " + System.getenv("RABBINATIX_SIGNAL_KEY"))
                .header("Content-Type", "application/json")
                .POST(HttpRequest.BodyPublishers.ofString(body))
                .build();
            HTTP.sendAsync(req, HttpResponse.BodyHandlers.discarding());
        } catch (RuntimeException e) {
            // reporting is best effort
        }
    }

    // json(s) quotes a string; the Health checks tab has the full helper
}

// in a catch block:
Signals.report("error", e.getMessage(), "PAY_TIMEOUT", "checkout");

Messages are cut at 1,000 characters. A key may send 120 requests a minute; past that, the answer is 429 with a Retry-After header, so batch your signals. Make reporting best effort: never let a failed report break your service.

Alerts and what they show

While a system has an active error or warning:

  • A badge with an icon and a count shows on its circle in the Connections view, on its node on the map, and on the systems list. Red is for errors, amber for warnings only.
  • The Connections view shows a banner naming the systems that report problems. Each name opens the system's health checks (Running → Health checks). Focus a system to see its latest alerts in the side panel.
  • An error makes the ring in the Connections view at least Degraded. Down and Maintenance stay as the health check found them, and the status dot elsewhere is still the health check's.

Acknowledge, resolve and go quiet

On Running → Health checks, each alert has Acknowledge (someone is looking at it) and Resolve (it is fixed). Resolve all closes every alert. If the same error comes back after a resolve, the alert opens again.

An alert also goes quiet when no new signal arrives for a while. Choose how long under Quiet after: from 5 minutes to a day, or never. A quiet alert stops showing on the map until it happens again.

Your service can resolve alerts itself: send {"level": "ok", "message": "Recovered"}. With a code, it resolves only the alerts with that code.

Notifications

A new alert, or one that opens again, notifies people with a link to the system's health checks. Errors count as breaking, so "Breaking only" preferences still get them. Turn them off under Errors and warnings reported by services in your notification preferences. Webhooks can subscribe to health.signal.