Run and observe
Report errors from your services
Let your running services report the errors a health check can't see, such as a failed payment, so the map and the Connections view show which systems are in trouble right now.
Open Systems in the appWhy use it
A health check only sees whether your service answers. It can't see a payment that failed or a queue that is backing up while the service still answers. Reported signals close that gap: your service tells Rabbinatix about those problems itself, as they happen.
You don't have to use it. Without a reporting key nothing is sent, and health checks work the same. Start with health checks for every system, and add reporting where a hidden failure would hurt, such as payments or checkout. On the Health checks tab, the ? next to Reported by the system explains the same.
What reported signals are
Each signal has a level:
Repeated signals are grouped into one alert: the same level, code, environment and message count together, even when numbers or ids in the message change. An alert shows how many times it happened and when it was first and last seen.
Make a reporting key
- Open the system and go to Running → Health checks.
- Under Reported by the system, click Make a reporting key.
- Copy the key. It is shown only once. Put it in your service's configuration, for example as an environment variable.
A key can only send signals for its own system. Rotate key makes a new one and stops the old one at once. Revoke key stops it without a new one; the alerts stay. You need permission to edit systems to manage keys and alerts.
Send a signal
Send POST /signal on this site with the header Authorization: Bearer <your key> and a JSON body:
{"level": "error", "message": "Payment provider timed out", "code": "PAY_TIMEOUT", "environment": "production", "source": "checkout"}Only level and message are required. You can also send details (an object of up to 4 KB, kept with the signal) and at (when it happened). Send a list to report up to 50 signals at once. The Health checks tab has ready-to-copy examples for curl, JavaScript or TypeScript, Python, C# and Java.
C#
A small helper for .NET 6 or later. Put the key in configuration rather than in code:
using System.Net.Http.Json;
using System.Text.Json;
using System.Text.Json.Serialization;
public static class Signals
{
private static readonly HttpClient Http = new() { Timeout = TimeSpan.FromSeconds(3) };
private static readonly JsonSerializerOptions Json = new() { DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull };
public static async Task ReportAsync(string level, string message, string? code = null, string? source = null, object? details = null)
{
try
{
using var req = new HttpRequestMessage(HttpMethod.Post, "https://<this site>/signal")
{
Content = JsonContent.Create(new { level, message, code, source, details }, options: Json),
};
req.Headers.Add("Authorization", "Bearer " + Environment.GetEnvironmentVariable("RABBINATIX_SIGNAL_KEY"));
await Http.SendAsync(req);
}
catch
{
// reporting is best effort
}
}
}
// in a catch block:
await Signals.ReportAsync("error", ex.Message, code: "PAY_TIMEOUT", source: "checkout");Java
A small helper for Java 11 or later, with no libraries. It sends in the background, so reporting never slows your service:
public final class Signals {
private static final HttpClient HTTP = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(3)).build();
public static void report(String level, String message, String code, String source) {
try {
String body = "{\"level\":" + json(level) + ",\"message\":" + json(message)
+ (code != null ? ",\"code\":" + json(code) : "") + (source != null ? ",\"source\":" + json(source) : "") + "}";
HttpRequest req = HttpRequest.newBuilder(URI.create("https://<this site>/signal"))
.timeout(Duration.ofSeconds(3))
.header("Authorization", "Bearer " + System.getenv("RABBINATIX_SIGNAL_KEY"))
.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(body))
.build();
HTTP.sendAsync(req, HttpResponse.BodyHandlers.discarding());
} catch (RuntimeException e) {
// reporting is best effort
}
}
// json(s) quotes a string; the Health checks tab has the full helper
}
// in a catch block:
Signals.report("error", e.getMessage(), "PAY_TIMEOUT", "checkout");Messages are cut at 1,000 characters. A key may send 120 requests a minute; past that, the answer is 429 with a Retry-After header, so batch your signals. Make reporting best effort: never let a failed report break your service.
Alerts and what they show
While a system has an active error or warning:
- A badge with an icon and a count shows on its circle in the Connections view, on its node on the map, and on the systems list. Red is for errors, amber for warnings only.
- The Connections view shows a banner naming the systems that report problems. Each name opens the system's health checks (Running → Health checks). Focus a system to see its latest alerts in the side panel.
- An error makes the ring in the Connections view at least Degraded. Down and Maintenance stay as the health check found them, and the status dot elsewhere is still the health check's.
Acknowledge, resolve and go quiet
On Running → Health checks, each alert has Acknowledge (someone is looking at it) and Resolve (it is fixed). Resolve all closes every alert. If the same error comes back after a resolve, the alert opens again.
An alert also goes quiet when no new signal arrives for a while. Choose how long under Quiet after: from 5 minutes to a day, or never. A quiet alert stops showing on the map until it happens again.
Your service can resolve alerts itself: send {"level": "ok", "message": "Recovered"}. With a code, it resolves only the alerts with that code.
Notifications
A new alert, or one that opens again, notifies people with a link to the system's health checks. Errors count as breaking, so "Breaking only" preferences still get them. Turn them off under Errors and warnings reported by services in your notification preferences. Webhooks can subscribe to health.signal.
Related
- Health checksTell Rabbinatix how to see whether a service is up in each environment. It calls each environment’s health URL while someone has the Connections view open, and shows the service as operational, degraded, down or in maintenance there.
- The system mapYour systems and the connections between them on one canvas, coloured by contract health, with a table and a Connections view.
- NotificationsHear about the events that matter to you: in the app or by email, with a level for each kind of event.
- IntegrationsSend contract and ticket events to your own endpoint or an email address with webhooks, and see each delivery and its retries.
