Notifications

Releval can report an evaluation run's outcome to a Slack channel as soon as its metrics are in, carrying the headline metrics and how each one moved against the previous run.

One Slack connection serves the whole deployment. Each notification then picks its own channel, so different evaluations can report to different teams without connecting Slack more than once.

Info

Releval uses a Slack bot token, not an incoming webhook. An incoming webhook URL is permanently bound to the channel it was created for and cannot be redirected, so one webhook could never serve more than one channel.

Create the Slack app

You create the app in your own Slack workspace, so no data passes through a third party and Releval needs no inbound access from the internet.

1. Go to api.slack.com/apps and choose Create New App, then From a manifest.
2. Pick the workspace to install into.
3. Paste the manifest below, then choose Next and Create.
4. On Install App, choose Install to Workspace and approve the permissions.
5. Copy the Bot User OAuth Token. It starts with xoxb-.

display_information:
  name: Releval
  description: Reports evaluation run outcomes from Releval
  background_color: "#1a202c"
features:
  bot_user:
    display_name: Releval
    always_online: false
oauth_config:
  scopes:
    bot:
      - chat:write
      - chat:write.public
      - channels:read
      - groups:read
settings:
  org_deploy_enabled: false
  socket_mode_enabled: false
  is_hosted: false
  token_rotation_enabled: false

The scopes do the following:

Scope	Why it is needed
chat:write	Post messages as the app.
chat:write.public	Post to any public channel without being invited to it first.
channels:read	List public channels, so Releval can show you a channel picker instead of asking for an id.
groups:read	List private channels the app has been invited to, so they appear in the picker too.

Note

chat:write.public covers public channels only. To report into a private channel, invite the app to it in Slack first with /invite @Releval.

If you omit groups:read, the picker still works but lists public channels only. Slack fails the whole channel listing when private channels are requested without it, so Releval asks again for public channels alone rather than showing you nothing.

Connect it to Releval

1. Sign in to Releval as an Admin.
2. Go to Settings → Notifications and choose Connect Slack.
3. Give the connection a name and paste the bot token.

Releval verifies the token with Slack before saving it, so a mistyped or revoked token is rejected there and then rather than failing silently at the first run.

Add a notification

With Slack connected, choose Create and pick:

• Channel — chosen from the channels the app can see.
• Applies to — a single evaluation, or every evaluation.
• Notify on — run completed, run failed, run stopped, in any combination.

Add one notification per route. Several notifications can share the same Slack connection, which is how you send one evaluation to #search-team and another to #ml-team.

Use Send test to post a sample message and confirm the channel, the permissions, and the message layout before a real run depends on them.

Notification settings

Messages link back to the run, the evaluation, the endpoint and the query set.

Info

Set Releval__BaseUrl to get those links. Notifications are sent from a background service, which has no incoming web request to derive your deployment's address from, so there is nothing to infer it from. Without it, messages are still delivered but every name appears as plain text and the View run button is omitted. A warning is logged each time a notification is built without it.

Variable	Default	Description
Releval__BaseUrl	(none)	The absolute base URL members use to reach this deployment. Required for the links in a message. Deployment-wide rather than a notification setting: invitation and password-reset emails are built from it too.
Notifications__MetricsSettleTimeout	00:05:00	How long to hold a completed-run notification while its metrics are still being calculated. Past this, the message is sent without them rather than not at all.
Notifications__MaxMetrics	6	How many run-level metrics to include in a message.
Notifications__DispatchInterval	00:00:10	How often pending notifications are checked for. Also how long one waiting on metrics sleeps between checks.
Notifications__SlackApiBaseUrl	https://slack.com/api/	The Slack Web API address. Change it only to route Slack traffic through an egress proxy.

Example

Environment Variable

environment:
  - Releval__BaseUrl=https://${RELEVAL_HOST}

appsettings.json

{
  "Releval": {
    "BaseUrl": "https://${RELEVAL_HOST}"
  }
}

Security

The bot token is encrypted at rest with the same data protection keys used for other stored credentials, and is never returned by the API. Read responses report only whether a token is configured.

When updating a connection, leaving the token field blank keeps the stored token; entering a value rotates it.