# OpenAI eval run finished notifications on your phone

> Get a push notification when an OpenAI eval run finishes, fails or is canceled.

- Company: OpenAI (https://www.justpush.io/recipes/openai)
- Event: `eval.run.succeeded` (eval.run.succeeded, eval.run.failed and eval.run.canceled)
- Tags: AI & LLMs, AI evals & experiments
- Install: https://studio.justpush.io/recipes/openai/eval-run
- Web page: https://www.justpush.io/recipes/openai/eval-run

## Setup

1. Click Install in Studio and sign in. The recipe is added to your account and you get a personal webhook URL.
2. In the OpenAI platform, open Settings > Project > Webhooks (platform.openai.com/settings/project/webhooks) and click Create.
3. Give the endpoint a name and paste your webhook URL (shown in Studio after install) as the URL.
4. Tick the eval.run.succeeded, eval.run.failed and eval.run.canceled event types and save. Webhooks are per project, so repeat this for each project you want pushes from.
5. Tick **eval.run.succeeded**, **eval.run.failed** and **eval.run.canceled** (or only the ones you want).
6. OpenAI's webhook only carries the run ID, not the eval it belongs to or its scores. "Succeeded" means the run finished, not that your tests passed, so open the eval to see the results. Via the API, list your evals and fetch the run with `GET /v1/evals/{eval_id}/runs/{run_id}`.
7. Only want to hear about failures? Add `?failures=1` to the URL:
8. ```
9. your webhook URL (shown in Studio after install)?failures=1
10. ```
11. OpenAI signs every delivery (Standard Webhooks: `webhook-id`, `webhook-timestamp`, `webhook-signature`), but JustPush doesn't check that signature yet, so keep your endpoint URL private.

## Code

Studio calls `handleRequest(request)` with the incoming webhook (`{ method, headers, body, query, raw }`) and sends the message object it returns. Returning `null` sends nothing.

```js
// OpenAI → eval.run.succeeded / eval.run.failed / eval.run.canceled
// Fires when an eval run finishes, fails or is canceled.
// Docs: https://developers.openai.com/api/reference/resources/webhooks
//
// OpenAI only sends the run ID (data.id), not the eval ID or the scores. "Succeeded" means the
// run finished, not that your tests passed; open the eval to see the results.

// Each event, how to word it, and how loud to be. Note OpenAI spells it "canceled" here.
const EVENTS = {
    "eval.run.succeeded": {
        title: "✅ Eval run finished",
        text: "finished. The results are ready to review",
        priority: -1, // Low: good news you were waiting for
        failure: false,
    },
    "eval.run.failed": {
        title: "❌ Eval run failed",
        text: "failed before it could score everything",
        priority: 1, // High: no results to act on
        failure: true,
    },
    "eval.run.canceled": {
        title: "🚫 Eval run canceled",
        text: "was canceled",
        priority: -1, // Low: someone canceled it on purpose
        failure: false,
    },
}

// Table lookup that ignores inherited names such as "toString".
function pick(table, key, fallback) {
    return typeof key === "string" && Object.prototype.hasOwnProperty.call(table, key) ? table[key] : fallback
}

// Only link to real http(s) URLs.
function link(url, cta) {
    return typeof url === "string" && /^https?:\/\//.test(url) ? [{ cta, url }] : []
}

// OpenAI IDs look like "evalrun_abc123"; anything else isn't used in text.
function cleanId(value) {
    return typeof value === "string" && /^[A-Za-z0-9_-]{1,128}$/.test(value) ? value : null
}

function handleRequest(request) {
    const body = request.body && typeof request.body === "object" ? request.body : {}
    const event = pick(EVENTS, body.type, null)

    // Guard: only eval run events, so other OpenAI events on this integration stay quiet.
    if (!event) {
        console.log(`Ignoring OpenAI "${typeof body.type === "string" ? body.type : "unknown"}" event`)
        return null
    }

    // ?failures=1 on the webhook URL only pushes failed runs.
    const failuresOnly = ["1", "true", "yes"].includes(String(request.query?.failures ?? "").toLowerCase())
    if (failuresOnly && !event.failure) {
        console.log(`Skipping ${body.type} (failures only)`)
        return null
    }

    const id = cleanId(body.data?.id)

    return {
        title: event.title,
        message: `${id ? `Run ${id}` : "An eval run"} ${event.text}.`,
        topic: "OpenAI",
        priority: event.priority,
        // The webhook has no eval ID, so link to the evaluations list rather than the run itself.
        buttons: link("https://platform.openai.com/evaluations", "Open evals"),
    }
}
```

## OpenAI eval.run.succeeded webhook payload (sample)

```json
{
  "id": "evt_685343a1381c819085d44c354e1b3311",
  "data": {
    "id": "evalrun_abc123"
  },
  "type": "eval.run.failed",
  "object": "event",
  "created_at": 1790668800
}
```

## FAQ

### Does this work on iPhone and Android?

Yes. Install the JustPush app from the App Store or Google Play and sign in. Every phone signed in to your account gets the notification.

### Do I need to write code?

No. Install the recipe in Studio, paste your webhook URL into OpenAI and you are done. The code is there if you want to change the text, the sound or the buttons.

### Can I change what the notification says?

Yes. After install the recipe's code is yours. Edit it in Studio and test it against the sample payload before you save.

### What does it cost?

JustPush is free for 30 days. After that it's $19.99 a year, or $39.99 once. There is no extra charge for recipes.