OpenAI eval run finished notifications on your phone.
Get a push notification when an OpenAI eval run finishes, fails or is canceled.
Monday 5 October
9:41
JustPushnow
Eval run finished
Get a push notification when an OpenAI eval run finishes, fails or is canceled.
Three steps to your first push
STEP 01
Install the recipe
One click in Studio gives you a personal webhook URL.
STEP 02
Paste your URL into OpenAI
Add it as a webhook for eval.run.succeeded, eval.run.failed and eval.run.canceled.
STEP 03
Get a push on your phone
With the text, sound and buttons from the recipe.
How to set up the OpenAI eval.run.succeeded, eval.run.failed and eval.run.canceled webhook
- Click Install in Studio and sign in. The recipe is added to your account and you get a personal webhook URL.
- In the OpenAI platform, open Settings > Project > Webhooks (platform.openai.com/settings/project/webhooks) and click Create.
- Give the endpoint a name and paste your webhook URL as the URL.https://••••••••/•••••••• your personal URL, shown after install
- Tick the eval.run.succeeded, eval.run.failed and eval.run.canceled event types and save. Webhooks are per project, so repeat this for each project you want pushes from.
- Tick **eval.run.succeeded**, **eval.run.failed** and **eval.run.canceled** (or only the ones you want).
- OpenAI's webhook only carries the run ID, not the eval it belongs to or its scores. "Succeeded" means the run finished, not that your tests passed, so open the eval to see the results. Via the API, list your evals and fetch the run with `GET /v1/evals/{eval_id}/runs/{run_id}`.
- Only want to hear about failures? Add `?failures=1` to the URL:
- ```
- your webhook URL?failures=1
- ```
- OpenAI signs every delivery (Standard Webhooks: `webhook-id`, `webhook-timestamp`, `webhook-signature`), but JustPush doesn't check that signature yet, so keep your endpoint URL private.
What runs when the webhook arrives
Studio calls handleRequest(request) with the incoming webhook and sends the message it returns. It's yours after install; change anything.
transform.js
1// OpenAI → eval.run.succeeded / eval.run.failed / eval.run.canceled
2// Fires when an eval run finishes, fails or is canceled.
3// Docs: https://developers.openai.com/api/reference/resources/webhooks
4//
5// OpenAI only sends the run ID (data.id), not the eval ID or the scores. "Succeeded" means the
6// run finished, not that your tests passed; open the eval to see the results.
7
8// Each event, how to word it, and how loud to be. Note OpenAI spells it "canceled" here.
9const EVENTS = {
10 "eval.run.succeeded": {
11 title: "✅ Eval run finished",
12 text: "finished. The results are ready to review",
13 priority: -1, // Low: good news you were waiting for
14 failure: false,
15 },
16 "eval.run.failed": {
17 title: "❌ Eval run failed",
18 text: "failed before it could score everything",
19 priority: 1, // High: no results to act on
20 failure: true,
21 },
22 "eval.run.canceled": {
23 title: "🚫 Eval run canceled",
24 text: "was canceled",
25 priority: -1, // Low: someone canceled it on purpose
26 failure: false,
27 },
28}
29
30// Table lookup that ignores inherited names such as "toString".
31function pick(table, key, fallback) {
32 return typeof key === "string" && Object.prototype.hasOwnProperty.call(table, key) ? table[key] : fallback
33}
34
35// Only link to real http(s) URLs.
36function link(url, cta) {
37 return typeof url === "string" && /^https?:\/\//.test(url) ? [{ cta, url }] : []
38}
39
40// OpenAI IDs look like "evalrun_abc123"; anything else isn't used in text.
41function cleanId(value) {
42 return typeof value === "string" && /^[A-Za-z0-9_-]{1,128}$/.test(value) ? value : null
43}
44
45function handleRequest(request) {
46 const body = request.body && typeof request.body === "object" ? request.body : {}
47 const event = pick(EVENTS, body.type, null)
48
49 // Guard: only eval run events, so other OpenAI events on this integration stay quiet.
50 if (!event) {
51 console.log(`Ignoring OpenAI "${typeof body.type === "string" ? body.type : "unknown"}" event`)
52 return null
53 }
54
55 // ?failures=1 on the webhook URL only pushes failed runs.
56 const failuresOnly = ["1", "true", "yes"].includes(String(request.query?.failures ?? "").toLowerCase())
57 if (failuresOnly && !event.failure) {
58 console.log(`Skipping ${body.type} (failures only)`)
59 return null
60 }
61
62 const id = cleanId(body.data?.id)
63
64 return {
65 title: event.title,
66 message: `${id ? `Run ${id}` : "An eval run"} ${event.text}.`,
67 topic: "OpenAI",
68 priority: event.priority,
69 // The webhook has no eval ID, so link to the evaluations list rather than the run itself.
70 buttons: link("https://platform.openai.com/evaluations", "Open evals"),
71 }
72}
The OpenAI eval.run.succeeded webhook
This is what OpenAI sends to your URL for eval.run.succeeded, eval.run.failed and eval.run.canceled. It is a sample, trimmed to the fields recipes use.
eval.run.succeeded · sample.json
1{
2 "id": "evt_685343a1381c819085d44c354e1b3311",
3 "data": {
4 "id": "evalrun_abc123"
5 },
6 "type": "eval.run.failed",
7 "object": "event",
8 "created_at": 1790668800
9}
OpenAI eval run finished notifications: questions
Does this work on iPhone and Android?
Yes. Install the JustPush app from the App Store or Google Play and sign in. Every phone signed in to your account gets the notification.
Do I need to write code?
No. Install the recipe in Studio, paste your webhook URL into OpenAI and you are done. The code is there if you want to change the text, the sound or the buttons.
Can I change what the notification says?
Yes. After install the recipe's code is yours. Edit it in Studio and test it against the sample payload before you save.
What does it cost?
JustPush is free for 30 days. After that it's $19.99 a year, or $39.99 once. There is no extra charge for recipes.
Ready when you are
OpenAI on your phone in two minutes.
Install the recipe, paste one URL, done. Free for 30 days, no credit card required.