LangSmith automation rule notifications on your phone.
Get a push notification when runs match a LangSmith automation rule, such as runs that error or get negative feedback.
Tuesday 29 September
9:41
JustPushnow
Automation rule
Get a push notification when runs match a LangSmith automation rule, such as runs that error or get negative feedback.
Three steps to your first push
STEP 01
Install the recipe
One click in Studio gives you a personal webhook URL.
STEP 02
Paste your URL into LangSmith
Add it as a webhook for Automation rule.
STEP 03
Get a push on your phone
With the text, sound and buttons from the recipe.
How to set up the LangSmith automation rule webhook
- Click Install in Studio and sign in. The recipe is added to your account and you get a personal webhook URL.
- Follow the setup notes below: Automation rule is configured in a different part of LangSmith than the other recipes.
- Paste your webhook URL (plus any query parameters from the notes) as the webhook URL and save.https://••••••••/•••••••• your personal URL, shown after install
- In your tracing project, go to **Automations → + Rule**, set a filter (for example *Error = true*, or a negative feedback score), and add the **Trigger webhook** action. Use this URL, with your rule's name so the push can show it:
- ```
- your webhook URL?rule=Failed%20runs
- ```
- (`%20` is a space.) Keep the **sampling rate** low for busy filters: LangSmith batches matching runs per polling window, and each batch is one push.
- Links go to `smith.langchain.com`. For the EU region or a self-hosted LangSmith, change `LANGSMITH_URL` at the top of the code. LangSmith doesn't sign these webhooks, so keep your endpoint URL private.
What runs when the webhook arrives
Studio calls handleRequest(request) with the incoming webhook and sends the message it returns. It's yours after install; change anything.
transform.js
1// LangSmith → automation rule
2// Fires when a LangSmith automation rule with a "Trigger webhook" action matches runs
3// (for example: every run that errors, or every run with negative feedback).
4// Docs: https://docs.langchain.com/langsmith/webhooks
5//
6// LangSmith batches: one request can carry many runs, collected over a polling window.
7// The payload only has the rule's ID, so name it in the URL: ?rule=Failed%20runs
8
9// Where "View run" links go. Change this for the EU region (https://eu.smith.langchain.com)
10// or a self-hosted LangSmith.
11const LANGSMITH_URL = "https://smith.langchain.com"
12
13// Only link to real http(s) URLs.
14function link(url, cta) {
15 return typeof url === "string" && /^https?:\/\//.test(url) ? [{ cta, url }] : []
16}
17
18function handleRequest(request) {
19 const body = request.body || {}
20 const runs = Array.isArray(body.runs) ? body.runs : []
21 const threads = Array.isArray(body.threads) ? body.threads : []
22
23 // Guard: only automation rule payloads with something in them.
24 if (!body.rule_id || (!runs.length && !threads.length)) {
25 console.log("Not a LangSmith automation payload (or an empty batch) — skipping")
26 return null
27 }
28
29 const rule = typeof request.query?.rule === "string" && request.query.rule.trim()
30 ? request.query.rule.trim()
31 : "Automation rule"
32
33 // Thread rules send conversations instead of runs.
34 if (!runs.length) {
35 return {
36 title: `🧵 ${rule}`,
37 message: `${threads.length} thread${threads.length === 1 ? "" : "s"} matched`,
38 topic: "LangSmith",
39 priority: 0,
40 }
41 }
42
43 const failed = runs.filter((run) => run.status === "error")
44 const first = failed[0] ?? runs[0]
45 const what = [first.name, first.run_type ? `(${first.run_type})` : null].filter(Boolean).join(" ") || "A run"
46 const more = runs.length > 1 ? ` + ${runs.length - 1} more` : ""
47 const url = typeof first.app_path === "string" && first.app_path.startsWith("/")
48 ? LANGSMITH_URL + first.app_path
49 : null
50
51 return {
52 title: `${failed.length ? "❌" : "🔎"} ${rule}`,
53 message: `${what} ${first.status === "error" ? "failed" : "matched"}${more}`,
54 topic: "LangSmith",
55 priority: failed.length ? 1 : 0, // High when runs are failing
56 buttons: link(url, "View run"),
57 }
58}
The LangSmith automation webhook
This is what LangSmith sends to your URL for automation rule. It is a sample, trimmed to the fields recipes use.
automation · sample.json
1{
2 "runs": [
3 {
4 "id": "6ab80f10-d79c-4fa2-b441-922ed6feb630",
5 "name": "AgentExecutor",
6 "inputs": null,
7 "status": "error",
8 "is_root": true,
9 "outputs": null,
10 "app_path": "/o/2ebda79f-1c2d-4e5f-8a9b-0c1d2e3f4a5b/projects/p/6a3be6a2-9a8c-4fc8-b4c6-a8983b286cc4/r/6ab80f10-d79c-4fa2-b441-922ed6feb630?trace_id=6ab80f10-d79c-4fa2-b441-922ed6feb630",
11 "end_time": "2026-09-29T09:28:54.085649",
12 "run_type": "chain",
13 "trace_id": "6ab80f10-d79c-4fa2-b441-922ed6feb630",
14 "session_id": "6a3be6a2-9a8c-4fc8-b4c6-a8983b286cc4",
15 "start_time": "2026-09-29T09:28:51.085646",
16 "total_cost": 0.0041,
17 "total_tokens": 1830,
18 "feedback_stats": null
19 },
20 {
21 "id": "7bc91a21-e8ad-4fb3-c552-033fe7ffc741",
22 "name": "AgentExecutor",
23 "inputs": null,
24 "status": "error",
25 "is_root": true,
26 "outputs": null,
27 "app_path": "/o/2ebda79f-1c2d-4e5f-8a9b-0c1d2e3f4a5b/projects/p/6a3be6a2-9a8c-4fc8-b4c6-a8983b286cc4/r/7bc91a21-e8ad-4fb3-c552-033fe7ffc741?trace_id=7bc91a21-e8ad-4fb3-c552-033fe7ffc741",
28 "end_time": "2026-09-29T09:28:55.2",
29 "run_type": "chain",
30 "trace_id": "7bc91a21-e8ad-4fb3-c552-033fe7ffc741",
31 "session_id": "6a3be6a2-9a8c-4fc8-b4c6-a8983b286cc4",
32 "start_time": "2026-09-29T09:28:52.1",
33 "total_cost": 0.0019,
34 "total_tokens": 920,
35 "feedback_stats": null
36 }
37 ],
38 "rule_id": "d75d7417-0c57-4655-88fe-1db3cda3a47a",
39 "end_time": "2026-09-29T09:28:56.492563+00:00",
40 "start_time": "2026-09-29T09:28:54.734491+00:00"
41}
LangSmith automation rule notifications: questions
Does this work on iPhone and Android?
Yes. Install the JustPush app from the App Store or Google Play and sign in. Every phone signed in to your account gets the notification.
Do I need to write code?
No. Install the recipe in Studio, paste your webhook URL into LangSmith and you are done. The code is there if you want to change the text, the sound or the buttons.
Can I change what the notification says?
Yes. After install the recipe's code is yours. Edit it in Studio and test it against the sample payload before you save.
What does it cost?
JustPush is free for 30 days. After that it's $19.99 a year, or $39.99 once. There is no extra charge for recipes.
Ready when you are
LangSmith on your phone in two minutes.
Install the recipe, paste one URL, done. Free for 30 days, no credit card required.