# LangSmith automation rule notifications on your phone

> Get a push notification when runs match a LangSmith automation rule, such as runs that error or get negative feedback.

- Company: LangSmith (https://www.justpush.io/recipes/langsmith)
- Event: `automation` (Automation rule)
- Tags: AI, Monitoring
- Install: https://studio.justpush.io/recipes/langsmith/automation-rule
- Web page: https://www.justpush.io/recipes/langsmith/automation-rule

## Setup

1. Click Install in Studio and sign in. The recipe is added to your account and you get a personal webhook URL.
2. Follow the setup notes below: Automation rule is configured in a different part of LangSmith than the other recipes.
3. Paste your webhook URL (shown in Studio after install) (plus any query parameters from the notes) as the webhook URL and save.
4. In your tracing project, go to **Automations → + Rule**, set a filter (for example *Error = true*, or a negative feedback score), and add the **Trigger webhook** action. Use this URL, with your rule's name so the push can show it:
5. ```
6. your webhook URL (shown in Studio after install)?rule=Failed%20runs
7. ```
8. (`%20` is a space.) Keep the **sampling rate** low for busy filters: LangSmith batches matching runs per polling window, and each batch is one push.
9. Links go to `smith.langchain.com`. For the EU region or a self-hosted LangSmith, change `LANGSMITH_URL` at the top of the code. LangSmith doesn't sign these webhooks, so keep your endpoint URL private.

## Code

Studio calls `handleRequest(request)` with the incoming webhook (`{ method, headers, body, query, raw }`) and sends the message object it returns. Returning `null` sends nothing.

```js
// LangSmith → automation rule
// Fires when a LangSmith automation rule with a "Trigger webhook" action matches runs
// (for example: every run that errors, or every run with negative feedback).
// Docs: https://docs.langchain.com/langsmith/webhooks
//
// LangSmith batches: one request can carry many runs, collected over a polling window.
// The payload only has the rule's ID, so name it in the URL: ?rule=Failed%20runs

// Where "View run" links go. Change this for the EU region (https://eu.smith.langchain.com)
// or a self-hosted LangSmith.
const LANGSMITH_URL = "https://smith.langchain.com"

// Only link to real http(s) URLs.
function link(url, cta) {
    return typeof url === "string" && /^https?:\/\//.test(url) ? [{ cta, url }] : []
}

function handleRequest(request) {
    const body = request.body || {}
    const runs = Array.isArray(body.runs) ? body.runs : []
    const threads = Array.isArray(body.threads) ? body.threads : []

    // Guard: only automation rule payloads with something in them.
    if (!body.rule_id || (!runs.length && !threads.length)) {
        console.log("Not a LangSmith automation payload (or an empty batch) — skipping")
        return null
    }

    const rule = typeof request.query?.rule === "string" && request.query.rule.trim()
        ? request.query.rule.trim()
        : "Automation rule"

    // Thread rules send conversations instead of runs.
    if (!runs.length) {
        return {
            title: `🧵 ${rule}`,
            message: `${threads.length} thread${threads.length === 1 ? "" : "s"} matched`,
            topic: "LangSmith",
            priority: 0,
        }
    }

    const failed = runs.filter((run) => run.status === "error")
    const first = failed[0] ?? runs[0]
    const what = [first.name, first.run_type ? `(${first.run_type})` : null].filter(Boolean).join(" ") || "A run"
    const more = runs.length > 1 ? ` + ${runs.length - 1} more` : ""
    const url = typeof first.app_path === "string" && first.app_path.startsWith("/")
        ? LANGSMITH_URL + first.app_path
        : null

    return {
        title: `${failed.length ? "❌" : "🔎"} ${rule}`,
        message: `${what} ${first.status === "error" ? "failed" : "matched"}${more}`,
        topic: "LangSmith",
        priority: failed.length ? 1 : 0, // High when runs are failing
        buttons: link(url, "View run"),
    }
}
```

## LangSmith automation webhook payload (sample)

```json
{
  "runs": [
    {
      "id": "6ab80f10-d79c-4fa2-b441-922ed6feb630",
      "name": "AgentExecutor",
      "inputs": null,
      "status": "error",
      "is_root": true,
      "outputs": null,
      "app_path": "/o/2ebda79f-1c2d-4e5f-8a9b-0c1d2e3f4a5b/projects/p/6a3be6a2-9a8c-4fc8-b4c6-a8983b286cc4/r/6ab80f10-d79c-4fa2-b441-922ed6feb630?trace_id=6ab80f10-d79c-4fa2-b441-922ed6feb630",
      "end_time": "2026-09-29T09:28:54.085649",
      "run_type": "chain",
      "trace_id": "6ab80f10-d79c-4fa2-b441-922ed6feb630",
      "session_id": "6a3be6a2-9a8c-4fc8-b4c6-a8983b286cc4",
      "start_time": "2026-09-29T09:28:51.085646",
      "total_cost": 0.0041,
      "total_tokens": 1830,
      "feedback_stats": null
    },
    {
      "id": "7bc91a21-e8ad-4fb3-c552-033fe7ffc741",
      "name": "AgentExecutor",
      "inputs": null,
      "status": "error",
      "is_root": true,
      "outputs": null,
      "app_path": "/o/2ebda79f-1c2d-4e5f-8a9b-0c1d2e3f4a5b/projects/p/6a3be6a2-9a8c-4fc8-b4c6-a8983b286cc4/r/7bc91a21-e8ad-4fb3-c552-033fe7ffc741?trace_id=7bc91a21-e8ad-4fb3-c552-033fe7ffc741",
      "end_time": "2026-09-29T09:28:55.2",
      "run_type": "chain",
      "trace_id": "7bc91a21-e8ad-4fb3-c552-033fe7ffc741",
      "session_id": "6a3be6a2-9a8c-4fc8-b4c6-a8983b286cc4",
      "start_time": "2026-09-29T09:28:52.1",
      "total_cost": 0.0019,
      "total_tokens": 920,
      "feedback_stats": null
    }
  ],
  "rule_id": "d75d7417-0c57-4655-88fe-1db3cda3a47a",
  "end_time": "2026-09-29T09:28:56.492563+00:00",
  "start_time": "2026-09-29T09:28:54.734491+00:00"
}
```

## FAQ

### Does this work on iPhone and Android?

Yes. Install the JustPush app from the App Store or Google Play and sign in. Every phone signed in to your account gets the notification.

### Do I need to write code?

No. Install the recipe in Studio, paste your webhook URL into LangSmith and you are done. The code is there if you want to change the text, the sound or the buttons.

### Can I change what the notification says?

Yes. After install the recipe's code is yours. Edit it in Studio and test it against the sample payload before you save.

### What does it cost?

JustPush is free for 30 days. After that it's $19.99 a year, or $39.99 once. There is no extra charge for recipes.