openclaw-langsmith

LangSmith tracing plugin for OpenClaw. Automatically traces agent turns, tool calls, and LLM invocations to LangSmith for observability, debugging, and cost tracking.

Features

• Agent turn tracing — Each agent turn becomes a LangSmith run with prompt, response, and token usage
• Token tracking — Prompt tokens, completion tokens, and total tokens displayed in LangSmith dashboard
• Smart tagging — Auto-tags traces with source (cron, discord, slack, telegram), job names, channel IDs
• Tool call tracing — Tool calls nested under their parent agent run
• Engram LLM tracing — Memory extraction/consolidation calls appear as LLM runs with full prompts
• Batch queue — Operations batched for efficient API usage (configurable interval and size)
• Per-feature toggles — Enable/disable each trace type independently
• Zero runtime dependencies — Uses native fetch and crypto.randomUUID()
• Error isolation — Tracing errors never affect gateway operation

Quick Start

1. Get a LangSmith API Key

1. Sign up at smith.langchain.com
2. Go to Settings > API Keys
3. Create a new API key (starts with lsv2_pt_...)

2. Install the Plugin

cd ~/.openclaw/extensions
git clone https://github.com/joshuaswarren/openclaw-langsmith.git
cd openclaw-langsmith
npm install && npm run build

3. Add API Key to Gateway Environment

The gateway needs the API key in its environment. Choose your platform:

Edit ~/Library/LaunchAgents/ai.openclaw.gateway.plist and add inside EnvironmentVariables:

<key>LANGSMITH_API_KEY</key>
<string>lsv2_pt_your_key_here</string>

Edit ~/.config/systemd/user/openclaw-gateway.service and add to the [Service] section:

Environment="LANGSMITH_API_KEY=lsv2_pt_your_key_here"

Or create an environment file at ~/.config/openclaw/env:

LANGSMITH_API_KEY=lsv2_pt_your_key_here

Then reference it in the service file:

EnvironmentFile=%h/.config/openclaw/env

Add to your docker-compose.yml or pass via -e:

environment:
  - LANGSMITH_API_KEY=lsv2_pt_your_key_here

4. Enable in openclaw.json

{
  "plugins": {
    "allow": ["openclaw-langsmith"],
    "entries": {
      "openclaw-langsmith": {
        "enabled": true,
        "config": {
          "langsmithApiKey": "${LANGSMITH_API_KEY}",
          "projectName": "openclaw"
        }
      }
    }
  }
}

5. Restart Gateway

macOS:

launchctl kickstart -k gui/$(id -u)/ai.openclaw.gateway

Linux:

systemctl --user restart openclaw-gateway

Docker:

docker compose restart openclaw-gateway

Verify (all platforms):

tail -f ~/.openclaw/logs/gateway.log | grep langsmith
# Should see: [langsmith] langsmith tracing active

Configuration

Filtering Traces

Traces are automatically tagged for easy filtering in LangSmith:

How It Works

Agent Turns

Hooks into before_agent_start and agent_end. Creates LangSmith runs with:

• Prompt content
• Response messages
• Token usage (prompt, completion, total)
• Duration
• Auto-generated tags based on session source

Tool Calls

Hooks into before_tool_call and after_tool_call. Tool runs are nested under the parent agent run using LangSmith's trace_id and dotted_order for proper hierarchy.

Engram LLM Calls

The engram memory plugin emits LlmTraceEvent objects via globalThis.__openclawEngramTrace. This plugin subscribes and creates LLM runs with:

• Full prompt text (no truncation)
• Full output text (no truncation)
• Model and operation type
• Token usage and duration

Error Isolation

• No API key? No problem. If you install the plugin without configuring an API key, it simply logs a warning and disables itself — OpenClaw continues running normally
• All LangSmith API calls wrapped in try/catch
• Network failures log warnings but never affect gateway operation
• Invalid API keys or LangSmith outages won't break your agents

Development

npm install
npm run build    # Build with tsup
npm run dev      # Watch mode

Related Projects

• OpenClaw — The AI agent gateway
• openclaw-engram — Local-first memory plugin
• LangSmith — LLM observability platform

License

MIT © Joshua Warren