LangSmith

Connect LangSmith so CloudThinker can find failed or costly LLM runs, inspect traces, and review dataset and experiment results.

Connect a LangSmith workspace and ask CloudThinker why an LLM run failed, which runs cost the most yesterday, or how two experiments compare on latency, cost, and feedback. You need three values: an API key, a workspace ID, and your region’s API URL.

Prerequisites

• A LangSmith account with access to the workspace you want CloudThinker to read.
• Permission to create an API key in LangSmith. Only organization admins can create service keys.
• The Admin role in your CloudThinker workspace, which can add and edit connections. See Roles.

Setup

1. Create an API key

   In LangSmith, open Settings → API Keys. Pick the key type from Required permissions. For a workspace-scoped service key, select the workspace. Set an expiration, then click Create API Key.

2. Copy the key

   Copy the key now. LangSmith shows it only once.

3. Copy the workspace ID

   If you belong to more than one workspace, switch to the one CloudThinker should read first. Then, on the Settings page, copy the Workspace ID under General.

4. Add the connection in CloudThinker

   Open More → Workspace settings → Connections, stay on the Builtin Connections tab, and select LangSmith. Fill in all three fields from Connection details, including your region’s API URL, and click Connect.

   Success state: the LangSmith card shows Connected.

   Connected only means CloudThinker saved the connection. A bad key, workspace ID, or URL shows up only when CloudThinker first reads LangSmith, so run the verify prompt before you rely on it.

Connection details

Field	Required	Description	Example
LANGSMITH_API_KEY	Yes	Personal access token or service key from Settings → API Keys	<your-api-key>
LANGSMITH_WORKSPACE_ID	Yes	The workspace ID from Settings → General	<your-workspace-id>
LANGSMITH_ENDPOINT	Yes	Your region’s API URL, from the table below	https://api.smith.langchain.com

All three fields are required. LangSmith needs the workspace ID when a key covers more than one workspace, and it does no harm for a single-workspace key.

Region	API URL
GCP US (default)	https://api.smith.langchain.com
GCP EU	https://eu.api.smith.langchain.com
GCP APAC	https://apac.api.smith.langchain.com
AWS US	https://aws.api.smith.langchain.com

For a self-hosted LangSmith, use that deployment’s API URL. It must accept connections from the internet; behind a firewall, see Reaching private services.

Required permissions

Key type	Who it acts as	Use it when
Personal access token	The user who created it, with all of that user’s access	You’re testing on your own
Service key, workspace-scoped	Only the workspaces you pick	Recommended for a shared CloudThinker workspace
Service key, organization-scoped	Every workspace in the organization	Avoid unless CloudThinker needs every workspace

Create a workspace-scoped service key for the one workspace CloudThinker should read. On LangSmith Enterprise, give it the Workspace Viewer role for read-only access. Without a read-only role, the key can write, and only your Approvals settings decide whether a write runs without asking you.

Agent capabilities

Capability	What you get
Projects and runs	Failed, slow, or costly runs in a named project, ranked by errors, latency, or cost
Traces and threads	One trace, or the message history of a conversation thread
Datasets and examples	Your datasets and the examples inside one dataset
Experiments	Experiment results for a dataset, including latency, cost, and feedback
Prompts	Your prompts and one prompt’s template

CloudThinker summarizes what it finds. It doesn’t print full run inputs, outputs, or message bodies, and it doesn’t read LangSmith billing or usage data on this connection.

Verify the connection

List my LangSmith projects and datasets

Success state: you get project and dataset names that match your workspace in LangSmith. An empty list with no error is still a working connection: the workspace may just be empty.

Example prompts

Name the exact project or dataset. Ranking runs needs a project name, and experiments and examples need a dataset name.

Show the five most expensive runs in the support-bot project from the last day and what drove their cost
Find failed runs in the support-bot project and group them by error
Compare the latest experiments on the support-eval dataset by latency, cost, and feedback

Write access

CloudThinker reads by default. It writes only when you ask for that exact change by name in the conversation: pushing a prompt, creating or updating a dataset or its examples, or starting an experiment.

Every change an agent makes goes through your workspace’s Approvals first. On the default Auto · Rules + AI · Ask me mode, AI judgment can let a safe change run without asking — and an Allow Rule can too, once CloudThinker turns on Rule enforcement for your workspace. Choose Manual to approve every change yourself.

To rule out writes entirely, use a key with a read-only role (see Required permissions).

Troubleshooting

401 Unauthorized

The key is missing, mistyped, expired, or deleted. Expired LangSmith keys can’t be reactivated. Create a new key, then open More → Workspace settings → Connections, select LangSmith, and replace LANGSMITH_API_KEY.

403 Forbidden

The key can’t read that workspace. An organization-scoped key needs the right LANGSMITH_WORKSPACE_ID, and a workspace-scoped key only works for the workspaces you picked. Check both, plus the key’s role.

Requests fail or lists come back empty, but the key is right

LANGSMITH_ENDPOINT may not match your workspace’s region. Set the endpoint to the URL for your region in Connection details.

404, or a project or dataset isn't found

The name doesn’t exist in this workspace. Ask CloudThinker to list your projects or datasets first, then use an exact name.

429 Too Many Requests

LangSmith limits how many requests a key can make per minute. Wait a minute, then ask a narrower question, for example one project and a shorter time window.

Security

• Least privilege — grant only the permissions the agents need for your use case; start read-only and widen later.
• Read-only by default — use read-only credentials unless you want agents to make changes through this connection.
• Rotate credentials — rotate keys and tokens on your normal schedule; CloudThinker picks up the new value when you update the connection.
• Revoke on offboarding — remove the credential at the provider when you delete a connection or a teammate leaves.

• Workspace-scoped key: limit the key to one workspace so others stay out of reach.
• Expiration: set one on the key and replace the key before it lapses.

Related

Langfuse

Add LLM traces and prompts from Langfuse

Commands and Prompt Syntax

Point a question at one connection with #