Braintrust

Connect Braintrust so CloudThinker can compare experiment scores, read monitor views, and check project settings with one API key.

Connect your Braintrust organization and ask CloudThinker how an experiment scored against its baseline, what a saved monitor view shows, or how a project is configured. The connection uses one Braintrust API key and is read-only: CloudThinker can look, never change.

Prerequisites

• A Braintrust organization on Braintrust’s US data plane. CloudThinker always connects to api.braintrust.dev, so EU data plane and self-hosted Braintrust deployments are not supported.
• Permission to create an API key in that organization.
• The Admin role in your CloudThinker workspace, which you need to add connections. See Roles.

A Braintrust API key carries the permissions of the user who created it and can’t be scoped below that user, so create it from an account that only needs to read. If your Braintrust plan offers permission groups, put that account in the Viewers group, which gives read access to the organization’s projects. A key created by an owner works, but carries owner access.

Setup

1. Open API keys in Braintrust

   Sign in to Braintrust with the read-only account you picked above and go to Settings → API keys.

2. Create the key

   Click + API key, name it cloudthinker, and choose an expiration. It can’t be changed later, so put a reminder in your calendar to rotate the key before that date (see Security). Click Create.

3. Copy the key

   Copy the key now. Braintrust shows it only once. If you lose it, create a new one.

4. Add the connection in CloudThinker

   Open More → Workspace settings → Connections, stay on the Builtin Connections tab, and select Braintrust. Paste the key into BRAINTRUST_API_KEY, and click Connect.

   Success state: the Braintrust card shows Connected.

Connection details

Field	Required	Description	Example
BRAINTRUST_API_KEY	Yes	An API key from Settings → API keys in your Braintrust organization	<your-api-key>

There’s no data plane or URL to enter: CloudThinker always calls api.braintrust.dev.

Required permissions

The key’s reach is its creator’s reach. Braintrust can’t scope a key below the account that made it, so the account you choose is the permission boundary.

Create the key from an account with read access to the projects CloudThinker should see, and nothing more. Even with a broader key, CloudThinker runs only the six reads below.

Agent capabilities

CloudThinker allows exactly six Braintrust actions on this connection. Everything else Braintrust offers is blocked.

Capability	What you get
Recent objects	Up to 10 recent projects, experiments, datasets, prompts, or functions of one type
Experiment summary	One experiment’s scores and metrics, optionally compared with a baseline experiment
Object lookup	Resolve a name, ID, or Braintrust URL to the object it points to
Share link	A Braintrust link to one resolved object, ready to paste into a ticket or chat
Monitor view	One saved monitor view and its charts, by exact name
Project settings	One project’s settings

CloudThinker reports names and counts. It shows an object ID only when you ask for one.

CloudThinker can’t run SQL over your logs, list automations, browse all monitor views, or create, edit, or run anything in Braintrust. Do those in Braintrust itself.

Verify the connection

List my 5 most recent Braintrust projects and report their names only

Success state: you get a short list of project names that match what you see in Braintrust. A 401 error means the key is wrong or expired; see Troubleshooting.

Example prompts

Use exact names from Braintrust. CloudThinker looks up the object you name; it doesn’t guess.

How did the support-bot-v3 experiment score compared with the support-bot-v2 experiment?
Show the saved monitor view called production-latency and list its charts
What are the settings of the support-bot project?

Troubleshooting

Every request fails with 401 Unauthorized

Braintrust didn’t accept the key. It may be mistyped, deleted, or expired. Create a new key, then open More → Workspace settings → Connections, select Braintrust, and replace the value.

403 Forbidden, or a project or experiment isn't found

The key only sees what its creator’s account sees. Check that account’s access to the project or experiment in Braintrust, and check the exact name you used.

It worked for months, then every request started failing

The key expired. Braintrust keys can’t be renewed, so create a new key and update the connection. Set a calendar reminder before the next expiration.

My organization uses the EU data plane or self-hosts Braintrust

This connection only talks to api.braintrust.dev. It can’t reach EU data plane or self-hosted deployments.

CloudThinker won't run a query or change something

That’s by design. The connection allows only the six reads listed in Agent capabilities.

Security

• Least privilege — grant only the permissions the agents need for your use case; start read-only and widen later.
• Read-only by default — use read-only credentials unless you want agents to make changes through this connection.
• Rotate credentials — rotate keys and tokens on your normal schedule; CloudThinker picks up the new value when you update the connection.
• Revoke on offboarding — remove the credential at the provider when you delete a connection or a teammate leaves.

• Account scope: the key acts as its creator, so create it from an account with read-only access.
• Expiration: pick an expiration you’ll track, and rotate the key before it lapses.

Related

Langfuse

Add LLM traces and prompts from Langfuse

LangSmith

Add LLM runs, datasets, and experiments from LangSmith

Commands and Prompt Syntax

Point a question at one connection with #