Splunk Observability

Connect Splunk Observability Cloud so agents can read APM service health, exemplar traces, metrics, and SignalFlow results

Connect a Splunk Observability Cloud organization to let CloudThinker agents break a service’s latency and errors down by tag, map its dependencies, pull exemplar traces, and run SignalFlow (Splunk’s streaming analytics language) against your metrics. The connection is read-only and uses Splunk’s hosted MCP Gateway, so you only supply an access token, your realm, and a gateway region.

Prerequisites

• A Splunk Observability Cloud organization that Splunk has authorized for its Observability MCP tools. Check this first: an organization that isn’t answers Org Not Allowed, and no setting in CloudThinker fixes that. Ask your Splunk administrator or Splunk support to enable it before you start.
• Observability Cloud is separate from Splunk Enterprise and Splunk Cloud Platform, which use the Splunk Platform connection.
• Your realm, the Splunk deployment your organization runs in. It’s the first label in your Observability URL: app.<realm>.signalfx.com or app.<realm>.observability.splunkcloud.com, for example us0, eu0, or au0.
• A Splunk admin who can create an org access token.
• APM instrumentation, if you want service and trace answers. An organization that only sends metrics returns zero APM environments, but metric and SignalFlow questions still work.
• The Admin role in the CloudThinker workspace, which you need to add connections. See Roles.

Setup

1. Create an access token

   In Splunk Observability Cloud, open Settings → Access Tokens and select New Token. Name it cloudthinker, choose the API authorization scope, and attach the read_only role.

2. Set its expiration

   Choose who can see the token and set an expiration date that matches your rotation policy. Splunk expires a token 30 days after creation unless you change it here.

3. Copy the token

   Select the token name, select Show Token, then Copy.

4. Add the connection in CloudThinker

   In the sidebar, click More → Workspace settings → Connections, stay on the Builtin Connections tab, and select Splunk Observability. Fill in:

   • Access token: the token you copied.
   • Realm: the realm only, in lowercase as it appears in your URL, such as us0. Don’t paste the full URL.
   • Gateway region: the Splunk Cloud Services region your requests should pass through (see Connection details). EU or APAC organizations with residency rules should change the US East default.

   Click Connect.

   Success state: Splunk Observability shows Connected. Run the verify prompt below to confirm the token matches the realm.

Connection details

Field	Description	Example
Access token	An API-scope org access token issued with the read_only role	<your-access-token>
Realm	The realm that holds your organization, typed as free text. Realm only, not a URL	us0
Gateway region	The Splunk Cloud Services region that serves the MCP Gateway. Default iad10 (US East)	fra10 (Germany)

The gateway region doesn’t limit which realm you can reach. Any region reaches any realm. It decides which Splunk infrastructure your requests and their results travel through, so pick the one your data residency requires.

Gateway region	Location
iad10	US East
pdx10	US West
dub10	Ireland
fra10	Germany
lon10	United Kingdom
tyo10	Japan
syd10	Australia
sin10	Singapore

A user API access token from your Splunk profile also works, but it expires when you log out of Splunk Observability Cloud. The connection then stops with no warning. Use an org access token for anything beyond a quick trial.

Required permissions

The read_only role is enough for every tool this connection uses, including running SignalFlow programs. A broader role such as power grants more than agents need.

Splunk enforces the role at its own API, and this connection has no write tools. Agents can’t create, edit, mute, or delete a detector, dashboard, chart, or alert. When an investigation points to a change, the agent tells you what to change and you make it in Splunk.

Use an org access token named cloudthinker with the read_only role, so you can revoke CloudThinker’s access on its own without touching anyone else’s.

Agent capabilities

Capability	What agents can do	Approval
Environments and services	List APM environments and the services reporting in each one	No
Service health	Break a service’s latency, errors, and request counts down by tag	No
Dependencies	Read a service’s inbound and outbound dependencies with their request counts and latencies	No
Traces	Pull exemplar traces for errors or latency buckets, then open every span of one trace by ID	No
Metrics	Search the metric catalog and read a metric’s dimensions, properties, and sample values	No
SignalFlow	Write a SignalFlow program from real metric names and run it for a time series	No
Alerts	Search triggered alerts and incidents	No

Agents write SignalFlow themselves and check metric names against your catalog first. They don’t use Splunk’s built-in program generator.

Verify the connection

Summarize my Splunk Observability setup: realm, APM environments, services, and whether APM is reporting

Expect your realm, a list of APM environments with their services, and a note on whether APM instrumentation is reporting. Zero environments with no error means the organization sends metrics but no APM data.

Example prompts

Which service in the prod environment has the worst error rate over the last 24 hours?
Why is checkout slow? Break its latency down by tag and show an exemplar error trace
Run a SignalFlow query for the busiest kafka metric over the last 6 hours

Troubleshooting

Org Not Allowed

Your organization isn’t authorized for Splunk’s Observability MCP tools. Changing the token, realm, or region won’t help. Ask your Splunk administrator or Splunk support to enable it.

Authentication fails, or the connection lists no tools

The token is wrong, expired, or was issued in a different organization than Realm names. A token that doesn’t match the realm often returns an empty tool list instead of an error. Check the realm in your Observability URL, then create a new token in that organization and update Access token.

The connection stopped working after about a month

Splunk tokens expire 30 days after creation by default, and a user API access token also expires when you log out. CloudThinker doesn’t know the token’s expiry date, so it can’t warn you ahead of time. Create an org access token with a longer expiration, update the connection, and set a reminder to rotate it.

An APM answer comes back empty

The organization may have no APM instrumentation, or the service name doesn’t match exactly. Ask the agent to list services first and use a name from that list. If the agent says the search didn’t finish, ask again over a shorter time range.

Exemplar traces time out

Splunk’s exemplar trace search can run longer than one tool call allows. The rest of the service health answer is still valid. Ask again over a shorter window, such as the last hour.

A metric you know exists isn't found

The catalog search matches loosely and returns only a few names per call. Give the agent the exact metric name, or try different search terms.

Security

• Least privilege — grant only the permissions the agents need for your use case; start read-only and widen later.
• Read-only by default — use read-only credentials unless you want agents to make changes through this connection.
• Rotate credentials — rotate keys and tokens on your normal schedule; CloudThinker picks up the new value when you update the connection.
• Revoke on offboarding — remove the credential at the provider when you delete a connection or a teammate leaves.

• Read-only, enforced by Splunk: a read_only token is the narrowest credential that runs every tool here, and Splunk rejects any write from it server-side.
• The gateway region is a data path: requests and results travel through the region you choose. The realm and region are routing values, not credentials, so agents can name them in answers.

Related

Splunk Platform

Search Splunk logs and review index volume.

Honeycomb

Trace queries, BubbleUp, and approval-gated trigger changes.