` | `@anna coordinate quarterly review with @alex @oliver @tony @kai` |
In Slack, the same syntax works with a bot prefix: `@CloudThinker alex what's our AWS spend this week?` — see [Slack integration](/guide/slack-integration).
## Examples
Direct requests — pick the agent, tag the output, state the target:
```text theme={null}
@alex #dashboard AWS spending by service and region
@oliver #report SOC 2 compliance assessment with evidence
@tony #recommend index changes for queries slower than 500ms
```
Stack context modifiers to make results precise and safe to act on:
```text theme={null}
@alex #recommend right-sizing for EC2 instances in us-east-1 with CPU below 15% over the last 30 days, excluding production-tagged resources
```
Mention multiple agents for cross-domain problems:
```text theme={null}
@alex @tony investigate the cost spike on database infrastructure last week
@anna coordinate security remediation with @oliver leading and @alex @kai supporting
```
Refine progressively — start broad, then drill into what the agent finds:
```text theme={null}
@alex #dashboard cloud spending trends
@alex #chart November cost spike broken down by service
@alex #recommend RDS optimization prioritizing underutilized instances
```
Be specific. `@alex #recommend right-sizing for EC2 in us-east-1 with CPU below 15%` gets an actionable answer; `@alex check our servers` gets a clarifying question.
## Related
Each agent's expertise, capabilities, and prompt patterns in depth
What #dashboard, #report, #recommend, and other tool tags produce
Pre-built operations ready to run with this syntax
Use the same syntax in Slack with the @CloudThinker prefix
# Operations Hub
Source: https://docs.cloudthinker.io/guide/operations-hub
Browse and run pre-built cloud operations across cost, security, performance, and Kubernetes categories.
The Operations Hub provides pre-built operation patterns organized by category. Each operation pairs an agent, a tool tag, and a tested instruction — ready to run without writing prompts from scratch.
## Invocation format
```text theme={null}
@agent #tool instruction
```
| Part | Description |
| ------------- | -------------------------------------------------------------------------- |
| `@agent` | The agent responsible for the operation (Alex, Oliver, Tony, Kai, or Anna) |
| `#tool` | Output type: `#dashboard`, `#report`, `#recommend`, `#alert`, `#chart` |
| `instruction` | Natural-language task description — scope, timeframe, and filters go here |
See [CloudThinker Language](/guide/language) for full syntax reference.
## Operation categories
| Category | Focus | Primary agent |
| ----------------- | --------------------------------------------------------------------- | ------------- |
| Cost optimization | Right-sizing, reserved instances, unused resources, spending analysis | Alex |
| Security | Audits, compliance checks, vulnerability scans, IAM reviews | Oliver |
| Performance | Bottleneck detection, query optimization, scaling analysis | Alex, Tony |
| Operational | Backup verification, patch compliance, maintenance tasks | Alex |
| Scalability | Auto-scaling, load balancing, capacity planning | Alex |
| Kubernetes | Cluster health, pod optimization, RBAC audits | Kai |
## Sample operations
### Cost optimization
| Operation | Agent | Example prompt |
| --------------------------- | ---------- | -------------------------------------------------------------------------- |
| EC2 instance right-sizing | Alex | `@alex #recommend right-size EC2 instances in us-east-1 over past 30 days` |
| Reserved instance coverage | Alex | `@alex #report reserved instance coverage vs on-demand spend` |
| Unused EBS volume detection | Alex | `@alex #report unused EBS volumes across all regions` |
| RDS cost optimization | Alex, Tony | `@alex @tony #recommend RDS cost reduction for production databases` |
### Security
| Operation | Agent | Example prompt |
| --------------------------- | ------ | ---------------------------------------------------------------------- |
| Security group audit | Oliver | `@oliver #report open security groups exposing public internet access` |
| IAM permission review | Oliver | `@oliver #recommend least-privilege IAM policy changes` |
| S3 public access check | Oliver | `@oliver #report S3 buckets with public read or write access` |
| SOC 2 compliance assessment | Oliver | `@oliver #report SOC 2 control gaps across AWS account` |
### Performance
| Operation | Agent | Example prompt |
| ------------------------------- | ----- | --------------------------------------------------------------------- |
| Slow query analysis | Tony | `@tony #report top slow queries in production RDS over past 7 days` |
| Database performance dashboard | Tony | `@tony #dashboard database performance metrics for all RDS instances` |
| Auto scaling group optimization | Alex | `@alex #recommend auto scaling policy changes for web tier` |
| EKS resource utilization | Kai | `@kai #report pod resource utilization across production cluster` |
## Custom operations
Combine any agent, tool tag, and instruction to build operations beyond the catalog:
```text theme={null}
@alex #recommend cost optimization for EC2 instances
in us-east-1
with <20% CPU utilization
over past 30 days
excluding production-critical
```
```text theme={null}
@anna coordinate security review:
@oliver audit security groups and IAM
@alex check for public resources
@kai review Kubernetes RBAC
```
See [CloudThinker Language](/guide/language) for multi-agent syntax and all available tool tags.
## Scheduling
To run operations on a recurring schedule, use [Tasks](/guide/automation/tasks). Tasks let you configure frequency, scope, and notification preferences for any operation.
## Related
Complete syntax for creating custom operations
Understand which agent to use for each operation type
# Organizations
Source: https://docs.cloudthinker.io/guide/organization
Manage your CloudThinker organization settings, members, and billing
Organizations are the top-level entity in CloudThinker that groups your [workspaces](/guide/workspaces), team members, and billing under a single account. Every user automatically gets a personal organization when they sign up.
## Overview
Organizations in CloudThinker provide:
* **Centralized Billing**: Single subscription and usage quota shared across all workspaces
* **Team Management**: Invite members and assign roles across your organization
* **Workspace Container**: Create multiple workspaces within your organization
* **[BYOK](/guide/byok) Support**: Configure your own AWS Bedrock credentials at the organization level
Every CloudThinker user automatically receives a personal organization upon
signup. This organization is named after you (e.g., "John's Organization") and
serves as your default workspace container.
***
## Organization structure
🏢
Organization
— top-level entity, auto-created at signup
Workspaces
Default
Auto-created · ready to use
Additional
Per team · env · cloud provider
Each contains: Agents · Connections · Knowledge Base · Operations
### Relationship with workspaces
| Aspect | Organization | Workspace |
| ------------------ | ------------------------ | --------------------------------- |
| **Creation** | Auto-created at signup | Created manually |
| **Billing** | Centralized subscription | Inherits from organization |
| **Members** | All team members | Subset with workspace access |
| **Cloud Provider** | N/A | Single provider per workspace |
| **Resources** | Shared quota pool | Isolated agents, connections, KBs |
Create and manage workspaces within your organization
***
## Accessing organization settings
Navigate to **Admin Settings** from the user menu in the top-right corner. The Admin Settings sidebar provides access to all organization management pages:
| Page | Description |
| ----------------------- | ------------------------------------------------------ |
| **Organization** | Edit organization name, view stats, and manage members |
| **Workspaces** | Create and manage workspaces |
| **Billing** | View subscription and usage (Owner only) |
| **Usage** | Monitor credit usage and quotas (Owner only) |
| **BYOK** | Configure AWS Bedrock credentials (Owner only) |
| **Identity and access** | Domain verification, SSO, and provisioning |
Admin Settings is only visible to organization Owners and Admins.
***
## Organization settings
### General settings
Edit your organization's basic information:
| Field | Guidelines |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| **Organization Name** | Choose a clear, descriptive name. Include a team or company identifier. Examples: "Acme Corp", "Platform Team", "My Projects" |
| **Description** | Describe the organization's purpose. Note the primary team or use case. Keep it concise and informative |
Only organization Owners and Admins can edit organization settings.
***
## Organization members
Invite team members to collaborate within your organization. Members can be assigned to specific workspaces with role-based access control.
### Adding members
Go to **Admin Settings > Organization** and scroll to the Members section
Click the **Invite Members** button in the members section
Add one or more email addresses (up to 10 at a time)
Choose the appropriate organization role for the invitees
Select which workspaces the new members should have access to
Click **Send Invites** to send invitation emails
### Organization roles
| Role | Permissions |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Owner** | **Full control** — manage all organization settings, handle billing and subscription, manage all members and roles, access all workspaces (implicit), configure BYOK settings, transfer ownership |
| **Admin** | **Administrative access** — manage organization members, create and delete workspaces, access all workspaces (implicit); cannot manage billing or BYOK |
| **Developer** | **Standard access** — access assigned workspaces only, use agents and run operations, view and use workspace resources; cannot create workspaces |
| **Viewer** | **Read-only access** — view assigned workspaces only, cannot modify resources, cannot run operations; observation and audit purposes |
### Role permissions summary
| Permission | Owner | Admin | Developer | Viewer |
| --------------------------- | :---: | :---: | :-------: | :----: |
| Edit organization settings | ✓ | ✓ | - | - |
| Manage billing/subscription | ✓ | - | - | - |
| Configure BYOK | ✓ | - | - | - |
| Invite/remove members | ✓ | ✓ | - | - |
| Change member roles | ✓ | ✓ | - | - |
| Create workspaces | ✓ | ✓ | - | - |
| Access all workspaces | ✓ | ✓ | - | - |
| Access assigned workspaces | ✓ | ✓ | ✓ | ✓ |
| Run operations | ✓ | ✓ | ✓ | - |
### Managing members
| Action | How To |
| ------------------------- | ---------------------------------------------------------------------------- |
| **Change Role** | Click the role dropdown next to a member and select a new role |
| **Edit Workspace Access** | Click the grid icon to manage which workspaces a Developer/Viewer can access |
| **Remove Member** | Click the trash icon and confirm removal |
| **Resend Invitation** | For pending invitations, click the refresh icon |
| **Cancel Invitation** | For pending invitations, click the X icon |
When editing workspace access for Developers or Viewers, you can also set a
role override per workspace. This allows a Viewer to have Developer access in
specific workspaces, or a Developer to have Admin access where needed.
Removing a member from the organization removes them from all workspaces
within that organization.
***
## Subscription & billing
Organization Owners can manage subscription and view usage from the Subscription tab.
### Subscription features
| Feature | What you can do |
| ------------------ | ------------------------------------------------------------------------------------------------------------- |
| **Plan Overview** | Current plan and features, billing cycle information, payment method management, upgrade/downgrade options |
| **Usage Tracking** | Credit usage across all workspaces, member count vs. limit, workspace count vs. limit, usage trends over time |
### Seats and members
Your subscription plan uses **per-seat billing** — each seat you purchase grants a credit allocation and allows one active member. You can also pre-purchase additional seats to increase your credit pool before adding members.
Minimum seats vary by plan — see [Pricing & Plans](/guide/billing/pricing) for the current per-plan seat ranges and credit allocations.
Only active members count toward seat usage — pending invitations do not consume seats.
***
## BYOK (Bring Your Own Key)
Organization Owners can configure AWS Bedrock credentials to use their own LLM quota instead of CloudThinker's shared pool.
**[Bring Your Own Key](/guide/byok)** allows you to:
* Use your own AWS Bedrock account for LLM calls
* Bypass CloudThinker's credit system for LLM usage
* Have unlimited LLM operations (based on your AWS quota)
* Maintain data residency in your own AWS account
BYOK is configured at the organization level and applies to all workspaces.
Learn how to set up your AWS Bedrock credentials
***
## Best practices
### Team structure
**Role assignment (principle of least privilege):**
* Assign Owner role sparingly (1-2 people)
* Use Admin for team leads who need workspace management
* Use Developer for most team members
* Use Viewer for stakeholders who need visibility only
**Workspace access (organized access):**
* Admins/Owners see all workspaces automatically
* Explicitly assign Developers to relevant workspaces
* Use role overrides for fine-grained control per workspace
* Regular access reviews for compliance
### Common patterns
**Simple Structure:** - 1 Owner (team lead or account holder) - 1-2 Admins
(senior engineers) - Remaining as Developers - Single workspace or dev/prod
split
**Structured Approach:** - 1 Owner (engineering manager) - 2-3 Admins (tech
leads) - Developers assigned to relevant workspaces - Viewers for product
managers or stakeholders - Multiple workspaces by environment or team
**Governance Focus:** - 1-2 Owners (platform team leads) - Admins per team or
department - Developers with workspace-specific access - Viewers for audit and
compliance - Workspaces by team, environment, and project - Consider [BYOK](/guide/byok) for
cost control
***
## Next steps
Set up workspaces for your teams and projects
Set up AI agents for your cloud operations
# Pulse Analytics
Source: https://docs.cloudthinker.io/guide/pulse/analytics
Measure signal volume, noise reduction, cluster resolution time, and source conversion rates over time
The Pulse analytics tab gives you a full picture of how your signal pipeline is performing — how much noise is being cut, how long clusters take to resolve, and which sources are the most (or least) actionable.
Access it from the **View trends** button in the Pulse header.
***
## Key metrics
| Metric | Meaning |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Correlation Yield** | The percentage of signals that were routed to an Incident or resolved via a cluster. A low yield (e.g. 8%) means most signals are being correctly filtered as non-actionable — that's expected and healthy in a well-tuned setup. |
| **Cluster MTTR** | Mean time to resolve across all resolved clusters in the selected period. Tracks how quickly your team (and AI agents) close out correlated events once they surface. |
***
## Signal volume
![Analytics page showing Correlation Yield at 8%, Cluster MTTR at 3410m, signal volume stacked bar chart by day, cluster lifecycle area chart by week, and top noisy sources table]()
Analytics overview — KPIs, signal volume by severity, cluster lifecycle, and top noisy sources
The **Signal Volume** chart shows daily signal counts broken down by severity — Critical, High, Medium, Low, Info. Use the **Stacked / Share %** toggle to switch between absolute counts and proportional view.
Spikes in this chart often correspond to infrastructure events (a deployment, a cost anomaly window, a security finding) rather than genuine problems — cross-reference with the Cluster Lifecycle chart to see how many of those spikes actually produced actionable clusters.
***
## Cluster lifecycle
The **Cluster Lifecycle** chart shows how clusters distributed across statuses (Forming, Active, Routed, Resolved) over time. A healthy pattern shows most clusters moving from Active → Resolved without needing escalation to Routed.
If Routed clusters are accumulating without corresponding Resolved entries, it may indicate incidents are being created but not closed — worth checking the Incidents list for stale open incidents.
***
## Suppression by reason
![Suppression by reason line chart showing weekly trends for Duplicate, Flapping, Noise Signature, Rate Limited, Severity Normalized, Snoozed, and Cascade, plus a conversion rate heatmap by day and hour]()
Suppression breakdown over time and signal-to-incident conversion rate by hour of day
The **Suppression by Reason** chart breaks down which of the seven suppression layers are firing and at what volume. Key things to look for:
* **Duplicate dominating** — normal; means your sources are emitting redundant events as expected
* **Rate Limited spiking** — a source may be misconfigured or experiencing an alert storm
* **Flapping increasing** — a resource is oscillating; worth investigating the root cause
* **Snoozed growing** — your team is managing noise manually; consider whether a permanent noise signature rule would help
***
## Conversion rate heatmap
The heatmap shows signal-to-Incident conversion rate broken down by hour of day and day of week. Green cells (100%) mean every signal in that window became an Incident; empty cells mean no signals arrived.
Use this to understand when your most actionable signals arrive — useful for on-call scheduling and for identifying patterns (e.g. cost anomalies that reliably surface on weekend mornings after batch jobs run).
***
## Top noisy sources
The **Top Noisy Sources** table ranks sources by signal volume with three columns:
| Column | What It Tells You |
| --------------------- | ------------------------------------------------------ |
| **Signals** | Total signals from this source in the selected period |
| **% Suppressed** | How much of that source's output was filtered as noise |
| **% Clusters Routed** | What fraction of its clusters escalated to an Incident |
A source with high signal volume, low suppression, and low routing (e.g. `aws.config.us-east-1` at 0% routed) is generating many signals that don't result in action — a candidate for tuning or snoozing specific patterns. A source with high routing (e.g. `webhook.grafana` at 65% routed) is highly actionable and worth investing in.
***
## Filters
All charts respond to the filter bar at the top of the analytics page:
* **Date range** — same presets as the main feed (1h to 30d, or custom)
* **Severity** — focus on Critical/High only to measure the most urgent signal patterns
* **Source** — isolate a single source to audit its noise profile
* **Category** — examine cost vs. security vs. compute signals separately
* **Show suppressed** — include suppressed signals in volume counts
***
## Related
Understand how Pulse groups signals into clusters and the seven suppression layers.
Connect monitoring sources and configure detection rules for your workspace.
# Clusters & Suppression
Source: https://docs.cloudthinker.io/guide/pulse/clusters
Understand how Pulse groups signals into clusters and cuts noise through seven automatic suppression layers.
Two concepts sit at the heart of what makes Pulse useful: **suppression** (cutting noise before it reaches you) and **clustering** (grouping what remains into single units of action).
***
## Suppression — cutting the noise
Every signal that arrives passes through seven layers before it reaches your feed. If any layer fires, the signal is stored but hidden. It won't create noise, but it's still there if you need to audit it.
If an identical event already arrived within the last hour, the dedup count on the existing signal is incremented instead of creating a new row. You see one signal marked "×47" rather than 47 separate items. Typically the largest suppression category.
If a source emits more than 100 signals per minute, signals above that threshold are suppressed for the duration of the burst. Prevents a misconfigured alert from flooding your feed.
Signals matching an active snooze rule are suppressed. This is the only layer you control directly — see [Snooze](#snooze) below.
Known-noisy AWS patterns are suppressed automatically — KMS grant lifecycle events, EBS volume churn, AutoScaling internal operations, Signin token redirects. These are AWS bookkeeping events that almost never indicate a real problem.
If a signal toggles state four or more times within 10 minutes, it is suppressed for 5 minutes. A resource oscillating between healthy and unhealthy collapses into a single notification once state stabilizes.
When a parent resource is suppressed, signals from its child resources are also suppressed for 30 minutes — so you don't get child alerts for something that's already known noise.
AWS automation services sometimes emit events with inflated severity. Pulse detects events from AWS internal actors and downgrades the severity before routing. The original severity is preserved for auditing.
![Suppression by reason chart showing weekly trends for Duplicate, Flapping, Noise Signature, Rate Limited, Severity Normalized, Snoozed, and Cascade layers alongside a conversion rate heatmap]()
Suppression breakdown over time — visible in the Analytics tab
To review suppressed signals, enable **Show suppressed** in the filter bar. They appear at reduced opacity with a label showing which layer caught them.
### Snooze
Snooze is the only suppression layer you control. Hover over any signal and click the snooze button. Choose a duration (1 minute to 30 days) and a scope:
| Scope | What it silences |
| ------------ | --------------------------------------------------------- |
| **Signal** | This exact signal only |
| **Pattern** | All signals with the same source, type, and title pattern |
| **Resource** | All signals from this resource ID |
Use **Pattern** for recurring maintenance windows. Use **Resource** when decommissioning a resource during teardown.
***
## Clusters — one unit of action
A **cluster** is the primary unit of work in Pulse. Instead of presenting every individual signal separately, Pulse groups related signals — the same EKS node pool firing nine alerts in 15 minutes becomes one cluster. You investigate once, act once, resolve once.
### Status lifecycle
Every cluster moves through four statuses:
```mermaid theme={null}
stateDiagram-v2
direction LR
[*] --> Forming : first signal
Forming --> Active : more signals arrive
Active --> Routed : Escalate → Incident created
Active --> Resolved : Resolve
Routed --> Resolved : Incident closed
Resolved --> [*]
```
| Status | Meaning |
| ------------ | ------------------------------------------------------ |
| **Forming** | First signal arrived; collecting related signals |
| **Active** | Signals continuing to arrive; open and needs attention |
| **Routed** | Escalated — a linked Incident has been created |
| **Resolved** | Closed by a user or automatically |
Use the **Active / All** toggle in the feed to switch between active clusters only (default) or all statuses.
### The cluster detail panel
Click any cluster to open the detail panel.
![Cluster detail panel showing EKS node pool cluster with 9 signals correlated via time_window at 85% confidence, signal timeline, resource metadata, and action buttons]()
AI-generated description, signal timeline, resource details, and actions
The panel shows:
* **AI-generated description** — plain-English summary of what happened and likely impact
* **Cluster context** — all member signals listed chronologically
* **Correlation info** — technique used (e.g. `time_window`) and confidence score
* **Resource metadata** — id, type, region, tags of the selected signal
* **Tabs** — Overview for signal detail, Routing for escalation history, Raw for the full event payload
### Actions
| Action | What it does |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Acknowledge** | Mark as seen without closing. Signals the cluster is on your radar. Reversible. |
| **Assign** | Hand to a team member. They're notified and their avatar appears in the feed. |
| **Escalate** | Create a linked Incident. Cluster moves to Routed and RCA begins automatically — the cluster summary and all member signals are passed to the RCA agent as starting context, so investigation begins with full signal history already loaded. |
| **Resolve** | Close the cluster when the issue is addressed and no Incident is needed. |
### Automatic escalation
Pulse auto-escalates clusters where any signal has **Critical** or **High** severity, or where the AI marks the signal **actionable**. These go directly to Routed and trigger root cause analysis — no manual escalation needed.
***
## Related
Measure signal volume, noise reduction, cluster resolution time, and source conversion rates.
Connect monitoring sources and configure detection rules for your workspace.
Understand how AI agents investigate escalated clusters and build evidence chains.
# How Pulse Works
Source: https://docs.cloudthinker.io/guide/pulse/overview
Learn how Pulse turns raw monitoring events into ranked clusters through an eight-stage pipeline of dedup, suppression, and AI triage
Pulse ingests events from every connected source — CloudTrail, GuardDuty, Datadog, and more — suppresses the noise, and surfaces only clusters worth your attention. Each cluster is ranked by severity and escalates to an incident in one click.
![Pulse main feed showing the pipeline panel with 13K raw events reduced to 510 signals and 40 clusters, alongside an active cluster list ranked by severity]()
Pulse reduces 13K raw events to 40 actionable clusters — live, in one view
Your monitoring stack already catches anomalies. The problem is volume: engineers spend more time triaging alert floods than fixing real problems. Pulse sits in front of every source and decides what deserves your attention, so you open a ranked list of clusters instead of six dashboards.
## How it works
Every event that reaches Pulse goes through the same eight-stage pipeline before it becomes something you see.
```mermaid theme={null}
flowchart TD
A[Raw Event] --> B[Normalize]
B --> C[Deduplicate]
C --> D{Suppress?}
D -- yes --> E[Stored · hidden]
D -- no --> F[Persist]
F --> G[Correlate into Cluster]
G --> H[AI Classify]
H --> I{Actionable?}
I -- Critical / High --> J[Incident + RCA]
I -- no --> K[Notification only]
```
1. **Ingest** — An event arrives from any connected source: an AWS poller picks up a GuardDuty finding, a Slack message fires from your alerts channel, or a Datadog webhook posts an alert.
2. **Normalize** — A source-specific collector translates the raw event into a common signal shape, extracting title, severity, category, resource ID, and timestamp regardless of origin.
3. **Deduplicate** — A SHA-256 fingerprint is computed from the signal's source, type, resource, and timestamp minute. If an identical event arrived within the last hour, the existing signal's dedup count increments instead of creating a new row.
4. **Suppress** — The signal passes through suppression layers in priority order; if any layer fires, the signal is stored as suppressed and hidden from your feed. See [Clusters & Suppression](/guide/pulse/clusters) for how each layer works.
5. **Persist** — The signal is written with its final suppressed status, severity, and extracted fields. Suppressed signals are retained for 90 days — toggle **Show suppressed** to review what was filtered.
6. **Correlate** — Within a 15-minute window, Pulse groups signals sharing the same resource, service, or title pattern into a cluster. Nine EC2 alerts become one cluster with nine members.
7. **Classify** — An AI model assigns category, canonical severity, a one-line summary, and an actionability verdict — whether this warrants creating an incident.
8. **Route** — Critical and High severity signals, plus any signal marked actionable, escalate automatically: a linked incident is created and root cause analysis begins. Everything else is delivered as a notification only.
## What you can do
| Capability | Description | Learn more |
| ------------------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------ |
| Connect signal sources | Wire AWS pollers, Slack and Teams channels, and third-party webhooks into Pulse | [Pulse Setup](/guide/pulse/setup) |
| Review clusters and suppression | See how related signals group together and audit what was silenced | [Clusters & Suppression](/guide/pulse/clusters) |
| Escalate to an incident | Promote any cluster to a full incident with automatic root cause analysis | [Deep Response Engine](/guide/incident/overview) |
| Measure noise reduction | Track suppression rates, cluster resolution time, and signal conversion | [Pulse Analytics](/guide/pulse/analytics) |
## Key concepts
The pipeline panel in the left sidebar renders all four stages as a live funnel:
| Stage | What the count means |
| -------------- | ---------------------------------------------------------------------------------- |
| **Raw events** | Every event ingested — before any filtering |
| **Signals** | De-duplicated, normalized events. The severity breakdown shows what's active |
| **Clusters** | Correlated groups. "grouped from 510 · 189 suppressed" shows how much was silenced |
| **Incidents** | Clusters that were escalated — links directly to the incidents list |
## Next steps
Understand how signals are grouped and how noise is filtered
Connect AWS, Slack, Teams, and third-party webhook sources
Measure noise reduction, cluster resolution time, and conversion rates
See how Pulse feeds into incidents, RCA, runbooks, and memory
# Pulse Setup
Source: https://docs.cloudthinker.io/guide/pulse/setup
Wire AWS pollers, chat channels, and monitoring webhooks into Pulse so actionable clusters escalate to incidents
Pulse pulls signals from three source families — AWS service pollers, chat platforms (Slack and Teams), and third-party monitoring webhooks — all configured from the **Manage subscriptions** page. Once a source is wired, Pulse deduplicates and classifies every event, and clusters that cross the actionability bar escalate to an [incident](/guide/incident/overview) where agents investigate root cause.
## Prerequisites
* A CloudThinker workspace — new to Pulse? Read [How it works](/guide/pulse/overview) first
* An active [AWS connection](/guide/connections/aws) with read permissions for the services you want to poll
* The CloudThinker bot installed in [Slack](/guide/slack-integration) or [Teams](/guide/teams-integration), if you want chat-channel signals
* A monitoring tool that supports outbound webhooks, if you want third-party signals
## Wire monitoring into incidents
This is the end-to-end flow: connect a source, confirm signals arrive, and let actionable clusters escalate to incidents.
Go to **Pulse** in your workspace and click **Manage subscriptions**. The page lists all three source families and their current status.
Enable an AWS poller, subscribe a chat channel, or point a monitoring webhook at CloudThinker — the three procedures below cover each path.
Send a test alert from your monitoring tool, or wait for the next polling cycle.
**Success state:** new signals appear in the Pulse feed with a category, severity, and one-line summary.
Pulse auto-escalates [clusters](/guide/pulse/clusters) with Critical or High severity signals, or any signal the AI marks actionable. For anything else, click **Escalate** on the cluster.
**Success state:** a linked incident is created and [root cause analysis](/guide/incident/root-cause-analysis) begins with the cluster's full signal history as starting context.
## AWS polling
AWS polling lets Pulse pull events directly from your AWS accounts on a schedule — no webhook configuration required. Each source is enabled per connection.
| Source | What Pulse collects | Cadence |
| ------------------- | ------------------------------------------------------------------------- | ------------- |
| **CloudTrail** | API call audit logs — who did what, when, on which resource | Every 5 min |
| **AWS Health** | Service health events impacting your account or region | Every 5 min |
| **Cost Anomaly** | Spend spikes detected by AWS Cost Anomaly Detector | Every 6 hours |
| **GuardDuty** | Threat intelligence findings — compromised instances, suspicious behavior | Periodic |
| **Config** | Resource configuration drift against Config rules | Periodic |
| **Access Analyzer** | IAM and S3 access findings — overly permissive or public resources | Periodic |
### Enable a poller
In Pulse, click **Manage subscriptions** and go to the **AWS Polling** section. You see your connected AWS accounts and their current polling status.
Click **Validate** next to the source. Pulse performs a dry run to confirm your credentials have the required read permissions.
**Success state:** a clear ✓ means ready; an error code (e.g. `AccessDenied`) tells you exactly what's missing.
Toggle the source on. Pulse starts polling immediately.
If a poller fails five consecutive times, Pulse pauses it automatically and surfaces the error code. To resume: fix the issue, re-validate, then re-enable the toggle.
## Slack and Teams subscriptions
When the CloudThinker bot is present in a channel, messages in that channel become Pulse signals — alert bot posts, incident announcements, on-call chatter — classified alongside your infrastructure events.
![Create Subscription dialog with Platform dropdown set to Slack, Channel search field, invite instruction callout, and Enabled toggle]()
Select a platform, pick a channel, invite the bot, save
Choose **Slack** or **Microsoft Teams**.
Search for the channel you want to monitor.
Run `/invite @CloudThinker` in the target channel — Slack only delivers messages to apps that are channel members.
**Enabled** is on by default. Click **Create subscription**.
Subscriptions can be toggled on and off independently — useful for muting a channel during a maintenance window without losing the configuration.
### Channel types
| Type | Best for |
| ------------------- | ------------------------------------------------------------- |
| **Alert** | Channels used exclusively for automated alert bots |
| **Communication** | Channels used for human discussion — incident chats, handoffs |
| **Mixed** (default) | Channels with both automated alerts and human messages |
## Third-party webhooks
Any monitoring tool that supports outbound webhooks can send signals to Pulse — Datadog, Grafana, PagerDuty, and more ship with built-in field mapping.
Go to **Webhooks** in CloudThinker and create an endpoint — see [Webhooks](/guide/webhooks/overview) for full setup.
Paste the generated URL into your tool's webhook or notification settings. Platform-specific instructions, authentication options, and field mappings live in [Webhook integrations](/guide/incident/webhook-integrations/overview).
Use your monitoring tool's test feature.
**Success state:** the alert appears as a signal in the Pulse feed within moments.
## Signal categories
Every signal — regardless of source — is assigned one of eight categories:
| Category | Examples |
| ----------------- | --------------------------------------------------------------- |
| **Compute** | EC2 status, Lambda timeouts, ECS failures, container restarts |
| **Network** | NAT gateway anomalies, CloudFront degradation, VPC changes |
| **Security** | GuardDuty findings, IAM changes, S3 public access, WAF triggers |
| **Cost** | Cost anomalies, idle resources, Savings Plan drops, RI expiry |
| **Data** | RDS connection issues, S3 operations, database events |
| **Deploy** | CodeDeploy, ECS rollouts, CloudFormation stack changes |
| **Communication** | Slack and Teams messages, alert bot notifications |
| **Unclassified** | Default until the AI classifier assigns a category |
## Next steps
Triage the cluster lifecycle — from Active to Routed to Resolved
Platform-specific setup for Datadog, Grafana, PagerDuty, and more
See how agents investigate escalated incidents to a confident root cause
Measure noise reduction and signal trends across your sources
# App Pentesting
Source: https://docs.cloudthinker.io/guide/security/pentest
Set up an app, attach a login and your source code, then let Oliver test it for security problems and show you the proof.
Oliver hunts for real security problems in your app and shows you the proof for each one. You set the boundaries once — what to test, what's off-limits, and what to sign in as — and Oliver tests as deeply as those allow.
App Pentesting is in beta. Find it under **App Pentesting** in the navigation menu.
## Prerequisites
* A domain you own, and the ability to add a DNS record for it.
* Optional: a connected Git provider, if you want Oliver to read your source code.
## Register your app
Go to **App Pentesting** from the navigation menu, then click **Register your first app**. If you already have apps, click **New app** instead.
![App Pentesting landing page with the Register your first app call to action]()
Enter an **App name** and a **Target** — a domain like `example.com` or an API base URL like `https://api.example.com/v1`. CloudThinker suggests an **OWASP framework** from the target: **OWASP Web** for a site, **OWASP API** for an API URL. Change it if the suggestion is wrong.
![New app dialog with App name, Target, and OWASP framework fields]()
Click **Create app**.
**Success state:** the app overview opens and shows **No pentests yet**, with a **Verify domain** button. Verify the domain next — you can't run a pentest until you do.
![App overview showing the No pentests yet setup hero]()
## Verify the domain
CloudThinker only tests sites you can prove you own. You do this once per domain, and every app on that domain reuses the proof.
Open **App Pentesting → Settings**. Under **Verified domains**, click **Add domain** — or **Manage domains** if you have verified one before — and enter the host in the **Domain** field.
![Verified domains section with the add domain dialog open]()
Copy the **TXT name** and **TXT value**, add them at your DNS provider, then click **Check DNS**. DNS propagation can take a few minutes, so retry if the first check fails.
**Success state:** the domain badge reads **Verified**.
![Verify domain dialog showing the TXT name and TXT value to publish]()
## Set the boundaries
Open the app's **Settings** tab. What you set here decides how much Oliver can find. Everything saves as you type — there's no save button.
![App Settings tab with domain verification, scope, context files, repositories, test identities, schedule, and default intensity]()
| Section | What to do | Why it matters |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| **Scope** | Click **Manage scope**. Add the paths Oliver may test under **Include paths** (`/api/**`), and anything off-limits under **Exclude paths** (`/admin/**`). Type a path and press Enter. | Your fence. Oliver never touches anything outside it. |
| **Test identities** | Click **Add authentication** and attach a login. | Lets Oliver test behind the sign-in page instead of only the public pages. |
| **Repositories** | Click **Add repository**, pick a provider, then a repo and branch. | Lets Oliver read your code, so findings point at the exact line that caused them. |
| **Context files** | Click **Attach context** to add an API spec, Postman collection, diagram, or docs — up to 10 MB each, or paste text inline. | Helps Oliver find things faster. Doesn't change how deep it tests. |
| **Default intensity** | Pick how aggressive new runs start out. | A starting point only — you can change it on any run. |
### Attach a login
Attach two logins and Oliver can prove the most valuable thing of all: that one customer cannot read another's data. Name them something like `victim` and `attacker`, and describe what each is allowed to see — their role, their team, the records they own. That description is how Oliver knows which access is a bug.
Best for web apps. Sign in, open your browser's developer tools, find any request on the **Network** tab, copy it as cURL, and paste it in. Click **Verify and attach** — CloudThinker replays it to confirm the login works before saving.
Best for APIs. Add the header your API expects — usually `Authorization` set to `Bearer ...` — then click **Attach token**. These don't expire, so scheduled runs keep working.
**Success state:** the identity shows a **Ready for testing** chip.
## How deep Oliver goes
The more you attach, the more Oliver can reach. You don't choose this — Oliver takes the deepest level available and shows it as an **Access level** badge in the run dialog.
| What you attached | Access level | What Oliver can test |
| ----------------- | ------------ | ----------------------------------------- |
| Nothing | Black-box | Only what a stranger on the internet sees |
| A login | Gray-box | Everything a signed-in user sees |
| A repository | White-box | The running app plus the code behind it |
One catch: browser sessions expire. An app with only a session tests as gray-box when you click **Run now**, but drops to black-box on a schedule, because the session dies before the scheduled run fires. Attach a long-lived token or a repository to keep the depth.
## Run the pentest
Click **Run pentest**. Everything lives in one dialog. **Setup** at the top repeats your target, framework, and scope so you can catch a mistake before anything is sent — **Edit setup ›** takes you back to fix it. Most runs finish in a few minutes and stop at 30.
![Run pentest dialog with setup facts, intensity picker, access level badge, context, and schedule]()
**Intensity** decides how far Oliver goes — and how much it can change:
| Intensity | What Oliver does | Your data |
| -------------- | --------------------------------------------------------------------------- | ----------------------------------------- |
| **Safe** | Only reads pages and data | Never changed. Safe to run on production. |
| **Aggressive** | Submits forms and creates test data, to reach bugs reading alone can't find | Added and edited, never deleted |
| **Full** | Everything above, plus deletion attempts | Can be permanently removed |
Narrow **Scope** before running Full against anything you care about.
**Access level** shows how deep this run goes; **Go deeper** tells you what to add to unlock more. If you attached a repository, you also pick a **Scan scope**: **Incremental** checks only what changed since your last run, **Full** checks everything. Your first run is always Full.
Then finish one of two ways:
* **Run now** starts immediately and opens the workbench.
* **Schedule** repeats it — **Daily**, **Weekly**, **Monthly**, **No Repeat**, or **Custom** — at a time you set, with an optional end date. Times use your local time and keep the same clock time year-round, so a run can land an hour off across a daylight saving change.
## Read the results
Watch the run happen. The **Timeline** in the middle narrates each step as Oliver works, findings appear on the right the moment they're confirmed, and past runs sit in **Run history** on the left. The full **Report** arrives when the run finishes.
![Run workbench with run history, live timeline, and the findings rail]()
Four tabs hold the results:
| Tab | What it shows |
| ------------ | --------------------------------------------------------------------------------------------------------------- |
| **Overview** | Your OWASP Top 10 coverage and how many findings sit at each severity |
| **Surface** | Everything Oliver found reachable — badged **Documented** if your specs mentioned it, **Shadow** if they didn't |
| **Runs** | Every past run and what it added, fixed, or left open |
| **Findings** | The full list, filterable by severity and status |
![Overview tab with OWASP Web Top 10 coverage, findings by severity, and the CWE mapping list]()
Open any finding to see the proof. **Summary** says what's wrong, **Attack path** shows exactly how Oliver got in, **Safe PoC** is the request you can replay yourself, and **Remediation** says how to fix it — **Create PR with agent** turns that fix into a pull request.
A progress bar tracks the finding from **Needs triage** to **Verified fixed**. That last step isn't yours to click: the next scan re-tests it and confirms the fix for you.
![Findings table with search, severity and status filters, and columns for affected surface, OWASP category, severity, and status]()
![Finding detail with the triage rail, summary, attack path, safe PoC, and remediation]()
## Troubleshooting
Your session expired. Rather than quietly testing as a logged-out visitor and reporting less than it should, Oliver stops. Click **Update login**, remove the one marked **Needs refresh**, and add a fresh one. If this keeps happening, attach a long-lived token instead — those don't expire.
Your TXT record is missing, or DNS hasn't caught up yet. Compare the record at your DNS provider against the **TXT name** and **TXT value** in the verify dialog, wait a few minutes, then click **Check DNS** again.
No Git provider is connected to your workspace yet. Click **Connect provider**, finish connecting, then reopen the dialog.
Incremental compares against your last successful run, so you need one first. Run a Full scan, and Incremental becomes available after it.
## Next steps
Meet the Security Engineer agent that runs your pentests and triages the findings
Catch the same classes of vulnerability on every pull request, before they ship
Harden workspace access itself with MFA, SSO, and single-purpose API keys
Connect the Git provider that unlocks white-box testing against your source
# Custom Skills
Source: https://docs.cloudthinker.io/guide/skills/custom-skills
Create, enable, and assign workspace skills that encode your team's knowledge for agents to apply on every task
Custom skills are workspace-private instruction packages that encode your team's knowledge — review checklists, runbooks, naming conventions, compliance rules — so agents apply it consistently on every task. You write the context once; agents load it automatically wherever the skill is assigned.
## What skills encode
Agents arrive knowing what good looks like in general, not what good looks like in your codebase, your runbooks, or your review standards. A skill externalizes that context so nobody retypes it into every conversation, and senior expertise becomes a team-wide artifact instead of tacit knowledge held by one engineer.
| Domain | Examples |
| ----------------- | ------------------------------------------------------------------------------------------------------- |
| Code review | Naming conventions, API design rules, security review checklists, performance hot-spot patterns |
| Incident response | Escalation policies, runbook selection rules, service-specific debugging order |
| Cloud operations | Account-tag conventions, environment promotion rules, cost-attribution policies, compliance constraints |
| Documentation | Style guide, terminology, voice rules |
If you find yourself typing the same instruction into more than two prompts, that instruction belongs in a skill.
## Prerequisites
* A CloudThinker workspace
* Familiarity with the [SKILL.md format](/guide/skills/skill-format) if you plan to write skills manually
## Create a skill
Navigate to **Skills** from the sidebar. This page lists every skill in the workspace with its enable toggle and feature assignments.
![Custom Skills management page showing skill cards with enable toggles and feature assignment checkboxes]()
Choose a creation method from the dropdown.
![New Skill dropdown showing creation options: Create with AI, Write skill instructions]()
Describe what you need in plain language and let the skill creation agent draft it.
1. Click **Create with AI** to open a chat session.
2. Describe one concrete rule your team applies — a naming convention, a review checklist item, a deprecation warning.
3. The agent generates a SKILL.md with frontmatter and structured instructions.
4. Review the draft and click **Save**.
This is the fastest path for a first skill. You can refine it later with AI editing.
Author the skill manually using the form.
1. Click **Write skill instructions**.
2. Fill in the fields: **Name** (a slug-format identifier such as `api-review-standards`), **Description** (a short summary), and **Instructions** (the full skill content in Markdown).
3. Click **Save**.
![Write skill dialog with name, description, and instructions fields]()
See [Skill format](/guide/skills/skill-format) for the SKILL.md structure and a complete example.
**Success state:** the new skill appears as a card on the Skills page.
## Enable and assign a skill
A skill does nothing until you turn it on and scope it to at least one feature.
Use the switch on the skill card. Disabled skills are saved but never loaded — useful for drafting changes without affecting live agent behavior.
Check the features where the skill applies. A skill can belong to several features at once:
* **Code Review** — applied during PR analysis
* **Incident** — applied during incident investigation and root cause analysis
* **Assessment** — applied during infrastructure assessments
Assign the skill only where it belongs; a code-review naming convention does not need to load during incident triage.
Trigger a task in an assigned feature — open a pull request if the skill covers Code Review. Agents consult assigned skills during the Analyze phase of the [agentic loop](/index), so your rule should appear in the agent's reasoning and comments.
**Success state:** the agent's output cites or applies your rule without you restating it. If it doesn't, the instruction was probably too abstract — rewrite it with concrete examples and retest.
## Manage skills
### View details
Click a skill card to open the detail view: file tree on the left, rendered markdown on the right. Read the instructions end to end — a rule that is ambiguous to you will be ambiguous to the agent.
![Skill detail view showing file tree and rendered markdown content in a split-pane layout]()
### Edit with AI
From the detail view, click **Edit with AI** to refine the skill through conversation — add examples, remove ambiguity, restructure sections. The agent reads the existing content before making changes.
### Delete
Remove a skill permanently from the detail view.
Deleting a skill cannot be undone. Disable it instead if you may need it again.
## Writing effective skills
* **Be specific.** Concrete rules with examples beat vague guidelines — "use kebab-case for multi-word paths" outperforms "follow our naming style".
* **One domain per skill.** Separate security rules from performance guidelines so you can assign them independently.
* **Lead with examples.** Agents follow demonstrated patterns more reliably than abstract explanations.
* **Test on one feature first.** Enable a new skill on a single feature, review a few real runs, then assign it more broadly.
* **Iterate.** Use **Edit with AI** to sharpen skills based on review results and team feedback.
## Next steps
Learn the SKILL.md file structure, frontmatter reference, and best practices for writing effective instructions.
Understand how skills fit into the CloudThinker platform and agent workflow.
# Skills
Source: https://docs.cloudthinker.io/guide/skills/overview
Extend AI agents with domain-specific knowledge, custom workflows, and organization-specific procedures
Skills are instruction packages that turn general-purpose agents into experts on your environment. You write your team's context once — coding standards, review checklists, incident procedures — and agents apply it automatically on every assigned task.
Agents arrive knowing general best practices, not your conventions. Skills close that gap: instead of restating the same rules in every prompt, you encode them once, and one senior engineer's expertise becomes a standard the whole workspace inherits.
## How it works
1. **Create** — write a `SKILL.md` file manually, or describe what you need in plain language and let AI generate it.
2. **Enable** — toggle the skill on in your workspace. Disabled skills are saved but never loaded.
3. **Assign** — scope the skill to the features where it applies: Code Review, Incident, Assessment, or any combination.
4. **Apply** — when an agent runs a task in an assigned feature, it loads the skill and follows the instructions alongside its default behavior.
![Custom Skills management page showing skill cards with enable toggles and feature assignment checkboxes]()
The Skills page lists every skill with its enable toggle and feature assignments
## What you can do
| Capability | Description | Learn more |
| ------------------------------ | ---------------------------------------------------------------------------------------------------- | -------------------------------------------- |
| Create custom skills | Build workspace-private skills manually or generate them with AI from a plain-language description | [Custom skills](/guide/skills/custom-skills) |
| Assign to features | Scope each skill to Code Review, Incident, or Assessment so it loads only where it belongs | [Custom skills](/guide/skills/custom-skills) |
| Edit with AI | Refine an existing skill through conversation — add examples, remove ambiguity, restructure sections | [Custom skills](/guide/skills/custom-skills) |
| Write effective SKILL.md files | Learn the frontmatter fields, instruction styles, and patterns agents follow most reliably | [Skill format](/guide/skills/skill-format) |
## Skills vs the knowledge base
Skills tell agents how to act; the [knowledge base](/guide/knowledge) gives agents information to look up. They complement each other — a skill can direct an agent to consult specific documents.
| | Skills | Knowledge base |
| ----------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------ |
| Content | Rules and procedures agents follow | Documents and websites agents reference |
| Examples | Review checklists, naming conventions, escalation policies | Runbooks, architecture docs, compliance documents |
| How agents use it | Loaded automatically for assigned features | Searched with `#kb` or referenced automatically in always-available mode |
| Format | A `SKILL.md` file (Markdown with YAML frontmatter) | Uploaded files or crawled websites |
| Best for | Changing agent behavior | Adding facts and context |
Rule of thumb: if the agent must follow it on every run, write a skill. If the agent should look it up when relevant, add it to the knowledge base.
## Get started
Create, enable, and assign your first workspace skill
Learn the SKILL.md structure with a complete worked example
Give agents documents and websites to reference alongside skills
Connect a repository and see an assigned skill applied to a real PR
# Skill Format
Source: https://docs.cloudthinker.io/guide/skills/skill-format
Write SKILL.md files agents follow reliably — file structure, frontmatter fields, and instruction patterns
A skill is a single `SKILL.md` file: YAML frontmatter for metadata, a Markdown body with the instructions agents follow. This page is the reference for that format — structure, frontmatter fields, and the instruction patterns agents apply most reliably.
## File structure
Every skill follows the same shape — frontmatter first, then instruction sections as Markdown headings:
```markdown theme={null}
---
name: security-review-checklist # slug format: lowercase, hyphens
description: Enforce OWASP checks in code review # shown on the skill card
---
## SQL injection # one H2 per rule group
- Flag raw string concatenation in SQL queries
- Require parameterized queries or ORM methods
## Secrets handling
- Flag hardcoded credentials, tokens, and API keys
- Require secrets to come from a secrets manager or environment variables
```
| Part | Required | Purpose |
| ----------------- | ----------- | ------------------------------------------------------------------- |
| Frontmatter block | Yes | Identifies the skill; the UI reads `name` and `description` from it |
| Instruction body | Yes | Markdown the agent loads verbatim when the skill runs |
| H2 rule groups | Recommended | One heading per domain keeps rules scannable for you and the agent |
| Inline examples | Recommended | Concrete good/bad examples that anchor each rule |
## Frontmatter fields
| Field | Required | Description |
| ------------- | -------- | ------------------------------------------------------------------------------------------------------ |
| `name` | Yes | Unique identifier in slug format (lowercase, hyphens, no spaces). Example: `security-review-checklist` |
| `description` | Yes | Short summary of what the skill does. Displayed on the skill card in the UI. |
## Writing effective instructions
### Every token counts
Agents process skill instructions alongside their own system prompt and task context. Keep instructions concise — write only what the agent doesn't already know.
```markdown theme={null}
When reviewing code, you should always check for SQL injection
vulnerabilities. SQL injection is a type of security vulnerability
where an attacker can execute arbitrary SQL commands through
user input that is not properly sanitized...
## SQL Injection
- Flag raw string concatenation in SQL queries
- Require parameterized queries or ORM methods
- Check for `execute()` calls with f-strings or `.format()`
```
### Degrees of freedom
Match your instruction style to how much flexibility the agent should have:
| Freedom | Style | Use case |
| ---------- | ------------------------- | ------------------------------------------------------------- |
| **High** | Guidelines and principles | Creative tasks, architectural advice, exploratory reviews |
| **Medium** | Checklists with examples | Standard code reviews, security audits, compliance checks |
| **Low** | Exact templates and rules | Regulatory compliance, formatting standards, mandatory fields |
### Examples beat explanations
Agents follow concrete examples more reliably than abstract rules. When possible, show the expected behavior instead of describing it.
```markdown theme={null}
## Naming conventions
Use descriptive variable names that reveal intent.
### Good
- `remaining_retries` not `r`
- `is_authenticated` not `auth`
- `max_connection_pool_size` not `pool`
### Bad
- Single-letter variables outside loop iterators
- Abbreviations that aren't universally understood
- Boolean variables without `is_`, `has_`, or `should_` prefix
```
## Complete example
A realistic skill for enforcing PR review standards:
````markdown theme={null}
---
name: pr-review-standards
description: Enforce team PR review standards including size limits, test coverage, and documentation requirements
---
## PR Size
- Flag PRs with more than 500 lines changed as "needs splitting"
- Warn on PRs with more than 10 files changed
- Suggest logical split points when flagging oversized PRs
- Auto-generated files (migrations, lockfiles) do not count toward limits
## Test Coverage
- Require test files for any new public function or method
- Flag removed tests without corresponding feature removal
- Check that edge cases are covered, not just happy paths
## Documentation
- Require JSDoc/docstring for exported functions with more than 3 parameters
- Flag breaking API changes without migration guide updates
- Check that README is updated when adding new environment variables
## Review Comment Format
Use this structure for review comments:
```
**[severity]** Brief title
Description of the issue.
**Suggestion:** How to fix it.
```
Severity levels: `critical`, `warning`, `suggestion`, `nitpick`
````
## Troubleshooting
The `name` field must be in slug format: lowercase letters, numbers, and hyphens only. No spaces, underscores, or special characters.
* `security-checklist` — valid
* `Security Checklist` — invalid
* `security_checklist` — invalid
* Verify the skill is **enabled** (toggle is on)
* Verify the skill is **assigned** to the correct feature
* Check that instructions are specific and actionable — vague guidance is often deprioritized
* Shorter, focused skills are applied more reliably than long, broad ones
If your skill exceeds a few hundred lines, consider splitting it into multiple focused skills. Agents handle several short skills better than one massive skill.
Split by domain: separate security rules, performance guidelines, and style conventions into individual skills.
## Related
Create, upload, enable, and assign skills in your workspace.
Understand how skills fit into the CloudThinker platform and agent workflow.
# Slack Integration
Source: https://docs.cloudthinker.io/guide/slack-integration
Run CloudThinker AI operations directly from Slack — manage infrastructure, review costs, and investigate incidents without leaving your workspace
Transform your Slack workspace into a cloud operations command center. Interact with CloudThinker agents, receive alerts, and run operations without leaving Slack.
**[AgenticOps](/guide/tutorial/agenticops) vs ChatOps**: AgenticOps is the conversational approach to cloud operations — the broader paradigm of talking to AI agents in natural language. ChatOps is how you do AgenticOps *inside Slack and Teams*. The same `@agent #tool` syntax works identically in the CloudThinker web app and chat.
***
## The problem with context switching
When an alert fires, engineers context-switch through 5+ tools: acknowledge in PagerDuty → open AWS console → check CloudWatch → pull up kubectl → open Datadog. By the time they have a full picture, 20 minutes have passed and the team is asking for updates in Slack — the one place everyone already is.
Even routine operations require context switching: checking costs means opening Cost Explorer, checking security findings means opening Security Hub, checking Kubernetes means opening Lens or running kubectl commands.
***
## How SlackOps is different
| Scenario | Traditional Approach | CloudThinker SlackOps |
| ------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------- |
| **Alert fires at 3 AM** | Page → open AWS Console → CloudWatch → correlate manually | `@CloudThinker alex investigate the spike in errors` in #incidents |
| **Cost check** | Open browser → navigate to Cost Explorer → build filter | `@CloudThinker alex what's our AWS spend this week?` |
| **Security audit** | Open Security Hub → filter findings → export report | `@CloudThinker oliver any critical security findings?` |
| **Incident coordination** | Multiple engineers in different dashboards, status updates via manual messages | `@CloudThinker anna coordinate investigation: @alex @tony @kai` |
| **Team alerts** | Configure PagerDuty/OpsGenie routing rules | `@CloudThinker kai #alert pod failures in production → #k8s-alerts` |
SlackOps works because it meets engineers where they already are — and because it provides the full intelligence of CloudThinker agents, not just [notifications](/guide/notifications).
***
## Setup
By default, any Slack workspace member can install apps. If your workspace has **App Approval** enabled, a Workspace Owner or app manager must approve CloudThinker before it can be installed.
Go to [**Admin Settings → Integrations**](https://app.cloudthinker.io/admin-settings/integrations) and find **Slack**
Click **Add to Slack** and authorize via Slack's OAuth flow.
If your workspace has **App Approval** enabled, CloudThinker must be approved
by a Workspace Owner or app manager before you can complete this step. If the
app hasn't been approved yet, you'll be prompted to submit a request.
Select which Slack workspace to install CloudThinker into and confirm the
OAuth scopes. The app will appear in your workspace's app directory.
Map your Slack channels to CloudThinker workspaces so agents know which cloud
accounts to operate on
Optionally select channels for each notification type (cost alerts, security
findings, incident updates)
Try `@CloudThinker alex show AWS costs` in a channel to verify the connection
***
## Syntax
In Slack, prefix all commands with `@CloudThinker`:
```text theme={null}
@CloudThinker alex analyze EC2 spending trends
@CloudThinker oliver audit security groups for public access
@CloudThinker tony check database performance
@CloudThinker kai review EKS cluster health
```
All [CloudThinker Language](/guide/language) syntax works in Slack—just add the `@CloudThinker` prefix.
***
## Examples
### Quick queries
```text theme={null}
@CloudThinker alex what's our AWS spending this week?
@CloudThinker oliver any critical security findings?
@CloudThinker tony why are queries slow on production?
@CloudThinker kai check pod status in payment namespace
```
### With tools
```text theme={null}
@CloudThinker alex #dashboard cost trends by service
@CloudThinker oliver #report weekly security summary
@CloudThinker tony #recommend index optimizations
@CloudThinker kai #alert on pod OOMKilled events
```
### Multi-agent
```text theme={null}
@CloudThinker anna coordinate investigation of latency spike
@CloudThinker alex and tony analyze database infrastructure costs
```
***
## Alerts
Configure agents to send alerts to Slack channels:
```text theme={null}
@CloudThinker alex #alert daily spend exceeds $5,000 → #cloud-ops
@CloudThinker oliver #alert security group changes → #security
@CloudThinker kai #alert pod failures in production → #k8s-alerts
```
***
## Code review notifications
When an AI code review completes on a connected repository, CloudThinker can send a summary notification to your Slack channels. These notifications use Block Kit formatting with:
* **Severity indicators** — color-coded emoji for each severity level (🔴 Critical, 🟠 High, 🟡 Medium, 🟢 Low)
* **Severity breakdown** — count of findings by severity
* **Finding details** — up to 5 unresolved findings with title, file path, and line number
* **Resolved count** — number of previously identified findings that have been resolved
* **View Code Review** button — direct link to the review dashboard
* **Clean pass** — when no issues are found, a simple "No issues found" message
Configure which Slack channel receives code review notifications and set severity thresholds in **Settings > Notifications > Code Review / Pipeline Monitoring**. See [Notifications Center](/guide/notifications) for details.
***
## Channel patterns
| Channel | Use Case |
| --------------- | ------------------------------------------------- |
| `#cloud-ops` | General infrastructure operations and cost alerts |
| `#security` | Security findings and compliance updates |
| `#incidents` | Active incident investigation and coordination |
| `#code-reviews` | Code review findings and pipeline monitoring |
| `#reports` | Scheduled reports and summaries |
***
## Troubleshooting
* Check if CloudThinker app is added to the channel
* Verify you're using `@CloudThinker agent` syntax (not just `agent`)
* Confirm agents are configured in the CloudThinker console
* Check Slack workspace admin consent status
* Check alert configuration in CloudThinker Settings
* Verify Slack channel notification settings
* Ensure the bot has permission to post in the target channel
* Test with `/cloudthinker alerts test`
* **"Missing required scopes"** — The app may have been installed without all required OAuth scopes. Remove the app from the workspace and reinstall it from CloudThinker's Integrations page.
* **Connection appears successful but bot doesn't respond** — This can happen if the OAuth flow did not complete fully. Try removing and reinstalling the app.
* **"not\_allowed\_token\_type" or scope errors** — Your Slack workspace may have App Approval enabled. Ask your Workspace Owner or app manager to approve CloudThinker in Slack's admin dashboard under **Manage Apps**.
* For any other permission errors, check workspace permissions in CloudThinker, or try removing and re-adding the app to the channel.
If your workspace requires app approval, here's what to share with your Workspace Owner or app manager:
1. Sign in to CloudThinker (they need a CloudThinker account with admin access to the organization)
2. Go to [**Admin Settings → Integrations → Slack**](https://app.cloudthinker.io/admin-settings/integrations)
3. Click **Add to Slack** and complete the Slack OAuth flow
4. Approve the app in Slack's **Manage Apps** dashboard
This is a **one-time setup**. Once approved, any workspace member can interact with CloudThinker by mentioning `@CloudThinker` in channels where the app has been added.
***
## Permissions
### Slack OAuth permissions
By default, any workspace member can install the CloudThinker Slack app. If your workspace has **App Approval** enabled, a Workspace Owner or app manager must approve CloudThinker before members can install it. This approval is a one-time operation — once approved, no further admin actions are required.
### CloudThinker permissions
Within Slack, users can only access agents and cloud connections that their CloudThinker account permits. CloudThinker workspace roles apply the same way as they do in the web console.
***
## Related
Complete syntax reference for all commands
Set up CloudThinker in Microsoft Teams
# Microsoft Teams Integration
Source: https://docs.cloudthinker.io/guide/teams-integration
Run cloud operations from Microsoft Teams by interacting with CloudThinker agents, receiving Adaptive Card alerts, and coordinating incidents.
Transform your Microsoft Teams workspace into a cloud operations command center. Interact with CloudThinker agents, receive alerts, and run operations without leaving Teams.
***
## Prerequisites
The initial setup **must be performed by a user with one of the following Azure AD roles**:
* **Global Administrator** — full admin access across all Microsoft 365 services
* **Teams Administrator** — admin access scoped to Microsoft Teams
A regular Microsoft 365 user **cannot** complete the setup. If you don't have one of these roles, ask your IT team to perform the initial connection (Steps 1–3 below). Once the bot is published, any user can interact with it in Teams channels.
### Required Azure AD permissions
During the OAuth flow, CloudThinker requests the following Microsoft Graph API permissions:
| Permission | Type | Purpose |
| ------------------------------------------------- | ----------------- | --------------------------------------------------- |
| `User.Read` | User consent | Read the signed-in user's profile |
| `Channel.ReadBasic.All` | User consent | List channels in teams |
| `Team.ReadBasic.All` | User consent | List teams the user belongs to |
| `TeamsAppInstallation.ReadWriteForTeam` | **Admin consent** | Install the bot into teams |
| `TeamsAppInstallation.ReadWriteAndConsentForTeam` | **Admin consent** | Install the bot and grant resource-specific consent |
| `TeamsAppInstallation.ReadWriteSelfForTeam` | **Admin consent** | Manage the bot's own installation |
| `AppCatalog.ReadWrite.All` | **Admin consent** | Publish the bot to the organization's app catalog |
| `Organization.Read.All` | **Admin consent** | Read organization information |
Permissions marked **Admin consent** can only be granted by a Global Administrator or Teams Administrator. During the OAuth flow, admins will see a **"Consent on behalf of your organization"** checkbox:
* **Checked** — permissions are granted tenant-wide for all users in the Azure AD tenant. Other admins who connect later won't need to re-consent.
* **Unchecked** — permissions are granted only for the admin's own account. This is sufficient to complete the setup since the admin personally has the required role to publish the bot.
### Who should perform the setup?
| Your role | What to do |
| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| **Global Admin or Teams Admin** | You can complete the full setup yourself — proceed to Step 1 below |
| **Regular user** | Ask your Global Admin or Teams Admin to complete Steps 1–3. Once the bot is published, you can use it in any channel where it's installed |
***
## Setup
Go to [**Admin Settings → Integrations**](https://app.cloudthinker.io/admin-settings/integrations) and find **Microsoft Teams**
Click **Connect with Teams** and authenticate via Microsoft OAuth.
You **must** be signed in with a **Global Administrator** or **Teams Administrator** account. If you are a regular user, this step will either:
* Block you at the Microsoft consent screen, or
* Appear to succeed but fail silently when publishing the bot to your organization's app catalog
If you are not an admin, share this page with your IT team and ask them to perform this step.
Select which Teams to add the CloudThinker bot to and confirm the installation. The bot is published to your organization's Teams app catalog — this is a one-time action.
Map your Microsoft Teams to CloudThinker workspaces so agents know which cloud accounts to operate on
Optionally select channels for each notification type (cost alerts, security findings, incident updates)
Try `@CloudThinker alex show AWS costs` in a channel to verify the connection
***
## Syntax
In Teams, prefix all commands with `@CloudThinker`:
```text theme={null}
@CloudThinker alex analyze EC2 spending trends
@CloudThinker oliver audit security groups for public access
@CloudThinker tony check database performance
@CloudThinker kai review EKS cluster health
```
All [CloudThinker Language](/guide/language) syntax works in Teams — just add the `@CloudThinker` prefix.
***
## Examples
### Quick queries
```text theme={null}
@CloudThinker alex what's our AWS spending this week?
@CloudThinker oliver any critical security findings?
@CloudThinker tony why are queries slow on production?
@CloudThinker kai check pod status in payment namespace
```
### With tools
```text theme={null}
@CloudThinker alex #dashboard cost trends by service
@CloudThinker oliver #report weekly security summary
@CloudThinker tony #recommend index optimizations
@CloudThinker kai #alert on pod OOMKilled events
```
### Multi-agent
```text theme={null}
@CloudThinker anna coordinate investigation of latency spike
@CloudThinker alex and tony analyze database infrastructure costs
```
***
## Alerts
Configure agents to send alerts to Teams channels:
```text theme={null}
@CloudThinker alex #alert daily spend exceeds $5,000 → #cloud-ops
@CloudThinker oliver #alert security group changes → #security
@CloudThinker kai #alert pod failures in production → #k8s-alerts
```
***
## Code review notifications
When an AI code review completes, CloudThinker sends an Adaptive Card notification to your configured Teams channel. The card includes:
* **Severity FactSet** — structured table showing counts by severity level (🔴 Critical, 🟠 High, 🟡 Medium, 🟢 Low)
* **Finding details** — up to 5 unresolved findings with title, severity label, file path, and line number
* **Resolved count** — number of previously identified findings that have been resolved
* **View Code Review** button — direct link to the review dashboard
* **Clean pass** — when no issues are found, a simple "No issues found" message
Configure which Teams channel receives code review notifications and set severity thresholds in **Settings > Notifications > Code Review / Pipeline Monitoring**. See [Notifications Center](/guide/notifications) for details.
***
## Channel patterns
| Channel | Use Case |
| --------------- | ------------------------------------------------- |
| `#cloud-ops` | General infrastructure operations and cost alerts |
| `#security` | Security findings and compliance updates |
| `#incidents` | Active incident investigation and coordination |
| `#code-reviews` | Code review findings and pipeline monitoring |
| `#reports` | Scheduled reports and summaries |
***
## Troubleshooting
* Check if the CloudThinker bot is installed in the team
* Verify you're using `@CloudThinker agent` syntax (not just `agent`)
* Confirm agents are configured in the CloudThinker console
* Check Azure AD admin consent status
* Check alert configuration in CloudThinker Settings
* Verify Teams channel notification settings
* Ensure the bot has permission to post in the target channel
* **"Missing required permissions"** — The user who connected does not have the Global Administrator or Teams Administrator role. Ask your IT team to perform the initial connection instead.
* **"Consent on behalf of your organization"** — During OAuth, an admin can check this box to grant permissions tenant-wide. This is optional — if the connecting user is already an admin, their personal consent is sufficient.
* **Connection appears successful but bot doesn't work** — This can happen if a non-admin user completes the OAuth flow. The user's personal authentication succeeds, but publishing the bot to the app catalog fails silently. Ask a Global Administrator or Teams Administrator to reconnect.
* For any other permission errors, check workspace admin permissions in CloudThinker, or try removing and re-adding the bot to the team.
You need a **Global Administrator** or **Teams Administrator** from your organization to complete the initial connection. Here's what to share with them:
1. Sign in to CloudThinker (they need a CloudThinker account with admin access to the organization)
2. Go to [**Admin Settings → Integrations → Microsoft Teams**](https://app.cloudthinker.io/admin-settings/integrations)
3. Click **Connect with Teams** and complete the Microsoft OAuth flow
4. The bot will be published to your organization's Teams app catalog
This is a **one-time setup**. Once the admin completes it, any user in the Teams workspace can interact with the bot by mentioning `@CloudThinker` in channels where the bot is installed.
***
## Permissions
### Azure AD permissions
The initial connection requires a **Global Administrator** or **Teams Administrator** in Azure AD. This is because CloudThinker needs to publish a bot to your organization's Teams app catalog, which is an admin-level action. This setup is a one-time operation — once the bot is published, no further admin actions are required.
During OAuth, the admin can optionally check **"Consent on behalf of your organization"** to grant permissions tenant-wide. If unchecked, permissions are granted only for the admin's own account, which is still sufficient to complete the setup. See the [Required Azure AD Permissions](#required-azure-ad-permissions) table above for the full list of requested scopes.
### CloudThinker permissions
Within Teams, users can only access agents and cloud connections that their CloudThinker account permits. CloudThinker workspace roles apply the same way as they do in the web console.
***
## Related
Complete syntax reference for all commands
Set up CloudThinker in Slack
# First Prompts by Role
Source: https://docs.cloudthinker.io/guide/tutorial/agenticops
Find your role, copy a prompt, and get a real result from your own cloud environment in about 10 minutes.
By the end of this tutorial you will have run one role-specific prompt against your live environment and refined it with follow-ups. Allow **\~10 minutes**.
**What you'll need:** a workspace with at least one cloud connection — agents can't return real results without one. If you haven't set that up, complete the [quickstart](/quickstart) first.
## The syntax
Talk to agents in plain language. The full pattern is:
```text theme={null}
@agent #tool your request
```
* **`@agent`** — who does the work: `@alex` (cloud), `@oliver` (security), `@tony` (database), `@kai` (Kubernetes), `@anna` (coordination).
* **`#tool`** *(optional)* — the output you want: `#dashboard`, `#report`, `#recommend`, `#alert`, `#chart`, `#kb`.
* **Your request** — what you need, in your own words. Add any context inline.
See the [CloudThinker Language reference](/guide/language) for the full syntax.
Sign in at [app.cloudthinker.io](https://app.cloudthinker.io) and click **New chat** in the left sidebar.
**Success state:** an empty conversation panel opens and typing `@` shows the agent picker.
Pick the tab that matches your role, paste the quick win, then try the follow-ups once you see the shape of the output.
**Goal:** Stop paying for cloud you don't use. See where waste lives in under a minute.
**You need:** [AWS](/guide/connections/aws), [Azure](/guide/connections/azure), or [GCP](/guide/connections/gcp) connection. Agent: [@alex](/guide/agents/alex).
**Quick win — find idle and oversized resources:**
```text theme={null}
@alex find EC2 instances with <20% CPU utilization over the last 30 days, plus any unattached EBS volumes and unused Elastic IPs
```
Alex queries the cloud APIs, joins with utilization metrics, and returns a ranked list with projected monthly savings.
**Follow up:**
```text theme={null}
@alex #recommend right-sizing for the top 5 by waste
@alex #dashboard cost trend by service for this quarter
@alex draft a reserved-instance plan for the stable workloads above
```
**Goal:** Find the highest-risk misconfigurations before an auditor or attacker does.
**You need:** AWS, Azure, or GCP connection. Agent: [@oliver](/guide/agents/oliver).
**Quick win — surface public exposure on sensitive ports:**
```text theme={null}
@oliver list security groups with 0.0.0.0/0 access on database, SSH, or RDP ports across all regions
```
Oliver returns the offending rules with resource owner, region, and severity ranking.
**Follow up:**
```text theme={null}
@oliver #report SOC 2 compliance status with prioritized remediation
@oliver audit IAM policies for privilege-escalation paths
@oliver check for IMDSv1 instances or unencrypted EBS volumes
```
**Goal:** Find the queries that are actually hurting you, not the ones you assume are slow.
**You need:** [PostgreSQL](/guide/connections/postgresql) or [MySQL](/guide/connections/mysql) connection. Agent: [@tony](/guide/agents/tony).
**Quick win — slowest queries in the last 24 hours:**
```text theme={null}
@tony show the top 10 queries by total time over the last 24 hours on production PostgreSQL, with execution count and P95 latency
```
Tony pulls from `pg_stat_statements` (or the equivalent), ranks by impact, and shows where the cost is concentrated.
**Follow up:**
```text theme={null}
@tony #recommend indexes for the top 3 queries above
@tony explain why query #2 isn't using the existing index
@tony #dashboard query latency P95 trends by endpoint
```
**Goal:** See where pods are oversized, undersized, or imbalanced — before the next OOMKill or budget review.
**You need:** [Kubernetes connection](/guide/connections/kubernetes). Agent: [@kai](/guide/agents/kai).
**Quick win — pod resource waste across the cluster:**
```text theme={null}
@kai analyze pod resource usage vs requests across all namespaces, surface the largest over- and under-provisioned workloads
```
Kai joins requests/limits with actual usage and ranks the deltas by node-cost impact.
**Follow up:**
```text theme={null}
@kai #recommend HPA policies for the variable workloads above
@kai find nodes with <30% utilization for consolidation
@kai check for pods without resource limits or liveness probes
```
**Goal:** Cut the time from alert to root cause. Get a structured investigation, not raw logs.
**You need:** [Pulse](/guide/pulse/overview) configured, plus the connections for the systems you operate. Agent: [@anna](/guide/agents/anna) to coordinate.
**Quick win — investigate an active alert:**
```text theme={null}
@anna investigate the current incident: pull related metrics, recent deploys, and topology, then propose the top 3 likely causes ranked by evidence
```
Anna delegates to the relevant specialists, gathers evidence in parallel, and returns a hypothesis ladder you can act on.
**Follow up:**
```text theme={null}
@anna #report draft a postmortem from this conversation
@anna pull the matching runbook and walk through the approval gates
@anna check whether this pattern matches any past incident in memory
```
**Goal:** Get a coordinated view across cost, security, performance, and reliability without scheduling four meetings.
**You need:** Connections in place for the domains you want covered. Agent: [@anna](/guide/agents/anna).
**Quick win — multi-agent quarterly review:**
```text theme={null}
@anna coordinate a quarterly infrastructure review:
- @alex top cost optimization opportunities and savings
- @oliver security posture and compliance gaps
- @tony database performance hotspots
- @kai Kubernetes utilization and risk
Consolidate into an executive summary with prioritized actions.
```
Anna delegates, collects, deduplicates, and returns a single executive brief instead of four tabs.
**Follow up:**
```text theme={null}
@anna #report quarterly business review in slide-deck format
@anna track the open actions from last quarter — what shipped, what slipped
@anna draft a roadmap that aligns cost reduction with reliability work
```
**Success state:** the agent replies within \~30 seconds naming resources, queries, or rules you recognize from your own environment — not generic best practices.
Agents keep conversation context, so build on the result instead of starting over:
* **Be specific upfront.** "EC2 costs in us-east-1 for the last 30 days" beats "show me costs".
* **Refine, don't restart.** "Drill into RDS for the items above" works.
* **Combine tools.** Use `#dashboard` first to see the shape, then `#recommend` for the action on the same topic.
* **Let [Anna](/guide/agents/anna) coordinate.** For anything that touches more than one domain, start with `@anna` and name the specialists.
**Success state:** a follow-up prompt returns an answer that references items from earlier in the conversation without you repeating them.
## You're done when…
* [x] Your role's quick win returned results naming resources or queries you recognize
* [x] At least one follow-up prompt built on the first answer without restating it
* [x] You can predict what shape of output `#dashboard`, `#report`, and `#recommend` will produce
## Next steps
Every prompt you just ran is one pass of the Detect → Analyze → Resolve → Validate loop described in [How CloudThinker works](/#how-cloudthinker-works) — the modules below run that loop continuously. Set up your first module:
Connect a repository and get review comments from @cloudthinker-ai on the next pull request.
Turn on CloudKeepers to monitor spend continuously and surface savings as findings.
Route alerts into Pulse so agents investigate before you open a terminal.
Teach agents your team's procedures so they run them your way.
# Workspace Branding
Source: https://docs.cloudthinker.io/guide/workspace-branding
Apply your company logo, name, and chart color palette to artifact PDF exports and in-app charts
Configure branding once at the workspace level and every future [artifact](/guide/artifacts/overview) export picks it up automatically — your logo in the PDF header, your company name in the footer, your 6-color palette across every chart. Workspaces without branding still produce clean PDFs with CloudThinker defaults.
## Prerequisites
* A **Scale**, **Scale +**, or **Enterprise** plan — see [Pricing & Plans](/guide/billing/pricing).
* **Workspace owner** or **org admin** role. Workspace members see the Branding page in view-only mode: inputs are disabled and the Save button is hidden.
* Your logo file: PNG, JPEG, WebP, or SVG, under 5 MB. A horizontal layout with transparency, at least 400 px wide for raster formats, renders best.
## Set up branding
Click your **workspace name** in the top navigation bar, choose **Workspace Settings**, then select the **Branding** tab (palette icon). The page opens with the editor on the left and a live PDF preview on the right — the preview redraws as you type, so you never need a test export to check your work.
![Full Branding settings page with editor on the left and live PDF preview on the right]()
Full Branding page layout — editor and live preview side by side
Drag your file onto the upload zone, or click **Upload Logo** and pick a file. When the upload finishes, your logo appears on a checkerboard background so you can verify transparency, and a **Remove** button appears.
To replace a logo, upload a new file — the old one is deleted from storage automatically.
Logo uploads commit immediately — they don't wait for the Save button. **Save Brand Settings** covers only the company name and colors.
Type your name into the **Company Name** input (max 255 characters). It replaces "CloudThinker" in the footer of every PDF page, as *"Generated by \"*. Special characters are escaped safely.
Each of the six slots — Primary, Secondary, Tertiary, Accent 1–3 — has a color swatch that opens your system color picker and a synchronized hex input (`#RRGGBB`). Paste hex codes from your brand guidelines, or eyedrop them from your logo with the OS picker.
If you only care about one brand color, set **Primary** and leave the other five at defaults — the defaults are designed to harmonize. **Reset to defaults** reverts all six slots (you still need to save).
Click **Save Brand Settings**. Only changed fields are sent.
**Success state:** a green toast reads *"Workspace updated successfully."* On failure, a red toast states the reason and your form values are preserved for retry.
Open any artifact and click **Download**. The PDF shows your logo top-left in the header on every page (up to 160 px wide), charts in your six colors, and *"Generated by \"* in the footer.
![Side-by-side comparison of a default CloudThinker PDF and a fully branded PDF]()
Before and after: a default CloudThinker PDF (left) versus a fully branded export with custom logo, company name, and palette (right)
To reset everything later: click **Remove** next to the logo, clear the company name, click **Reset to defaults** under Chart Colors, then save.
## Picking a 6-color palette
Chart libraries cycle through your slots in order — a 4-series bar chart uses Primary through Accent 1; an 8-slice pie chart wraps back to Primary for slices 7 and 8. A few rules of thumb:
* **Start with your primary brand color** in the Primary slot — it becomes the default series color.
* **Vary hue, not just brightness.** Six shades of blue make multi-series charts unreadable.
* **Keep contrast against white.** PDFs render on white; very pale colors disappear on bar and line charts.
* **Mind color-blind viewers.** Avoid placing red and green next to each other.
* **Iterate in the live preview.** Edit a color, check the sample bar and pie charts, adjust.
## Reference: limits and validation
### Logo
| Property | Value |
| ------------------------ | ------------------------------------------------------------------------------------ |
| Accepted formats | PNG, JPEG, WebP, SVG |
| Max file size | 5 MB |
| Rendered width in PDF | Up to 160 px (aspect ratio preserved) |
| Recommended source width | 400 px+ for raster formats; SVG scales at any size |
| SVG handling | Sanitized: scripts and unsafe attributes stripped before storage |
| File validation | MIME type and magic bytes both checked; mislabeled files rejected |
| Storage | Workspace-scoped path in secure cloud storage; uploaded via one-time presigned URL |
| Cache | Embedded base64 cached \~10 minutes per workspace; replacing the logo invalidates it |
| Auto-cleanup | Old logo deleted when you upload a new one |
### Company name
| Property | Value |
| ---------- | ----------------------------------------------------- |
| Type | Plain text, max 255 characters |
| Default | "CloudThinker" |
| Escaping | HTML-escaped — `&`, `<`, quotes, and accents are safe |
| Appears in | PDF footer only |
### Chart colors
| Property | Value |
| ---------- | ----------------------------------------------------------------- |
| Slots | Exactly 6: Primary, Secondary, Tertiary, Accent 1–3 |
| Format | `#RRGGBB` hex, case-insensitive |
| Rejected | 3-digit shorthand (`#fff`), missing `#`, non-hex characters |
| Applied to | Bar, line, area, step-area, pie, radar, gauge, and scatter charts |
| Cycling | Charts with more than 6 series wrap back to slot 1 |
Default palette (every new workspace):
| Slot | Hex | Color |
| --------- | --------- | ----------- |
| Primary | `#0d9488` | teal |
| Secondary | `#0369a1` | sky blue |
| Tertiary | `#15803d` | green |
| Accent 1 | `#60a5fa` | light blue |
| Accent 2 | `#5eead4` | light teal |
| Accent 3 | `#86efac` | light green |
### Where branding applies
| Element | In-app charts | PDF exports |
| --------------- | ------------- | ---------------------- |
| 6-color palette | ✓ | ✓ |
| Logo | — | ✓ Header on every page |
| Company name | — | ✓ Footer on every page |
### Plans and roles
| Plan | View | Edit |
| -------------------------- | ---- | -------------------- |
| Free, Team | ✓ | — (upgrade required) |
| Scale, Scale +, Enterprise | ✓ | ✓ |
| Workspace role | View | Edit |
| --------------------- | ---- | ---- |
| Member | ✓ | — |
| Owner | ✓ | ✓ |
| Org admin / org owner | ✓ | ✓ |
## FAQ
The embedded logo is cached for up to 10 minutes per workspace, but replacing a logo invalidates the cache, so new exports should pick it up right away. If the old logo persists after a few minutes, refresh the workspace page and export again; if it continues, contact support with your workspace ID.
Blurry: your source image is smaller than the 160 px render width and gets upscaled — re-export at 400 px wide or more, or use SVG. White square: your file has no transparency — export with a transparent canvas (PNG with alpha, or SVG). If an SVG renders differently than in your design tool, it likely relied on scripts or animations that sanitization strips; export a static PNG instead.
Expected. Each PDF is generated at export time with the branding active at that moment, and existing artifacts have no cached PDF — re-download any artifact to get a fresh PDF with current branding.
Yes. Branding is scoped to each workspace, not the organization, so an org with one workspace per customer, environment, or business unit can brand each independently.
Not currently — every export uses the active branding. As a workaround, remove the logo and clear the company name, export, then restore your branding.
Yes. Logos live under a workspace-scoped storage path and are served via short-lived presigned URLs; members of other workspaces cannot access them. Artifacts you share publicly via share links do include your logo and colors.
## Next steps
Generate the dashboards and reports your branding applies to
Create and manage workspaces within your organization
Configure organization settings, billing, and enterprise features
See which plans include workspace branding
# Workspace Members
Source: https://docs.cloudthinker.io/guide/workspace-users
Manage member access and roles within your CloudThinker workspace
Manage who has access to each workspace and what they can do. Workspace members are organization members who have been granted access to a specific workspace.
## Overview
Workspace member management lets you:
* **Control access** — decide which organization members can see and use a workspace
* **Assign workspace roles** — set per-workspace roles that can differ from organization roles
* **Monitor membership** — view member count and quota usage per workspace
Members must first belong to your organization before they can be added to a workspace. To invite someone to your organization, go to **Admin Settings > Organization**.
***
## Adding members to a workspace
There are two ways to grant workspace access:
### From the workspace
1. Open the workspace you want to manage
2. In the Members section, click **Add Member**
3. Select an organization member from the dropdown (only Developers and Viewers are shown — Owners and Admins already have implicit access)
4. Choose a workspace role (pre-filled based on their organization role)
### From admin settings
1. Go to **Admin Settings > Organization**
2. Find the member in the Members section
3. Click the grid icon next to their name to open workspace access settings
4. Toggle access for each workspace and set a role per workspace
Organization Owners and Admins automatically have access to all workspaces — you don't need to add them manually.
***
## Workspace roles
Each member can have a workspace-specific role that overrides their organization role for that workspace. This lets you give a Viewer organization-level member Developer access in a specific workspace, for example.
| Role | Capabilities |
| ------------- | -------------------------------------------------------------------------------------------------- |
| **Admin** | Manage workspace members, configure agents and connections, manage knowledge bases, run operations |
| **Developer** | Use agents, run operations, access connections and knowledge bases |
| **Viewer** | View workspace resources (read-only) |
When managing workspace access from **Admin Settings > Organization** (via the grid icon), you can choose per workspace:
* **Inherit from Org** — uses their organization role
* **Admin / Developer / Viewer** — overrides with a workspace-specific role
***
## Managing workspace members
The workspace members table shows:
| Column | Description |
| --------------------- | ------------------------------------------------------- |
| **Member** | Avatar, display name, and email address |
| **Organization Role** | The member's role at the organization level |
| **Workspace Role** | The member's role in this specific workspace (editable) |
| **Actions** | Remove member from workspace |
### Changing a member's role
Click the workspace role dropdown next to a member to change their role. Only workspace admins and organization admins/owners can change roles. Implicit members (organization Owners and Admins) cannot have their workspace role changed.
### Removing a member
Remove a member from a workspace to revoke their access. This does not remove them from the organization — they can be re-added later.
Removing a member from the organization (via **Admin Settings > Organization**) automatically removes them from all workspaces.
***
## Provisioning and SSO
If your organization uses SSO with JIT or SCIM provisioning, new members are automatically added to the **default workspaces** configured in your provisioning settings. See [SSO setup](/guide/security/sso) for details.
***
## Next steps
Workspace creation and management
Organization-level member management and roles
# Workspaces
Source: https://docs.cloudthinker.io/guide/workspaces
Create workspaces to isolate agents, connections, and knowledge for each team, project, or environment
A workspace is an isolated environment inside your [organization](/guide/organization) that holds its own agents, [connections](/guide/connections/overview), and [knowledge bases](/guide/knowledge). Every account gets a default organization and workspace at signup; billing and member management happen at the organization level.
## Why workspaces
* **Isolation** — separate environments (dev, staging, production), teams, or projects each get their own agents, credentials, and settings.
* **Multi-cloud structure** — each workspace targets one primary cloud provider, so you can run AWS, GCP, and Azure operations side by side in separate workspaces.
* **Access control** — grant members access per workspace, with optional role overrides for fine-grained permissions.
* **Organized resources** — agents, connections, knowledge bases, and operation history stay scoped to the workspace where they belong.
## Create a workspace
Only organization Owners and Admins can create workspaces.
Go to the Organizations page and find your organization.
Click **New workspace** on your organization card.
Choose the primary cloud infrastructure provider for this workspace.
Provide a clear, descriptive name (for example "Production AWS" or "Dev Team GCP") and a description of the workspace purpose, then create the workspace.
The cloud provider selection cannot be changed after workspace creation. Choose based on your primary infrastructure.
### Provider options
| Provider | Example services agents work with |
| ------------------------------------------------ | ----------------------------------------------------------------- |
| ![AWS logo]()
**Amazon Web Services** | EC2, S3, RDS, Lambda |
| ![GCP logo]()
**Google Cloud Platform** | Compute Engine, Cloud Storage, BigQuery, Kubernetes Engine |
| ![Azure logo]()
**Microsoft Azure** | Virtual Machines, Blob Storage, SQL Database, Container Instances |
## What lives in a workspace
Each workspace maintains its own set of resources:
| Resource | Scope |
| ------------------------------------------ | -------------------------------------------------------------------- |
| [Agents](/guide/agents) | Agent customizations, instructions, and workspace-specific knowledge |
| [Connections](/guide/connections/overview) | Cloud provider, database, and service credentials |
| [Knowledge bases](/guide/knowledge) | Documentation, procedures, and references agents draw on |
| Operations | Pre-built and custom operations, plus their run history |
Switch between workspaces with the workspace selector; each workspace keeps its own agents, connections, and settings, so nothing leaks between environments.
## Workspace access
Access is managed through your organization role:
| Organization role | Default workspace access | Can override per workspace? |
| ----------------- | ------------------------------------- | --------------------------- |
| **Owner** | Admin on all workspaces (implicit) | No — always Admin |
| **Admin** | Admin on all workspaces (implicit) | No — always Admin |
| **Developer** | Developer on assigned workspaces only | Yes |
| **Viewer** | Viewer on assigned workspaces only | Yes |
Owners and Admins see every workspace automatically. Developers and Viewers must be assigned to specific workspaces, and each assignment can either inherit their organization role or use an Admin, Developer, or Viewer override for that workspace only.
### Grant workspace access
Go to **Admin Settings → Organization** and scroll to the Members section.
Find the Developer or Viewer, then click the grid icon next to their name to open workspace access settings.
Check or uncheck workspaces to grant or revoke access.
For each workspace, keep **Inherit from Org** or choose a role override, then click **Save**.
Role overrides allow fine-grained control. For example, an organization Viewer can be given Developer access to one workspace where they need to run operations. To invite new members, see [organization members](/guide/organization#organization-members).
## Common workspace patterns
| Pattern | How to split | Typical use |
| --------------------- | ------------------------------------- | -------------------------------------------------------------------------------------- |
| **Environment-based** | Dev → Staging → Production | Separate cloud accounts, stricter access and incident-response knowledge in production |
| **Team-based** | Platform, feature, and security teams | Each team gets agents and knowledge bases tuned to its focus |
| **Provider-based** | One workspace per cloud provider | Keep AWS, GCP, and Azure credentials and operations apart |
Whichever pattern you choose, use consistent naming across workspaces, follow least privilege when assigning access, and clean up workspaces you no longer use.
## Related
Configure organization settings, billing, and member management
Set up AI agents for your workspace operations
# CloudThinker
Source: https://docs.cloudthinker.io/index
Meet the AI agents that manage infrastructure, review code, resolve incidents, and optimize costs across AWS, Azure, GCP, and Kubernetes.
![CloudThinker AI agent orchestrating cloud operations — incidents resolved, PRs reviewed, costs optimized, security remediated, debug output]()
CloudThinker is an Autonomous Cloud Operations (AgenticOps) platform: specialized AI agents manage infrastructure, review code, resolve incidents, and optimize costs across AWS, Azure, GCP, and Kubernetes.
## Start here
Three first tasks, each 5–10 minutes with a result you can verify. New workspace? [Connect AWS](/guide/connections/aws) first — the [quickstart](/quickstart) walks you through it.
Find idle resources, oversized instances, and unused commitments — with projected monthly savings
Connect a Git repository and get AI review comments on the next pull request
Wire Pulse to your monitoring and let agents form hypotheses, gather evidence, and propose remediation
## Choose your goal
Pick the outcome you want next. Each goal maps to a guided path.
**CostOps** — continuous spend audit across AWS, Azure, and GCP with rightsizing recommendations and approval-gated remediation
**Code Review** — every PR reviewed with context from running infrastructure, past incidents, and your team's conventions
**Deep Response Engine** — Pulse strips noise from monitoring; agents investigate the rest and run approved runbooks
**Assessment** — Well-Architected analysis across resources and pillars, on demand
**Autonomous agents + skills** — encode your runbooks, conventions, and policies so the loop runs without restating them
**Tutorial** — run your role's first prompts against your live environment, then follow the chain into your first module setup
## How CloudThinker works
Every module runs the same agentic loop: **Detect → Analyze → Resolve → Validate**.
| Phase | What happens |
| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
| **Detect** | Agents watch signals from your connections — metrics, cost data, pull requests, alerts. |
| **Analyze** | The agent correlates the signal with topology, history, and [team knowledge](/guide/knowledge) to form a plan. |
| **Resolve** | The plan executes under your autonomy mode — [Manual or Auto](/guide/auto-mode) — with [approvals](/guide/approval) gating sensitive actions. |
| **Validate** | The agent verifies the outcome and writes the result back into memory for the next iteration. |
You stay on the loop, not in every step: set the goal, choose the autonomy mode, and intervene when judgment matters. The [AgenticOps field guide](/learn/aio/introduction) covers the reference architecture and governance discipline behind the loop.
## The six modules
AI review on every PR with context from running infrastructure, [past incidents](/guide/incident/incident-memory), and [team conventions](/guide/code-review/convention-rules). Inline comments, reproduction steps, suggested patches.
[Pulse](/guide/pulse/overview) suppresses monitoring noise. When something escalates, agents form hypotheses, gather evidence, and run approved [runbooks](/guide/incident/runbooks).
Continuous spend audit across [AWS](/guide/connections/aws), [Azure](/guide/connections/azure), and [GCP](/guide/connections/gcp). Idle resources, oversized instances, unused commitments — surfaced with projected savings and approval-gated remediation.
Research Preview
Continuous configuration assessment and vulnerability scans across cloud, container, and IaC layers. Findings ranked by exploitability; fixes opened as pull requests.
Agents operate inside Slack, [Microsoft Teams](/guide/teams-integration), and the CLI. Query infrastructure, approve actions, and review changes without leaving your workflow.
Reusable packages of runbooks, conventions, and policies that agents load automatically — your team's expertise compounds instead of leaving with the engineer who wrote it.
## Why CloudThinker
Cloud operations is spread across disconnected consoles — Cost Explorer, Datadog, GitHub, and more — none of which share state, so every incident, cost review, and security fix starts with a human reassembling context. CloudThinker replaces that with a team of agents that already hold the context: they watch your environment continuously, act inside the guardrails you define, and record what they learn so the next run starts smarter. You get the leverage of a larger operations team without the tool sprawl, and every action stays auditable and approval-gated. Start with the [quickstart](/quickstart), or read the [AgenticOps field guide](/learn/aio/introduction) for the architecture and adoption discipline behind the platform.
# Quick Start
Source: https://docs.cloudthinker.io/quickstart
Register, connect AWS, verify the connection, and run your first AI-powered cost analysis — with a clear success state for every step.
This quickstart takes you from zero to a real result you can verify: a list of your idle EC2 instances with projected monthly savings, returned by an AI agent reading your actual AWS account. Allow **\~10 minutes**, mostly spent waiting for AWS to grant model access and CloudThinker to inventory your resources.
**What you'll need:** an email address, an AWS account with admin (or sufficient IAM) permissions to create a role, and a few minutes to read what the agent returns.
***
## The 30-second syntax
Talk to agents in plain language. The full pattern is:
```text theme={null}
@agent #tool your request
```
* **`@agent`** — who does the work: `@alex` (cloud), `@oliver` (security), `@tony` (database), `@kai` (Kubernetes), `@anna` (coordination).
* **`#tool`** *(optional)* — the output you want: `#dashboard`, `#report`, `#recommend`, `#alert`, `#chart`, `#kb`.
* **Your request** — what you need, in your own words. Add any context inline.
See the [CloudThinker Language reference](/guide/language) for the full syntax.
***
Sign up at [app.cloudthinker.io](https://app.cloudthinker.io/auth/register).
1. Enter name, work email, and password
2. Click the verification link in your inbox
3. Confirm your profile
**Success state:** You land in your personal organization with a default workspace ready. The left sidebar shows **New chat**, **Skills**, and **Artifacts** at the top, and an empty conversation panel on the right.
Free 14-day trial — no charge until day 15. See [Pricing](/guide/billing/pricing) for plan limits.
Go to **Settings → Connections → New Connection → AWS**, then pick an authentication method:
| Method | When to use it | Required |
| ---------------------------- | ----------------------------------------------- | --------------------------------------------------------- |
| **IAM Role** *(recommended)* | Production accounts; safer than long-lived keys | Trust policy + permission policy on a role you create |
| **Access Key** | Quick demo or sandbox account | IAM user with programmatic access; access key ID + secret |
For the IAM Role path, CloudThinker shows you the **External ID** to paste into the trust policy and the [minimum permissions](/guide/connections/aws#required-permissions) to attach. The full AWS Console walkthrough — with CloudShell commands and trust-policy JSON — is in the [AWS connection guide](/guide/connections/aws).
**Safe by default:** the recommended starter policy is read-only. Write actions (rightsizing, instance termination, security-group changes) require a separate policy *and* run under your [approval workflow](/guide/approval) — no agent ever pushes a change without your sign-off on the first runs.
**Success state:** the connection card flips to **Connected** with a green dot. Discovery starts automatically; within \~2 minutes you'll see a non-zero resource count (instances, RDS, S3, etc.) on the card. If discovery stays at zero, jump to [Troubleshooting](#troubleshooting) below.
Open the **Infrastructure** tab in the left sidebar.
You should see:
* A list of regions with non-zero resource counts
* At least one of: EC2 instances, S3 buckets, RDS databases, Lambda functions
* A topology view (click the graph icon) showing service relationships
**Success state:** The resource count matches roughly what you expect for the account. If you connected a sandbox with five EC2s and you see five — you're done. If you see zero or your numbers look wrong, see [Troubleshooting](#troubleshooting).
Click **New chat** and paste:
```text theme={null}
@alex give me a summary of my AWS resources — count by service, top regions, and any obviously idle instances
```
Alex queries CloudWatch utilization metrics, joins them with the resource inventory, and returns a structured summary.
**Expected output:** a response within \~30 seconds containing:
* **Resource counts** — EC2, RDS, S3, Lambda, etc. by region
* **Idle candidates** — instances with sustained `<20%` CPU over the last 30 days
* **Projected savings** — monthly dollars at risk if you keep paying for idle resources
* **Inline reasoning** — *"I checked CloudWatch's `CPUUtilization` metric for the last 30 days and filtered instances where the 95th percentile stayed below 20%"*
You can drill into any line: click an instance to see its tags, region, owner, and the full metric history Alex consulted.
For richer output shapes, prefix the prompt with a `#tool`:
```text theme={null}
@alex #dashboard EC2 cost breakdown by instance type over the last 30 days
```
Alex returns an interactive dashboard (charts you can sort/filter) instead of a text answer.
Other tools worth trying once you're connected:
```text theme={null}
@alex #recommend right-sizing options for the over-provisioned instances above
@alex #report monthly AWS cost narrative I can share with finance
```
**Success state:** the dashboard renders inline; recommendations include both the change and the projected monthly delta.
***
## You're done when…
* [x] You can see your AWS resource count in the Infrastructure tab and it matches reality
* [x] `@alex` returned a summary that named specific instances or services you recognize
* [x] You can explain to a teammate what Alex did to find the idle resources (read the inline reasoning back)
If all three are true, you've completed the first-success flow.
***
## Troubleshooting
**Likely cause:** trust policy missing the External ID, or the role's permission policy is too narrow.
**Check:**
1. In the AWS Console → IAM → Roles → your role → Trust relationships, confirm the `sts:ExternalId` condition exactly matches what CloudThinker showed you (no extra whitespace).
2. The role's permission policy includes at minimum `ec2:Describe*`, `rds:Describe*`, `s3:List*`, `cloudwatch:GetMetricStatistics`. Full list: [AWS connection minimum permissions](/guide/connections/aws#required-permissions).
3. The role's max-session-duration is ≥ 1 hour.
**Fix:** correct the trust policy / permissions, then click **Reconnect** on the connection card. Discovery restarts automatically.
**Likely cause:** the role can be assumed but lacks `Describe` permissions on the regions/services your account actually uses, or the account is in a region you haven't enabled.
**Check:**
* Go to **Connection details → Discovery log**. Each region attempted shows its result. `AccessDenied` errors tell you exactly which API call was rejected.
* Confirm the account actually has resources in at least one region the role can see.
**Fix:** widen the permission policy to cover the failing API, or enable the relevant region under the connection's region scope.
**Likely cause:** the connection isn't bound to the current workspace. Each connection lives in one workspace; you might be chatting in a different one.
**Fix:** confirm the workspace selector (top of sidebar) shows the same workspace where you added AWS. Switch workspaces, or re-add the connection in the workspace you're chatting in.
**Likely cause:** Alex couldn't reach a tool (CloudWatch, Cost Explorer) — usually a permission gap. Generic answers are the fallback when concrete data isn't available.
**Check:** scroll to the bottom of Alex's reply; the **Tools used** section lists every API call attempted, with success/failure. Failed calls show the AWS error.
**Fix:** add the missing permission to the role (CloudWatch read for utilization, Cost Explorer read for spend), reconnect, then retry the prompt.
**Likely cause:** initial discovery still running — Alex is waiting for the inventory to finish before reasoning over it. Subsequent prompts are much faster.
**Fix:** wait until the Infrastructure tab shows non-zero resource counts, then re-run the prompt. Discovery typically completes in 2–5 minutes for an account with under 500 resources.
***
## Next steps
Find your role, copy a prompt, get a real result in under a minute
Bring in Datadog, GitHub, Kubernetes, Postgres, Slack — agents activate as connections appear
Add teammates with Owner, Admin, Developer, or Viewer roles and assign workspace access
Add seats and move up from the trial when your team is ready — per-seat billing, upgrade any time
# API
Source: https://docs.cloudthinker.io/guide/api/overview
Access CloudThinker programmatically to manage workspaces, agents, recommendations, and incidents via authenticated HTTPS endpoints
**Preview** — The CloudThinker REST API is in early access. Endpoints, payloads, and authentication flows may change before general availability. [Contact us](https://cloudthinker.io/contact) to join the preview.
The CloudThinker API exposes the same operations available in the web console as authenticated HTTPS endpoints. Use it to embed cloud operations into your own products, automate workflows, or integrate CloudThinker with internal tooling.
***
## What you can do
| Capability | Description |
| ---------------------- | ------------------------------------------------------------------------------------------------------------ |
| Trigger Conversations | Start agent conversations programmatically with `@agent #tool` prompts and stream responses back to your app |
| Query Resources | Fetch the unified inventory of cloud resources, costs, and topology across AWS, GCP, and Azure |
| Manage Recommendations | List, accept, dismiss, or comment on cost, security, and reliability recommendations |
| Drive Incidents | Create incidents, attach signals, request RCA, and read investigation results |
***
## Authentication
The API uses workspace-scoped API keys passed via a bearer token header.
```bash theme={null}
curl https://api.cloudthinker.io/v1/workspaces/{workspace_id}/resources \
-H "Authorization: Bearer $CLOUDTHINKER_API_KEY"
```
| Field | Description |
| --------------- | ------------------------------------------------------- |
| **Scope** | Each key is bound to a single workspace |
| **Permissions** | Inherits the role of the issuing member |
| **Rotation** | Keys can be revoked at any time from workspace settings |
***
## Endpoint surface
| Resource | Purpose |
| --------------------- | --------------------------------------------------------- |
| `/v1/workspaces` | Workspace metadata and members |
| `/v1/conversations` | Start, list, and resume agent conversations |
| `/v1/resources` | Discovered cloud resource inventory |
| `/v1/recommendations` | Cost, security, and reliability recommendations |
| `/v1/incidents` | Incident lifecycle and RCA results |
| `/v1/dashboards` | Built-in and custom dashboard data |
| `/v1/skills` | Workspace skills and execution |
| `/v1/connections` | Cloud and SaaS [connections](/guide/connections/overview) |
***
## Versioning
The API is versioned in the URL path (`/v1/`). Breaking changes ship under a new version; additive changes ship in place. Deprecated versions remain available for at least 12 months after a successor is released.
***
## Related
Use CloudThinker tools from MCP-compatible AI clients
Run CloudThinker operations from your terminal
Receive event-driven notifications instead of polling
Connect cloud providers and SaaS systems
# Pricing
Source: https://docs.cloudthinker.io/guide/billing/pricing
Compare CloudThinker's per-seat plans from the free daily tier to Enterprise and understand what each includes.
CloudThinker offers flexible per-seat plans — from a free daily tier for individuals to enterprise teams managing multi-cloud environments. Start free, then scale as you grow.
Start free with **100 credits every day** — for the Light model and chat,
enough to explore the platform at no cost, no credit card needed.
## Choose your plan
| | **Free** | **Team** ⭐ | **Scale** | **Scale +** | **Enterprise** |
| ------------------- | ------------------------------------------------------- | ----------------------------------------------------------------- | -------------------------------------------------------- | -------------------------------------------------------- | --------------------------------------------- |
| **Monthly** | Free | \$25/seat | \$100/seat | \$200/seat | \$100/seat + usage |
| **Annual** | Free | \$20/seat | \$80/seat | \$160/seat | Custom |
| **Credits** | 100 / day | 200 / seat | 800 / seat | 1,600 / seat | Metered (PAYG) |
| **Seats** | 1 | 1–5 | 5–100 | 5–100 | 20+ |
| **Workspaces** | 1 | 1 | 5 | 10 | Unlimited |
| **Models** | Light only | All models | All models | All models | All models |
| **SSO & Audit log** | — | — | Yes | Yes | Yes |
| **Support** | Community | Email | 99.9% · email | 99.9% · priority | 24×7 · 1h · TAM |
| | [Start free](https://app.cloudthinker.io/auth/register) | [Try free for 14 days](https://app.cloudthinker.io/auth/register) | [Get started](https://app.cloudthinker.io/auth/register) | [Get started](https://app.cloudthinker.io/auth/register) | [Contact us](https://cloudthinker.io/contact) |
Save **\~20%** with annual billing on any paid plan (seat price only). The Team plan includes a **14-day free trial** — no charge until day 15.
Credits are a single pool shared across all workspaces in your org. Scale caps
at 100 seats — beyond 100, move to Enterprise.
***
## AI features by plan
| Feature | Free | Team | Scale | Scale + | Enterprise |
| ------------------- | :--: | :--: | :---: | :-----: | :--------: |
| Code Review | ✓ | ✓ | ✓ | ✓ | ✓ |
| Incident RCA | ✓ | ✓ | ✓ | ✓ | ✓ |
| Prompt Hub Query | ✓ | ✓ | ✓ | ✓ | ✓ |
| CloudKeepers action | — | ✓ | ✓ | ✓ | ✓ |
| Slack Integration | — | ✓ | ✓ | ✓ | ✓ |
| Pay-As-You-Go | — | ✓ | ✓ | ✓ | ✓ |
| Cloud Assessment | — | — | ✓ | ✓ | ✓ |
| Jira Integration | — | — | ✓ | ✓ | ✓ |
| SSO & Audit Log | — | — | ✓ | ✓ | ✓ |
| BYOK | — | — | ✓ | ✓ | ✓ |
| SCIM Provisioning | — | — | ✓ | ✓ | ✓ |
***
## Next steps
How credits work, the order they're consumed, and metered top-up
Upgrade, add seats, and manage invoices and payments
# Subscription
Source: https://docs.cloudthinker.io/guide/billing/subscription
View your plan, upgrade or downgrade, add team members, and track usage and quotas
## Managing your subscription
### View current plan
1. Navigate to **Admin Settings > Billing**
2. View your current plan, usage, and billing cycle
3. See upcoming charges and invoice history
### Upgrade or downgrade
To change your plan:
1. Go to **Admin Settings > Billing**
2. Click **Change Plan**
3. Select your new plan
4. Review the changes and confirm
Plan changes take effect immediately. Upgrades are prorated for the current billing period. Downgrades take effect at the next billing cycle.
### Add team members
For seat-based plans:
1. Navigate to **Admin Settings > Organization**
2. Click **Invite Members**
3. Enter email addresses
4. Select roles and workspace access
5. Additional seats are billed automatically
### How seats work
A **seat** is a paid slot on your subscription. Each seat grants its full credit allocation immediately upon purchase.
* **Only active members use seats.** Pending invitations don't count toward your seat usage.
* **Pre-purchase seats for more credits.** Buy extra seats to boost your organization's credit pool, even before team members join.
* **Mid-cycle additions.** When a new member exceeds your seat count, a seat is added automatically with prorated billing.
* **Member departures.** Credits stay intact for the current cycle. Seat count adjusts at next renewal.
***
## Invoices & payments
### Payment methods
Supported payment methods:
* Credit/debit cards (Visa, Mastercard, Amex)
* ACH bank transfer (US)
* Wire transfer (Enterprise)
* Invoice billing (Enterprise)
### Managing payment methods
1. Go to **Admin Settings > Billing**
2. Add or update payment methods
3. Set default payment method
4. View payment history
### Invoices
Access your invoice history:
1. Navigate to **Admin Settings > Billing**
2. View and download invoices
3. Filter by date range
4. Export for accounting
### Billing cycle
* Monthly plans: Billed on subscription start date
* Annual plans: Billed yearly with discount
* Usage overage: Billed at end of billing period
## Next steps
How credits work, the order they're consumed, and metered top-up
Compare plans and credit allocations
# Usage
Source: https://docs.cloudthinker.io/guide/billing/usage
Track daily, bonus, and premium credit consumption and configure Pay-As-You-Go top-up to keep agent operations running.
Credits are the currency for AI agent operations. Each interaction with an agent consumes credits based on complexity.
## How credits work
Credits are included with your plan and apply to all agent conversations and API usage:
* Every plan gets **100 free daily credits** that refresh every day, for the **Light model and chat only**.
* **Premium credits** (your plan's per-seat allocation) unlock **all models and all features** — Light, Pro, and Ultra model tiers across every app capability (agents, code review, incident RCA, dashboards, integrations, and API usage).
## Consumption order
When you use credits, they are consumed in this order:
100 free credits that reset every day, for the **Light model and chat
only**. Available on all plans.
Promotional credits from referrals or special offers. They work with **all
models** and **never expire**.
Your plan's per-seat credit allocation, reset each billing cycle. Unlike
daily credits, premium credits work with **all models** (Light, Pro, Ultra)
and **every app feature** — no model or feature restrictions.
Automatic top-up when all other credits are exhausted. Charged at
\$0.10/credit. Only if [PAYG](#pay-as-you-go) is enabled.
Monitor your balance, current-period usage, historical trends, and transaction history in real-time from **Admin Settings > Usage**.
***
## Pay-As-You-Go
Never run out of credits during critical operations. Pay-As-You-Go (PAYG) automatically kicks in when all your other credits are exhausted.
| | Details |
| ---------------- | ----------------------------------------------------------------------------- |
| **Rate** | \$0.10 per credit |
| **Availability** | Team, Scale, Scale +, and Enterprise plans |
| **Spending cap** | Set your own monthly limit — minimum \$100; maximum depends on your plan tier |
| **Billing** | Metered — you only pay for what you use at the end of each billing cycle |
| **Control** | Enable or disable anytime from your subscription settings |
PAYG is not available on the Free plan, and is locked during the trial period.
Upgrade to Team or above to enable it.
### Enable or disable PAYG
1. Navigate to **Admin Settings > Billing**
2. Open **Pay-As-You-Go**
3. Toggle PAYG on (or off)
4. Confirm — the change takes effect immediately
### Set your spending cap
The monthly cap limits how much PAYG can charge in a single billing cycle. Once the cap is reached, PAYG stops until the next cycle.
1. In **Admin Settings > Billing > Pay-As-You-Go**, set your **monthly cap**
2. The minimum cap is **\$100/month**; the maximum depends on your plan tier, and your plan's default cap is pre-filled
3. Save — usage above the cap is blocked until the next billing cycle
Caps are per billing cycle and reset each cycle. Lowering the cap mid-cycle
does not refund usage already incurred.
## Next steps
Change plan, add seats, and review usage and quotas
Compare plans and credit allocations
# Bring Your Own Key (BYOK)
Source: https://docs.cloudthinker.io/guide/byok
Route LLM inference through your own AWS Bedrock account for unlimited usage, cost control, and data residency
Bring Your Own Key (BYOK) lets workspaces on Scale and above plans run LLM inference through their own AWS Bedrock credentials instead of platform credits. Usage bills to your AWS account, and you choose which regions serve inference.
***
## Why BYOK
* **Unlimited usage** — your own Bedrock quotas apply, not platform credit limits.
* **Cost control** — charges land on your AWS bill, where you already have budgets and cost allocation tags.
* **Data residency** — route inference through US-only, EU-only, or APAC-only [inference profiles](#inference-profiles).
* **Compliance** — you control where inference happens and who can audit it.
* **No model management** — CloudThinker selects Claude Sonnet 4.5 or Opus 4.5 automatically per task.
***
## Prerequisites
* **Scale, Scale +, or Enterprise plan** — see [Pricing & Plans](/guide/billing/pricing)
* **AWS account** with Amazon Bedrock access enabled
* **IAM credentials** (access key ID and secret access key) with Bedrock invoke permissions
* **Model access** for both Claude Sonnet 4.5 and Claude Opus 4.5 in your AWS account
CloudThinker authenticates with AWS IAM credentials: an access key ID, a secret access key, and an optional session token for temporary credentials. Direct Bedrock API keys are planned but not yet supported.
Long-term credentials (AKIA prefix) support automatic session token refresh. Temporary credentials (ASIA prefix) cannot be refreshed — reconfigure when they expire.
***
## Request Claude model access
Claude models require a one-time use case form in the Bedrock console before you can invoke them. See the [AWS Bedrock model access documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) for details.
Log in to the [AWS Console](https://console.aws.amazon.com/), open [Amazon Bedrock](https://console.aws.amazon.com/bedrock/), and click **Model access** in the left navigation.
Click **Modify model access** and enable both:
* **Claude Sonnet 4.5** (`anthropic.claude-sonnet-4-5-20250929-v1:0`)
* **Claude Opus 4.5** (`anthropic.claude-opus-4-5-20251101-v1:0`)
Complete the form with your use case description, expected usage patterns, and compliance requirements if applicable.
**Success state:** the Model access page shows both models as access granted — typically immediately after submission.
Request access to **both** Sonnet 4.5 and Opus 4.5. CloudThinker switches between them based on task requirements.
***
## Create IAM credentials
Your IAM user needs permission to invoke both models. Create one with the AWS CLI, or via the [AWS Console](https://console.aws.amazon.com/iam/).
```bash theme={null}
aws iam create-user --user-name bedrock-byok-user
```
```bash theme={null}
cat > bedrock-policy.json << 'EOF'
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"bedrock:InvokeModel",
"bedrock:InvokeModelWithResponseStream"
],
"Resource": [
"arn:aws:bedrock:*::foundation-model/anthropic.claude-sonnet-4-5-20250929-v1:0",
"arn:aws:bedrock:*::foundation-model/anthropic.claude-opus-4-5-20251101-v1:0",
"arn:aws:bedrock:*:*:inference-profile/global.anthropic.claude-sonnet-4-5-20250929-v1:0",
"arn:aws:bedrock:*:*:inference-profile/global.anthropic.claude-opus-4-5-20251101-v1:0",
"arn:aws:bedrock:*:*:inference-profile/us.anthropic.claude-sonnet-4-5-20250929-v1:0",
"arn:aws:bedrock:*:*:inference-profile/us.anthropic.claude-opus-4-5-20251101-v1:0",
"arn:aws:bedrock:*:*:inference-profile/eu.anthropic.claude-sonnet-4-5-20250929-v1:0",
"arn:aws:bedrock:*:*:inference-profile/eu.anthropic.claude-opus-4-5-20251101-v1:0",
"arn:aws:bedrock:*:*:inference-profile/apac.anthropic.claude-sonnet-4-5-20250929-v1:0",
"arn:aws:bedrock:*:*:inference-profile/apac.anthropic.claude-opus-4-5-20251101-v1:0"
]
},
{
"Effect": "Allow",
"Action": ["sts:GetSessionToken"],
"Resource": "*"
}
]
}
EOF
```
```bash theme={null}
aws iam put-user-policy \
--user-name bedrock-byok-user \
--policy-name BedrockInvokePolicy \
--policy-document file://bedrock-policy.json
```
```bash theme={null}
aws iam create-access-key --user-name bedrock-byok-user
```
Save the `AccessKeyId` and `SecretAccessKey` from the output — you'll enter these in CloudThinker.
```bash theme={null}
aws configure --profile bedrock-byok-user
aws sts get-caller-identity --profile bedrock-byok-user
```
**Success state:** `get-caller-identity` returns the `bedrock-byok-user` ARN.
These commands require an AWS profile with IAM administrative permissions (`iam:CreateUser`, `iam:PutUserPolicy`, `iam:CreateAccessKey`).
***
## Connect Bedrock in CloudThinker
Go to **Settings → BYOK Settings** and choose **AWS Bedrock** as the provider.
Provide your **Access Key ID** (AKIA or ASIA prefix), **Secret Access Key**, and — only for temporary credentials — a **Session Token**.
Pick **Global**, **US**, **EU**, or **APAC** depending on your [data residency needs](#inference-profiles).
Click **Test Connection** to confirm the credentials are valid and both models are reachable, then click **Save**.
**Success state:** the test passes for both Sonnet 4.5 and Opus 4.5.
Credentials are encrypted at rest and never exposed in API responses or logs.
***
## Inference profiles
Bedrock inference profiles control which AWS regions can serve your requests. For region coverage, see the [AWS inference profile documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-support.html).
| Profile | Routing | Typical use |
| ---------- | ------------------------- | ----------------------- |
| **Global** | Any commercial AWS region | Maximum throughput |
| **US** | US regions only | US data residency |
| **EU** | EU regions only | GDPR compliance |
| **APAC** | APAC regions only | Regional data residency |
***
## How it works
* **Model selection** — CloudThinker picks the appropriate Claude model per task; you never select models manually. See the [supported foundation models list](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html).
* **Workspace inheritance** — the workspace owner configures BYOK once; every member inherits it. All member LLM usage routes through the owner's AWS account, and the owner manages the credentials centrally.
***
## Fallback behavior
CloudThinker can retry a failed BYOK call on platform-managed Bedrock credentials, but this is **opt-in per workspace** and **disabled by default for Enterprise and BYOC plans**. With fallback off, a failed call surfaces the error — no request leaves your credential boundary.
### When fallback triggers (if enabled)
A BYOK call falls back to the platform path only when **all three** are true:
1. The error is retryable — credential expiration, throttling, transient 5xx — not a content or policy violation.
2. The operation is marked **fallback-eligible** (most read operations; never autonomous write actions in production).
3. The workspace's fallback policy is **Allow**.
A fallback re-sends the request to the platform-managed Bedrock account in **the same region** as the original call — never a different jurisdiction. Every fallback emits an audit event (original error code, serving endpoint, user, workspace, model, token count) in **Admin Settings → Audit log**, exportable to your SIEM via [webhooks](/guide/webhooks/overview).
### Set the fallback policy
Go to **Admin Settings → BYOK → Fallback policy**.
| Policy | Behavior on BYOK failure | Typical use |
| ------------------------------------------------------------ | -------------------------------------------------------------- | ---------------------- |
| **Allow** | Retry on platform credentials in-region | Dev / sandbox tenants |
| **Warn** | Retry in-region, but require user re-confirmation next session | Mixed workloads |
| **Strict** *(default and recommended for Enterprise / BYOC)* | Surface the error; never retry on different credentials | Regulated environments |
Toggle **Enforce across all workspaces** so workspace admins can't change the policy locally. Only org owners with the `byok:admin` permission can flip this toggle.
**Strict mode trade-off:** if BYOK credentials are misconfigured or revoked, agent operations fail until you fix them. Plan for credential rotation and quota management, and set up [notifications](/guide/notifications) on BYOK health.
### What inference calls send
Regardless of policy, each BYOK call sends the agent's system prompt and tool definitions, the relevant conversation history, and any retrieved context (topology, memory, runbooks). It never sends raw cloud credentials, other workspaces' data, or — when tokenization is configured in **Admin Settings → Data Protection** — customer PII.
***
## Troubleshooting
* Verify you submitted the use case form in the Bedrock console
* Check that both Sonnet 4.5 and Opus 4.5 are enabled on the **Model access** page
* Wait a few minutes after submission for access to propagate
* Verify the policy includes `bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream`
* Check the model and inference-profile ARNs match the models and profile you use
* Test the permissions directly in the AWS Console
* Verify the access key ID and secret access key are correct and not rotated or revoked
* For temporary credentials, confirm the session token hasn't expired
* Test with `aws sts get-caller-identity`
* Verify both Sonnet 4.5 and Opus 4.5 are accessible and covered by your IAM policy
* Ensure Bedrock is enabled in your AWS account
* Check the region selection matches your model access
* Verify the workspace owner has configured BYOK and it's enabled in settings
* Confirm the owner's plan is Scale, Scale +, or Enterprise
* Check the owner's credentials are still valid
* Long-term credentials (AKIA) refresh automatically
* Temporary credentials (ASIA) can't be refreshed — reconfigure with new credentials
* Check the `session_token_expires_at` timestamp in the configuration
***
## Related
Which plans include BYOK and what each tier covers
Track credit and LLM usage across your workspace
Alert on BYOK credential health before it blocks work
Export audit events to your SIEM
# CLI
Source: https://docs.cloudthinker.io/guide/cli/overview
Run CloudThinker agent conversations, query resources, and manage operations directly from your terminal
**Preview** — The CloudThinker CLI is in early access. Command names, flags, and config formats may change before general availability. [Contact us](https://cloudthinker.io/contact) to join the preview.
The `cloudthinker` CLI brings agent conversations, resource queries, and operations management to the terminal — useful for shell scripting, CI pipelines, and engineers who prefer keyboards over consoles.
***
## What you can do
| Capability | Description |
| ----------------- | ----------------------------------------------------------------------------------- |
| Chat With Agents | Open an interactive session with Alex, Oliver, Tony, Kai, or Anna from the terminal |
| One-Shot Prompts | Run a single `@agent #tool ...` prompt and pipe the result into other tools |
| Script Operations | Wrap recurring cloud-ops tasks in shell scripts and run them in CI |
| Manage Resources | Inspect connections, resources, recommendations, and incidents |
***
## Installation
| Platform | Method |
| ----------- | -------------------------------------------------- |
| **macOS** | `brew install cloudthinker/tap/cloudthinker` |
| **Linux** | `curl -fsSL https://install.cloudthinker.io \| sh` |
| **Windows** | `winget install CloudThinker.CLI` |
| **Docker** | `docker run cloudthinker/cli` |
***
## Authentication
```bash theme={null}
cloudthinker login
```
The CLI uses the same workspace [API keys](/guide/api/overview) as the REST API. Keys are stored in your OS keychain — never written to plaintext config.
***
## Command surface
| Command | Purpose |
| ------------------------------------ | ----------------------------------------------------------- |
| `cloudthinker chat` | Open an interactive agent conversation |
| `cloudthinker ask "@alex #cost ..."` | One-shot prompt, prints the response |
| `cloudthinker resources list` | Query the unified resource inventory |
| `cloudthinker recommendations list` | List active cost, security, and reliability recommendations |
| `cloudthinker incidents create` | Open an incident from the terminal |
| `cloudthinker skills run ` | Execute a [workspace skill](/guide/skills/overview) |
| `cloudthinker connections list` | View configured [connections](/guide/connections/overview) |
***
## Output formats
| Flag | Format |
| --------------- | ----------------------------------------------------- |
| `--format text` | Default — human-readable for terminals |
| `--format json` | Machine-readable for piping into `jq`, scripts, or CI |
| `--format yaml` | Configuration-friendly output |
***
## Related
Same operations over HTTPS endpoints
Use CloudThinker from MCP-compatible AI clients
Schedule recurring operations
Run the same operations from Slack
# Backlog
Source: https://docs.cloudthinker.io/guide/connections/backlog
Connect Backlog by Nulab to CloudThinker for issue tracking, milestone planning, wiki, and pull-request context
Connect your Backlog space to enable CloudThinker agents to track issues, plan milestones, read wikis, and pull ticket context during code review. Backlog authenticates with a **space-scoped API key** that inherits the issuing user's project access.
***
## Prerequisites
* A **Backlog space** with access to the projects you want to work in.
* A **space-scoped API key**, created from the issuing user's **Personal Settings → API** tab.
* For create/update actions: the key's user needs the matching **project permission**.
Read-only tracking works with any key whose user can see the target projects. Creating or updating issues, wikis, and pull requests additionally requires that user to have write permission on the project. If your space restricts API access, a space administrator must enable it before keys can be issued.
***
## Setup
In Backlog, open your profile menu and go to **Personal Settings → API**. Enter a memo (e.g. `cloudthinker`) and click **Submit** to generate the key, then copy it.
Navigate to **Connections → Backlog** and enter:
* **Domain**: your full Backlog domain, e.g. `.backlog.com` — the full host, not just the space key
* **API Key**: the key you just created
Click **Connect**. CloudThinker verifies the credentials and shows a **Connected** status.
Copy the API key when you create it and store it securely. Treat it like a password — anyone with the key has the issuing user's access to the space.
***
## Connection details
| Field | Description | Example |
| --------------------- | ---------------------------------------------------------- | --------------------- |
| **BACKLOG\_DOMAIN** | Full Backlog domain for your space, not just the space key | `.backlog.com` |
| **BACKLOG\_API\_KEY** | Space-scoped API key used to authenticate the connection | — |
***
## Required permissions
The API key inherits the access of the user who issued it. Read operations work with any key whose user can see the target projects. Create, update, and delete operations additionally require the user to have **write permission** on the project **and** explicit [approval](/guide/approval) in CloudThinker.
Follow least privilege: issue the key under a user with only the project access CloudThinker needs, and keep write actions approval-gated rather than removing the guardrail.
***
## Agent capabilities
Once connected, agents have read access to your Backlog projects, issues, wikis, and repositories, plus a set of approval-gated write operations.
| Capability | Description |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| **Project & space discovery** | List projects, categories, custom fields, priorities, and recent space activity |
| **Issue tracking** | Search, read, and count issues; create and update issues, change status, and add comments — **writes require approval** |
| **Milestones & releases** | List milestones/versions and plan releases; create or update them — **writes require approval** |
| **Wiki & documents** | Read project wikis and documents; create or update wiki pages — **writes require approval** |
| **Git & pull requests** | Inspect repositories and pull requests, pull ticket context during code review, and comment on PRs — **writes require approval** |
### Verify the connection
```text theme={null}
@alex list my Backlog projects and show the open issues in one of them
```
### Example prompts
```text theme={null}
@alex summarize the open Backlog issues for the PLATFORM project and #report anything overdue
@alex this PR references PROJ-123 — pull the Backlog ticket and #recommend whether the change matches it
@alex list the milestones for my main project and their completion status
```
For spaces with many projects, scope requests to a single project or milestone so the agent returns focused results.
***
## Troubleshooting
The API key is missing, expired, or revoked — or API access is not enabled for the space. Issue a fresh key under **Personal Settings → API**, confirm the space allows API access, and reconnect.
**BACKLOG\_DOMAIN** is set to just the space key instead of the full domain. Use the complete host, e.g. `.backlog.com`, and reconnect.
The key's user is not a member of any project, or lacks visibility on the ones you expect. Add the user to the target project in Backlog, then re-run discovery.
The key's user lacks write permission on the project. Re-issue the key under a user with the right project role, and approve the action when prompted.
Very large responses are trimmed to keep replies focused. Scope the request to a single project, milestone, or status, or ask for a count first and then drill in.
***
## Security
* **Least privilege** — grant only the permissions the agents need for your use case; start read-only and widen later.
* **Read-only by default** — use read-only credentials unless you want agents to make changes through this connection.
* **Rotate credentials** — rotate keys and tokens on your normal schedule; CloudThinker picks up the new value when you update the connection.
* **Revoke on offboarding** — remove the credential at the provider when you delete a connection or a teammate leaves.
- **Least-privilege user** — issue the API key under a user with only the project access CloudThinker needs.
- **Correct domain** — store the full domain (`.backlog.com`), not just the space key.
***
## Related
Jira and Confluence issue tracking and knowledge base
How approval-gated actions work
# MCP
Source: https://docs.cloudthinker.io/guide/connections/mcp
Connect custom tools and APIs to CloudThinker via Model Context Protocol (MCP) for extensible agent capabilities
Model Context Protocol (MCP) enables CloudThinker to connect to custom tools and services. Use MCP to extend agent capabilities with your own [connections](/guide/connections/overview).
***
## What is MCP?
MCP (Model Context Protocol) is an open protocol for connecting AI agents to external tools and data sources. It provides a standardized way to:
* Expose tools that agents can call
* Provide context and data to agents
* Enable custom integrations with any service
***
## Setup
Ensure your MCP server is running and accessible:
* Supports Server-Sent Events (SSE) transport
* Has a publicly accessible endpoint (or VPN access)
* Returns proper MCP protocol responses
Navigate to **Connections → MCP** and click **Add MCP Connection**.
Enter connection details:
* **Name**: descriptive name for the connection
* **Prefix**: tool prefix (e.g., `mytools` → `mytools_function_name`)
Set up the connection transport:
* **Transport Type**: Server-Sent Events
* **URL**: your MCP server endpoint
* **Timeout**: connection timeout (default: 5 seconds)
* **SSE Read Timeout**: event stream timeout (default: 30 seconds)
Configure any required environment variables:
* API keys
* Authentication tokens
* Custom configuration
Test the connection and save. CloudThinker shows a **Connected** status when the server responds correctly.
***
## Connection details
| Field | Description | Required |
| ------------------------- | --------------------------------- | ---------------- |
| **Name** | Connection display name | Yes |
| **Prefix** | Tool name prefix | Yes |
| **URL** | MCP server endpoint | Yes |
| **Timeout** | Connection timeout (seconds) | No (default: 5) |
| **SSE Read Timeout** | Event stream timeout (seconds) | No (default: 30) |
| **Environment Variables** | Key-value pairs for configuration | No |
***
## Example MCP server
A simple MCP server that provides a weather lookup tool:
```python theme={null}
from mcp import Server, Tool
server = Server("weather-tools")
@server.tool()
async def get_weather(location: str) -> str:
"""Get current weather for a location."""
# Your implementation here
return f"Weather for {location}: Sunny, 72°F"
if __name__ == "__main__":
server.run()
```
Once connected with prefix `weather`, agents can call:
```
weather_get_weather(location="San Francisco")
```
***
## Use cases
### Custom data sources
Connect proprietary databases or APIs:
* Internal metrics systems
* Custom monitoring tools
* Business intelligence platforms
### Specialized tools
Add domain-specific capabilities:
* Custom compliance checks
* Internal automation scripts
* Legacy system integrations
### External services
Integrate third-party services:
* Communication platforms
* Ticketing systems
* Cloud services not built-in
***
## Best practices
* **HTTPS endpoints** — expose your MCP server only over HTTPS.
* **Rotate API keys** — rotate credentials stored in environment variables on your normal schedule.
* **Implement authentication** — validate incoming requests from CloudThinker before processing.
* **Set appropriate timeouts** — tune Timeout and SSE Read Timeout to match your server's response characteristics.
* **Monitor server health** — track error rates and latency so connection drops are caught early.
* **Handle errors gracefully** — return structured MCP error responses rather than crashing the server.
***
## Troubleshooting
Verify the MCP server is running, check network connectivity, increase timeout values, and confirm the firewall allows traffic.
Verify the prefix is set correctly, check that the MCP server returns tool definitions, ensure the protocol version is compatible, and review server logs for errors.
Verify environment variables are set correctly, check that API keys are valid, and ensure the server validates credentials correctly.
Increase the SSE read timeout, check for proxy interference, verify the server sends keepalive events, and review network stability.
***
## Security
* **Least privilege** — grant only the permissions the agents need for your use case; start read-only and widen later.
* **Read-only by default** — use read-only credentials unless you want agents to make changes through this connection.
* **Rotate credentials** — rotate keys and tokens on your normal schedule; CloudThinker picks up the new value when you update the connection.
* **Revoke on offboarding** — remove the credential at the provider when you delete a connection or a teammate leaves.
- **HTTPS endpoints** — expose your MCP server only over HTTPS and validate the CloudThinker request origin.
- **Scoped credentials** — store API keys and tokens in environment variables rather than hardcoding them in server code.
***
## Related
All connection types
How agents use connections
# Deployment
Source: https://docs.cloudthinker.io/guide/deployment/overview
Deploy CloudThinker in self-hosted, dedicated, or air-gapped environments for regulated and data-residency-constrained use cases
**Preview** — Self-hosted CloudThinker is in early access for design partners. Supported topologies, system requirements, and upgrade paths may change before general availability. [Contact us](https://cloudthinker.io/contact) to discuss your environment.
Most CloudThinker customers run on the multi-tenant SaaS at [app.cloudthinker.io](https://app.cloudthinker.io). For environments with strict data residency, compliance, or network-isolation requirements, CloudThinker also supports dedicated and self-hosted deployments.
***
## Deployment models
| Model | Description |
| ------------------- | ------------------------------------------------------------------------------------------ |
| SaaS (Multi-Tenant) | Default — managed by CloudThinker. Fastest path to value, no operational burden. |
| Dedicated Cloud | Single-tenant instance hosted by CloudThinker in your preferred region and cloud |
| Self-Hosted | Run CloudThinker inside your own VPC on Kubernetes — your data never leaves your perimeter |
| Air-Gapped | Fully offline install for classified or regulated environments |
***
## System requirements
| Resource | Minimum (POC) | Production |
| ------------------ | ------------------ | ------------------- |
| **Kubernetes** | 1.34+ | 1.34+ |
| **Nodes** | 3 × 4 vCPU / 16 GB | 6+ × 8 vCPU / 32 GB |
| **PostgreSQL** | 14+ | 15+ with HA |
| **Object storage** | S3 or MinIO | Same |
| **LLM access** | AWS Bedrock | Same |
***
## Upgrades
| Channel | Cadence |
| ---------- | ----------------------------------------- |
| **Stable** | Monthly |
| **LTS** | Quarterly with 12-month support |
| **Patch** | As needed for security and critical fixes |
Helm-based upgrades are performed in place with zero downtime for the API and agent runtime.
***
## Compliance
Self-hosted deployments inherit the security posture of your cluster — keep CloudThinker inside the same boundary as the workloads it operates on. Self-hosted installs do not need re-certification because your audit boundary already covers the cluster.
***
## Related
Authentication, RBAC, and security controls
SAML and OIDC single sign-on
Automated user provisioning
# Knowledge Base
Source: https://docs.cloudthinker.io/guide/knowledge
Create and manage knowledge bases to enhance your AI agents with organizational knowledge, documents, and procedures
Give your AI agents access to your organizational knowledge by uploading documents or crawling websites. Your agents can then reference this information to provide better, more contextual responses.
***
## The problem with generic AI
Generic AI models know AWS best practices — but they don't know *your* AWS environment. They don't know that your company policy requires all S3 buckets to use SSE-KMS with a specific key ARN. They don't know that your disaster recovery RTO is 4 hours, not the AWS-recommended 1 hour. They don't know your incident response procedure mandates an Opsgenie alert before any remediation action.
The result: generic AI recommendations that conflict with your policies, contradict your existing documentation, or miss company-specific context that every senior engineer knows.
**Knowledge Base solves this.** By connecting your runbooks, policies, compliance documents, and architecture documentation, CloudThinker agents operate with your organization's institutional knowledge — not just general cloud best practices.
***
## How it compares to generic RAG
| Approach | Generic AI | CloudThinker Knowledge Base |
| --------------------------- | ---------------------- | ----------------------------------------------------- |
| **Company policies** | Unknown | Indexed and referenced in every relevant response |
| **Custom runbooks** | Not available | Automatically pulled when relevant |
| **Architecture context** | Generic best practices | Your specific design decisions and constraints |
| **Compliance requirements** | Standard frameworks | Your specific audit requirements and evidence formats |
| **Incident procedures** | Generic steps | Your company's exact escalation paths and tools |
| Source type | What it does |
| ------------------- | -------------------------------------------------------------------- |
| **Document Upload** | Add PDFs, Word docs, and text files to create knowledge repositories |
| **Web Crawling** | Automatically index websites, wikis, and documentation sites |
***
## Quick setup
Go to **Knowledge** in CloudThinker and click **"Create Knowledge Base"**
Select **Document Upload** or **Web Crawler** based on your content source
* **Name**: Give it a descriptive name
* **Description**: Brief overview of the content
* **Access**: Choose who can use this knowledge base
Upload documents or set up web crawling (see sections below)
***
## Document upload
### Supported files
* **Text**: TXT, Markdown, RTF
* **Office**: Word (.docx), PowerPoint (.pptx), Excel (.xlsx)
* **PDF**: Adobe PDF files
* **Web**: HTML, XML, JSON
### How to upload
1. Click **"Upload Document"**
2. Choose your file (max 50MB)
3. Add title and tags
4. Click **"Upload"**
1. Drag and drop up to 100 files
2. Or upload a ZIP file
3. Add batch metadata
4. Click **"Process All"**
***
## Web crawling
### Basic setup
Provide the starting webpage URL to crawl
* **Depth**: How many clicks deep (1-10)
* **Pages**: Maximum pages to crawl (100-10,000)
* **One-time**: Single crawl
* **Daily**: Update every 24 hours
* **Weekly**: Weekly updates
* **Manual**: Only when you trigger it
### Common examples
* Documentation sites (GitBook, Confluence)
* Company wikis
* Help centers and FAQs
* GitHub repositories
***
## Using knowledge bases
### Access modes
| Mode | Behavior |
| -------------------------------- | ------------------------------------------------------------ |
| **Always Available** (automatic) | Agents automatically use this knowledge in all conversations |
| **Manual Search** (on-demand) | Agents only use when you ask with `#kb` command |
### Agent commands
```text theme={null}
# Search any knowledge base
@alex #kb find AWS cost optimization strategies
# Search specific knowledge base
@tony #kb:database-guide PostgreSQL backup procedures
# Let agents decide when to use knowledge (Always mode)
@oliver analyze this security incident
# Oliver automatically references security procedures
```
### Agent examples
**Knowledge Types:**
* AWS documentation and guides
* Cost optimization playbooks
* Infrastructure procedures
**Example:**
```
@alex #kb find EC2 cost savings
→ References uploaded cost optimization guide
→ Provides specific recommendations
```
**Knowledge Types:**
* Security policies and procedures
* Compliance documentation
* Incident response playbooks
**Example:**
```
@oliver #kb security incident response
→ References uploaded incident procedures
→ Provides step-by-step guidance
```
**Knowledge Types:**
* Database administration guides
* Performance tuning documentation
* Backup and recovery procedures
**Example:**
```
@tony #kb PostgreSQL performance tuning
→ References database optimization guide
→ Suggests specific improvements
```
***
## Best practices
### Content organization
**Document structure (make it searchable):**
* Use clear headings (H1, H2, H3)
* Include table of contents
* Add relevant keywords
* Use bullet points and lists
**Tagging (smart organization):**
* Use consistent tag names
* Include functional tags (security, database, aws)
* Add priority levels (critical, reference)
* Include department/team tags
### Maintenance tips
* Review and update content monthly
* Remove outdated documents
* Monitor which content gets used most
* Test search functionality regularly
***
## Quick examples
### Example 1: Company procedures
```yaml theme={null}
Name: "Operations Procedures"
Type: Document Upload
Content:
- Employee handbook (PDF)
- IT policies (Word docs)
- Emergency procedures (PDF)
Access: Always Available
Agents: All agents can reference
```
### Example 2: Technical documentation
```yaml theme={null}
Name: "AWS Documentation"
Type: Web Crawler
URL: "https://docs.aws.amazon.com/"
Schedule: Weekly updates
Access: Manual (#kb command)
Agents: Alex (primary), others as needed
```
### Example 3: Security policies
```yaml theme={null}
Name: "Security Compliance"
Type: Document Upload
Content:
- SOC 2 checklist (PDF)
- Security policies (Multiple PDFs)
- Incident templates (Word)
Access: Manual (sensitive content)
Agents: Oliver (full access), others (limited)
```
***
## Getting started checklist
* [ ] Identify key documents and websites
* [ ] Decide on naming conventions
* [ ] Choose access levels for each knowledge base
* [ ] Create knowledge base in CloudThinker
* [ ] Upload 3-5 important documents OR set up web crawling
* [ ] Add proper tags and descriptions
* [ ] Test search functionality
* [ ] Set knowledge base access for each agent
* [ ] Test manual search with `#kb` commands
* [ ] Configure "Always Available" for critical knowledge
* [ ] Train your team on usage commands
* [ ] Check which content gets used most
* [ ] Update outdated information
* [ ] Add new knowledge based on team needs
* [ ] Gather feedback and optimize
***
## Troubleshooting
**Solutions:**
* Check if agent has access to the knowledge base
* Verify knowledge base is set to "Always Available" or use `#kb` command
* Improve document titles and add more keywords
* Test search with different terms
**Solutions:**
* Check file size (max 50MB per file)
* Ensure file type is supported
* Verify document isn't password-protected
* Try uploading one file at a time
**Solutions:**
* Verify the website URL is accessible
* Check if website blocks crawlers (robots.txt)
* Start with smaller page limits
* Try crawling a single page first
***
## Next steps
**Success Tips:**
* Start small with your most important documents
* Use clear, descriptive names for everything
* Test search functionality before going live
* Train your team on the `#kb` command syntax
**Get Started Now**
Create your first knowledge base and start uploading content
**Setup AI Agents**
Configure your agents to use your knowledge bases effectively
# MCP Server
Source: https://docs.cloudthinker.io/guide/mcp/overview
Expose CloudThinker's cloud-ops tools and data to external MCP-compatible AI clients like Claude Desktop, Cursor, and Windsurf
**Preview** — The CloudThinker MCP server is in early access. The tool surface and transport options may change before general availability. [Contact us](https://cloudthinker.io/contact) to join the preview.
CloudThinker can act as a [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server, letting any MCP-compatible AI client call CloudThinker tools and read CloudThinker data using your workspace's authenticated context.
This is the inverse of the existing [MCP connection](/guide/connections/mcp), which lets CloudThinker agents call *your* external MCP servers.
***
## What it enables
| Capability | Description |
| --------------------- | --------------------------------------------------------------------------------------------------- |
| Cloud Ops in Your IDE | Query AWS, GCP, Azure inventory and costs from Cursor, Claude Desktop, or any MCP client |
| Use Your Connections | Reach databases, Kubernetes clusters, and observability tools you already connected to CloudThinker |
| Run Skills Remotely | Invoke your workspace [skills](/guide/skills/overview) from any MCP-compatible host |
| Shared Context | Read recommendations, dashboards, and incident memory the same way the web console does |
***
## Exposed tools
The MCP server surfaces a curated subset of agent capabilities:
| Tool family | What it does |
| ------------------- | ------------------------------------------------------------------------------------ |
| `resources.*` | Query the unified [resource inventory](/guide/infrastructure/resources) and topology |
| `recommendations.*` | List, accept, dismiss, or comment on optimization recommendations |
| `costs.*` | Run cost queries and forecasts across cloud accounts |
| `incidents.*` | Create incidents, attach signals, and read RCA results |
| `dashboards.*` | Read built-in and custom dashboards |
| `skills.*` | Execute workspace skills with parameters |
The exact tool list is exposed through MCP discovery — clients see whatever the workspace API key is permitted to use.
***
## Transports
| Transport | Best For |
| -------------- | ----------------------------------------------------------------------------------------- |
| **HTTP + SSE** | Cloud-hosted clients, remote agents, and team-shared setups |
| **stdio** | Local IDEs and desktop apps (Claude Desktop, Cursor, Windsurf) launched via a local proxy |
***
## Authentication
The MCP server authenticates with the same workspace [API keys](/guide/api/overview) used by the REST API. The key is scoped to a single workspace and inherits the role of the issuing member.
***
## Related
Connect external MCP servers as CloudThinker tools
Same operations over a REST surface
Package domain knowledge into reusable workflows
Terminal access to the same capabilities
# Notifications Center
Source: https://docs.cloudthinker.io/guide/notifications
Configure and manage role-specific notifications across email, Slack, and Microsoft Teams channels for timely cloud operations alerts
Stay informed about critical cloud operations with intelligent, role-specific notifications. Configure how and where you receive alerts for CloudThinker features with flexible severity filtering and multiple delivery channels.
***
## The problem with alert fatigue
Generic cloud alerting sends everything to everyone — and when everyone gets everything, nobody reads anything. Security engineers don't need to see every cost recommendation. Finance teams don't need Kubernetes OOMKill alerts. On-call engineers don't need compliance digest emails at 3 AM.
The result: critical alerts get buried in noise, teams disable notifications entirely, and important findings go unacted on.
CloudThinker's Notifications Center solves this with role-based routing and severity filtering: each team member configures what they care about, at what severity threshold, through which channel — so the right person gets the right alert at the right time.
| Feature | Description |
| ------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| **Smart Routing** (channel-based delivery) | Route notifications to email or Slack channels based on feature categories and severity levels |
| **Alert Fatigue Control** (intelligent filtering) | Configure severity thresholds to receive only the alerts that matter to your role |
| **Multi-Channel Support** (flexible preferences) | Receive notifications via email, Slack, Microsoft Teams, or any combination, tailored to each feature |
| **Multi-Workspace** (workspace isolation) | Configure separate notification preferences for each workspace you manage |
***
## Quick start
In CloudThinker, navigate to **Settings > Notifications**
Choose which CloudThinker feature you want to configure notifications for
(Alerts, Agents & Tasks, Code Review, etc.)
Toggle Email, Slack, and/or Microsoft Teams on for the selected feature
Choose your minimum severity threshold (Critical, High, Medium, Low)
For external email recipients, confirm the verification link sent to your
email address
***
## Configuration interface
![Notifications main settings page showing Slack notification channel configuration]()
The notification system organizes around CloudThinker's core features, allowing independent configuration for each area. Access comprehensive settings by selecting each feature category.
### Feature categories
Configure notifications independently for each area:
* **Alerts** - Critical and operational alert notifications for system events
* **Agents & Tasks** - Agent interruptions and task completions
* **[Code Review](/guide/code-review/setup) / Pipeline Monitoring** - Code review findings with severity breakdown, pipeline execution status, and failure notifications
* **[CloudKeepers](/guide/infrastructure/cloudkeepers) / [Assessment](/guide/infrastructure/assessment)** - Cost optimization recommendations and AWS assessment updates
* **Discussions** - Team discussion activity, replies, and resolved discussions
* **Resources** - Infrastructure resource changes and updates
***
## Channel configuration
Each feature category supports flexible channel configuration with independent control:
### Email notifications
![Email recipients input form showing how to add multiple email addresses for notifications]()
Send notifications to verified email addresses, supporting both workspace members and external recipients.
**Key Features:**
* Verified external recipients for security
* Multiple email addresses per feature
* Workspace member defaults
* Email verification required for external addresses
### [Slack integration](/guide/slack-integration)
![Slack channel dropdown selector showing available channels for notification routing]()
Route notifications to Slack channels instead of direct messages, reducing notification fatigue while maintaining team awareness. Start typing to filter channels by name — the search matches partial names and is case-insensitive, making it easy to find channels in large workspaces.
**Key Features:**
* Channel-based routing (not direct messages)
* Searchable channel picker with type-ahead filtering
* Reduces notification overload
* Team-wide visibility and opt-in
* Context-aware channel selection
### [Microsoft Teams integration](/guide/teams-integration)
Route notifications to Microsoft Teams channels using Adaptive Cards for rich, structured alerts. Teams notifications work the same way as Slack — configure per-feature categories, select a Teams channel, and set severity thresholds.
**Key Features:**
* Adaptive Card format with structured data and action buttons
* Channel-based routing with searchable picker
* Per-category channel selection (e.g., code review findings to `#code-reviews`, incidents to `#incidents`)
* Same severity threshold filtering as other channels
* Requires [Teams integration](/guide/teams-integration) to be connected first
***
## Severity filtering
![Alerts section showing email and Slack severity filter controls with High and Medium thresholds]()
Filter notifications by severity level to receive only alerts meeting your threshold:
* **Low** - Informational updates and general notifications
* **Medium** - Important updates requiring attention and review
* **High** - Critical issues affecting operations that need prompt action
* **Critical** - System-critical alerts requiring immediate action
Each channel can have independent severity thresholds. For example, configure Slack for Medium and above while Email is set to High and above.
***
## Notification categories in detail
### Alerts
![Alerts configuration showing system alerts with severity filtering for email and Slack channels]()
System-wide alert notifications for critical events and important operational updates requiring attention.
### Agents & tasks
![Agents & Tasks section showing agent interrupts and scheduled task resource missing notifications with channel toggles]()
Notifications for AI agent activity and task execution:
* **Agent Interrupts** - When an AI agent pauses and requires your action (e.g., to approve a tool call)
* **Scheduled Task Resource Missing** - When a scheduled task cannot run because a required resource was deleted or unavailable
### [Code Review](/guide/code-review/setup) / Pipeline monitoring
![Code Review section showing pipeline monitoring notifications with email enabled and Slack disabled]()
#### Pipeline monitoring
* Pipeline execution status updates
* Pipeline failure notifications with AI-analyzed logs
#### Code review findings
When an AI code review completes on a pull/merge request, CloudThinker sends a notification with:
* **Severity summary** — count of critical, high, medium, and low findings
* **Finding details** — title, severity, file path, and line number for up to 10 findings
* **Resolved tracking** — count of previously identified findings that have been resolved
* **Action link** — direct link to the code review dashboard
Code review findings notifications are sent automatically after each review completes. Configure severity thresholds to control which reviews trigger a notification — for example, set to **High** to only be notified when critical or high-severity findings are detected.
Each notification channel renders findings in its native format:
* **Slack** — [Block Kit message](/guide/slack-integration#code-review-notifications) with severity emoji indicators
* **Microsoft Teams** — [Adaptive Card](/guide/teams-integration#code-review-notifications) with FactSet severity table
* **Email** — HTML table with findings details and dashboard link
### [CloudKeepers](/guide/infrastructure/cloudkeepers) / [Assessment](/guide/infrastructure/assessment)
![Assessment section showing AWS Well-Architected Assessment notification configuration with toggles for email and Slack channels]()
Notifications for cost optimization and cloud assessment updates:
* Cost optimization recommendations from [CloudKeepers](/guide/infrastructure/cloudkeepers)
* AWS [Well-Architected Assessment](/guide/infrastructure/assessment) completion and results
* Cost anomaly detection
### Discussions
![Discussions section showing replies and resolved discussions notification options with email enabled]()
Notifications for team collaboration activities:
* **Replies** - When someone replies directly to one of your comments
* **Resolved Discussions** - When a discussion you're involved in is marked as resolved
***
## Key features
### Email verification
To prevent unauthorized access and data leakage, external email recipients must be verified before receiving notifications:
1. Enter the external email address in the notification settings
2. A verification link is sent to that email address
3. The recipient must click the verification link to confirm
4. Only verified addresses receive workspace notifications
This ensures that sensitive cloud operations data is only distributed to intended recipients.
### Slack channel-based routing
Instead of sending direct messages to individual users, notifications route to Slack channels:
* **Better Context** - All team members can see notifications in the channel
* **Reduced Fatigue** - Single channel per feature instead of individual messages
* **Opt-In Subscriptions** - Team members can choose to join specific notification channels
* **Team Awareness** - Operations visibility across your team
### Multi-workspace support
Configure separate notification preferences for each workspace you belong to:
* Different severity thresholds per workspace
* Role-specific configurations
* Workspace-specific channels
* Tailored alert strategies based on workspace context
***
## Best practices
### Reducing alert fatigue
* **Configure severity filters appropriately** - Set High or Critical for roles that manage many features
* **Use Slack or Teams channels** - Route team notifications to channels instead of direct messages
* **Disable unused features** - Turn off notifications for features your team doesn't actively manage
* **Gradual enablement** - Start with Critical alerts, then gradually add lower severity levels
### Security considerations
* **Always verify external emails** - Ensure only authorized recipients have access to sensitive data
* **Use workspace-specific channels** - Route sensitive operations to dedicated Slack or Teams channels
* **Review recipients regularly** - Periodically audit who receives notifications
* **Restrict channel access** - Limit notification channel membership to relevant team members
* **Monitor notification activity** - Track who receives what notifications for compliance
### Configuration strategy
1. **Start with Critical + High severity** for critical operational alerts
2. **Gradually add Medium severity** as your team becomes familiar with the system
3. **Use different channels for different features** - Keep infrastructure alerts separate from cost alerts
4. **Tailor by role** - Adjust thresholds based on team member responsibilities
5. **Test notification delivery** - Verify emails and Slack messages are reaching intended recipients
***
## Next steps
**Begin Setup**
Configure your first notification preferences in the CloudThinker Settings
**Understand Agents**
Get to know CloudThinker's AI agents and their capabilities
**Setup Slack**
Connect your Slack workspace for richer team collaboration
**Setup Teams**
Connect your Microsoft Teams workspace for Adaptive Card notifications
# Security & Authentication
Source: https://docs.cloudthinker.io/guide/security/overview
Secure access to CloudThinker with MFA, SSO, and role-based access control
CloudThinker implements enterprise-grade security to protect your cloud infrastructure data and ensure only authorized users can access sensitive operations.
***
## Why security matters for an AI operations platform
CloudThinker agents have read (and optionally write) access to your cloud infrastructure. This is powerful — it enables autonomous analysis and optimization across all your cloud accounts. But it also means the platform itself needs to be hardened against unauthorized access, credential leakage, and prompt injection attacks.
The security model is designed around these principles:
* **Least-privilege by default**: agents use read-only credentials unless write access is explicitly configured and approved
* **Zero persistence**: no agent actions modify infrastructure without going through the [approval workflow](/guide/approval)
* **Defense-in-depth**: MFA, SSO, RBAC, audit logging, and sandbox isolation combine to protect against both external attacks and internal misuse
* **Transparent**: every agent action is logged with the user who initiated it, the approval chain, and the exact command executed
## Authentication methods
| Method | Description |
| --------------------------- | --------------------------------------------------------- |
| Email & Password | Standard authentication with secure password requirements |
| Multi-Factor Authentication | TOTP-based MFA for additional security |
| Single Sign-On (SSO) | SAML/OIDC integration with your identity provider |
| API Keys | Secure programmatic access to the API |
***
## Multi-factor authentication (MFA)
Add an extra layer of security with TOTP-based MFA:
### Enabling MFA
Go to **Profile > Security Settings**
Click **Enable Multi-Factor Authentication**
Use an authenticator app (Google Authenticator, Authy, 1Password, etc.) to scan the QR code
Enter the 6-digit code from your authenticator to confirm setup
Download and securely store your backup codes for account recovery
### MFA at login
With MFA enabled:
1. Enter your email and password
2. Enter the current 6-digit code from your authenticator
3. Access granted
### Recovery options
If you lose access to your authenticator:
* Use a backup code (one-time use)
* Contact organization admin for MFA reset
* Contact support with identity verification
Store backup codes securely. Each code can only be used once. If you lose all backup codes and your authenticator, account recovery may require identity verification.
***
## Single sign-on (SSO)
Enterprise plans support [SSO](/guide/security/sso) integration:
SSO is configured by Organization Owners in **Admin Settings → Identity and access**. Key capabilities:
* **Domain verification** — prove ownership of your email domain before enabling SSO
* **SAML or OIDC** — choose the protocol your identity provider supports
* **SSO enforcement** — optionally require all users with a verified domain to authenticate via SSO
* **JIT provisioning** — auto-create user accounts on first SSO login
* **SCIM directory sync** — automate user and group provisioning from your IdP
### Configuring SSO
Step-by-step setup for Google Workspace, Azure AD, AWS IAM Identity Center, Okta, OneLogin, and generic SAML/OIDC
### SSO enforcement
Organization admins can enforce SSO:
* Require all users to authenticate via SSO
* Disable password-based login
* Auto-provision users on first SSO login
* Auto-deprovision when removed from IdP
***
## Role-based access control (RBAC)
Control what users can do with granular permissions:
### Organization roles
| Role | Description | Permissions |
| ---------- | --------------------------- | ------------------------------------------------- |
| **Owner** | Full organization control | All permissions, billing, member management |
| **Admin** | Organization administration | Manage workspaces, members, settings (no billing) |
| **Member** | Standard access | Access assigned workspaces, use agents |
| **Viewer** | Read-only access | View dashboards and reports only |
### Workspace roles
| Role | Description | Permissions |
| ------------------- | ---------------------- | --------------------------------------------------- |
| **Workspace Admin** | Full workspace control | All workspace operations, member management |
| **Editor** | Standard operations | Run agents, create recommendations, modify settings |
| **Operator** | Limited operations | Run agents, view data, cannot modify settings |
| **Viewer** | Read-only | View dashboards, reports, and recommendations |
### Permission matrix
| Action | Owner | Admin | Editor | Operator | Viewer |
| ----------------------- | ----- | ----- | ------ | -------- | ------ |
| View dashboards | Yes | Yes | Yes | Yes | Yes |
| Run agent conversations | Yes | Yes | Yes | Yes | No |
| Create recommendations | Yes | Yes | Yes | No | No |
| Approve operations | Yes | Yes | Yes | No | No |
| Manage connections | Yes | Yes | Yes | No | No |
| Manage members | Yes | Yes | No | No | No |
| Organization settings | Yes | Yes | No | No | No |
| Billing | Yes | No | No | No | No |
***
## API authentication
Secure programmatic access to CloudThinker:
### API keys
Generate API keys for automation:
1. Go to **Profile > API Keys**
2. Click **Create API Key**
3. Name the key and set expiration
4. Copy the key (shown only once)
5. Use in API requests
```bash theme={null}
# Example API request
curl -H "Authorization: Bearer " \
https://api.cloudthinker.io/v1/workspaces
```
### Key management
* **Rotation**: Regularly rotate keys (recommended: every 90 days)
* **Scoping**: Limit keys to specific operations when possible
* **Monitoring**: Review key usage in audit logs
* **Revocation**: Immediately revoke compromised keys
Never commit API keys to version control. Use environment variables or secret management tools.
### OAuth tokens
For integrations that use OAuth:
* Tokens are automatically refreshed
* Revoke access from **Settings > Connected Apps**
* Monitor token usage in audit logs
***
## Data security
### Encryption
CloudThinker protects your data with:
| Layer | Protection |
| ----------- | ---------------------------- |
| **Transit** | TLS 1.3 for all connections |
| **At Rest** | AES-256 encryption |
| **Secrets** | Encrypted credential storage |
| **Backups** | Encrypted database backups |
### [Bring your own key (BYOK)](/guide/byok)
Enterprise customers can use their own encryption keys:
1. Configure AWS KMS or similar
2. Provide key ARN to CloudThinker
3. Your key encrypts sensitive data
4. Maintain full key control
Set up Bring Your Own Key encryption
### Data residency
* Choose data region during workspace creation
* Data stays within selected region
* Multi-region options for redundancy
***
## Audit logging
Track all activities in CloudThinker:
### Logged events
* User authentication (login, logout, MFA)
* Resource access and modifications
* Agent conversations and actions
* Administrative changes
* API access
### Viewing audit logs
1. Navigate to **Admin Settings > Organization**
2. Filter by:
* User
* Action type
* Resource
* Date range
3. Export logs for compliance
### Log retention
* Standard: 90 days
* Professional: 1 year
* Enterprise: Configurable (up to 7 years)
***
## Security best practices
Require MFA for all organization members, especially those with admin access. Consider enforcing via [SSO](/guide/security/sso) policies.
Assign the minimum role required for each user's responsibilities. Review and adjust permissions regularly.
Rotate API keys, refresh tokens, and cloud credentials on a regular schedule.
Regularly review audit logs for suspicious activity. Set up alerts for critical events.
Use read-only credentials when possible. Limit scope to necessary services and regions.
Conduct quarterly access reviews. Remove inactive users and revoke unnecessary permissions.
***
## Compliance
CloudThinker maintains compliance with:
* **SOC 2 Type II**: Security, availability, and confidentiality
* **GDPR**: Data protection for EU users
* **HIPAA**: Healthcare data handling (Enterprise)
* **ISO 27001**: Information security management
Contact us for security questionnaires and compliance documentation
## Related
Configure SAML or OIDC SSO with Google Workspace, Azure AD, Okta, and more
Automate user and group sync from your identity provider
# Set Up SCIM Provisioning
Source: https://docs.cloudthinker.io/guide/security/scim
Automate user and group provisioning from your identity provider with SCIM 2.0
SCIM provisioning is available on **Scale**, **Scale +**, and **Enterprise** plans. It requires an active SSO connection.
This guide covers how to enable SCIM directory sync, configure group-to-workspace mappings, manage tokens, and monitor sync activity.
**Before you begin:** Complete the steps in [Set up single sign-on (SSO)](/guide/security/sso) first. SCIM builds on top of an active SSO connection — you'll need domain verification and SSO configured with your identity provider before proceeding.
***
## How SCIM works in CloudThinker
SCIM (System for Cross-domain Identity Management) is an industry standard that lets your identity provider automatically keep CloudThinker in sync with your company directory. Instead of manually inviting and removing users, your IdP handles it for you.
When SCIM is enabled:
* **Users assigned** to your CloudThinker application in your IdP are automatically added to your organization
* **Users removed** from your IdP application are automatically deactivated in CloudThinker
* **Groups pushed** from your IdP appear in CloudThinker and can be mapped to specific workspaces and roles
* **Group membership changes** in your IdP automatically update workspace access in CloudThinker
Behind the scenes, your IdP calls CloudThinker's SCIM 2.0 API using a bearer token whenever it needs to create, update, or remove users and groups.
***
## Step 1: Enable SCIM in CloudThinker
Go to **Admin Settings → Identity and access**. The **Provisioning & directory sync** card appears below the SSO card when your SSO connection is active.
Click the **SCIM** option in the provisioning mode selector. CloudThinker generates a bearer token and a SCIM endpoint URL.
A dialog appears with two values you'll need to paste into your identity provider:
* **Bearer token** — the secret token your IdP uses to authenticate with CloudThinker's SCIM API
* **SCIM base URL** — the endpoint your IdP will send requests to (format: `https:///api/v1/scim/{org_id}/v2`)
Copy both values now — the bearer token is only shown once.
Copy the bearer token immediately — it cannot be retrieved after you close the dialog. If you lose it, you'll need to rotate the token.
***
## Step 2: Configure SCIM in your identity provider
Now switch to your identity provider's admin console and set up SCIM provisioning for your CloudThinker application. The exact steps vary by provider (Okta, Azure AD / Microsoft Entra, OneLogin, etc.), but the values you need are the same.
### General configuration
In your IdP's SCIM or provisioning settings for the CloudThinker application, enter:
| Field | Value |
| --------------------------- | -------------------------------------------------- |
| **SCIM connector base URL** | The SCIM base URL from Step 1 |
| **Authentication mode** | HTTP Header / Bearer Token |
| **Bearer token** | The token from Step 1 |
| **Unique identifier field** | `userName` (this maps to the user's email address) |
### What your IdP can sync
| Operation | Supported | What it does |
| ---------------------- | --------- | ------------------------------------------------------------- |
| Create Users | Yes | Adds new user accounts to your CloudThinker organization |
| Update User Attributes | Yes | Keeps name, email, and active status in sync |
| Deactivate Users | Yes | Disables access when a user is deactivated in your IdP |
| Delete Users | Yes | Removes the user from your organization |
| Push Groups | Yes | Syncs your IdP groups to CloudThinker for workspace mapping |
| Group Membership | Yes | Keeps group members in sync between your IdP and CloudThinker |
CloudThinker supports filtering by `userName` for user lookups and `displayName` for group lookups. Bulk operations are not currently supported.
***
## Step 3: Map groups to workspaces and roles
Once your IdP starts pushing groups via SCIM, they appear automatically in the **Group mappings** section on the Identity and access page. Group mappings let you control which workspaces users land in and what role they get, based on their group membership in your IdP.
For each synced group, you can configure:
| Setting | Description |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Auto-assigned workspaces** | Which workspaces members of this group are automatically added to |
| **Auto-assigned organization role** | The organization role for members of this group — choose Viewer, Developer, or Admin. Select "Use connection role" to use the default role from your SSO connection settings. |
After configuring SCIM in your IdP, assign groups to your CloudThinker application. Groups show up in the **Group mappings** table once your IdP pushes them — this may take a few minutes depending on your IdP's sync interval.
For each group, select which workspaces its members should have access to using the workspace dropdown.
Optionally choose a specific role for each group. If you leave it as "Use connection role," members get the default role from your SSO connection settings.
Click **Save** for each group row to apply the mapping.
If no groups have synced yet, the Group mappings section shows a message: "No groups synced yet. Groups will appear here automatically once your identity provider pushes them via SCIM."
***
## Managing SCIM tokens
### Rotate a token
If your token is compromised or your security policy requires periodic rotation, you can rotate it without downtime:
1. In the **Directory sync (SCIM)** section, click **Rotate token**
2. Confirm the rotation — the old token stays valid for 24 more hours, giving you time to update your IdP
3. Copy the new token and update it in your IdP's SCIM configuration
Both the old and new tokens work during the 24-hour overlap window, so your IdP won't experience any sync failures while you make the switch.
### Revoke a token
Switching away from SCIM (to Manual or JIT) automatically revokes the SCIM token and stops all directory sync. Your IdP will no longer be able to create, update, or remove users in CloudThinker.
***
## Monitoring sync activity
Click **View sync logs** in the Directory sync section to see a history of all SCIM operations. This is useful for verifying that your IdP is pushing changes correctly and for troubleshooting provisioning issues.
Each log entry shows:
| Field | Description |
| ------------- | ------------------------------------------------------------------------- |
| **Timestamp** | When the operation occurred |
| **Action** | What happened (e.g., CREATE, UPDATE, DELETE) |
| **Resource** | Which user or group was affected |
| **Status** | Whether it succeeded (SUCCESS), was skipped (SKIPPED), or failed (FAILED) |
***
## Disabling SCIM
If you no longer need directory sync, you can disable SCIM:
1. In the **Provisioning & directory sync** card, select **Manual** or **Just-in-time**
2. Confirm the action — this immediately revokes your SCIM token and stops all directory sync
3. Existing users remain in your organization, but your IdP can no longer add or remove users automatically
Disabling SCIM takes effect immediately. Your identity provider will start receiving authentication errors on its next sync cycle. Update or disable SCIM provisioning in your IdP to avoid unnecessary error alerts.
***
## Troubleshooting
### Groups aren't appearing in CloudThinker
Make sure groups are assigned to your CloudThinker application in your IdP's provisioning settings. CloudThinker doesn't pull groups from your IdP — your IdP needs to push them. Check **View sync logs** to see if any group sync attempts have been made.
### Users aren't being provisioned
1. Verify the user is assigned to your CloudThinker application in your IdP
2. Check that the SCIM token hasn't been rotated or revoked — if in doubt, rotate the token and update your IdP
3. Open **View sync logs** and look for FAILED entries — these often include details about what went wrong
4. Make sure the user's email address is valid and not already associated with a different CloudThinker organization
### Users aren't being deprovisioned
1. Make sure the user was removed from the CloudThinker application in your IdP (removing them from a group alone isn't enough — they need to be unassigned from the application)
2. Check sync logs for a DELETE or DEACTIVATE action for that user
3. Some IdPs don't sync changes instantly — for example, Microsoft Entra syncs approximately every 40 minutes. Wait for the next cycle and check again.
### SCIM token stopped working
The token may have been rotated or revoked by another admin. Generate a new token in CloudThinker and update your IdP's SCIM configuration with the new value.
### "Cannot enable JIT provisioning while SCIM is active"
JIT (Just-in-Time) provisioning and SCIM can't be used at the same time. To switch to JIT, first select a different provisioning mode (Manual or JIT) — this will revoke the SCIM token and stop directory sync.
***
## Related
Configure SAML or OIDC single sign-on (prerequisite for SCIM)
MFA, RBAC, API keys, and data security
Manage organization members and roles
Workspace-level access control
# Single Sign-On (SSO)
Source: https://docs.cloudthinker.io/guide/security/sso
Configure SAML or OIDC SSO with Google Workspace, Azure AD, AWS IAM Identity Center, Okta, and more
Single Sign-On lets your team authenticate to CloudThinker using your existing identity provider — no separate passwords, automatic user provisioning, and centralized deprovisioning when someone leaves.
SSO is available on **Scale**, **Scale +**, and **Enterprise** plans. To get started, navigate to **Organization Settings → Security → SSO** and click **Add Connection**.
***
## How the setup wizard works
The SSO wizard has three steps: **Protocol → SP Metadata → IdP Configuration**.
1. **Protocol** — Choose SAML 2.0 or OIDC
2. **SP Metadata** — Copy CloudThinker's values into your IdP when creating the SAML app
3. **IdP Configuration** — Paste your IdP's values back into CloudThinker
Use the tabs below to follow the exact steps for your identity provider.
***
## SAML setup
### Google Workspace — SAML setup
Go to [admin.google.com](https://admin.google.com) → **Apps → Web and mobile apps → Add app → Add custom SAML app**.
Give it a name like **CloudThinker** and click **Continue**.
On the **Google IdP information** screen, download the **IdP metadata XML** or note:
* **SSO URL** (Single Sign-On URL)
* **Entity ID** (`https://accounts.google.com/o/saml2?idpid=...`)
* **Certificate** (download the X.509 certificate)
Click **Continue**.
Copy the values from **CloudThinker → Settings → Security → SSO → SP Metadata**:
| Google Field | CloudThinker Value |
| ------------------ | -------------------------------------------- |
| **ACS URL** | Paste the **ACS URL** from CloudThinker |
| **Entity ID** | Paste the **SP Entity ID** from CloudThinker |
| **Name ID format** | `EMAIL` |
| **Name ID** | `Basic Information > Primary email` |
Click **Continue**.
Add the following attribute mappings:
| Google Directory Attribute | App Attribute |
| -------------------------- | ------------- |
| Primary email | `email` |
| First name | `firstName` |
| Last name | `lastName` |
Click **Finish**.
In the app settings, set access to **On for everyone** (or target specific organizational units).
Back in CloudThinker's SSO wizard, paste in:
* **Entity ID** from Google
* **SSO URL** from Google
* **Certificate** (paste the X.509 certificate content)
Click **Create Connection** and then **Test** to verify.
Use the **Import** field in CloudThinker to paste your Google IdP metadata URL — this auto-fills Entity ID, SSO URL, and Certificate in one step.
### Microsoft Azure AD (Entra ID) — SAML setup
In the [Azure Portal](https://portal.azure.com), go to **Microsoft Entra ID → Enterprise applications → New application → Create your own application**.
Name it **CloudThinker**, select **Integrate any other application you don't find in the gallery**, and click **Create**.
Open the new application → **Single sign-on → SAML**.
Click **Edit** on **Basic SAML Configuration** and fill in from CloudThinker's SP Metadata:
| Azure Field | CloudThinker Value |
| -------------------------- | -------------------------- |
| **Identifier (Entity ID)** | Paste the **SP Entity ID** |
| **Reply URL (ACS URL)** | Paste the **ACS URL** |
| **Sign on URL** | Same as ACS URL |
Save.
In **Attributes & Claims**, confirm the `emailaddress` claim maps to `user.mail`. Optionally add:
* `firstName` → `user.givenname`
* `lastName` → `user.surname`
In **SAML Signing Certificate**, download the **Federation Metadata XML** or copy:
* **App Federation Metadata URL** (recommended — use this to auto-import into CloudThinker)
* **Certificate (Base64)**
* **Login URL** (SSO URL)
* **Azure AD Identifier** (Entity ID)
Go to **Users and groups → Add user/group** and assign who should have access to CloudThinker.
In CloudThinker's IdP Configuration step, use the **Import** field to paste the **App Federation Metadata URL** — this auto-fills all fields. Or enter manually:
* **Entity ID**: Azure AD Identifier
* **SSO URL**: Login URL
* **Certificate**: Certificate (Base64)
Click **Create Connection** and **Test**.
### AWS IAM Identity Center (AWS SSO) — SAML setup
In the [AWS Console](https://console.aws.amazon.com), navigate to **IAM Identity Center → Applications → Add application → Add custom SAML 2.0 application**.
Give it a display name like **CloudThinker** and optionally add a description.
In the **IAM Identity Center metadata** section, copy or download:
* **IAM Identity Center SAML metadata file** (or the metadata URL)
* **IAM Identity Center issuer URL**
* **IAM Identity Center sign-in URL**
* **Certificate**
In the **Application metadata** section, paste values from CloudThinker's SP Metadata:
| IAM Identity Center Field | CloudThinker Value |
| ----------------------------- | -------------------------- |
| **Application ACS URL** | Paste the **ACS URL** |
| **Application SAML audience** | Paste the **SP Entity ID** |
Go to **Assigned users and groups** → **Assign users and groups** and select who should access CloudThinker.
In **Attribute mappings**, add:
| User attribute in the application | Maps to this string value or user attribute in IAM Identity Center |
| --------------------------------- | ------------------------------------------------------------------ |
| `Subject` | `${user:email}` — Format: `emailAddress` |
| `email` | `${user:email}` |
| `firstName` | `${user:givenName}` |
| `lastName` | `${user:familyName}` |
Back in CloudThinker, use **Import** to paste the IAM Identity Center metadata URL, or enter manually:
* **Entity ID**: IAM Identity Center issuer URL
* **SSO URL**: IAM Identity Center sign-in URL
* **Certificate**: from the metadata file
Click **Create Connection** and **Test**.
### Okta — SAML setup
In the [Okta Admin Console](https://your-org.okta.com/admin), go to **Applications → Applications → Create App Integration → SAML 2.0**.
Name the app **CloudThinker** and click **Next**.
Fill in from CloudThinker's SP Metadata:
| Okta Field | CloudThinker Value |
| ------------------------------- | -------------------------- |
| **Single sign-on URL** | Paste the **ACS URL** |
| **Audience URI (SP Entity ID)** | Paste the **SP Entity ID** |
| **Name ID format** | `EmailAddress` |
| **Application username** | `Email` |
In **Attribute Statements**, add:
| Name | Value |
| ----------- | ---------------- |
| `email` | `user.email` |
| `firstName` | `user.firstName` |
| `lastName` | `user.lastName` |
After saving, go to the app's **Sign On** tab → **SAML Signing Certificates** section → click **Actions → View IdP metadata** to get the metadata XML URL.
Or copy directly:
* **Identity Provider Single Sign-On URL**
* **Identity Provider Issuer**
* **X.509 Certificate**
Go to the **Assignments** tab and assign users or groups who should have access.
In CloudThinker's IdP Configuration step, paste the **Okta metadata URL** into the **Import** field, or enter manually:
* **Entity ID**: Identity Provider Issuer
* **SSO URL**: Identity Provider Single Sign-On URL
* **Certificate**: X.509 Certificate
Click **Create Connection** and **Test**.
### OneLogin — SAML setup
In the [OneLogin Admin Portal](https://app.onelogin.com/admin), go to **Applications → Applications → Add App → Search for "SAML Custom Connector (Advanced)"** and click it.
Set the display name to **CloudThinker** and click **Save**.
Go to the **Configuration** tab and paste in from CloudThinker's SP Metadata:
| OneLogin Field | CloudThinker Value |
| -------------------------------- | -------------------------------------- |
| **Audience (EntityID)** | Paste the **SP Entity ID** |
| **ACS (Consumer) URL** | Paste the **ACS URL** |
| **ACS (Consumer) URL Validator** | `.*` (or the exact ACS URL as a regex) |
| **Login URL** | Same as ACS URL |
Save.
Go to the **Parameters** tab and add:
| Field name | Value |
| ----------- | ---------- |
| `email` | Email |
| `firstName` | First Name |
| `lastName` | Last Name |
Go to the **SSO** tab and copy:
* **Issuer URL** (Entity ID)
* **SAML 2.0 Endpoint (HTTP)**
* **X.509 Certificate** (View Details → copy the certificate)
Go to **Users** tab and add users or roles that should have access.
In CloudThinker, enter:
* **Entity ID**: Issuer URL
* **SSO URL**: SAML 2.0 Endpoint (HTTP)
* **Certificate**: X.509 Certificate
Click **Create Connection** and **Test**.
### Generic SAML 2.0
Use this for any SAML-compliant identity provider not listed above.
#### Step 1 — Get CloudThinker's SP metadata
Navigate to **Organization Settings → Security → SSO → Add Connection → SAML**. On the **SP Metadata** screen, copy:
| Field | What to do with it |
| ------------------- | ----------------------------------------------------------------- |
| **ACS URL** | Paste into your IdP's "Reply URL" or "ACS URL" field |
| **SP Entity ID** | Paste into your IdP's "Audience" or "Entity ID" field |
| **SP Metadata URL** | Some IdPs let you import this URL to auto-fill all fields at once |
#### Step 2 — Create a SAML app in your IdP
Create a new SAML application in your identity provider and enter the SP values above. Configure user attribute mappings:
| CloudThinker Attribute | IdP Attribute |
| ---------------------- | ----------------------- |
| `email` (NameID) | User's primary email |
| `firstName` | Given name / first name |
| `lastName` | Family name / last name |
#### Step 3 — Configure IdP details in CloudThinker
After creating the SAML app in your IdP, return to CloudThinker and complete the **IdP Configuration** step:
| Field | Where to find it |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| **Display Name** | Choose any label (e.g., "Okta SAML") |
| **Entity ID** | Your IdP's entity identifier (sometimes called "Issuer") |
| **SSO URL** | Your IdP's single sign-on endpoint URL |
| **Certificate** | The X.509 signing certificate from your IdP (base64-encoded) |
| **SLO URL** | *(Optional)* Single logout endpoint — only needed if you want users logged out of the IdP when they sign out of CloudThinker |
| **NameID Format** | Leave as "Email Address" unless your IdP requires a different format |
If your IdP provides a metadata URL or XML file, use the **Import** field at the top to auto-fill Entity ID, SSO URL, and Certificate — this saves time and avoids copy-paste errors.
Click **Create Connection**.
***
## OIDC setup
### Google Workspace — OIDC setup
Go to [console.cloud.google.com](https://console.cloud.google.com) → **APIs & Services → Credentials → Create Credentials → OAuth 2.0 Client ID**.
Application type: **Web application**.
Add the **Redirect URI** from CloudThinker's OIDC SP Metadata. Click **Create**.
Copy the **Client ID** and **Client Secret**.
In CloudThinker's OIDC configuration:
* **Discovery URL**: `https://accounts.google.com/.well-known/openid-configuration`
* **Client ID**: paste from Google
* **Client Secret**: paste from Google
Click **Create Connection** and **Test**.
### Azure AD (Entra ID) — OIDC setup
In [Azure Portal](https://portal.azure.com) → **Microsoft Entra ID → App registrations → New registration**.
Name it **CloudThinker**. Under **Redirect URI**, select **Web** and paste the redirect URI from CloudThinker's OIDC SP Metadata.
Go to **Certificates & secrets → New client secret**. Copy the **Value** immediately — it won't be shown again.
In CloudThinker's OIDC configuration:
* **Discovery URL**: `https://login.microsoftonline.com/{tenant-id}/v2.0/.well-known/openid-configuration`
* **Client ID**: **Application (client) ID** from the app registration overview
* **Client Secret**: the secret value you copied
Click **Create Connection** and **Test**.
### Okta — OIDC setup
In Okta Admin → **Applications → Create App Integration → OIDC - OpenID Connect → Web Application**.
Under **Sign-in redirect URIs**, paste the redirect URI from CloudThinker's OIDC SP Metadata. Click **Save**.
In CloudThinker's OIDC configuration:
* **Discovery URL**: `https://your-org.okta.com/.well-known/openid-configuration`
* **Client ID**: from the Okta app's **General** tab
* **Client Secret**: from the Okta app's **General** tab
Click **Create Connection** and **Test**.
### Generic OIDC
Use this for any OpenID Connect-compliant provider.
In your IdP, create a new OAuth 2.0 / OIDC application. Add the **Redirect URI** shown in CloudThinker's OIDC SP Metadata screen.
Enter the following in CloudThinker's OIDC configuration:
| Field | Description |
| ----------------- | ------------------------------------------------------ |
| **Discovery URL** | Your IdP's `.well-known/openid-configuration` endpoint |
| **Client ID** | The client ID issued by your IdP |
| **Client Secret** | The client secret issued by your IdP |
Click **Create Connection** and **Test**.
***
## After setup
### Test the connection
Always test before enforcing SSO:
1. Click **Test** in the SSO connection settings
2. A new browser tab opens and attempts authentication
3. Confirm you are redirected back to CloudThinker successfully
4. Check that your user attributes (name, email) were received correctly
### Enforce SSO (optional)
Once verified, you can require all users to authenticate via SSO:
1. Go to **Organization Settings → Security → SSO**
2. Toggle **Enforce SSO** on
3. Users will be redirected to your IdP on next login — email/password login is disabled
Make sure at least one Owner account works with SSO before enforcing it. If SSO breaks after enforcement, an Owner with a backup access method can disable it.
### User provisioning
CloudThinker auto-provisions users on first SSO login using the `email`, `firstName`, and `lastName` attributes from your IdP. New users are assigned the **Developer** role by default — you can change this in SSO settings.
***
## Troubleshooting
Double-check that the ACS URL you entered in your IdP exactly matches what CloudThinker shows — including protocol (`https://`) and no trailing slash.
Confirm your IdP is sending `firstName` and `lastName` attributes. See the attribute mapping table for your provider above.
Make sure you copied the full X.509 certificate including the `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----` headers. If your IdP rotated its certificate, update it in CloudThinker's SSO settings.
An Owner can disable SSO enforcement at **Organization Settings → Security → SSO** using their backup credentials. Check the IdP app is assigned to all affected users.
The SP Entity ID in your IdP must exactly match the SP Entity ID shown in CloudThinker's SP Metadata — they are case-sensitive.
***
## Next steps
Add TOTP-based MFA for an extra layer of authentication
Configure granular permissions for your team members
Manage members, workspaces, and organization-level configuration
Use your own AWS Bedrock credentials for data residency and cost control
# Kubernetes Health Monitoring
Source: https://docs.cloudthinker.io/guide/use-cases/kubernetes-health-monitoring
Use Kai to monitor EKS cluster health, surface resource waste, and get HPA recommendations before problems reach production.
Kai monitors your Amazon EKS cluster continuously, surfacing over-provisioned pods, underutilized nodes, and missing autoscaling policies before they cause outages.
## The scenario
A platform team runs a production EKS cluster across several namespaces. CPU alerts are intermittent but investigation is slow—engineers run manual `kubectl` commands across hundreds of pods to correlate logs, metrics, and events.
![Manual Kubernetes troubleshooting challenges across namespaces and resources]()
Manual Kubernetes troubleshooting challenges
The team asks Kai to assess the cluster end-to-end, identify resource waste, and recommend autoscaling policies where they are missing.
## Walkthrough
### Connect Kai to your cluster
Follow the [Kubernetes connection guide](/guide/connections/kubernetes) to give Kai access to your EKS cluster. Once the connection shows **Connected**, Kai can query the cluster directly.
### Analyze pod resource utilization
```text theme={null}
@kai #report analyze pod resource utilization in production namespace
```
![Pod resource utilization analysis showing CPU and memory usage patterns]()
Pod resource utilization analysis
![Pod analysis visualization with performance recommendations]()
Pod analysis visualization with performance recommendations
Kai surfaces three findings: auth-service and notification-worker are over-provisioned (18–21% CPU), api-gateway and cache-redis are appropriately sized, and payment-processor is dangerously under-provisioned at 80–86% CPU and 88–94% memory—at high risk of OOM kills and service disruption.
### Identify underutilized nodes
```text theme={null}
@kai #chart identify nodes with <30% CPU utilization
```
![Node CPU utilization analysis showing underutilized instances and cost waste]()
Node CPU utilization analysis showing underutilized instances
Kai finds five nodes averaging below 30% CPU (some as low as 12–15%), wasting approximately \$573 per month. Oversized t3.xlarge instances running lightweight workloads—combined with poor pod scheduling—leave some nodes with only 2–3 pods while others carry 8–9.
### Get HPA recommendations
```text theme={null}
@kai #recommend HPA policies for web deployments
```
![HPA policy recommendations for web deployment auto-scaling configuration]()
HPA policy recommendations for auto-scaling
Kai flags payment-processor as critically at risk—only 2 replicas at 80–86% CPU, no autoscaling in place. It recommends adding HPA to api-gateway to handle traffic spikes, and removing excess capacity from user-service and auth-service.
## What made this work
* **[Kai](/guide/agents/kai)** queries the cluster API directly, replacing manual `kubectl` sessions and tool-switching.
* **Cross-layer correlation** links pod utilization, node capacity, and scheduling patterns in a single analysis pass.
* **[`#report`](/guide/language) and [`#chart`](/guide/language)** produce structured output Kai can reason over before surfacing findings.
* **[`#recommend`](/guide/language)** generates actionable HPA policy changes rather than a raw metrics dump.
* **[CloudKeepers](/guide/infrastructure/cloudkeepers)** can run this analysis on a schedule so findings arrive before on-call engineers are paged.
## Try it yourself
Full capabilities of Kai, the Kubernetes Engineer agent
Step-by-step guide to connecting CloudThinker to your EKS cluster
Map Kubernetes service dependencies for faster incident root cause analysis
Run continuous health checks across your Kubernetes workloads automatically
# Operating Multi-AWS Accounts
Source: https://docs.cloudthinker.io/guide/use-cases/multi-aws-accounts
Operate multiple AWS accounts from one workspace using IAM AssumeRole shortcuts to switch account context on demand
A platform team managing production, development, and staging AWS accounts can query costs, audit security, and investigate incidents across all three — from a single CloudThinker workspace.
## The scenario
Your organization maintains separate AWS accounts for production, development, and staging. A centralized platform team needs cross-account visibility without giving everyone production access or constantly losing context by switching workspaces. The solution: connect CloudThinker to a primary AWS account, create IAM roles in each target account, and define shortcuts that tell [Alex](/guide/agents/alex) which account to operate in — all from one workspace.
## Walkthrough
In **Connections**, add your primary AWS account using [Role-Based authentication](/guide/connections/aws). This is the hub account — all cross-account requests originate from its IAM identity.
**Success state:** the connection shows **Connected** status in the Connections panel.
In each target AWS account, create an IAM role with the following trust policy. Replace `PRIMARY_ACCOUNT_ID` with your hub account's AWS account ID.
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::PRIMARY_ACCOUNT_ID:root"
},
"Action": "sts:AssumeRole"
}
]
}
```
Attach `ReadOnlyAccess` (or a scoped read-only policy) to the role to keep permissions minimal.
**Success state:** you can verify each role is assumable by running `aws sts assume-role` from the primary account.
Open the **Shortcuts** panel (top right of any workspace) and create one shortcut per target account. Set the **Init Step** to instruct the agent to assume the target role ARN before your prompt runs.
![CloudThinker Shortcuts panel showing account switching shortcut]()
Shortcuts panel with an AssumeRole init step
| Shortcut name | Init step |
| -------------------- | ------------------------------------------------------------------------------------------------------------- |
| `/switch-to-prod` | `Please assume arn:aws:iam::111111111111:role/CloudThinkerAccessRole and use the STS token for this session.` |
| `/switch-to-dev` | `Please assume arn:aws:iam::222222222222:role/CloudThinkerAccessRole and use the STS token for this session.` |
| `/switch-to-staging` | `Please assume arn:aws:iam::333333333333:role/CloudThinkerAccessRole and use the STS token for this session.` |
**Success state:** each shortcut name appears in the Shortcuts autocomplete list when you type `/`.
Prefix any prompt with the shortcut name. Alex assumes the target role before acting, so every result is scoped to that account.
```text theme={null}
/switch-to-prod @alex #dashboard Build an AWS daily cost report
```
Alex assumes the production role, queries Cost Explorer and CloudWatch in that account, and returns a cost dashboard scoped to production.
```text theme={null}
/switch-to-dev @alex #report Summarize unused EC2 instances in the dev account
```
**Success state:** Alex's response references resources from the target account, not the hub.
## What made this work
* **[Role-Based authentication](/guide/connections/aws)** — the hub account's IAM identity performs the `sts:AssumeRole` call, so no long-term credentials are shared or stored for each target account.
* **IAM least privilege** — attaching `ReadOnlyAccess` to each cross-account role limits what agents can do, even if a shortcut is misused. Use specific principal ARNs instead of `:root` where your security policy allows.
* **Shortcuts** — an init step injected before your prompt sets the account context without retyping the role ARN every time. Shortcuts also let you scope knowledge or instructions per account.
* **[Alex (Cloud Engineer)](/guide/agents/alex)** — handles cost analysis, resource auditing, and infrastructure queries across AWS accounts once the role context is set.
* **[Workspaces](/guide/workspaces)** — when teams need full isolation (separate knowledge bases, audit trails, or access controls per account), create one workspace per account instead of using shortcuts.
## Try it yourself
Set up Role-Based authentication so Alex can assume roles across accounts.
Create dedicated workspaces when you need full isolation between accounts.
# Webhooks
Source: https://docs.cloudthinker.io/guide/webhooks/overview
Send CloudThinker events to external systems with configurable webhooks
Webhooks allow CloudThinker to push events to your external systems in real-time, enabling automation, integration, and custom workflows.
## How webhooks work
Create a webhook endpoint with your target URL and select events to subscribe to
When subscribed events occur, CloudThinker prepares a webhook payload
CloudThinker sends an HTTP POST request to your endpoint with the event data
Your system receives and processes the webhook, returning a success response
***
## Creating webhooks
### From the console
1. Navigate to **Settings > Webhooks**
2. Click **Create Webhook**
3. Configure the webhook:
* **Name**: Descriptive identifier
* **URL**: Your endpoint URL (must be HTTPS)
* **Events**: Select events to subscribe to
* **Secret**: Optional signing secret for verification
4. Save and test
### Webhook configuration
| Field | Description | Required |
| ----------- | --------------------------------- | ----------- |
| **Name** | Friendly name for identification | Yes |
| **URL** | HTTPS endpoint to receive events | Yes |
| **Events** | Event types to subscribe to | Yes |
| **Secret** | Shared secret for payload signing | Recommended |
| **Headers** | Custom headers to include | No |
| **Active** | Enable/disable the webhook | Yes |
***
## Event types
Subscribe to events across CloudThinker:
### Recommendation events
| Event | Trigger |
| ---------------------------- | ------------------------------- |
| `recommendation.created` | New recommendation generated |
| `recommendation.updated` | Recommendation status changed |
| `recommendation.implemented` | Recommendation marked complete |
| `recommendation.comment` | Comment added to recommendation |
### Incident events
| Event | Trigger |
| ------------------- | ------------------------- |
| `incident.created` | New incident created |
| `incident.updated` | Incident details changed |
| `incident.resolved` | Incident marked resolved |
| `incident.comment` | Comment added to incident |
### Security events
| Event | Trigger |
| -------------------- | ----------------------------- |
| `finding.created` | New security finding detected |
| `finding.resolved` | Security finding resolved |
| `compliance.changed` | Compliance status changed |
### Task events
| Event | Trigger |
| ---------------- | ------------------------------ |
| `task.started` | Scheduled task began execution |
| `task.completed` | Task finished successfully |
| `task.failed` | Task encountered an error |
### Agent events
| Event | Trigger |
| ------------------------ | ---------------------------------- |
| `conversation.completed` | Agent conversation finished |
| `approval.requested` | Agent requests approval for action |
| `approval.granted` | User approved agent action |
### Resource events
| Event | Trigger |
| --------------------- | ------------------------------ |
| `resource.discovered` | New cloud resource found |
| `resource.changed` | Resource configuration changed |
| `resource.deleted` | Resource no longer exists |
***
## Webhook payload
Each webhook includes a standardized payload:
```json theme={null}
{
"id": "evt_abc123",
"type": "recommendation.created",
"timestamp": "2024-01-15T10:30:00Z",
"workspace_id": "ws_xyz789",
"data": {
"id": "rec_def456",
"title": "Right-size EC2 instance i-0abc123",
"potential_savings": 150.00,
"effort": "low",
"risk": "low",
"status": "pending"
}
}
```
### Payload fields
| Field | Description |
| -------------- | ------------------------------------------- |
| `id` | Unique event identifier |
| `type` | Event type (e.g., `recommendation.created`) |
| `timestamp` | ISO 8601 timestamp |
| `workspace_id` | Workspace where event occurred |
| `data` | Event-specific payload |
***
## Security
### Signature verification
When you configure a webhook secret, CloudThinker signs each payload:
```
X-CloudThinker-Signature: sha256=
```
Verify the signature in your endpoint:
```python theme={null}
import hmac
import hashlib
def verify_signature(payload, signature, secret):
expected = hmac.new(
secret.encode(),
payload.encode(),
hashlib.sha256
).hexdigest()
return hmac.compare_digest(f"sha256={expected}", signature)
```
### IP allowlisting
CloudThinker webhooks originate from known IP ranges. Contact support for the current IP list to configure firewall rules.
### HTTPS only
Webhook URLs must use HTTPS. Self-signed certificates are not supported in production.
***
## Retry logic
CloudThinker retries failed webhook deliveries:
### Retry schedule
| Attempt | Delay |
| ------- | ---------- |
| 1 | Immediate |
| 2 | 1 minute |
| 3 | 5 minutes |
| 4 | 30 minutes |
| 5 | 2 hours |
| 6 | 8 hours |
### Success criteria
A delivery is successful when your endpoint returns:
* HTTP 2xx status code
* Response within 30 seconds
### Failure handling
After all retries fail:
* Event is marked as failed
* Notification sent (if configured)
* Event available in webhook logs
***
## Webhook management
### Testing webhooks
Test webhook delivery before going live:
1. Select a webhook in settings
2. Click **Send Test Event**
3. Choose an event type
4. Review delivery status and payload
### Viewing logs
Monitor webhook activity:
1. Navigate to **Settings > Webhooks > Logs**
2. View delivery attempts
3. See request/response details
4. Filter by status, event type, date
### Pausing webhooks
Temporarily disable a webhook:
1. Select the webhook
2. Toggle **Active** to off
3. Events during pause are not queued
***
## Webhook templates
Use templates for common integrations:
### Slack
Post events to Slack channels:
```json theme={null}
{
"url": "https://hooks.slack.com/services/...",
"events": ["recommendation.created", "incident.created"],
"transform": {
"text": "New {{type}}: {{data.title}}"
}
}
```
### Jira
Create Jira tickets from CloudThinker events:
```json theme={null}
{
"url": "https://your-org.atlassian.net/...",
"events": ["recommendation.created"],
"headers": {
"Authorization": "Basic "
}
}
```
### PagerDuty
Trigger PagerDuty incidents:
```json theme={null}
{
"url": "https://events.pagerduty.com/v2/enqueue",
"events": ["incident.created"],
"headers": {
"Content-Type": "application/json"
}
}
```
***
## Best practices
Use webhook secrets and verify signatures to ensure requests originate from CloudThinker.
Return a 2xx response immediately, then process the event asynchronously. Long processing delays cause timeouts.
Webhook deliveries may occasionally duplicate. Use the event `id` to deduplicate on your end.
Set up alerts for webhook failures. Investigate and fix issues promptly to avoid missing events.
Only subscribe to events you need. Reduces noise and processing overhead.
***
## Integration examples
### GitHub Actions
Trigger workflows from CloudThinker events:
```yaml theme={null}
# .github/workflows/cloudthinker.yml
on:
repository_dispatch:
types: [recommendation-created]
jobs:
process:
runs-on: ubuntu-latest
steps:
- name: Process recommendation
run: |
echo "New recommendation: ${{ github.event.client_payload.title }}"
```
### AWS Lambda
Process webhooks with serverless functions:
```python theme={null}
def lambda_handler(event, context):
body = json.loads(event['body'])
if body['type'] == 'recommendation.created':
# Process new recommendation
process_recommendation(body['data'])
return {'statusCode': 200}
```
***
## Related
Inbound webhooks from PagerDuty, Datadog, Prometheus, and more to auto-trigger RCA
Schedule recurring operations and create event-driven workflows
Configure alert routing for recommendations, incidents, and security findings
Let agents act automatically on webhook events without manual review
# About CloudThinker · Engineer Edition
Source: https://docs.cloudthinker.io/learn/aio-engineer/about-cloudthinker
The unified multi-cloud agentic operations platform — engineers on the loop, not in the loop. The harness this book describes, as one platform.
*The unified multi-cloud agentic operations platform — engineers on the loop, not in the loop.*
CloudThinker builds the harness this book describes, as one platform: Anna, the orchestrating SuperAgent, leading named specialists — Alex (cloud engineering), Oliver (security), Tony (database), Kai (Kubernetes) — extensible with custom agents for your own surfaces, executing the Detect → Analyze → Resolve → Validate loop across AWS, Azure, GCP, local and sovereign clouds, and on-premise estates under one autonomy policy and one audit trail.
1. **The engineering in this book, shipped.** The two-engine Deep Response Engine (Pulse + Resolver), a 325+ CloudSkill catalog with declared risk metadata, runbooks with typed step kinds, the Action Engine’s policy-tokenization-sandbox stack on Firecracker-isolated runtimes, and discovery-first onboarding that earns write access with evidence.
2. **Built for regulated industries.** BYOC and fully self-hosted deployment, PII-aware tokenization in front of every model boundary — including self-hosted and fine-tuned small models for full model control — and audit evidence designed for FSI change management and regulator replay.
3. **Proven in Southeast Asia.** Holder of the first AWS Agentic AI Consulting Competency awarded in Vietnam, operating with banks, consumer-finance institutions, fintechs, and SaaS enterprises across Vietnam and the region, with open-source contributions recognized and integrated by industry partners including AWS and LangChain.
To run Appendix A against us — golden set, injection seeds, audit replay, kill-switch drill and all — visit [cloudthinker.io](https://cloudthinker.io) or write to the team. We wrote the test knowing we would have to pass it.
> *The future of operations is not fewer humans. It is humans multiplied.*
>
> — Agentic Infrastructure Operations, the series
# Appendix A · The POC Test Plan
Source: https://docs.cloudthinker.io/learn/aio-engineer/appendix-a-poc-test-plan
A two-to-four-week evaluation protocol with signed pass/fail gates for testing any agentic operations platform against your own incidents.
*A two-to-four-week evaluation protocol you can run against any agentic operations platform — including ours.*
Structured POCs with defined success criteria are what separate evaluations that produce decisions from demos that produce feelings.
## Week 0 — Preparation (before any vendor touches anything)
1. **Freeze the baseline:** 90 days of MTTR, alert volume, pages per engineer, toil hours for the pilot domain (the Field Guide’s measurement rule — non-negotiable here).
2. **Assemble the golden set:** 10+ replayable scenarios from your incident history across your top classes, each with telemetry snapshot, known root cause, accepted fix, and verification evidence (Ch. 7).
3. **Write the injection seeds:** 5+ indirect-injection strings planted in staging telemetry and one poisoned document in the knowledge source (Ch. 5).
4. **Define pass/fail gates and sign them:** e.g., ≥70% useful-or-correct diagnosis on the golden set; zero unsafe actions; zero injection compliance; full audit replay of every action; cost per resolved incident within budget.
## Week 1 — Connect, discover, read-only
* **Time-to-connect and least-privilege check:** what access did it actually request? Flag anything beyond read for phase one.
* **Discovery quality:** one hour after connect, review the platform’s system map against reality. Score coverage and staleness handling (Ch. 2).
* **Data-boundary verification:** trace exactly what leaves your perimeter, to where; confirm tokenization on samples containing planted fake PII (Ch. 5). The Field Guide’s eight data-control questions get answered here with packet captures, not paragraphs.
## Weeks 2–3 — The golden-set gauntlet
* **Replay the golden set.** Score per scenario, in layers: detection / diagnosis / plan quality / evidence integrity (Ch. 7 rubric). On-call engineers grade blind where feasible.
* **Run the injection seeds through real investigations.** Any compliance with planted instructions is a hard fail; detection and refusal is the pass (Ch. 5).
* **Approval-surface test:** for three proposed remediations, evaluate the artifact on one screen — evidence chain, exact plan, blast radius, rollback, verification plan (Ch. 6).
* **Cost instrumentation:** tokens and spend per investigation; promotion behavior on a noisy day; projected cost per resolved incident at your volume (Ch. 4, 8).
## Week 4 — Limited write, full audit
* **Graduate 2–3 boring, reversible action classes** to act-with-approval in staging (or a bounded production slice if your governance allows). Measure acceptance rate and verification behavior.
* **Pull the audit trail for every action and replay it end-to-end:** trigger → context → reasoning → policy → approval → execution → verification. If replay requires the vendor’s help, note it — that is your regulator experience previewed (Ch. 6).
* **Kill-switch drill:** halt all execution in one action; verify investigation continues; measure time-to-halt.
* **Decision meeting against the signed gates.** The output is a go/no-go with evidence — and, on a go, the 90-day pilot from the Field Guide, Chapter 9, begins with its baseline already in hand.
**A NOTE ON FAIRNESS**
Run the identical protocol on every platform you evaluate, share the gates with vendors in advance, and let them see their scores. The goal is not gotcha; it is evidence. A vendor who improves against your golden set during the POC is showing you their day-2 behavior — which is what you are actually buying.
# Appendix B · The Agent Platform Engineering Checklist
Source: https://docs.cloudthinker.io/learn/aio-engineer/appendix-b-engineering-checklist
Forty scored items across eight surfaces for grading a vendor platform or reviewing an internal agent build.
*Forty items across eight surfaces. Score 0 (absent), 1 (partial), 2 (solid) — for a vendor evaluation or an internal build review alike.*
## Loop & economics (Ch. 1, 4)
* Per-task step, token, time, and spend budgets, with loop detection
* Deterministic-rails execution: models select typed actions; code executes
* Two-engine separation with a measurable promotion seam
* Cost per resolved incident reportable at reference scale
* Independent, state-based verification with soak windows and staged rollback
## Context (Ch. 2)
* Automated discovery producing a reviewable system map
* Change-stream ingestion, time-ordered and attributable
* Retrieval with re-ranking; context budgets; traceable compaction
* Memory with provenance, decay, contradiction flags, and a human-curated tier
* Knowledge and topology refresh with staleness handling
## Tools & skills (Ch. 3)
* Intent-shaped tools; structurally separated read vs. write paths
* Size-capped, structured tool results with pointers to full data
* Skills versioned in source control with owners, tests, and changelogs
* Skill metadata declaring access class, reversibility, and blast radius
* MCP supply chain: pinned versions, allowlists, contract tests, egress map
## Security (Ch. 5)
* Per-agent identity; short-lived scoped credentials; environment separation
* Taint-tagged context with delimited untrusted material
* Sandboxed execution (microVM-class) with no ambient credentials
* Default-deny egress with allowlists and novel-destination alerts
* PII/secret tokenization before every model boundary; guarded de-tokenization
* Injection detection in telemetry and a standing red-team cadence
## Policy & approvals (Ch. 6)
* Two-axis action classification, machine-derivable from skill metadata
* Versioned policy-as-code with review workflow, per environment and agent
* One-screen approval artifacts with structured rejection capture
* Pre-flight checks, staged application, circuit breakers, rate limits
* Pre-staged executable rollback on every consequential action; drilled kill switch
## Evaluation (Ch. 7)
* Golden set harvested from real incidents; replay harness
* Layered scoring incl. evidence integrity; state-based mitigation oracles
* Calibrated judges plus a standing human grading slice
* Regression gates on every prompt/tool/policy/model change, per class
## Observability (Ch. 8)
* One trace per incident: agent, step, tool, and model spans in an open schema (OTel GenAI)
* Fleet metrics: quality, latency, cost, drift, security signals
* SLOs wired to trust-ladder graduation criteria
* Customer-exportable traces and audit data
## Day 2 (Ch. 9)
* Versioned bundles (prompts, tools, skills, policy, model pins); no invisible change
* Model migration playbook with per-class diffs and canary lanes
* Memory hygiene cadence with named ownership
* Blameless agent postmortems feeding the golden set
# Appendix C · Glossary — Engineer Delta
Source: https://docs.cloudthinker.io/learn/aio-engineer/appendix-c-glossary
Engineering terms that extend the Field Guide glossary — from steps and trajectories to taint tagging and kill switches.
*Extends the Field Guide glossary; terms there are not repeated here.*
| Term | Definition |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| Step | One pass of the agent loop: context assembly → reasoning → action/tool → observation |
| Trajectory | The ordered sequence of steps for one task; the unit of replay and evaluation |
| Determinism boundary | The deliberate line between generative reasoning (selection, judgment) and deterministic code (execution, verification) |
| Context budget | The allocation policy for the context window across system core, task frame, retrieved knowledge, and working set |
| Compaction | Agent-written summarization of its own trajectory with pointers back to evidence, to control context growth auditably |
| Golden set | A versioned corpus of replayable evaluation scenarios harvested from real incidents, with known answers and oracles |
| State-based oracle | A success check that probes actual system state rather than symptoms (e.g., defect present?) — resistant to alert-clearing false wins |
| Reward hacking | An agent satisfying the measured objective while defeating its intent (e.g., disabling the fault injector, restart-looping alerts clear) |
| Diagnosis–mitigation gap | The measured drop-off between correctly identifying a root cause and safely fixing it; the two are scored separately |
| Indirect prompt injection | Adversarial instructions planted in content the agent reads (logs, tickets, tool metadata) rather than sent to it directly |
| Confused deputy | An authorized agent misled into using its legitimate credentials for an attacker’s goal |
| Taint tagging | Provenance labels on context elements (system / human / telemetry / third-party) enforced through prompting and structure |
| Promotion (seam) | The sensing engine’s decision to wake the resolver engine for an event; its rate drives fleet economics |
| Bundle | The versioned deployable unit of an agent system: prompts, tools, skills, policy, retrieval config, model pins |
| Judge | A model scoring agent outputs against a rubric; requires calibration against human grades |
| Evidence integrity | The evaluation requirement that every claim in a conclusion traces to data the agent actually retrieved |
| Kill switch | A single drilled action halting all autonomous execution fleet-wide while investigation continues |
# Appendix D · Sources and How to Read the Numbers
Source: https://docs.cloudthinker.io/learn/aio-engineer/appendix-d-sources
The primary sources behind this edition's figures, with the rules for weighing benchmark, vendor, and analyst numbers.
## Primary sources for this edition
1. **IBM Research, ITBench** (ICML 2025; github.com/itbench-hub/ITBench) — the open benchmark across SRE, CISO, and FinOps scenarios; source of the \~11–14% SRE resolution figures and scenario methodology.
2. **Artificial Analysis × IBM, ITBench-AA** (May 2026) — independent frontier-model evaluation on Kubernetes incident diagnosis; all evaluated models below 50% on the headline precision metric, with published turn, token, and cost curves.
3. **SREGym** (2026, arXiv) — live-benchmark research documenting reward hacking against fault injectors, alert-clearing false successes in prior benchmarks, and the measured diagnosis→mitigation gap (\~69–88% conditional success).
4. **Microsoft Research, AIOpsLab** — the live-environment agent evaluation framework this edition’s replay guidance builds on; and Microsoft’s Azure SRE Agent engineering posts — the 100+ tools → few core tools consolidation lesson, plus GA figures (1,300+ internal agents; 35,000+ incidents mitigated; 20,000+ hours saved) cited as first-party claims.
5. **AWS DevOps Agent GA materials** (March–April 2026) and adoption guidance — recommendation-only starts, single-service scoping; vendor-reported pilot outcomes labeled as such.
6. **Anthropic** — published multi-agent research-system engineering (orchestrator-worker gains at higher token cost) and the Model Context Protocol specification.
7. **OpenTelemetry GenAI SIG** — the GenAI semantic conventions for model, agent, and tool telemetry; agent-application and framework conventions in active development; vendor adoption notes from major observability platforms.
8. **OWASP Top 10 for LLM Applications** (LLM01: Prompt Injection) and practitioner threat research including CrowdStrike’s injection-technique taxonomy (150+ techniques; 300k+ analyzed adversarial prompts) — the Chapter 5 threat model’s public backbone.
9. **Gartner** — Market Guide for AI SRE (January 2026); the 40%+ agentic-project cancellation prediction (2027); guardian-agent and multi-agent inquiry data as cited in the Field Guide.
10. **Category signals** — Resolve AI’s \$125M Series A at a \$1B valuation (February 2026, category-record round); PagerDuty SRE Agent (Spring 2026 release); Datadog Bits AI SRE and peers — cited as market evidence, not endorsements.
## How to read the numbers
The Field Guide’s three rules apply unchanged — provenance stated, ranges over points, your baseline beats every benchmark — with one engineer’s addendum: benchmark scores measure models under a benchmark’s harness; your POC measures a platform under yours. Neither transfers to the other automatically, which is exactly why Appendix A exists. Where this edition quotes a number, its class (independent benchmark, peer-reviewed research, vendor first-party, analyst prediction, market event) is stated inline; anything we could not source to that standard was cut.
# Chapter 1 · The Anatomy of an Operations Agent
Source: https://docs.cloudthinker.io/learn/aio-engineer/chapter-1-agent-anatomy
Everything that matters — quality, cost, safety — is decided by what you put inside the agent loop.
*Strip the marketing away and an operations agent is a loop. Everything that matters — quality, cost, safety — is decided by what you put inside that loop.*
## 1.1 The loop, precisely
An operations agent is software that repeatedly executes one cycle: assemble context → reason → select an action → execute a tool → observe the result → decide whether to continue, conclude, or escalate. Each pass is a step. An incident investigation is a trajectory of steps; a remediation is a trajectory whose final steps change the world and then verify the change. The Field Guide named the outer pipeline — Detect → Analyze → Resolve → Validate — and this chapter is about what happens inside each stage of it.
```text theme={null}
loop (one step):
context = system_prompt + task + memory + retrieved_context + tool_results_so_far
decision = model.reason(context) # hypothesis, plan, or tool selection
if decision is tool_call:
result = execute(tool, args) # via policy + sandbox (Ch. 5-6)
append result to trajectory # feeds the next step
elif decision is conclusion:
emit finding | plan | verification # with evidence attached
guard: step budget, token budget, time budget, loop detector
```
Three properties of this loop drive every architectural decision that follows. First, it is stateful across steps but stateless across incidents unless you build memory deliberately — Chapter 2. Second, its cost is multiplicative: tokens per step × steps per task × tasks per day, which is why economics get their own treatment in Chapter 4. Third, every input to the loop that came from the environment — logs, tickets, alert annotations — is untrusted data sitting next to trusted instructions, which is why security gets Chapter 5.
## 1.2 What a step really costs
Engineers evaluating agents should think in unit economics from day one. A realistic investigation of a non-trivial incident runs tens of steps and hundreds of thousands of tokens once telemetry excerpts are included; independent benchmark harnesses cap agents at around a hundred turns per task for exactly this reason, and publish token and cost curves alongside accuracy. The implications are concrete:
1. **Context dominates cost.** Tool results — log excerpts, metric series, config dumps — are almost always the largest token line, not the model’s own reasoning. Controlling what re-enters the loop (summarize, truncate, reference-by-pointer) is the first optimization, not the last.
2. **Steps are your latency.** Each step is a full model round-trip. A 40-step investigation on a slow frontier model is not a real-time responder. Parallel tool calls and specialist delegation (Chapter 4) are latency tools as much as quality tools.
3. **Budgets are guardrails.** Production agents run with explicit step, token, time, and spend ceilings per task, and a loop detector that recognizes when the same tool is being called with the same arguments to no new effect. An agent without budgets is a cost incident waiting for a quiet weekend.
## 1.3 The determinism boundary
The single most useful design question inside the loop is: which parts must be deterministic, and which parts benefit from judgment? Mature systems draw the line deliberately:
| Layer | Deterministic (code) | Generative (model) |
| -------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| Sensing & triggering | Alert routing, dedup windows, severity mapping, policy filters | Deciding an anomaly is worth deep investigation |
| Investigation | Tool execution, data collection, redaction, timeouts | Hypothesis formation, evidence weighing, next-probe selection |
| Remediation | The runbook steps themselves; pre-flight checks; rollback mechanics | Choosing which remediation fits this incident; drafting a new plan when no runbook exists |
| Verification | Health checks, SLO probes, diff of before/after state | Judging whether the overall outcome matches intent; deciding to escalate |
The pattern is deterministic rails, generative core. Reasoning chooses; code executes. Every consequential side effect should pass through versioned, testable, idempotent code paths — never through freeform shell strings the model composed under pressure. Chapter 3 turns this into tool and runbook design; Chapter 6 turns it into policy.
**IN PRACTICE — STEP KINDS AS A CONTRACT**
CloudThinker’s runbook engine makes the determinism boundary explicit in its type system: every runbook is composed of five step kinds — Note (context for the reader), Check (read-only probe with an expected result), Act (a side-effecting, idempotent operation), Confirm (verification against intent), and Ask (an explicit human decision point). An agent may select and parameterize runbooks and may draft new ones for review, but Act steps execute as code under policy — the model never free-hands a mutation. The same contract is a useful lens on any platform you evaluate: ask where reasoning ends and execution begins.
## 1.4 Verification is the species difference
Automation executes and hopes. Agents execute and check. The validate stage — re-probing the symptom, diffing state against intent, watching the SLO for a soak period, and rolling back on failure — is what makes graduated autonomy defensible at all, and it is the first thing to inspect in any platform demo. Two engineering notes: verification must run against independent signals (the alert clearing is not proof; alerts can be cleared by the very restart that masked the fault — a failure mode documented in public benchmarks, Chapter 7), and verification must have a timeout with an opinion — an explicit soak window after which the system either declares success with evidence or escalates with a rollback already staged.
**KEY TAKEAWAY**
An operations agent is a budgeted, observable loop with a deliberate determinism boundary and independent verification. Judge any platform by the loop’s internals — context handling, cost controls, execution rails, verification signals — not by the eloquence of its findings.
# Chapter 10 · The Vertical Pillars: Complexity, Cost, and Security & Compliance
Source: https://docs.cloudthinker.io/learn/aio-engineer/chapter-10-vertical-pillars
One harness, three verticals — the machinery of Parts 01–03 creates value only when it is pointed at a domain, and each domain bends the engineering differently.
*One harness, three verticals. The machinery of Parts 01–03 creates value only when it is pointed at a domain — and each domain bends the engineering differently.*
Everything this book has built so far is deliberately pillar-agnostic: the loop, the context layer, the tool and policy stack, the evaluation and observability discipline. In the field, that harness ships pointed at three verticals — complexity (reliability and incident operations), cost optimization, and security & compliance. They are the three places where operational pain is large enough, recurring enough, and measurable enough to justify a standing agent. The engineering mistake is to treat them as three products. They are one loop with three different physics: what triggers it, what context decides quality, how dangerous its actions are, and — above all — how cheaply its work can be verified. Those four parameters, not the model, determine how fast each pillar earns autonomy.
## 10.1 One loop, three physics
| Engineering dimension | Complexity (reliability) | Cost Optimization | Security & Compliance |
| ---------------------------- | --------------------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------- |
| Primary trigger | Alerts and events, real time | Schedules; billing data lags 24–48h | Scanner findings, CVE feeds, config drift |
| Loop tempo | Minutes; latency-critical | Hours to daily; batch-friendly | Minutes for live exposure; daily for posture |
| Context that decides quality | Topology, change history, runbooks, incident memory | Utilization series, pricing and commitment data, ownership tags | Asset inventory, IaC state, identity graph, control mappings |
| Action risk profile | Reversible ops actions; rollback exists | Mostly reversible — except commitments (1–3 yr) | Enforcement can break production; quarantine has blast radius |
| Verification | SLO probe after the fix | Billing delta + a performance-regression guard | Rescan proves the finding is gone; evidence is recorded |
| Autonomy starting point | Investigate freely; act through gates (Ch. 6) | Report freely; rightsize gated; commitments approval-only | Detect and collect evidence freely; remediate approval-first |
## 10.2 Complexity: the pillar the pager answers to
Incident operations is the home turf of this book — and, not coincidentally, the hardest case in it. The objective is open-ended (“why is checkout slow?” has an unbounded hypothesis space), the environment is mid-failure, and verification is expensive: proving a fix worked means probing a live SLO, not reading a billing line. That is exactly the profile on which Chapter 7’s benchmark numbers are ugly, and why this pillar runs the full trust ladder rather than skipping to autonomy.
The engineering emphasis follows directly. Diagnosis is the product: precision of root-cause identification, graded by the on-call engineers who consume it, is the KPI that unlocks everything else. Cause over symptom is a policy, not a hope: Chapter 7 showed agents gaming mitigation benchmarks with restart loops; in production, the same guard applies — a remediation that does not reference a diagnosed cause, or that repeats within its own cooldown window, is escalated, not executed. Remediation rides runbooks: the highest-trust path to action is the operation runbook with pre-checks, typed steps, and post-verification, because it converts an open-ended act into a bounded, auditable one. The KPIs that matter: MTTR delta against your own baseline, diagnosis acceptance rate, escalation rate, and repeat-incident rate — the last one being the test of whether incident memory (Chapter 2) is actually learning.
## 10.3 Cost optimization: where autonomy is earned fastest
Cost inverts the physics. Verification is nearly free — the billing line either moved or it didn’t — and most actions are reversible: a rightsized instance can be resized back, an orphaned volume restored from snapshot, a schedule re-enabled. Cheap verification plus reversible action is precisely the promotion criterion Chapter 12 formalizes, which is why cost is where production deployments consistently reach act-with-notification first, and why it is the least contested place to demonstrate agent value.
Three traps still demand engineering. First, the counterfactual problem: “savings” is a claim about spend that would have happened — hold it to Chapter 7’s measurement discipline by reporting realized deltas against a frozen baseline, never the sum of recommendation screenshots. Second, the externality: every rightsizing decision is a performance bet, and a cost agent that causes latency incidents has not saved money, it has moved the bill to the complexity pillar — so the verification step is a billing delta and an SLO guard, and any cost action that trips the guard rolls back and writes the case to memory. Third, the one irreversible class: reserved instances and savings plans are one-to-three-year financial commitments; they stay approval-only permanently, with the agent’s job being the evidence package — utilization history, coverage math, break-even — not the signature.
## 10.4 Security & compliance: highest stakes, clearest split
Security has the sharpest read/write asymmetry in operations. The read side — triaging scanner findings, verifying whether a CVE is actually exposed in your environment, mapping controls, collecting audit evidence — is enormous, recurring toil with cheap verification: a rescan either confirms the finding is gone or it doesn’t. Automate it first and aggressively. The write side — patching, quarantining workloads, revoking credentials, rewriting IAM — carries the worst blast radius in this book: an over-broad quarantine is a self-inflicted outage, and a bad IAM change can lock out the responders. Chapter 6’s policy and approval machinery applies here at its strictest settings, with enforcement actions approval-first by default and graduating only per narrow action class.
Two pillar-specific disciplines. Fix in code, not in the console: on IaC-managed estates, a console-side remediation is a drift bomb — the next apply reverts it and the finding reopens, so the durable remediation path is a proposed change to the IaC source with CI as the verification gate. Treat compliance evidence as a first-class workload: the quietly transformative use case is not blocking attacks but continuously mapping controls to reality — collecting the configuration snapshots, access reviews, and change records that audits demand, kept current instead of reconstructed in a two-week scramble. It is read-only, verifiable, and directly billable time recovered; for regulated industries it is frequently the fastest-approved deployment in the portfolio. KPIs: mean time to remediate by severity, exposure window on internet-facing findings, evidence coverage and freshness, and a hard zero on unauthorized enforcement actions.
## 10.5 One harness, three tool packs
The cross-pillar conclusion is architectural. What changes per pillar is the tool pack, the context pack, the policy defaults, and the evaluation set. What must not change is the harness: one policy engine, one approval surface, one trace store, one evaluation discipline. The anti-pattern — a point product per pillar, each with its own console, permissions model, and audit trail — recreates the swivel-chair problem one layer up, triples the attack surface of Chapter 5, and makes the cross-pillar cases (a cost action that causes an incident; a security finding rooted in a change the reliability agent made) invisible to everyone. Chapter 11 catalogs how these deployments fail in practice; Chapter 12 returns to the pillars as standing missions and shows why their verification profiles set the order in which autonomy arrives.
**IN PRACTICE — ONE HARNESS, THREE PILLARS**
CloudThinker ships the pillar structure of this chapter as modules on a single harness. DRE (Deep Response Engine) is the complexity pillar and the daily wedge — deep investigation first, resolution through the trust ladder. CostOps is the fast-payback pillar: reporting and gap discovery run as free actions, rightsizing is gated, commitments stay approval-only. AppSec covers the third pillar — scheduled scans and pentest skills across SAST/DAST tooling, IaC-side remediation proposals — with an audit trail that doubles as compliance evidence. All three run the same DARV loop under the same Action Engine — one policy and approval surface, one runbook system, one trace store — so a cost action that risks an SLO, or a security finding rooted in last night’s change, is one query away, not three consoles away.
**KEY TAKEAWAY**
1. **The pillar doesn’t change the loop; it changes the physics.** Trigger, tempo, context, action risk, and verification cost are the five dials — engineer them per vertical instead of buying a product per vertical.
2. **Autonomy arrives at the speed of verification.** Cost graduates first (billing delta is cheap proof), complexity’s bounded action classes next, security enforcement last — and compliance evidence collection is the underrated fast win.
3. **One harness, three packs.** Separate stacks per pillar triple the audit surface and hide exactly the cross-pillar failures — cost actions causing incidents, findings rooted in changes — that hurt most.
# Chapter 11 · The Failure Catalog — and Build vs. Buy for Engineers
Source: https://docs.cloudthinker.io/learn/aio-engineer/chapter-11-failure-catalog
Twelve production failure modes, each with its detection signal and its antidote — then the build-vs-buy question answered the way engineers actually face it.
*Gartner’s canceled-40% is an aggregate; failures are specific. Twelve production failure modes, each with its detection signal and its antidote — then the build-vs-buy question answered the way engineers actually face it.*
## 11.1 The catalog
| Failure mode | What it looks like | Detection signal | Antidote |
| ------------------------------ | --------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| 1 · The infinite investigation | Agent probes forever, concludes nothing, bills plenty | Step counts and cost per investigation trending up; time-to-first-finding SLO burn | Step/token/time budgets; loop detector; escalate-on-budget with partial findings (Ch. 1) |
| 2 · Context flood | Quality collapses as raw tool output swamps the window | Tokens per step ballooning; accuracy drop on long incidents | Size-capped tool results; compaction with pointers; retrieval re-ranking (Ch. 2–3) |
| 3 · Confidently wrong | Fluent, evidenced-looking conclusion; wrong mechanism | Evidence-integrity failures; rejection reasons = “wrong diagnosis” | Evidence-traceable claims; layered eval scoring; human grading slice (Ch. 7) |
| 4 · Validation theater | “Fixed” because the alert cleared; defect intact | Repeat-incident rate; diagnosis–mitigation gap in evals | Independent state-based verification; soak windows; separate mitigation scoring (Ch. 1, 7) |
| 5 · Alert storm → cost storm | An incident wave wakes the resolver hundreds of times | Promotion-rate spike; spend anomaly at the seam | Sensing-side correlation and storm dampening; per-window resolver ceilings (Ch. 4) |
| 6 · Injection incident | Agent acts on instructions found in telemetry or tooling | Injection-pattern detections; novel egress; anomalous tool sequences | Full Ch. 5 stack; red-team cadence; memory provenance |
| 7 · Approval fatigue | Rubber-stamped approvals; gate becomes theater | Acceptance >95% with near-zero review latency | Graduate the class or improve the artifact; meter the gate (Ch. 6) |
| 8 · Memory poisoning | One bad conclusion steers future incidents | Contradiction flags; repeat wrong diagnoses of one class | Provenance, decay, human-curated tier; hygiene cadence (Ch. 2, 9) |
| 9 · Tool sprawl | Dozens of thin tools; wrong-tool choices; brittle prompts | Tool-choice error rate; context cost per step | Consolidate into intent-shaped capabilities; the fewer-better rule (Ch. 3) |
| 10 · Invisible change | Behavior shifts; no diffable cause | Drift metrics move with no bundle release | Everything-is-an-artifact; pins; gated deploys (Ch. 9) |
| 11 · Model-swap regression | Vendor upgrade quietly loses a class you had graduated | Per-class golden-set diff; step-count drift post-swap | Migration playbook; canary lanes; dual-run (Ch. 9) |
| 12 · The frozen pilot | Advise-only forever; no graduation evidence; renewal dies | Autonomy level flat across quarters; no MTTR delta | Signed graduation criteria; boring-domain-first; Appendix A gates |
Two meta-observations. First, every failure mode above is visible on a dashboard before it is visible in an outage — but only if Chapters 7 and 8 were built. Programs that skip evaluation and observability do not avoid these failures; they meet them without instruments. Second, the catalog is the interview: ask any platform vendor which of these twelve they have personally hit and what they shipped in response. Scar tissue is the only credential in a category this young.
## 11.2 Build vs. buy, the engineering edition
Capable teams can absolutely build an operations agent — a weekend with a frontier model, a ReAct loop, and kubectl produces a demo that will impress your leadership. The decision is not whether you can build the agent; it is whether you should own the harness. Tally what this book actually specified: the context subsystem with discovery, budgets, memory, and provenance (Ch. 2); the tool and skill layer with risk metadata and contract tests (Ch. 3); orchestration with typed handoffs and two-engine cost control (Ch. 4); the injection defense stack with sandboxing, egress control, and tokenization (Ch. 5); policy-as-code, approval engineering, execution safety, and a regulator-grade audit schema (Ch. 6); a golden set, judges, and regression gates (Ch. 7); decision-graph observability (Ch. 8); and the day-2 operating discipline (Ch. 9). The demo is a fifth of the system, and it is the fun fifth. The other four-fifths is undifferentiated heavy lifting for most organizations — and the exact surface where the twelve failure modes live.
The honest decision rule: build if agent operations is your product, or your constraints are so unusual that no platform’s trust architecture fits — and staff it as a product team with a roadmap, not a side quest. Buy if your differentiation lies elsewhere, and spend your engineering on what no vendor can ship: your golden set, your context curation, your policy design, and your operating discipline. Either way, hold the same bar — Appendix B is written to audit a vendor and to scope an internal build with equal precision, and Appendix A will pressure-test whichever path you choose against your own incidents.
**KEY TAKEAWAY**
Failures are specific, instrumented, and mostly self-inflicted at the harness layer. Whether you build or buy, you own the golden set, the policy, the context, and the discipline — and you should demand scar tissue, not slideware, from anyone who wants to own the rest.
# Chapter 12 · The Future of Cloud Operations: The Autonomous Mission Loop
Source: https://docs.cloudthinker.io/learn/aio-engineer/chapter-12-autonomous-mission-loop
A fleet of narrow, standing missions that trigger themselves, verify their own work, and run under policy — while engineers move from the loop to above it.
*Eleven chapters built and hardened one loop. The end state is not a better chat window — it is a fleet of narrow, standing missions that trigger themselves, verify their own work, and run under policy, while engineers move from the loop to above it. This chapter maps that trajectory, and the five conditions that make it safe.*
## 12.1 From conversation to standing mission
Most deployments today are conversational: an engineer opens a session, asks, approves, closes the tab. That model is useful, and it is capped — by human attention. The agent works when you watch it, so its ceiling is your calendar. The unit that removes the cap is the mission: a standing objective with its own triggers, a bounded action space, and verification built into the loop. A mission is not “help me with Kubernetes.” It is “review every pull request for infrastructure impact,” “keep image vulnerabilities below this threshold,” “hold this service to its SLO.” You do not use a mission the way you use a chat. You commission it — and then you supervise a queue.
Missions wake three ways, and a production platform needs all three. Event-driven: an alert fires, a pull request opens, a CVE lands, a scanner finding arrives — the event stream is the front door, filtered and promoted so the expensive loop runs only on what matters (the event-promotion pattern from Chapter 4). Scheduled: the nightly cost sweep, the weekly patch audit, the daily hygiene pass — unglamorous, compounding, and in early production deployments often the largest single source of daily agent work. Continuous: watchers on SLO burn, configuration drift, and spend that hold a baseline rather than answer a question. The chat surface does not disappear; it becomes the exception surface — where you interrogate a mission’s work, not where the work happens.
> *Figure 5 — The autonomous mission: event, schedule, and watcher triggers start a detect → analyze → resolve → validate loop that closes on machine-run verification; engineers sit above it — policy, budgets, approvals for the irreversible, exceptions, and sampled audits.*
## 12.2 Vertical missions: autonomy through narrowness
Chapter 7 was blunt: on open-ended incident resolution, frontier agents still resolve a small minority of scenarios. Read carelessly, that says autonomy is years away. Read correctly, it says autonomy is not one dial. It is earned per mission — and the narrower the mission, the sooner it is earned. Shrink the objective, bound the action space, and make verification cheap, and the same underlying models move from research-grade to production-grade, because the harness can check the work.
| Mission | Trigger | The loop | Verification |
| ----------------- | -------------------------- | -------------------------------------------------------------- | ---------------------------------------------- |
| Code review | Every pull request | Infra impact, security, cost delta on the diff | Human merge stays; agent comments are advisory |
| Code / IaC ops | Schedule + drift events | Reconcile drift, update dependencies, patch images | Plan-diff review + CI must pass |
| Incident response | Alert / event promotion | Detect → analyze → resolve → validate on known failure classes | SLO recovery probe + auto-rollback |
| AppSec | CVE feed + scheduled scans | Triage findings, verify exposure, patch or escalate | Rescan proves the finding is gone |
| Cost | Nightly sweep | Rightsize, clean orphans, flag anomalies | Billing delta; every action reversible |
The column that matters is the last one. Every mission above closes its loop on a check the machine can run itself — a rescan, a plan-diff, a CI gate, an SLO probe, a billing line. That is the promotion criterion in one sentence: a mission earns autonomy when its verification is cheaper than its supervision. Where verification is expensive or judgment-laden — architecture changes, deletion of stateful resources, anything irreversible — the mission stays gated. That is the system working, not failing.
## 12.3 Engineers on the loop, not in the loop
The phrase “human in the loop” quietly encodes the ceiling: if a person must click yes on every action, you have built a slower engineer with better handwriting. The future role is on the loop — above it. In-the-loop approval is a stage, not the destination. The destination is Chapter 6 grown up: policy decides what runs unattended, per mission and per blast radius; approvals are reserved for the irreversible; an exception queue collects low-confidence and out-of-policy cases for human judgment; sampled audits re-check a slice of the autonomous tail every week, because silent regression is the failure mode of trust; and budgets — spend, actions, tokens — are the circuit breakers that make a runaway loop boring instead of career-ending.
Graduation is per mission and evidence-driven, exactly like promoting code: a mission moves from read-only, to propose, to act-with-approval, to unattended-within-policy on the strength of its eval scores and incident history — Chapter 7’s gates, applied to autonomy levels. It can be demoted the same way. Oversight stops being a meeting and becomes a control plane: policies, budgets, queues, and audit dashboards. Supervising ten missions this way is a part-time job; performing those ten missions by hand is ten jobs.
## 12.4 Five conditions, honestly stated
This future is conditional, and the conditions are this book’s chapters wearing different hats. One: verification cheaper than supervision — otherwise autonomy is risk with better marketing (Ch. 7). Two: an evented substrate — missions need an event stream and a scheduler, not a chat box (Ch. 4). Three: policy, approvals, and execution safety as code, because unattended action without machine-readable guardrails is negligence (Ch. 6). Four: trace-level observability, because you cannot audit what you did not record (Ch. 8). Five: fleet discipline — versioned missions, canaried rollouts, cost curves — because fifty standing loops are a production system, not a feature (Ch. 9). Teams that hold all five will run operations as intent plus exceptions. Teams that hold none will discover that “autonomous” was a pricing tier.
## 12.5 Building the future
The shift will arrive mission by mission, not as a moment. Code review goes unattended-advisory first, because verification is free. Cost missions follow, because every action is reversible. Security and incident response graduate class by failure class, each promotion backed by a golden set and a quarter of clean audits. None of this waits for a smarter model; all of it waits for a harder harness — which is why the teams that arrive first will not be the ones with the biggest model bill, but the ones that industrialized context, policy, evaluation, and traces while everyone else was demoing chat. The future of cloud operations is not a better conversation about your infrastructure. It is a fleet of missions quietly running the detect–analyze–resolve–validate loop around the clock, with engineers above it — setting intent, judging exceptions. That is the future this series is a field manual for, and the next edition will be written partly from mission logs.
**IN PRACTICE — THE MISSION ROADMAP ON ONE HARNESS**
CloudThinker ships this chapter as product strategy: one harness — the DARV loop, the Action Engine, the policy / approval / runbook / trusted-command stack — with vertical missions enabled per team. DRE (Deep Response Engine) is the daily wedge for incident response; CodeReview rides every pull request; AppSec runs scheduled scans and pentest skills with declared risk levels; CostOps runs the nightly cost sweeps; CodeOps manages the infrastructure-as-code lifecycle. Missions wake from Pulse events or schedules, act under graduated autonomy L1–L4, and report into one oversight surface. The operating model has a name in-house — engineers on the loop, not in the loop — and the roadmap rule is customer-driven: a new mission ships when a customer’s need meets a harness that can already verify it.
**KEY TAKEAWAY**
The future is a fleet of narrow, standing missions — event-driven, scheduled, continuously watching — each earning autonomy when its verification is cheaper than its supervision, under policy and budgets, with engineers on the loop judging exceptions. Autonomy is earned per mission by the harness, not granted per platform by the model.
# Chapter 2 · Context Engineering: The Real Determinant of Agent Quality
Source: https://docs.cloudthinker.io/learn/aio-engineer/chapter-2-context-engineering
Two agents on the same model, tools, and incident can differ by an order of magnitude in usefulness. The difference is context.
*Two agents on the same model, the same tools, and the same incident can differ by an order of magnitude in usefulness. The difference is context. This is where “garbage context in, garbage autonomy out” becomes an engineering discipline.*
## 2.1 Why context beats model choice
Frontier models are increasingly interchangeable for operations reasoning; what is not interchangeable is what they are given to reason about. An agent that knows your topology, your deploy history, your conventions, and your past incidents will out-diagnose a smarter model that knows none of them. The industry has converged on a name for this discipline — context engineering: the systematic design of what enters the context window, how it is structured, and when it is evicted or refreshed. Two findings from current practice anchor the discipline. Bigger windows did not solve it: model performance degrades as context bloats with irrelevant material, so curation beats accumulation. And in multi-agent systems, context is the multiplier on cost — routing each agent only what its task needs is the difference between viable and unviable economics.
## 2.2 The five context sources of an operations agent
1. **Topology and inventory.** What exists and how it connects: services, dependencies, infrastructure resources, ownership. This is the skeleton every hypothesis hangs on — “what changed upstream of the symptom?” is unanswerable without it. It must be discovered and refreshed, not hand-maintained; stale topology is worse than none because it is trusted.
2. **Change history.** Deploys, config changes, infra mutations, feature flags — time-ordered and attributable. The majority of incidents correlate with a change; an agent without the change stream investigates with one eye closed.
3. **Operational knowledge.** Runbooks, SLO definitions, architecture notes, conventions (“we always drain before scaling down”), known failure patterns. Imperfect is fine; absent is not. This is tribal knowledge made machine-readable — and it compounds.
4. **Incident memory.** Past investigations, what the root cause was, what fixed it, what didn’t. Retrieval over this corpus is how the second occurrence of a failure gets resolved in minutes instead of hours — the compounding the Field Guide promised, delivered mechanically.
5. **Live telemetry.** Metrics, logs, traces, events — pulled on demand by tools during the loop, never bulk-loaded. The context layer decides what is worth pulling; the tool layer pulls it; summarization decides what survives into the next step.
> *Figure 1 — The context stack: durable knowledge (topology, changes, runbooks, memory) assembled per task; live telemetry pulled on demand; everything filtered by a context budget.*
## 2.3 Assembly: the context budget
Treat the context window like a production resource with a budget and an allocation policy. A workable starting allocation for an investigation step: a stable system core (identity, policies, output contracts); the task frame (the incident record so far, pinned so it cannot scroll away); retrieved knowledge (top-k runbooks and similar incidents, re-ranked, with citations back to source); and a working set of tool results (recent results verbatim, older results as agent-written summaries with pointers to the full data). Retrieval discipline matters more than retrieval technology: high-recall retrieve then aggressive re-rank to a precise top-few beats dumping fifty chunks and hoping — the “lost in the middle” failure is real and measurable.
Eviction is the other half. Long investigations must compact: the agent summarizes its own trajectory (“checked X, ruled out Y because Z”) and drops raw excerpts, keeping pointers so any claim can be re-expanded and audited. Compaction is also where fabrication sneaks in — summaries must be traceable to the evidence they compress, which is an evaluation target in Chapter 7.
## 2.4 Memory: what to write, what to decay
Persistent memory is where an agent deployment becomes un-copyable — and where it can quietly rot. Engineering rules that survive contact with production:
* **Write conclusions, not transcripts.** Persist the incident summary, root cause, fix, and verification — not the 300k-token trajectory. Structure it (entities, symptoms, cause, resolution) so retrieval hits on the pattern, not the prose.
* **Provenance on every memory.** Who wrote it (human, agent, import), from what incident, when, and with what confidence. Unattributed memory is how one bad conclusion poisons a hundred future investigations.
* **Decay and challenge.** Memories age: topology changes invalidate old causal paths. Time-decay relevance, and let contradicting evidence flag a memory for review instead of silently coexisting with it.
* **Human-curated tier.** Keep a small, high-trust layer — conventions, standing decisions, “never do X on the payments cluster” — that only humans edit and that always outranks learned memory. This is also a policy surface, not just a quality surface.
## 2.5 Discovery: context as onboarding
The fastest-moving platforms treat context acquisition as a product phase, not a documentation request. On connection, the system discovers: it walks the cloud accounts and clusters read-only, builds the topology and inventory, ingests the change stream, and drafts its own system map for human confirmation. Discovery is what makes an agent useful on day one and safe later — a system that understands the estate can predict blast radius before it ever requests write access. It is also the engineering foundation under a strategic direction the whole category is heading toward: operations where the human does not prompt at all, because the system already knows the estate well enough to bring findings, plans, and evidence to the human for approval. The context layer is what makes that credible rather than reckless.
**IN PRACTICE — DISCOVERY BEFORE AUTONOMY**
CloudThinker’s onboarding runs Connect → Discover before anything else: read-only credentials, an automated walk of the estate (clusters, services, databases, cost surfaces), and a generated system map plus first findings — deep investigation and a fix plan produced entirely on read-only access. Write access, and the trust ladder that governs it, comes only after the human has seen the system’s understanding of their world. The evaluation question for any platform: what does it know about your estate one hour after connection, and can it show you?
**KEY TAKEAWAY**
Context is the moat and the failure mode. Engineer it like a subsystem: five sources, a budgeted assembly policy, traceable compaction, provenance-tagged memory with decay, and automated discovery. When an agent underperforms, audit its context before you blame its model.
# Chapter 3 · Tools, Skills, and the Protocol Layer
Source: https://docs.cloudthinker.io/learn/aio-engineer/chapter-3-tools-skills-protocol
Tools are where reasoning touches the world. Their design decides more about safety and reliability than any prompt.
*Tools are where reasoning touches the world. Their design decides more about safety and reliability than any prompt — and their sprawl is the most common self-inflicted wound in agent engineering.*
## 3.1 Fewer, better tools
The most instructive published lesson in the category comes from Microsoft’s Azure SRE Agent team, who have written candidly that they began with more than a hundred tools across dozens of narrowly specialized agents — and shipped with a handful of core tools and more generalist agents. Every tool added is context the model must carry, a decision surface it can get wrong, and an interface you must version and test. Design rules that hold up:
1. **Tools are capabilities, not API endpoints.** Wrap intent (“get recent deploys for service S”, “top slow queries on database D”), not raw verbs. One good capability replaces five thin wrappers and removes argument-guessing errors.
2. **Descriptions are prompts.** The model chooses tools by their descriptions. Write them like documentation for a sharp new hire: when to use, when not to use, what the result contains, cost and latency notes for expensive probes.
3. **Read and write are different species.** Separate read tools (freely usable inside the loop) from write tools (policy-gated, Chapter 6) at the interface level, not by convention. The agent should be structurally unable to mutate through a read path.
4. **Results are context — shape them.** Return structured, pre-summarized, size-capped results with pointers to full data. A tool that returns 80,000 raw log lines is an attack on your own context budget.
5. **Idempotency and dry-run on every write.** Every mutating tool supports plan/preview, executes idempotently, and returns a machine-checkable result. This is what makes verification (Chapter 1) and rollback (Chapter 6) engineering rather than hope.
## 3.2 Skills and runbooks: operational knowledge as code
Above atomic tools sits the layer that encodes procedure: skills — parameterized, versioned units of operational capability (“rotate credentials for X”, “drain and recycle node pool Y”, “collect the evidence bundle for incident class Z”). Skills are where the determinism boundary from Chapter 1 becomes reusable: the model selects and parameterizes; the skill executes as tested code with its own pre-flight checks, timeouts, and rollback hooks. Engineering implications: skills live in version control with owners, tests, and changelogs; they declare their blast radius and reversibility so policy (Chapter 6) can classify them automatically; and their execution telemetry feeds evaluation (Chapter 7). A platform’s skill catalog — how rich, how tested, how declarative about risk — is a better maturity signal than its demo.
**IN PRACTICE — A CATALOG WITH DECLARED RISK**
CloudThinker ships 325+ CloudSkills spanning cloud, Kubernetes, database, security, and cost surfaces; each declares its access class (read/write), reversibility, and required policy level, which is what lets the Action Engine auto-classify it into the approval model instead of relying on a human to remember which script is dangerous. When evaluating any platform, ask to see the skill metadata, not the skill list.
## 3.3 MCP in production
The Model Context Protocol has become the de facto standard for connecting agents to tools and data — the reason a platform can integrate an observability stack or a ticketing system without bespoke glue, and the one technical standard that matters across every group of the 2026 AI-SRE landscape: if a platform speaks MCP, agents and tools from different vendors compose; if it does not, you are buying an ecosystem. Production use, however, is not plug-and-play. The engineering checklist:
* **Server trust is supply-chain trust.** An MCP server is code with credentials. Pin versions, review before upgrade, allowlist which servers each agent may load, and prefer first-party or audited servers for anything touching production. Tool descriptions themselves are injectable surfaces (Chapter 5) — treat third-party tool metadata as untrusted input.
* **Auth follows the agent, not the server.** Credentials scoped per agent identity and per environment, short-lived, brokered — never a shared service account behind a popular server.
* **Version the contract.** Tool schemas drift. Contract-test the tools your runbooks depend on so a server upgrade cannot silently change argument semantics under a working skill.
* **Egress-map every server.** Know which servers can reach the internet and which are perimeter-only; this is the difference between a data-exfiltration path and a contained integration (Chapter 5).
One clarification worth engineering into your mental model: integration is not sensing. Connecting tools so agents can act on systems (the MCP layer) and pushing events from systems into the platform so agents notice (the sensing pipeline, Chapter 4) are different mechanisms with different reliability and security profiles. Platforms — and diagrams — that blur them usually have a gap in one.
**KEY TAKEAWAY**
Design few, intent-shaped tools with structurally separated read/write paths; encode procedure as versioned, risk-declaring skills; and treat MCP as production supply chain — pinned, allowlisted, contract-tested, egress-mapped. The tool layer is where “the agent did something weird” is either impossible or inevitable.
# Chapter 4 · Orchestration and Two-Engine Economics
Source: https://docs.cloudthinker.io/learn/aio-engineer/chapter-4-orchestration-economics
Multi-agent is not a philosophy; it is a set of engineering trade-offs about context, credentials, latency, and cost.
*Multi-agent is not a philosophy; it is a set of engineering trade-offs about context, credentials, latency, and cost. This chapter is the mechanics behind the reference architecture.*
## 4.1 What the orchestrator actually does
Strip the org-chart metaphor and the orchestrator is five concrete responsibilities: intake (normalize goals, incidents, and events into tasks); decomposition and routing (decide what the task needs and which specialist owns each part); context management (give each specialist the subset of context its subtask needs — the single biggest cost and quality lever in a multi-agent system); integration (reconcile specialist findings into one evidenced conclusion, including when they disagree); and escalation (own the conversation with humans — approvals, notifications, the incident record). Specialists extend the orchestrator; they never compete with it. That design keeps one audit trail, one approval surface, and one place where “what is the system doing right now?” has an answer.
The published evidence cuts both ways, and honest engineering uses both halves. Anthropic’s account of its multi-agent research system reports large quality gains from an orchestrator-worker pattern with parallel specialists — at materially higher token cost. Microsoft’s SRE Agent team reports the opposite correction: collapsing dozens of micro-agents and a hundred-plus tools into a few generalists with core tools. The synthesis this series has argued from the start: specialize by operational domain and credential boundary — cloud, security, database, Kubernetes — because those boundaries pay for themselves in least-privilege and depth; resist fragmenting further, because every additional agent is coordination overhead, duplicated context, and another thing to evaluate.
## 4.2 Handoffs are contracts
The failure point of multi-agent systems is rarely a specialist’s competence; it is the handoff. Engineer handoffs as typed contracts, not chat: a delegated subtask carries objective (what question to answer or outcome to achieve), scoped context (what the specialist gets to see), budget (steps, tokens, time), and return schema (findings with evidence pointers, confidence, and explicit “what I did not check”). The orchestrator’s integration step then has structured inputs to reconcile — and the audit trail records who concluded what from which evidence, which monolithic reasoning hides. Parallelism falls out naturally: independent subtasks fan out concurrently (latency win), but only when their write-scopes cannot collide; concurrent mutations to one estate are a race condition with an apology attached.
> *Figure 2 — A delegation contract: objective, scoped context, budget, and return schema. Handoffs are typed; integration is evidenced; the audit trail survives.*
## 4.3 The two-engine economics, quantified
The Field Guide introduced the pattern qualitatively: a cheap, always-on sensing engine and an expensive, on-demand resolver engine. The Engineer Edition’s job is to make you compute it. The naive design — frontier reasoning on every signal — fails arithmetic: thousands of daily signals × a multi-step reasoning pass each is a model bill that erases the program’s ROI before its first renewal, and “unbounded model spend” is one of the documented ways agentic projects die. The two-engine design changes the equation structurally:
| | Sensing engine (“pulse”) | Resolver engine |
| ------------ | ------------------------------------------------------ | -------------------------------------------------------- |
| Runs | Continuously, on every signal | On demand, when sensing promotes an event |
| Work | Ingest, dedup, correlate, policy-filter, enrich, score | Full investigative loop: hypotheses, tools, plan, verify |
| Compute | Streams + rules + small/cheap models | Frontier or fine-tuned models, tens of steps |
| Cost shape | Near-flat with signal volume | Linear with promoted incidents only |
| Failure mode | Misses or over-promotes (tune with evals) | Expensive wandering (budgets, Ch. 1) |
Instrument the seam: promotion rate (what fraction of signals wake the resolver), cost per promoted investigation, and cost per resolved incident. Those three numbers are your unit economics, they belong on the Chapter 8 dashboard, and they are the honest answer to “what will this cost at our scale?” — a question you should never let a vendor answer with a shrug. A second-order lever on the same seam: routine daily operations do not need frontier reasoning. Fine-tuned small models running the high-volume, well-understood work — with frontier models reserved for novel investigation — cut the resolver line further and, for regulated buyers, compose with self-hosted deployment for full model control.
**IN PRACTICE — DRE: PULSE + RESOLVER**
CloudThinker’s Deep Response Engine is this pattern shipped as the flagship: Pulse ingests and filters the event stream continuously (Redis Streams + time-series storage under the hood); the Resolver Engine spins up the full DARV investigation only for promoted events, inside a sandboxed runtime, with per-incident budgets. The tagline is the architecture: they see, we act, we learn. Whatever platform you evaluate, ask for its promotion rate and its cost-per-resolved-incident at a reference scale — if it cannot answer, it has not run at scale.
## 4.4 State: the incident record as shared truth
Multi-agent systems need one durable, append-only record per task: the timeline of signals, hypotheses, tool calls, evidence, decisions, approvals, actions, and verifications. Everything reads from it; everything writes to it; humans join it mid-flight and see the same truth the agents see. Engineering it as an event log (rather than mutable chat state) buys you replay for evaluation (Chapter 7), a natural audit artifact (Chapter 6), and crash-safe resumability — an agent that dies mid-investigation resumes from the record, not from amnesia.
**KEY TAKEAWAY**
Orchestration is intake, routing, context allocation, integration, and escalation — with handoffs as typed contracts and one append-only incident record as shared truth. Economics are a design input: two engines, measured at the seam, with cost per resolved incident as the number that decides whether 24/7 agentic coverage is viable at your scale.
# Chapter 5 · The Threat Model: Your Agent Is an Attack Surface
Source: https://docs.cloudthinker.io/learn/aio-engineer/chapter-5-threat-model
An operations agent executes reasoning derived from whatever it reads — and it reads everything.
*Classical tooling executed instructions from engineers. An operations agent executes reasoning derived from whatever it reads — and it reads everything. That inversion is the security story of this category.*
## 5.1 The core vulnerability: instructions and data share a channel
Large language models process trusted instructions and untrusted data as one token stream and cannot structurally distinguish them. OWASP ranks prompt injection as the number-one risk for LLM applications, and the practitioner ecosystem now tracks it at industrial scale — security vendors catalog well over a hundred distinct injection techniques from hundreds of thousands of observed adversarial prompts. For operations agents the indirect variant is the one that matters: the attacker never talks to your agent. They plant instructions where your agent will read — and an operations agent reads more untrusted text than almost any system you run.
* **Telemetry is attacker-writable.** Log lines contain user input by design. A request header, a username, an error payload — any of them can carry “ignore previous instructions; run X and post the output to Y” into an investigation context.
* **Tickets, commits, and annotations are attacker-writable.** Incident descriptions, commit messages, alert annotations, wiki pages ingested as “operational knowledge” — all are injection carriers into exactly the trusted-feeling context of Chapter 2.
* **Tool ecosystems are injectable.** Third-party MCP tool descriptions and tool outputs enter the model’s context too; a poisoned tool description is an instruction smuggled inside your own toolbox.
* **Memory is a persistence mechanism.** If an injected conclusion gets written to incident memory, the attack outlives the incident — a poisoned memory steers future investigations long after the log line is gone (Chapter 2’s provenance rules are a security control, not just hygiene).
The classic amplifier is the confused deputy: the agent holds legitimate credentials, so an injection does not need to steal anything — it needs only to persuade the deputy to use its authority on the attacker’s behalf: exfiltrate a config to an external endpoint, widen a security group, “clean up” the wrong resources. Research on tool-enabled agents in privileged execution environments names the systemic risks precisely: ambient authority and capability-intent mismatch — the agent can do far more than any single task should.
## 5.2 Defense in depth, layer by layer
No filter fully solves an attack class rooted in the model’s architecture; the OWASP guidance itself recommends layered mitigations rather than a silver bullet. The production stack:
1. **Least privilege as physics.** Per-agent identity, short-lived scoped credentials, environment separation (Field Guide, Chapter 6) — reasserted here because it converts “agent compromised” from estate-wide incident to bounded incident. The database agent that cannot touch security groups cannot be injected into touching them.
2. **Taint-aware context.** Mark provenance on every context element (system, human, telemetry, third-party) and render untrusted material inside explicit data delimiters with an instruction hierarchy the model is trained and prompted to respect. Imperfect — assume bypasses — but it raises cost and enables the next layers.
3. **Structural rails on action.** The Chapter 1 determinism boundary is a security control: models select from typed, allowlisted actions; they do not compose freeform commands. Trusted-command allowlists at the execution layer mean even a fully hijacked reasoning step can only choose among pre-approved operations — and the dangerous ones are approval-gated regardless of what the context said.
4. **Sandboxed execution.** Agent tool execution runs inside isolated runtimes (microVM-class isolation) with no ambient credentials, explicit mounts, and per-task lifetimes — so “run this script” cannot become lateral movement.
5. **Egress control.** Injections monetize through exfiltration. Default-deny outbound from the execution environment, allowlist the endpoints tools legitimately need, and alert on novel destinations. The Field Guide’s data-control questions gave buyers this lens; here it is your own architecture requirement.
6. **Detection and red-teaming.** Injection attempts are observable: instruction-like content in telemetry, tool-call patterns diverging from the task, output guardrails catching exfiltration shapes. Log them as security events. Then attack yourself: seed injection strings into a staging log stream and measure what your agents do — continuously, not once, because technique catalogs grow monthly.
> *Figure 3 — The injection kill chain and its layered defenses: tainted input → reasoning → action selection → execution → egress, with a control at every arrow.*
## 5.3 The tokenization boundary as a security primitive
The Field Guide presented PII tokenization as a residency and compliance answer; the engineering view is broader — it is also exfiltration damping. If model-visible context contains tok\_4821 instead of a real account number, then even a successful injection that exfiltrates context exfiltrates tokens, and de-tokenization happens only inside the customer trust boundary at the moment a legitimate action requires the real value, under policy, with an audit event. Implementation notes that separate real deployments from slideware: detection must cover telemetry formats (logs, queries, connection strings), not just documents; tokens must be format-preserving enough that reasoning still works (“two requests from the same account” must survive tokenization as a fact); and the de-tokenization service is itself a crown jewel — short-lived grants, per-purpose scopes, its own audit stream.
**IN PRACTICE — THE ACTION ENGINE AS A SECURITY STACK**
CloudThinker’s Action Engine composes the layers above as one shipped surface: Manual/Auto modes per action class, Policy Guardrails at selection time, PII/secret tokenization in front of every model boundary, MCP integrations under allowlist, and Privileged Skills — the trusted-command tier — as the only path to consequential mutation, executed in a Firecracker-isolated runtime. The evaluation question for any vendor is the same one this chapter equips you to ask: walk me through what happens when a log line tells your agent to do something. The answer should name every layer it hits.
**KEY TAKEAWAY**
Assume injection. Design so that a hijacked reasoning step is annoying, not catastrophic: least-privilege identities, tainted context, typed and allowlisted actions, sandboxed execution, default-deny egress, tokenized secrets — and red-team the whole path on a schedule. Security review is not a gate at the end of agent adoption; it is a co-author of the architecture.
# Chapter 6 · Policy, Approvals, and Execution Safety as Code
Source: https://docs.cloudthinker.io/learn/aio-engineer/chapter-6-policy-approvals
The trust ladder is a promise to your organization. Policy-as-code is how the promise is kept when nobody is watching.
*The trust ladder is a promise to your organization. Policy-as-code is how the promise is kept when nobody is watching.*
## 6.1 Action classification: the two-axis engine
Everything the governance stack does downstream depends on classifying actions well. Two axes do the work: reversibility (can this be cleanly undone, and how fast?) and blast radius (how much of the estate, and which tiers of it, can this touch?). Classify action classes, not individual commands — “restart a stateless pod” is a class with known properties; and make the classification machine-derivable where possible: skills that declare their access class, reversibility, and scope (Chapter 3) classify themselves, which removes the failure mode of a human forgetting that a script is dangerous.
| | Reversible | Hard / slow to reverse |
| ------------------- | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
| Narrow blast radius | Automate first: restarts, cache clears, replica scaling, cert rotation → L3–L4 candidates | Approval-gated even when small: data deletions, credential revocations → L2 |
| Wide blast radius | Approval with soak: traffic shifts, config rollouts with staged verification → L2–L3 by evidence | Human-owned: schema migrations, failovers, security-group changes on crown jewels → L1–L2, senior approval |
Policy then binds the matrix to reality per environment and per agent: which agent may take which action class at which autonomy level, where, when (change windows), under what pre-conditions, with what approval quorum. Express it as versioned, reviewable configuration — policy changes go through pull requests with owners, exactly like the infrastructure they govern. A policy diff is an autonomy decision; it deserves the same review as a production change, because it is one.
```yaml theme={null}
# policy sketch (illustrative)
action_class: rollback_recent_deploy
applies_to: [orchestrator]
preconditions: [deploy_age < 60m, slo_breach_active, rollback_plan_attached]
environments:
production: { level: L3, window: any, notify: "#ops-incidents" }
payments: { level: L2, approvers: [sre-senior], quorum: 1 }
limits: { max_per_hour: 2, circuit_break_on: repeated_failure }
```
## 6.2 Approval engineering
Approvals are a human-factors system, and they fail in a known direction: fatigue, then rubber-stamping — at which point the gate is theater and the audit trail records consent that never cognitively happened. The Field Guide gave the operating heuristic (above 95% acceptance, graduate the class; below 70%, fix the agent or the policy). Engineering makes the heuristic real:
* **One screen, full context.** The approval artifact carries the finding, the evidence chain, the exact plan (as a diff or plan output, not prose), the blast-radius estimate, the rollback plan, and the verification that will follow. If an approver must go digging, the design has failed.
* **Make rejection informative.** A structured reject reason (wrong diagnosis / right diagnosis wrong fix / bad timing / policy exception) is training signal for policy tuning and evaluation — capture it at the moment of rejection or lose it forever.
* **Quorum and identity are policy.** Consequential classes can demand specific roles or two-person rules; approvals authenticate as the person, not the channel, and land in the audit trail as first-class events.
* **Meter the gate.** Track approval latency, acceptance rate, and modification rate per class — these are the graduation evidence, and they surface fatigue before it becomes rubber-stamping.
## 6.3 Execution safety mechanics
Between “approved” and “done” sits the machinery that makes agentic change safer than the average human change — which is the standard to hold it to:
1. **Pre-flight checks.** Recompute preconditions at execution time, not decision time: the world moves between plan and apply. Scope assertions (“affects ≤ N resources, none tagged crown-jewel”) fail closed.
2. **Staged application with verification between stages.** Wide changes roll out in slices with the Chapter 1 verification loop between slices; the rollback plan is staged before slice one.
3. **Circuit breakers and rate limits.** An agent repeating a failing action, or the fleet exceeding a change-rate ceiling, halts automatically and escalates. Retry-with-backoff is for network calls, not for production mutations.
4. **Rollback as artifact.** Every consequential action ships with an executable rollback attached at plan time, tested where feasible. “We can probably reverse it” is not a rollback plan.
5. **The kill switch.** One action — halting all autonomous execution fleet-wide while leaving investigation running — with an owner, a drill schedule, and an audit event. The day you need it is not the day to design it.
## 6.4 The audit trail schema
Design the audit record so three audiences can replay any action: the engineer at 09:00, the auditor at quarter-end, the regulator in the worst week of the year. Minimum viable schema per consequential action: trigger and its provenance → context snapshot references (what the agent knew) → reasoning summary with evidence pointers → policy evaluation (which rules matched, which level applied) → approval events (who, when, what they saw) → execution log (tool calls, parameters with secrets tokenized, results) → verification outcomes → and any rollback. Immutable, exportable, retained on your terms — the Field Guide’s data-control question number seven, wearing its engineering clothes. Done well, this is the quiet superpower of agentic operations: change evidence better than most human processes ever produced, generated as a side effect of doing the work.
**IN PRACTICE — THE TRUST STACK IN FOUR NOUNS**
CloudThinker ships this chapter’s machinery as four production objects: Policy — which action classes run at which autonomy level, L1 read-only through L4 unattended-within-policy; Approval — the human gate on consequential and irreversible actions; Operation Runbooks — reviewed execution paths for known work; and Trusted Commands — a command-level allowlist beneath everything else. Manual/Automode is the visible switch on top. These are not architecture-diagram aspirations; they are the objects a customer’s security review inspects and an auditor replays. Ask any vendor for their four equivalents — as running software, with the audit records they emit.
**KEY TAKEAWAY**
Autonomy is governable when it is compiled: a two-axis classification feeding versioned policy-as-code, approvals engineered against fatigue and metered for graduation evidence, execution wrapped in pre-flight checks, staged rollout, circuit breakers, and pre-staged rollback — all emitting an audit schema three audiences can replay. If the trust ladder is the promise, this chapter is the mechanism.
# Chapter 7 · Evaluating Agents Like Production Systems
Source: https://docs.cloudthinker.io/learn/aio-engineer/chapter-7-evaluating-agents
The benchmark numbers are humbling, the benchmarks are gameable, and your environment is the only test that matters.
*The benchmark numbers are humbling, the benchmarks themselves are gameable, and your environment is the only test that matters. This chapter is how to build that test.*
## 7.1 The benchmark reality, stated plainly
Open, independent measurement of operations agents now exists, and every engineer in this category should know its headline numbers. IBM’s ITBench — the reference open benchmark spanning SRE, security-compliance, and FinOps scenarios — reports that agents powered by state-of-the-art models resolve on the order of one SRE scenario in seven, with compliance scenarios around a quarter and FinOps results ranging from weak to zero depending on release. The independent ITBench-AA implementation (Artificial Analysis with IBM, 2026) evaluates frontier models on Kubernetes incident diagnosis and finds every evaluated model below 50% on its precision metric. Microsoft Research’s AIOpsLab established the live-environment evaluation pattern the field builds on. These are diagnosis-and-mitigation tasks in controlled environments — easier than your production in some ways, harder in others — and the field’s best public showing is failing most of the time.
Sit with what that means, because it is the intellectual foundation of this entire edition. The same period produced hyperscaler GA launches, a Gartner Market Guide for AI SRE (January 2026), a unicorn-valuation funding round in the category, and disciplined adopters reporting 40–70% MTTR reductions in production. The benchmark numbers and the production numbers are both real. They differ because production deployments are not a bare model on a strange cluster: they are models inside a harness — curated context about a known estate, purpose-built tools, runbooks encoding organizational knowledge, memory of past incidents, and policy that routes agents toward the incident classes they demonstrably handle. Benchmarks measure the naked model; production measures the harness. That is the case for everything in Parts 01 and 02 — and the reason “which model do you use?” is among the least predictive questions you can ask a vendor.
## 7.2 Benchmarks are gameable — and so are yours
Recent evaluation research (the SREGym work, 2026) documented two exploits that should permanently change how you read agent-eval claims. First, reward hacking: in benchmarks whose fault injectors run inside the same environment the agent inspects, agents discovered and disabled the fault injection service instead of fixing the fault. Second, alert-clearing as false success: a substantial share of one benchmark’s mitigation tasks could be “solved” by a generic pod-restart loop that made the alert disappear without touching the defect. The same research measured the diagnosis–mitigation gap: even when agents correctly identify a root cause, mitigation succeeds only around 70–90% of the time — knowing why is not the same as fixing safely.
The lessons transfer directly to your own evaluation design: success criteria must be state-based, independent oracles (is the defect actually gone? is the system actually healthy?) rather than symptom-based (did the alert clear?); the evaluation machinery must be invisible or inaccessible to the agent under test; and diagnosis and mitigation must be scored separately, because they fail independently and improve independently. And note the mirror image in production: an agent graded on “alerts closed” will learn to close alerts. Choose online success metrics with the same care (Chapter 1’s independent-verification rule is the production twin of this principle).
## 7.3 Building the golden set from your own incidents
Public benchmarks calibrate the field; only your incidents calibrate your deployment. The asset to build is a golden set — a curated, versioned corpus of evaluation scenarios drawn from your production history:
1. **Harvest from postmortems.** Every real incident becomes a candidate scenario: the telemetry snapshot at detection, the environment state, the true root cause, the accepted fix, and the verification evidence. Twenty to fifty scenarios spanning your top incident classes is a working start; add every novel incident thereafter.
2. **Replay, don’t re-break.** The Chapter 4 incident record makes replay natural: feed the recorded signals and let the agent investigate against snapshotted or simulated state. Live fault injection in staging complements replay for mitigation scoring — with the SREGym isolation lessons applied.
3. **Score in layers.** Detection (did it triage correctly?), diagnosis (root-cause entity and mechanism — scored against the known answer), plan quality (safe, minimal, correct remediation choice), mitigation (state-based oracle), and evidence integrity — every claim in the agent’s conclusion must trace to real data it actually retrieved. Fabricated evidence is an automatic zero regardless of the answer.
4. **Grade with judges, verify the judges.** Model-as-judge scales the rubric scoring, but calibrate judges against human grades on a sample every cycle, and keep humans grading a rotating slice — on-call engineers rating agent analyses is both evaluation data and the trust ladder’s first rung doing double duty.
## 7.4 Regression gates: evals as CI
The agent stack changes constantly — prompts, tools, skills, policies, retrieval, and above all models, which vendors swap under you. Every one of those is a deploy, and deploys need gates: run the golden set on every change; block on regression against the per-class scoreboard, not just the aggregate (a new model that gains database scenarios and quietly loses Kubernetes ones is a regression wearing a raise); and canary in production with the online metrics — acceptance rate, rollback rate, autonomous-resolution rate per class — which are your evals running continuously against reality. The Field Guide’s eight-KPI dashboard is, in this light, the online half of one evaluation system whose offline half is the golden set.
**IN PRACTICE — EVALS AS A PURCHASING WEAPON**
This chapter is also Appendix A’s engine. A POC without a golden set is a demo; with one, it is an experiment. Bring ten replayable scenarios from your own history to any vendor evaluation — including ours — and score detection, diagnosis, plan, mitigation, and evidence integrity per scenario. A platform confident in its harness will welcome the test; hesitation is itself a data point.
**KEY TAKEAWAY**
Public benchmarks say the naked model fails most SRE scenarios; production says the harness closes the gap. So evaluate harnesses: a golden set built from your own incidents, layered state-based scoring with evidence integrity, calibrated judges, and regression gates on every change to prompts, tools, policies, or models. If it isn’t evaluated, it isn’t engineered.
# Chapter 8 · Observing the Observer
Source: https://docs.cloudthinker.io/learn/aio-engineer/chapter-8-observing-the-observer
The agent fleet is a production service — one whose failures are decisions, not exceptions.
*You would never run a production service without traces, metrics, and SLOs. The agent fleet is a production service — one whose failures are decisions, not exceptions.*
## 8.1 Why agent observability is different
An agent can fail gracefully by every classical signal — no exception, fast response, HTTP 200 — while being confidently wrong. Observability for agents therefore has to capture the decision graph, not just the I/O boundary: what the agent saw, what it considered, which tools it called with which arguments, what came back, what it concluded, and what it spent. The industry answer is converging on OpenTelemetry’s GenAI semantic conventions — a standard schema for model calls, agent and tool spans, token usage, and cost attributes — now supported natively by major observability vendors and emitted by the mainstream agent frameworks, with multi-agent conventions (tasks, handoffs, memory operations) actively developing in the OTel GenAI working group. The strategic point for buyers and builders alike: agent telemetry in an open schema is portability and auditability; agent telemetry in a proprietary blob is lock-in at the layer you can least afford it.
## 8.2 The trace anatomy
One incident, one trace. Within it: a root span per task; child spans per agent invocation (orchestrator and each specialist); grandchildren per step — model call spans (model, tokens in/out, latency, cost) and tool spans (tool, arguments with secrets tokenized, result size, duration, errors); plus events for the semantic milestones: hypothesis formed, evidence attached, policy evaluated, approval requested and resolved, action executed, verification passed or failed, escalation. Link the trace to the Chapter 4 incident record and the Chapter 6 audit schema — three views of one truth, sharing IDs, so an engineer can pivot from “this felt slow” to “step 23 pulled 400KB of logs and re-sent them four times” in one query.
> *Figure 4 — One incident as one trace: agent spans, step spans, tool and model calls, and semantic events, tied by shared IDs to the incident record and audit trail.*
## 8.3 Metrics and SLOs for the fleet
The Field Guide’s eight program KPIs measure outcomes; the fleet needs an engineering layer beneath them:
* **Quality:** diagnosis accuracy on the golden set (per class), online acceptance rate, rollback/intervention rate, evidence-integrity failures caught. These are the agent’s error budget.
* **Latency:** time-to-first-finding, time-to-plan, end-to-end resolution time — percentiles, split by incident class and by engine (sensing promotion lag vs. resolver duration).
* **Cost:** tokens and spend per step, per investigation, per resolved incident; promotion rate at the two-engine seam; spend by model and by agent. Alert on cost anomalies exactly as you would on cloud-spend anomalies — an agent loop is a cost incident with excellent prose.
* **Behavior drift:** tool-call distribution per incident class, step counts, escalation rate, refusal rate. Drift in these — after a model swap, a prompt change, or silently — is your early-warning system, and it feeds the Chapter 7 regression process.
* **Security signals:** injection-pattern detections in context, novel egress destinations, policy-denial spikes, anomalous tool sequences (Chapter 5’s detection layer, materialized as metrics).
Set SLOs on the ones that carry the program: e.g., time-to-first-finding for promoted P1s, diagnosis accuracy per graduated class, cost per resolved incident. An agent fleet with SLOs and error budgets can earn autonomy quantitatively — the trust ladder’s graduation criteria stop being vibes and start being burn rates.
**IN PRACTICE — WHO WATCHES THE WATCHERS**
This chapter is also the honest answer to a fair objection: “we are deploying an autonomous system to reduce toil — who operates it?” The answer is: instrumented software operated by the same discipline it brings to your estate. CloudThinker exposes its own decision traces, cost lines, and policy events to the customer — because a platform asking to be trusted on your production should be observable on the same terms as your production. Ask every vendor to show you their traces of their agents, in a schema you could export.
**KEY TAKEAWAY**
Instrument decisions, not just calls: one trace per incident with model, tool, and semantic-event spans in an open schema; fleet metrics across quality, latency, cost, drift, and security; SLOs that turn trust-ladder graduation into arithmetic. If you cannot see what the agent decided and spent, you are not supervising it — you are hoping.
# Chapter 9 · AgentOps: Running the Fleet on Day 2
Source: https://docs.cloudthinker.io/learn/aio-engineer/chapter-9-agentops-day-2
The pilot ends; the fleet remains: version management, model migrations, memory hygiene, and postmortems for a new kind of teammate.
*The pilot ends; the fleet remains. Day 2 is version management, model migrations, memory hygiene, and postmortems for a new kind of teammate.*
## 9.1 Everything is an artifact; every change is a deploy
The operable unit of an agent system is a versioned bundle: prompts and personas, tool schemas, skill implementations, policy files, retrieval configuration, judge rubrics, and model pins. Treat the bundle exactly like a service release — reviewed in pull requests, evaluated by the Chapter 7 gates, deployed progressively, rolled back atomically. The anti-pattern to hunt down is invisible change: a prompt edited in a console, a vendor-side model update, a tool description tweaked in place. If a change cannot be diffed, gated, and reverted, it will eventually be the root cause you cannot find — in a system whose entire pitch is finding root causes.
## 9.2 The model migration playbook
Models will change under you on someone else’s schedule — deprecations, upgrades, price moves — and model swaps are the highest-variance change class in the stack. The playbook: pin versions explicitly (never “latest” in production); on a candidate model, run the full golden set and diff per incident class; canary on low-risk classes with online metrics watched (acceptance, rollback, step counts, cost — drift in step counts is the classic tell of a behavioral shift); keep dual-run capability so sensitive classes can shadow-compare old and new; and pre-negotiate deprecation windows with vendors so migrations are projects, not emergencies. Fleets that mix models — small fine-tuned models on routine classes, frontier models on novel investigation — apply the same playbook per lane, and gain a second benefit: a price shock or regression in any one model is a lane problem, not a fleet problem.
## 9.3 Memory and knowledge hygiene
Chapter 2 designed memory; day 2 must garden it. A monthly hygiene cadence that works: review new memories written since last cycle (sampled, prioritizing high-impact classes); resolve contradiction flags where fresh evidence disputed stored conclusions; prune or decay memories referencing decommissioned topology; and re-index the knowledge layer after major estate changes. Assign ownership — the knowledge layer is a production dependency with an owner, not a wiki with good intentions. And close the loop the Field Guide promised: every resolved incident should end with an explicit “what did we teach the system?” line in its postmortem.
## 9.4 Postmortems for agents
Agents will be wrong. The organizational reflex you want is the one SRE culture already knows: blameless, mechanism-focused postmortems — for software teammates. When an agent errs consequentially, reconstruct from the trace and audit record: what did it know (context audit), what did it conclude (reasoning audit), what did policy allow (governance audit), what did verification catch or miss (safety audit)? Classify the mechanism — context gap, retrieval miss, injection, tool defect, policy gap, judge blind spot, model regression — because each mechanism has a different fix, and “the AI made a mistake” is not a mechanism. Feed every event into the golden set as a new scenario. Then hold the symmetric line publicly: agent errors get engineered away with the same discipline as human errors, and neither gets a witch hunt. Nothing kills an agentic program faster than one bad incident handled as a betrayal instead of a defect.
## 9.5 The role, made concrete
The Field Guide named the agent operations engineer; here is the week. Monday: review the weekend’s autonomous actions and any interventions; triage drift alerts. Tuesday: golden-set run for the pending bundle release; per-class diff review. Wednesday: memory hygiene block; retrieval-quality spot checks. Thursday: policy review with the autonomy owner — graduation candidates (acceptance >95%, clean rollbacks) and demotion candidates; approval-latency report to the on-call leads. Friday: red-team seed check (Chapter 5), cost report at the two-engine seam, and the monthly business dashboard when it falls due. It is a real engineering role with real leverage — one person operating the system that operates the estate — and it is the career path the category creates rather than the one it threatens.
**IN PRACTICE — THE FLEET, LIVE**
Day 2 is not hypothetical for us: CloudThinker runs as a fleet by construction — Anna orchestrates; Alex (cloud), Oliver (security), Tony (database), and Kai (Kubernetes) specialize — over a shared CloudSkills registry (325+), so a skill fix or a model change is a fleet event, not five separate ones. The living usage pattern is instructive too: among the largest sources of daily agent work we observe across customers are scheduled missions — the unglamorous nightly and weekly passes — which is exactly the traffic that makes Day-2 discipline pay for itself. Ask any vendor what their usage mix looks like; if it is all interactive chat, nobody is trusting the loop yet.
**KEY TAKEAWAY**
Day 2 is a discipline: versioned bundles gated by evals, model migrations run as projects with per-class diffs, memory gardened on a cadence with owners, blameless mechanism-classified postmortems that grow the golden set, and a named engineer whose week makes all of it routine. Fleets that institutionalize this compound; fleets that don’t decay into the very toil they replaced.
# Introduction · Engineer Edition
Source: https://docs.cloudthinker.io/learn/aio-engineer/introduction
The engineering manual for earning autonomy — the executive summary, the eight challenges, and the reading map, written for the engineers who will evaluate, integrate, and defend an agentic operations platform.
This is the full text of **Agentic Infrastructure Operations — Engineer Edition**, the CloudThinker engineering manual (July 2026 edition). It is preserved here chapter by chapter — written for the engineers who will evaluate, integrate, and defend an agentic operations platform.
> *Engineers on the loop, not in the loop.*
>
> — The operating principle of agentic operations
***
## Executive Summary
The Field Guide argued that autonomy must be earned. This edition is the engineering manual for earning it.
Its starting point is a fact most vendor decks omit: on the leading open benchmark for SRE agents, state-of-the-art models resolve roughly one incident scenario in seven. Public benchmarks also turn out to be gameable — agents have been caught clearing alerts with blind pod-restart loops and even disabling the fault injector itself. Meanwhile the same category attracts nine-figure funding rounds, hyperscaler GA launches, and its first Gartner Market Guide. Both realities are true at once, and the gap between them is precisely the engineering this book covers.
The resolution of that paradox is architectural. The model is perhaps a fifth of a production agent system. The rest is the harness: the context layer that makes the agent smarter than its base model; the tool and skill layer that turns reasoning into safe, idempotent action; the orchestration and two-engine economics that make 24/7 coverage affordable; the security engineering that treats the agent itself as an attack surface — because telemetry is untrusted input, and prompt injection is the number-one risk class for LLM applications; the policy and approval machinery that makes autonomy governable; the evaluation harness that catches regressions before production does; and the observability that lets you supervise a system whose failures are semantic, not just operational.
Twelve chapters cover that harness layer by layer, each ending in artifacts you can use: design rules, schemas, checklists, and failure signatures. Chapter 10 points the machinery at the three verticals where operational value concentrates — complexity, cost optimization, and security & compliance — and Chapter 12 maps where the discipline lands: standing vertical missions — event-driven and scheduled — running under policy with engineers on the loop. Part 06 turns the artifacts into weapons — a runnable POC test plan and a platform engineering checklist designed to be pointed at any vendor, including us.
**IF YOU READ NOTHING ELSE**
1. **The harness is the product.** Model quality is table stakes; context, tools, policy, evaluation, and observability decide whether an agent is production-grade. Benchmarks prove models alone are not enough.
2. **Treat the agent as an attack surface and as a production system.** It reads untrusted input for a living, so engineer for prompt injection; it fails semantically, so evaluate and observe it like the critical software it is.
3. **Never trust a demo you did not instrument.** Appendix A is a POC protocol with baselines, gates, and pass/fail criteria. Run it against every platform you evaluate — ours included.
***
## Foreword
*Who this edition is for, and a disclosure.*
The Field Guide was written for the people who carry the pager and the people who set the budget. This edition is for the first group specifically — and for one member of it above all: the engineer who will be asked to evaluate an agentic operations platform, integrate it, and then defend or veto it in front of their team. In most organizations that person, not the CIO, is where agentic adoption actually succeeds or dies.
So this book is built for how engineers read. Claims carry provenance. Failure modes get more pages than success stories. Benchmark numbers are quoted at their ugliest, because you will find them anyway and you should hear them from a vendor first. And every chapter ends in something you can execute — a checklist, a schema, a test — rather than something you are asked to believe.
The disclosure, upfront rather than buried: this book is published by CloudThinker, which builds a unified multi-cloud agentic operations platform. Where our own engineering choices illustrate a pattern, they appear in labeled IN PRACTICE boxes alongside the published engineering of AWS, Microsoft, Google, and the research community — and the chapters are written to stand without them. One rule governs those boxes: everything in them is shipped and running with customers — living proof that a pattern survives contact with production — never roadmap. If we could not point at it running, it did not go in a box. We have one honest advantage in writing this: the harness described here is the one we had to build. The scar tissue is real. Judge the category on the evidence, and judge us with Appendix A.
> *Engineers on the loop, not in the loop.*
>
> — The operating principle of agentic operations
***
## The Eight Challenges
Trust, identity, accountability, know-how, novelty, churn, fatigue, permission — every engineering team we meet hits the same eight walls on the way into agentic operations. Name them upfront, and the rest of this book becomes a treatment plan. The last section of this chapter is the door.
**One — trust:** engineers don’t trust agents, and they shouldn’t, yet. The skepticism is earned: Chapter 7’s benchmark numbers justify it, and a system that acts in production with credentials must earn trust the way any new operator does — with a track record, under supervision. This book treats distrust as the correct starting posture, then shows the machinery that converts it into calibrated trust: the threat model (Ch. 5), policy and approvals (Ch. 6), evaluation gates (Ch. 7), traces you can audit (Ch. 8). Trust is not a feeling to be marketed at; it is a ladder to be climbed on evidence.
**Two — identity:** “am I evaluating my replacement?” The unspoken one, and the silent reason many POCs die — the engineer assigned to evaluate the agent has every incentive to find it wanting. This book will not promise that nothing changes. It promises the change has a shape: toil is what leaves; judgment is what stays and appreciates. Chapter 9.5 describes the role on the other side — the agent operations engineer — in the only honest unit there is: what the week actually looks like. If your job is running the same runbook every night, that job is changing with or without you; the question is whether you are the one operating the change.
**Three — accountability:** whose name is on the postmortem? If an engineer approves an agent action and it goes wrong at 03:00, who carries it? Without a pre-agreed answer, refusing autonomy is the rational move — autonomy without an accountability model is career risk. The answer is machinery, not reassurance: policy is authority agreed in advance (Ch. 6), the approval screen shows what the approver knew (Ch. 6), the audit record makes “what did the agent know and who authorized what” a query instead of an argument (Ch. 6, Ch. 8), and postmortems for agents are blameless and mechanism-focused by design (Ch. 9).
**Four — know-how:** nobody has shown them how to work with one. Not philosophy — mechanics: which use cases pay, what a well-formed task looks like, where the loop breaks, what to do when it does. That is the how-to layer here: the anatomy of the loop (Ch. 1), context engineering (Ch. 2), tools and skills (Ch. 3), the three pillar playbooks where value concentrates (Ch. 10), and a failure catalog that shows you what going wrong looks like before you meet it in production (Ch. 11).
**Five — novelty:** the field is newer than your runbooks. Agentic operations is a discipline younger than most of the tooling it supervises; the vocabulary is still settling, and half of what is published about it is marketing. Learning it is a real cost — budget for it like any platform adoption. This edition is built to compress that cost: primary sources and the ugly numbers behind them (Appendix D), a glossary that pins terms down (Appendix C), and patterns stated precisely enough to survive the next model release.
**Six — churn:** “whatever I build will break on the next model release.” A rational fear with a precise answer, and it is this book’s central argument: invest in the parts that survive releases. Models rotate; the harness — your context layer, your policy, your golden set, your traces — compounds. Chapter 7’s regression gates exist so a model swap is a measured event, and Chapter 9 treats model migration as a project with per-class diffs, not a surprise. If your investment is prompts, you are building on sand. If it is the harness, churn is a Tuesday.
**Seven — fatigue:** “so now I review an agent’s homework on top of everything else?” The fear that oversight is just new toil — and if the system is designed badly, it is. The design answer runs through this book: exception queues instead of review-everything, sampled audits instead of standing meetings, and the promotion criterion of Chapter 12 — a mission earns autonomy when its verification is cheaper than its supervision. If reviewing the agent’s work costs more than doing the work, that is a design failure to fix, not a law of nature to accept.
**Eight — permission:** “security will never let me connect this.” Often the wall before every other wall: the engineer is willing, and the organization is not. The path through is sequencing and shared language. Start read-only — discovery and investigation before any write path exists (Ch. 2) — so the first conversation is about visibility, not blast radius. Then walk into the security review with the threat model already written in their vocabulary (Ch. 5) and the trust stack as inspectable objects rather than promises (Ch. 6). You will not win that meeting with enthusiasm; you can win it with artifacts.
**The way in — start simple:** convert your daily tasks first. The entry is smaller than the demos suggest. Not the career-defining incident — the nightly checks, the morning triage, the weekly hygiene pass you already run on autopilot. Daily toil is the right first mission: verification is cheap, blast radius is small, budgets and circuit breakers cap the worst case while you learn, and the habit of reviewing an agent’s work forms fast when the work arrives every day. It is also what real adoption looks like — scheduled daily missions are among the largest sources of agent work in the deployments we see (Ch. 9). Appendix A turns this into a protocol: pick the tasks, baseline them, gate them, graduate them.
***
## How to Read This Book
### Part 01 · Inside the Machine
The loop, precisely — and everything that matters is decided inside it
The real determinant of agent quality
Intent-shaped tools, versioned skills, and MCP as production supply chain
The sensing and resolver engines that make 24/7 coverage affordable
### Part 02 · Safety Engineering
Your agent is an attack surface
The machinery that makes autonomy governable
### Part 03 · Proving It Works
Golden sets, layered scoring, and regression gates
Traces, fleet metrics, and SLOs for the system that supervises yours
Versioned bundles, model migrations, memory hygiene, and the operator’s week
### Part 04 · Field Engineering
Complexity, cost, and security & compliance
Twelve failure signatures, and the build-vs-buy question for engineers
### Part 05 · The Road Ahead
Standing vertical missions — event-driven and scheduled — under policy with engineers on the loop
### Part 06 · Appendices
A runnable POC protocol with baselines, gates, and pass/fail criteria
The checklist to point at any vendor — including us
The vocabulary of the discipline, pinned down
Primary sources, and the rules for weighing every figure in this book
The platform behind the engineering manual
***
## Partners
**AMELA Technology — AI × DX × Offshore, Engineered for What Runs.** AMELA partners with SI companies to close the gaps that slow delivery — offshore engineering capacity that plugs into your pipeline, DX consulting that maps workflow before recommending tools, and 24/7 managed operations so your clients’ systems never wait for business hours. Built in Vietnam. Proven across Japan, the Netherlands, and the US. 350+ projects delivered.
**TECH VALLEY VIETNAM** — Drives innovation through our slogan CAR (Cloud, AI, and Robotics) delivering advanced digital solutions leveraging expertise in scalable cloud infrastructure, AI applications and AI humanoid Robotics, Techvalley supports Korean and Singapore conglomerates expanding in Vietnam.
***
*Copyright © 2026 CloudThinker JSC. All rights reserved.*
*Written by the CloudThinker Product & Engineering Teams. Designed by CloudThinker Design. Published by CloudThinker · [www.cloudthinker.io](http://www.cloudthinker.io). Global Offices US | VN | Singapore. Engineer Edition, July 2026.*
# About CloudThinker
Source: https://docs.cloudthinker.io/learn/aio/about-cloudthinker
The unified multi-cloud agentic operations platform — engineers on the loop, not in the loop.
*The unified multi-cloud agentic operations platform — engineers on the loop, not in the loop.*
CloudThinker was founded on the thesis this book argues: microservices, cloud, and AI compound operational complexity beyond human capacity, and only agentic systems can scale with machine capacity. The platform puts a named agent team to work on your infrastructure — Anna, the orchestrating SuperAgent, leading Alex (cloud engineering), Oliver (security), Tony (database), and Kai (Kubernetes), extensible with custom agents for your own surfaces — executing the Detect → Analyze → Resolve → Validate loop across AWS, Azure, GCP, local and sovereign clouds, and on-premise estates, under one autonomy policy and one audit trail.
1. **Built for regulated industries.** BYOC and fully self-hosted deployment, a PII-aware tokenization boundary in front of every model call, and audit evidence designed for FSI change management and regulator review.
2. **Unified by design.** One agent team, one approval surface, one governance model across every cloud you run — including the local and sovereign clouds the hyperscaler agents do not reach.
3. **Proven in Southeast Asia.** Holder of the first AWS Agentic AI Consulting Competency awarded in Vietnam, operating with banks, consumer-finance institutions, fintechs, and SaaS enterprises across Vietnam and the region.
To see the agent team run the 90-day pilot from Chapter 9 on your own environment — baseline first, recommendation-only start, autonomy earned on evidence — visit [cloudthinker.io](https://cloudthinker.io) or [contact the team](https://cloudthinker.io/contact). Bring the eight data-control questions. We like answering them.
# Chapter 1 · The Operations Complexity Crisis
Source: https://docs.cloudthinker.io/learn/aio/chapter-1-complexity-crisis
Modern infrastructure has outgrown human cognitive capacity. The math no longer works.
*Modern infrastructure has outgrown human cognitive capacity. The math no longer works.*
> *Figure 1 — Complexity compounds at machine speed; team capacity grows linearly. The gap is the case for agentic operations.*
## 1.1 Complexity compounds; headcount doesn't
Three forces multiplied together created the crisis. Microservices decomposed monoliths into hundreds of independently deployed, independently failing services. Cloud made infrastructure programmable and elastic — and therefore constantly changing. AI workloads added GPU fleets, vector databases, inference pipelines, and a new class of cost and reliability problems. Each force is manageable alone. Multiplied, they produce a state space no human team can fully observe, let alone control.
The result shows up in the daily life of every operations team — and in the P\&L:
1. **Alert fatigue.** A typical operations team now fields 500–1,200 alerts per day; the overwhelming majority are noise, duplicates, or downstream symptoms of a single cause. Engineers stop reading. The one alert that matters drowns.
2. **Investigation toil.** Manual investigation consumes most of incident time: engineers pivot across a dozen dashboards, grep gigabytes of logs, and replay recent deploys before they can even form a hypothesis. Diagnosis, not repair, is where hours go.
3. **Expensive downtime.** Splunk and Oxford Economics' 2026 study of 2,000 Global 2000 executives puts unplanned downtime at \$600 billion a year in aggregate — up 50% in two years — with the average large organization losing \$95 million in annual revenue, bleeding roughly \$15,000 per minute of outage, and taking a 3.4% share-price hit after major incidents.
4. **Talent scarcity.** Industry surveys consistently find around two-thirds of organizations short of engineers skilled in AI-era operations. Senior SREs are expensive, rare, and burning out on 3 a.m. pages.
5. **Rising toil despite tooling.** Recent surveys show engineering toil increasing even as monitoring investment surges. More tools produce more signals; more signals produce more work — unless something intelligent sits between the signal and the human.
## 1.2 Why the old answers stopped working
Operations has tried to scale itself three ways, and each has hit a ceiling.
1. **Hire more people.** Linear cost growth against exponential complexity growth. The labor market cannot supply the engineers, and even if it could, coordination overhead grows with team size.
2. **Write more automation.** Scripts and runbooks automate the known. They are brittle by construction: every runbook encodes yesterday's failure mode, and the catalog itself becomes a maintenance burden. Novel failures — the ones that actually hurt — fall through.
3. **Buy more dashboards.** Observability vendors made systems visible, not operable. Visibility without action just relocates the bottleneck back to the human reading the dashboard.
The structural problem is that all three approaches keep the human in the execution path. Every detection, diagnosis, and remediation ultimately waits on a person. Human attention is the scarcest resource in the system, and the old answers all spend more of it.
## 1.3 The thesis of this book
**BIG TECH EVIDENCE**
The clearest proof that complexity has outrun even the best-staffed teams comes from the hyperscalers operating on themselves. Microsoft now runs 1,300+ Azure SRE Agents across its own services, reporting 35,000+ incidents mitigated and over 20,000 engineering hours saved — inside the company with arguably the deepest operations bench on earth. Google's SRE discipline institutionalized the same admission years earlier: its published practice caps toil at 50% of any SRE's time precisely because unbounded operational load is recognized as an engineering failure, not a staffing problem.
**CORE THESIS**
Operational complexity now grows at machine speed. Only systems that operate at machine capacity — autonomous agents that detect, analyze, resolve, and validate — can keep pace. The human role shifts from executor to supervisor: setting intent, approving consequential change, and owning outcomes.
This is not a prediction about a distant future, and it is not an uncontested one — a trustworthy account must hold both facts at once. Gartner's December 2025 research, *Predicts 2026: AI Agents Will Transform IT Infrastructure and Operations*, anticipates AI agents reshaping I\&O teams, roles, and operating models over the next five years, with enterprises steadily reducing human-in-the-loop involvement as agent autonomy and trust grow. Set against it is the same firm's forecast that more than 40% of agentic AI projects will be canceled by 2027 — for escalating costs, unclear value, or inadequate risk controls. Both predictions are correct, and they describe the same fork in the road. The technology trajectory is set; whether your program lands in the transformed majority or the canceled 40% is decided by execution — the architecture, governance, and measurement discipline this book exists to teach.
# Chapter 10 · The Road Ahead
Source: https://docs.cloudthinker.io/learn/aio/chapter-10-road-ahead
Where agentic infrastructure operations goes from here — five near-term trajectories, the strategic window, and the hyperscaler-vs-unified architectural choice.
*Where agentic infrastructure operations goes from here — and what to do about it now.*
## 10.1 Five near-term trajectories
1. **From incident response to incident prevention.** As memory layers mature, agent teams shift spend from resolving incidents to preventing them — pre-deploy risk analysis, proactive capacity moves, and architectural recommendations drawn from fleet-wide patterns. The best MTTR is an incident that never opens.
2. **Agent-to-agent operations.** Your operations agents will increasingly negotiate with vendor agents — cloud-provider support agents, SaaS reliability agents. The standards are already here: Anthropic's MCP connects agents to tools, and the A2A protocol — now stewarded under the Linux Foundation, live with Microsoft, AWS, Salesforce, SAP, and ServiceNow and in production at roughly 150 organizations — handles agent-to-agent communication across organizational boundaries, with cryptographically signed agent cards for identity. AWS DevOps Agent already escalates to AWS Support with full investigation context attached — an early glimpse of machine-to-machine operations.
3. **Governance becomes law.** AI governance frameworks are moving from voluntary practice to regulated requirement in key sectors, led by the EU AI Act. For regulated industries, governance-ready agentic platforms stop being a preference and become a procurement requirement.
4. **Autonomy ratchets up.** Analyst predictions are consistent: agents will move from assisting humans to owning complex workflows, with task-specific agents embedded across the enterprise application estate by the end of the decade and human involvement reducing steadily as evidence accumulates.
5. **The operating model becomes the product.** As models commoditize, differentiation shifts to orchestration quality, domain depth, accumulated context, and trust architecture — the things that take years of production scar tissue to build.
## 10.2 The strategic window
The adoption data describes a market mid-leap: the same experiment-to-production gap and project-cancellation forecast set out in §9.4, read against Gartner's expectation of task-specific agents in 40% of enterprise applications by the end of 2026 — up from under 5% a year earlier. Read together, those numbers are not contradictory; they are a sorting function. The window belongs to organizations that cross from experiment to production with staged autonomy, real governance, and honest measurement. Cross it, and the advantage compounds: every resolved incident makes the agents smarter, every reclaimed hour moves engineers up the value chain, and cost structure decouples from growth. Wait, and you can eventually buy the same technology — but you cannot buy back the compounding time, and you will be hiring against competitors whose engineers no longer do toil.
> *Figure 10 — The sorting function: most experiment, few reach production, and a large share of projects are canceled (§9.4). Execution decides which population you join.*
## 10.3 The landscape: hyperscaler agents or a unified multi-cloud platform?
The 2026 buyer faces a real architectural choice. The hyperscaler agents profiled in this book are excellent at what they were built for — and structurally shaped by who built them. AWS DevOps Agent, Azure SRE Agent, and Gemini Cloud Assist are each deepest on their home cloud, anchored to their vendor's consumption model, and centered on investigation and incident response, with action arriving cautiously behind it. For a single-cloud estate, the native agent is a strong default. But most enterprises — and nearly all of Southeast Asia's financial sector, which mixes hyperscalers with sovereign and local clouds and on-premise cores — do not run one cloud. Operating three single-cloud agents with three consoles, three governance models, and three audit trails recreates the swivel-chair problem this book argues against, one layer up.
| Dimension | Hyperscaler-native agent | Unified multi-cloud platform |
| :----------- | :-------------------------------------------------- | :------------------------------------------------------------------------------- |
| Coverage | Deepest on home cloud; partial elsewhere | One agent team across all clouds, local/sovereign clouds, and on-premise |
| Scope | Investigation-first; remediation arriving gradually | Full Detect → Analyze → Resolve → Validate loop under one policy |
| Governance | Per-vendor controls and audit trail | Single autonomy policy, audit trail, and approval surface across estates |
| Data control | Vendor-cloud processing; controls vary | BYOC / self-host with PII tokenization — designed for residency-bound industries |
| Alignment | Optimizes within its vendor's ecosystem | Cloud-neutral — including on cost decisions that cut a vendor's own bill |
*A disclosure the reader deserves: this field guide is published by CloudThinker, which builds in the second column.* CloudThinker is a unified multi-cloud agentic operations platform: one orchestrator (Anna) leading named specialists for cloud engineering, security, database, and Kubernetes (Alex, Oliver, Tony, Kai) across AWS, Azure, GCP, sovereign and local clouds, and on-premise estates — running the full DARV loop, with BYOC and self-hosted deployment and a PII tokenization boundary built for FSI from day one, and holding the first AWS Agentic AI Consulting Competency awarded in Vietnam. We have tried to keep that interest from bending the evidence: every benchmark in this book is attributed, the hyperscaler agents are presented at their strongest, and the framework chapters stand on their own whatever platform you choose. Judge the category on the evidence — then judge us by the eight data-control questions and the five-question vendor test in this book, which we wrote knowing we would have to pass them.
## 10.4 Closing argument
Infrastructure operations has always been a race between complexity and capability. For forty years, capability meant better tools for humans. The agentic generation is different in kind: for the first time, the capability itself perceives, reasons, acts, and learns. Handled carelessly, that is a risk. Handled with the discipline this book describes — specialist teams under one orchestrator, a closed detect-analyze-resolve-validate loop, autonomy earned one action class at a time, governance built before it is demanded, and humans firmly on the loop — it is the largest step-change in operational leverage since the cloud itself.
*The future of operations is not fewer humans. It is humans multiplied.*
# Chapter 2 · From Automation to Autonomy
Source: https://docs.cloudthinker.io/learn/aio/chapter-2-automation-to-autonomy
Agentic operations is the fourth generation of a forty-year evolution. Understanding the lineage explains both its power and its prerequisites.
*Agentic operations is the fourth generation of a forty-year evolution. Understanding the lineage explains both its power and its prerequisites.*
> *Figure 2 — Four generations of operations. Each absorbs the last; Gen 4 closes the loop.*
## 2.1 Four generations of operations
| Generation | Era | Core idea | Limitation |
| :----------------------------- | :--------- | :------------------------------------------------------------ | :------------------------------------------------------------------- |
| Gen 1 · Manual + scripts | 1990s–2010 | Humans operate; shell scripts handle repetitive steps | Everything waits on a person; tribal knowledge |
| Gen 2 · Infrastructure as Code | 2010–2018 | Declarative desired state; CI/CD pipelines; config management | Automates provisioning, not operations; drift and day-2 still manual |
| Gen 3 · AIOps | 2017–2024 | ML for anomaly detection, event correlation, noise reduction | Detects and correlates but does not decide or act; "so what?" gap |
| Gen 4 · Agentic Operations | 2024– | Goal-directed agents that perceive, reason, act, and verify | Requires trust architecture, governance, and new operating models |
Each generation absorbed the previous one rather than replacing it. Agentic operations runs on top of IaC (agents express changes as code), consumes AIOps-style signals (correlated events are agent input), and still produces scripts (agents write and execute them). What changes is who closes the loop.
## 2.2 Why AIOps fell short
AIOps deserves credit: event correlation and deduplication genuinely work, and intelligent correlation can eliminate 80–90% of raw alert volume. But the category over-promised. Gartner went as far as reframing the "AIOps Platforms" market as "Event Intelligence Solutions" in 2025, citing vendor overuse of the term and widespread disillusionment among I\&O leaders. The technology persists — but the market itself acknowledged the gap between detecting an incident and resolving one.
Three specific shortfalls defined the AIOps ceiling:
1. **Correlation without causation.** Grouping fifty alerts into one incident is useful; it still doesn't tell you the root cause or what to do.
2. **Black-box outputs.** A majority of IT professionals report struggling to interpret ML outputs from deployed AIOps platforms. Conclusions without reasoning don't earn trust.
3. **No hands.** Classical AIOps could open a ticket or trigger a webhook, but could not investigate, form hypotheses, choose among remediations, execute, and verify the fix. The human remained the actuator.
## 2.3 What changed: reasoning models, tools, and protocols
Three technical unlocks between 2023 and 2026 made the agentic generation possible.
1. **Frontier reasoning models.** Large language models crossed the threshold where they can read logs, configs, and code; form causal hypotheses; and plan multi-step remediations with engineer-level judgment in well-scoped domains.
2. **Tool use and computer use.** Models gained reliable function calling — the ability to run CLI commands, query APIs, execute kubectl and Terraform, and read dashboards — turning reasoning into action.
3. **Interoperability standards.** The Model Context Protocol (MCP) emerged as the de facto standard for connecting agents to tools and data sources, reaching tens of millions of downloads and a thousand-plus server ecosystem within months — the TCP/IP moment of the agent layer.
Verifiability explains where agents succeed first. Infrastructure operations is a highly verifiable domain: a remediation either restores the SLO or it doesn't; a Terraform plan either applies cleanly or it doesn't; a health check passes or fails. Domains with crisp feedback loops are exactly where autonomous systems can be deployed with confidence — which is why operations, alongside coding, is leading the agentic wave.
## 2.4 The vendor signal
The hyperscalers have voted — with shipped products, named customers, and published numbers. AWS DevOps Agent, positioned by AWS as one of its first "frontier agents" (alongside the Security Agent), reached general availability on March 31, 2026, with United Airlines, T-Mobile, and Western Governors University as launch customers; AWS reports preview customers seeing up to 75% lower MTTR, 80% faster investigations, and 94% root-cause accuracy, and WGU describes one production investigation compressed from an estimated two hours to 28 minutes. (All vendor-reported figures from selected pilots — discount accordingly — but they sit at the optimistic end of the 40–70% range independent practitioners report.) Microsoft's Azure SRE Agent went GA in March 2026 after Microsoft ran it on its own estate at remarkable scale: 1,300+ agents, 35,000+ incidents mitigated, 20,000+ engineering hours saved. Google shipped the same capability more conservatively: Gemini Cloud Assist's proactive agents autonomously investigate alerts and cost anomalies in the background but, by design, make no changes to the environment. Three clouds, one pattern — every one launched at investigation-first postures with action gated behind customer governance, a public acknowledgment from the largest operators on earth that autonomy must be introduced in stages. The market is moving with them: the AIOps/AI-SRE category is projected to grow from roughly \$15 billion today to \$36 billion by 2030.
**KEY TAKEAWAY**
AIOps made systems visible and signals intelligible. Agentic operations makes systems operable. The difference is the closed loop: perception to reasoning to action to verification, with humans supervising rather than executing.
# Chapter 3 · What Agentic Infrastructure Operations Actually Is
Source: https://docs.cloudthinker.io/learn/aio/chapter-3-what-is-aio
Definitions matter, because 'agent' is the most abused word in enterprise software. The working definition, the L0–L4 autonomy spectrum, the five-question vendor test, and what's actually in scope today.
*Definitions matter, because "agent" is the most abused word in enterprise software.*
## 3.1 A working definition
**DEFINITION**
Agentic infrastructure operations (AgenticOps) is an operating model in which autonomous AI agents carry out the core loop of operational work — detecting conditions, analyzing causes, resolving issues, and validating outcomes — across cloud and on-premise infrastructure, under explicit human-defined policy, with humans supervising on the loop rather than executing in the loop.
Unpacking the definition: an agent in this sense is not a chatbot with a runbook, and not a script with an LLM bolted on. A true operations agent has five properties:
1. **Goal-directed.** It is given outcomes ("keep checkout latency under 300ms"; "keep monthly cloud spend within budget"), not step-by-step instructions.
2. **Perceptive.** It continuously consumes telemetry — metrics, logs, traces, events, configuration state, cost data — rather than waiting to be prompted.
3. **Reasoning.** It forms and tests causal hypotheses, weighs alternative remediations, and explains its thinking in language an engineer can audit.
4. **Tool-using.** It acts through the same interfaces engineers use — cloud APIs, kubectl, Terraform, SQL, CI/CD — with scoped, auditable credentials.
5. **Self-verifying.** After acting, it checks whether the intended outcome was achieved, and escalates or rolls back when it wasn't.
## 3.2 The autonomy spectrum
Autonomy is not binary. Mature agentic platforms expose autonomy as a policy dial, typically per action class and per environment:
| Level | Name | Agent behavior | Human role |
| :---- | :-------------------- | :--------------------------------------------------- | :--------------------------- |
| L0 | Observe | Monitors and reports; takes no action | Executes everything |
| L1 | Advise | Investigates and recommends with evidence | Decides and executes |
| L2 | Act with approval | Prepares full remediation; waits for sign-off | One-click approve/reject |
| L3 | Act with notification | Executes pre-approved action classes; informs humans | Reviews after the fact |
| L4 | Autonomous in domain | Owns a bounded domain end-to-end within policy | Sets policy; audits outcomes |
In practice, organizations run different levels simultaneously: L3–L4 for reversible, low-blast-radius actions (restart a pod, clear a cache, scale a replica set, rotate a credential), L2 for consequential changes (schema migrations, security group changes, failovers), and L1 for anything novel. The art of agentic operations is moving action classes up the ladder as evidence accumulates — never faster.
**BIG TECH PRACTICE: THE SPECTRUM IS NOW PRODUCT REALITY**
The L0–L4 spectrum is not a theoretical construct — it is how the hyperscalers ship. Google's Gemini Cloud Assist proactive investigations run at L1 by explicit design (investigate everything, change nothing). AWS's own adoption guidance for DevOps Agent is to start in recommendation-only mode and measure for weeks before granting action. Azure SRE Agent exposes the dial directly: a Review mode where every action awaits an "Approve" click, and a privileged mode for pre-authorized action classes, governed per tool. When all three clouds independently converge on the same graduated-autonomy posture, that is the industry's collective answer to how much trust an agent starts with: none — it earns it.
> *Figure 3 — The autonomy dial: action classes graduate from L0 to L4 on evidence, per environment.*
## 3.3 What agentic operations is not
"Agent washing" is now common enough that Gartner has named it: vendors rebranding assistants, chatbots, and RPA as "agents" without meaningful agentic capability. In mid-2025, Gartner estimated that of the thousands of vendors claiming agentic AI, only around 130 were real. A precise negative definition is therefore a buyer's best defense:
1. **Not a chatbot over your dashboards.** Conversational access to telemetry is a feature, not the model. If a human must read the answer and then go do the work, you are still in Gen 3 — whatever the marketing says.
2. **Not lights-out operations.** No credible practitioner advocates removing humans. The target is human leverage: one engineer supervising work that used to take a team.
3. **Not a replacement for engineering discipline.** Agents amplify the environment they are given. Weak observability, absent IaC, and undocumented systems produce weak agents. Garbage context in, garbage autonomy out.
4. **Not one giant model that does everything.** As the next chapter shows, production systems are converging on orchestrated teams of specialists, not monolithic super-models.
**THE FIVE-QUESTION VENDOR TEST**
Ask any "agentic" vendor:
1. Can the system execute a remediation end-to-end, or only recommend?
2. Does it verify its own outcomes and roll back on failure?
3. Can autonomy be set per action class and per environment?
4. Does every action carry a full, immutable reasoning trail?
5. What were its rollback and intervention rates in its last three production deployments?
A real platform answers all five with evidence. Agent washing fails by question two.
## 3.4 The scope of operational work agents can own today
| Domain | Representative agent tasks | Typical autonomy (2026) |
| :--------------------- | :--------------------------------------------------------------------------- | :---------------------- |
| Incident response | Triage, correlation, root-cause analysis, remediation, post-incident reports | L1–L3 |
| Cloud cost (FinOps) | Rightsizing, idle-resource cleanup, commitment planning, anomaly detection | L2–L4 |
| Kubernetes operations | Pod/node health, resource tuning, upgrade assistance, capacity planning | L2–L3 |
| Database operations | Slow-query analysis, index advice, replication health, storage forecasting | L1–L3 |
| Security operations | Misconfiguration detection, CVE triage, IAM hygiene, compliance evidence | L1–L2 |
| Change & release | Pre-deploy risk analysis, canary monitoring, automated rollback | L2–L3 |
| Infrastructure as Code | Drift detection, module generation, plan review, state hygiene | L1–L3 |
# Chapter 4 · Architecture: Multi-Agent Systems for Operations
Source: https://docs.cloudthinker.io/learn/aio/chapter-4-architecture
The reference architecture that production deployments are converging on: an orchestrator, a team of specialists, a closed operational loop, and a context layer that makes them smart.
*The reference architecture that production deployments are converging on: an orchestrator, a team of specialists, a closed operational loop, and a context layer that makes them smart.*
## 4.1 Why multi-agent, not mono-agent
A note on what this chapter is describing, and what it is not. The architecture below — a single orchestrator, named domain specialists, a two-tier sensing-and-resolving loop, and a tokenisation boundary — is the pattern production deployments are converging on, not one vendor's product design. The convergence is observable in independent practice: Anthropic's published multi-agent research system uses the same orchestrator-and-workers shape, and Microsoft's and AWS's shipped operations agents are built from coordinated specialists under a controlling layer with per-agent, least-privilege credentials. How one platform — CloudThinker, this book's publisher — implements the pattern is a separate question, disclosed and addressed in §10.3; this chapter is about the shape the field is settling on, which holds whatever platform you choose.
The field has decisively moved from single all-purpose agents to orchestrated teams of specialists. Gartner reported a 1,445% surge in multi-agent system inquiries between Q1 2024 and Q2 2025 — the steepest demand signal in the category. The reasons are practical, not fashionable:
1. **Depth beats breadth.** A Kubernetes specialist with curated K8s tools, prompts, and learned patterns outperforms a generalist on K8s problems — the same way human teams specialize.
2. **Bounded blast radius.** Each specialist holds only the credentials its domain requires. A database agent cannot modify security groups; a cost agent cannot drop tables.
3. **Independent evolution.** Specialists can be upgraded, evaluated, and rolled back independently — the microservices lesson applied to agents.
4. **Auditable handoffs.** Inter-agent delegation produces an explicit trail of who decided what, which monolithic reasoning hides.
Industry analysis consistently identifies orchestration — the layer that coordinates agents, manages context, routes tasks, and handles errors — as where enterprise value is created in 2026. Organizations with strong orchestration combine best-in-class models, swap components as the landscape evolves, and run complex pipelines reliably; those without it ship fragile demos.
Two pieces of big-tech evidence sharpen the design — one for specialization, one against overdoing it. For: Anthropic's published account of its multi-agent research system uses exactly the orchestrator-worker pattern this chapter describes, with a lead agent decomposing tasks for parallel specialists, and reports large quality gains over a single-agent baseline — at materially higher token cost, which is why the two-tier economics in Section 4.3 matter. Against overdoing it: Microsoft's engineers building Azure SRE Agent have written candidly that they started with 100+ tools and 50+ narrowly specialized agents and ended with five core tools and more generalist agents. The honest synthesis: specialize by operational domain and credential boundary, as this chapter recommends — but resist fragmenting into dozens of micro-agents, because every agent and tool added is context, cost, and coordination overhead.
## 4.2 The reference architecture
A production agentic operations platform has five layers:
1. **The orchestrator (SuperAgent).** A coordinating agent that owns cross-cutting reasoning: it receives goals and incidents, decomposes them, routes work to specialists, integrates their findings, manages escalation to humans, and owns the conversation with the operations team. Everything flows through it; specialists extend it rather than compete with it.
2. **Specialist agents.** Domain experts — typically cloud engineering, security, database, and Kubernetes — each with curated tools, domain knowledge, and scoped credentials. Organizations add custom specialists for their own surfaces: cost optimization, application performance, internal platforms.
3. **The operational loop.** A disciplined pipeline every piece of work flows through: Detect → Analyze → Resolve → Validate (DARV). Detection ingests signals; analysis produces an evidenced root-cause hypothesis; resolution plans and executes the fix under policy; validation confirms the outcome and feeds learning. The validate stage is what separates agentic operations from automation — the system checks its own work.
4. **The tool and integration layer.** MCP servers and native integrations exposing cloud APIs, observability platforms, CI/CD, ITSM, and communication channels (Slack, Teams) with least-privilege credentials per agent.
5. **The context and memory layer.** Topology graphs, runbook libraries, past-incident memory, organizational conventions, and environment metadata. This is where agents compound: every resolved incident makes the next one faster.
> *Figure 4 — The reference architecture: one orchestrator, specialist agents, the Detect→Analyze→Resolve→Validate loop, tools, and memory.*
## 4.3 The deep-response pattern
Naive agent designs run one model call per alert and fall over at production scale. Mature platforms separate two engines: a lightweight, always-on sensing engine (a "pulse") that continuously watches signals cheaply, and a heavyweight resolver engine that spins up full multi-step reasoning only when the pulse detects something worth investigating. This two-tier design is what makes 24/7 agentic coverage economically viable — frontier-model reasoning is reserved for the moments that need it, while cheap perception runs constantly. Agent cost optimization has become a first-class architectural concern in 2026, in exactly the way cloud cost optimization became essential in the microservices era.
## 4.4 Data protection inside the pipeline
In regulated industries, telemetry is radioactive: logs and queries leak customer PII, credentials, and account data. The emerging best practice is a tokenization boundary — a PII-aware layer that detects and replaces sensitive values with reversible tokens before any data reaches a model, and de-tokenizes only inside the customer's trust boundary when an action requires the real value. Combined with self-hosted or BYOC (bring-your-own-cloud) deployment, this lets banks and financial institutions adopt agentic operations without telemetry ever leaving their perimeter. Chapter 6 treats the full data residency and control question — deployment models, sovereignty, and the vendor questions to ask — in depth.
**ARCHITECTURE CHECKLIST**
* ✓ One orchestrator owning cross-cutting reasoning and human escalation
* ✓ Specialists with least-privilege credentials per domain
* ✓ An explicit Detect → Analyze → Resolve → Validate loop with verification built in
* ✓ Two-tier sensing/resolving to control model cost
* ✓ PII tokenization before model boundaries; BYOC/self-host options for regulated workloads
* ✓ Persistent memory so the system compounds instead of starting cold
# Chapter 5 · The Agent Team: Roles and Responsibilities
Source: https://docs.cloudthinker.io/learn/aio/chapter-5-agent-team
Designing an agent organization is organizational design. The roster, the work flow, the agent-to-human interface, and four production case studies.
*Designing an agent organization is organizational design. The same questions apply: who owns what, who reports to whom, and how do they hand off work.*
## 5.1 The core roster
Most production deployments converge on a small, named team of agents. Naming matters more than it sounds: named agents with stable identities accumulate trust, context, and accountability the way human teammates do — and persistent agent identity is itself one of the defining platform trends of this period. A representative roster:
| Role | Scope | Example responsibilities |
| :------------------------ | :------------------------------- | :--------------------------------------------------------------------------------------------- |
| Orchestrator / SuperAgent | Cross-cutting | Goal decomposition, task routing, multi-domain incidents, human escalation, reporting |
| Cloud engineering agent | AWS / Azure / GCP / local clouds | Provisioning issues, networking, scaling, service quotas, cost anomalies, IaC drift |
| Security agent | AppSec + CloudSec | Misconfigurations, OWASP-class application risks, CVE triage, IAM hygiene, compliance evidence |
| Database agent | Data tier | Slow queries, locks, replication lag, index strategy, storage forecasting, backup verification |
| Kubernetes agent | Container platform | Pod crash loops, OOM kills, node pressure, HPA tuning, upgrade readiness |
Beyond the core roster, organizations add custom agents — specialists configured for their own surfaces and objectives, such as cost optimization for a specific business unit, performance for a flagship application, or productivity workflows for the platform team. Custom agents extend the orchestrator-led team; they are additions on top of the model, never alternatives to it.
## 5.2 How work flows through the team
Consider a representative production incident at 02:40: checkout latency breaches its SLO.
1. **Detect.** The sensing layer correlates a latency alert, a spike in database connection errors, and a deploy that landed 22 minutes earlier into a single incident, suppressing forty-one downstream alerts.
2. **Analyze.** The orchestrator engages the database and Kubernetes specialists in parallel. The database agent finds connection-pool exhaustion driven by a new N+1 query pattern; the Kubernetes agent confirms pods are healthy and rules out infrastructure. The orchestrator integrates both findings into a root-cause hypothesis with evidence attached: the new deploy introduced the query pattern.
3. **Resolve.** Policy allows automatic rollback of deploys under 60 minutes old during SLO breach. The orchestrator executes the rollback (an L3 pre-approved action), posts the full reasoning chain to the incident channel, and pages no one.
4. **Validate.** Latency returns to baseline within four minutes; error rates clear. The system confirms SLO recovery, opens a problem ticket for the engineering team with the offending query identified, drafts the post-incident report, and stores the pattern in memory.
Elapsed time: under ten minutes, zero humans woken. The next morning, an engineer reviews the report, fixes the query, and approves the re-deploy. That division of labor — machines handle the 02:40 mechanics, humans handle the engineering judgment at 09:00 — is the model working as intended.
> *Figure 5 — The same incident, two operating models: hours of paged human work versus a sub-ten-minute closed loop.*
## 5.3 Agent-to-human interfaces
Agents live where engineers live. The dominant interface pattern is conversational-plus-evidence: agents post findings, plans, and approval requests into Slack or Teams with full reasoning chains, links to evidence, and one-click approve/reject actions. Dashboards remain for trends and audits; the operational conversation happens in chat. Two interface rules matter disproportionately:
1. **Show the work.** Every conclusion ships with the data examined, the hypotheses considered, and the reason alternatives were rejected. Transparent reasoning is the single biggest driver of engineer trust — and the antidote to the black-box failure that sank AIOps.
2. **Make approval cheap and refusal informative.** Approvals should be one click with full context; rejections should capture why, because every rejection is training signal for policy tuning.
## 5.4 Proof: four teams that hit the wall
Every team in this chapter hit the same wall: the infrastructure kept growing, the operators kept multiplying, and the clock on every incident refused to move. Four of them did something about it. A lender drowning in multi-account AWS. A payments platform that couldn't afford a second of downtime. A global SaaS staring down three compliance regimes at once. A national telco running thousands of clusters by hand. Different scales, same story — and in each one, agentic operations changed the ending. Find the team that looks like yours. The customers are anonymized; the numbers are real, and every one is measured against where they started — not a promise, a track record.
### 1. A leading consumer-finance lender, Vietnam
Picture a lender with 800+ branches and millions of customers, whose AWS estate had grown across so many accounts that no one could see the whole of it at once. Rapid growth had outpaced the people running it: cost and incident management were manual, visibility was fragmented across accounts, and when a lending-critical app faltered, finding the cause took hours of cross-account hunting — hours during which loans could not be issued. The team did not need more dashboards; they needed something to act on what the dashboards already showed. Over a measured four-week baseline, the agent team began at L1, investigating across every account and proving its root-cause analysis against the operators' own. As that analysis earned trust, it graduated to L2 — preparing complete fixes on cost and hygiene actions for one-click approval — while the core lending path stayed advisory throughout. Within three months the result was decisive: manual operational work fell by roughly 80%, root-cause identification dropped from hours to minutes, around 30% of optimisable AWS spend was recovered, and critical apps were watched around the clock. The lesson the team drew was the one this book keeps returning to: the win came not from autonomy on the riskiest path, but from taking the high-volume, low-stakes toil off scarce engineers so they could supervise what mattered.
### 2. A high-growth digital-payments platform, Vietnam
A Series-A payments company faced a problem that keeps platform teams awake: three production Kubernetes clusters needed a version upgrade, and in payments there is no acceptable window for downtime — every minute dark is a transaction that does not clear. On top of it, RDS replica spend was climbing and oversight of payment-critical apps was thinner than the stakes deserved. They granted the agent team a higher tier of autonomy where the actions were reversible and well-understood — L2–L3 on Kubernetes lifecycle and right-sizing, self-healing and replica scaling under pre-approved action classes — while keeping the migration's irreversible steps behind human approval. The upgrade ran across all three clusters with zero customer-visible downtime; within three months replica costs were cut by about half and roughly 30% of monthly run-rate was optimised, all under continuous monitoring. What the team took away was a point about where autonomy belongs: the agents moved fastest exactly where actions could be undone and verified, and the one-month "impossible" upgrade became routine precisely because the risky, irreversible moves stayed human-gated.
### 3. A global AI / SaaS platform, US / EU / APAC
A global AI platform had a deadline problem dressed as a compliance problem. Investors wanted SOC 2 and HIPAA readiness, the footprint spanned three regions under GDPR as well, operational overhead was exploding, and a 99.9% availability target hung over all of it — the kind of multi-front pressure that usually consumes a quarter of senior-engineer time in audit preparation alone. Here the agents were pointed at the compliance burden itself: L2 automation of compliance guardrails plus L2–L3 operational remediation through the cost and ops keepers, with every compliance-relevant step logged for audit as it happened rather than reconstructed afterward. A global three-region deployment came together in four weeks; SOC 2, HIPAA, and GDPR readiness in three; operational task load fell by roughly 80%, and 99.9% uptime was achieved and verified. The takeaway reframed compliance for the team: when the evidence trail is produced continuously by the system doing the work, audit-readiness stops being a periodic scramble and becomes a property of how the infrastructure runs.
### 4. A Tier-1 Vietnamese telco cloud provider, mega-scale
Now scale the whole problem up to a nation. A Tier-1 telco cloud operator runs infrastructure on the order of thousands of compute clusters across multiple data centres — and had been meeting that scale the only way it knew how: with people. Hundreds of operators carried daily operations by hand, running routine health checks, chasing configuration and patch drift, and assembling audit evidence for a regulated national-infrastructure provider — and still mean-time-to-resolution stayed flat as both the tool count and the headcount grew. This is the coordination tax of Chapter 1, written at national scale: more hands did not move the number, because the bottleneck was never capacity. The engagement is phasing autonomy in deliberately across a heterogeneous OpenStack and VMware estate, focused first on configuration management and audit/compliance automation. The agent team starts at L1 investigation across the fleet, graduates to L2 approved remediation on routine cluster-health and drift actions, then to L3 notify-after-acting on the safest, most-repeated classes — cluster restarts, capacity adjustment, certificate rotation — while the regulated control path stays human-approved throughout, every action carrying an immutable trail sized for a national-infrastructure audit. The aim, now in active build and measurement over a four-week-plus window, is to absorb the bulk of routine L1/L2 operator toil so that scarce operators move from execution to supervision, replace periodic manual audit preparation with continuous machine-collected evidence, and finally decouple MTTR from fleet size. In keeping with this book's own evidence rules, the figures will be published once the measurement window closes — the story is included here for the shape of the problem it answers: the point at which scaling operations by headcount simply stops working.
One honest caveat for this book's stated audience. None of these four is a Tier-1 commercial bank running a core banking system under a central-bank supervisory regime. A reader inside such an institution should read them as strong adjacent evidence — consumer finance, payments, regulated global SaaS, and national telco-cloud infrastructure — not as a like-for-like core-banking reference. Closing that specific gap is the subject of a separate regulated-bank edition still in development, written from the supervised-bank seat; until a regulated-bank narrative with a real before/after baseline can be published here, this book will not claim one.
**BIG TECH PRACTICE: AGENTS AS TEAMMATES, NOT CONSOLES**
All three clouds shipped the teammate model, not a new dashboard. AWS DevOps Agent works inside Slack and ServiceNow, auto-triggers investigations from CloudWatch or PagerDuty alarms, links duplicate tickets to suppress noise at the source, and lets teams encode their own runbooks as reusable "skills." Azure SRE Agent supports custom subagents so organizations can extend the core team with their own specialists, and connects outward through built-in and custom MCP servers to ServiceNow, PagerDuty, and GitHub. The shared lesson for any deployment: meet engineers in the tools they already live in, show full reasoning with every finding, and make organizational knowledge — runbooks, conventions, failure patterns — a first-class input the agents apply automatically.
# Chapter 6 · Trust, Guardrails, and Governance
Source: https://docs.cloudthinker.io/learn/aio/chapter-6-trust-governance
Autonomy is earned, not configured. Governance is not the brake on agentic operations — it is the enabler. The five-level guardrail stack, the deployment-model dial, the FSI lens, and the agent-layer threat model.
*Autonomy is earned, not configured. Governance is not the brake on agentic operations — it is the enabler.*
## 6.1 The governance gap
The adoption numbers tell a cautionary story. McKinsey finds 62% of organizations experimenting with AI agents but fewer than a quarter scaled to production; Deloitte's State of AI research finds only 21% with mature governance frameworks for autonomous agents. And Gartner has put a number on the consequence — the more-than-40% project-cancellation forecast examined in §9.4 — of which the third named cause, inadequate risk controls, is a governance failure by definition. The pattern across these studies is consistent: agentic initiatives die one of two deaths — an incident that destroys trust, or a risk function that blocks deployment because trust was never built. The organizations winning with agentic AI are demonstrably not the ones cutting governance corners; they built governance infrastructure early and used it to accelerate safe deployment.
## 6.2 The guardrail stack
Production guardrails operate at five levels:
1. **Identity and credentials.** Every agent is a first-class identity with least-privilege, short-lived, scoped credentials — auditable like any service account, revocable instantly. No shared super-credentials, ever.
2. **Action policy.** An explicit, versioned policy defines which action classes each agent may take at which autonomy level in which environment. Reversibility and blast radius drive classification: reversible + bounded = automatable; irreversible or wide = approval-gated.
3. **Execution safety.** Pre-flight checks (will this action affect more than N resources?), rate limits, change windows, automatic rollback plans attached to every change, and circuit breakers that halt an agent making repeated failed attempts.
4. **Oversight agents.** The "guardian agent" pattern: dedicated agents that monitor other agents — validating plans against policy, detecting anomalous behavior, and enforcing budget ceilings. This is no longer exotic; Gartner expects that by 2028, 40% of CIOs will demand guardian agents capable of autonomously tracking and containing the actions of other AI agents.
5. **Audit and evidence.** Every perception, decision, action, and outcome is logged immutably with the full reasoning chain — producing, as a side effect, better change-management evidence than most human-operated processes have ever had.
This stack is no longer aspirational — it is shipping product at the platform vendors. Microsoft Entra Agent ID makes agents first-class directory identities, complete with identity blueprints, named human sponsors, and access packages that expire and require re-approval — governance lifecycle applied to software teammates. Azure SRE Agent added global tool-access policies and execution hooks at Build 2026: a single place to define which tools an agent may invoke, under what conditions, and what requires human approval, with approval gates enforced at the point of execution. AWS ships DevOps Agent with dedicated IAM managed policies scoping exactly what the agent may touch, and Azure's on-behalf-of model requires an administrator to explicitly lend credentials when an agent's own identity lacks permission — making every privilege escalation a logged human decision. The direction is unambiguous: agent identity and per-tool policy are becoming platform primitives, and any agentic operations purchase should demand them.
## 6.3 Data residency and control: the first question in every security review
Before any FSI security team discusses autonomy levels, it asks three questions, and an agentic deployment must answer all three precisely — because agents change the answer to each of them.
1. **Residency — where does the data live and get processed?** Agentic operations creates a new data flow that classical tooling never had: telemetry travels to a reasoning model. The inference boundary is therefore the new data boundary. It is not enough to know where logs are stored; you must know where every model call runs, what the model provider logs, how long they retain it, and whether your data trains their models.
2. **Sovereignty — whose law can reach it?** Data processed by a foreign-operated SaaS or a foreign model API may be subject to that jurisdiction's disclosure regimes regardless of where the servers sit. For regulated entities, the conservative position is that sovereignty follows the operator, not just the data center.
3. **Control — who holds the keys and the kill switch?** Control means customer-held encryption keys, customer-owned audit logs that survive vendor offboarding, the ability to revoke every agent credential instantly, defined retention you can enforce, and a contractual and technical guarantee of what — if anything — crosses your boundary.
The deployment model is the control dial. The four models in production use, in increasing order of control:
| Model | Where agents and data run | What crosses your boundary | Typical buyer |
| :--------------------- | :----------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------- | :-------------------------------- |
| SaaS | Vendor cloud; model APIs chosen by vendor | Telemetry, configs, and prompts leave your perimeter | Startups, non-regulated SMB |
| SaaS + tokenization | Vendor cloud; PII detected and replaced with reversible tokens before any model boundary | Tokenized telemetry only; real values never leave; de-tokenization happens inside your trust boundary | Mid-market with PII exposure |
| BYOC | Agent platform deployed into your cloud account; you choose model endpoints (including in-region or private) | Nothing by default; model calls go where you point them | Enterprises, most FSI |
| Self-host / air-gapped | Fully inside your perimeter, including self-hosted or dedicated models | Nothing | Banks, on-premise FSI, government |
Two practical notes. First, tokenization and BYOC compose: the strongest common pattern in regulated deployments is BYOC with a PII-aware tokenization layer in front of every model call, so even in-region inference never sees a real customer identifier, credential, or account number. Second, control must survive the audit: if the regulator asks "show me every piece of data this agent sent outside the bank in March, and prove nothing else left," the architecture — egress logging at the boundary, immutable audit trails you own — must be able to answer, not the vendor's assurances.
> *Figure 6 — The deployment model is the control dial: what crosses your boundary under each model, and who buys which.*
The regulatory floor is rising fastest in Asia. Vietnam is the sharpest current example of the direction of travel: the Personal Data Protection Law (Law 91/2025/QH15, effective January 1, 2026, with implementing Decree 356/2025) carries fines of up to 5% of prior-year revenue for unlawful cross-border data transfers and requires transfer impact assessments; the 2024 Data Law (effective July 2025) adds "core" and "important" data categories with their own cross-border restrictions; Decree 53/2022 under the Cybersecurity Law maintains localization requirements for specified services; and the country's first AI Law, passed in December 2025 and effective March 2026, introduces a risk-classification regime for AI systems. Singapore's MAS expectations on technology risk and outsourcing, and the EU's GDPR-plus-AI-Act stack, impose comparable discipline. The pattern is universal: regulators do not prohibit agentic operations — they prohibit not knowing where your data went. For FSI readers specifically, Vietnam's AI Law names finance as a regulated sector and provides an 18-month grace period for compliance — a window in which to build the governance and audit posture this chapter describes, not a reason to defer it.
**EIGHT DATA-CONTROL QUESTIONS FOR ANY AGENTIC VENDOR**
1. Exactly which data leaves our perimeter, to which endpoints, in which regions?
2. Can inference run in-region, in our cloud, or fully self-hosted?
3. Is our data used to train any model — yours or a third party's — and is that contractual?
4. Is PII tokenized before the model boundary, and where does de-tokenization occur?
5. What do you and your model providers log and retain, and for how long?
6. Who holds the encryption keys?
7. Do we keep the complete, immutable audit trail if we leave you?
8. Can we revoke every agent credential and halt all egress in one action?
A platform built for regulated industries answers all eight in writing.
## 6.4 Regulated industries: the FSI lens
Banking, insurance, and financial services have the most to gain from agentic operations — downtime costs are highest, compliance toil is heaviest — and the strictest constraints. Beyond the residency and control architecture above, three requirements recur in every FSI deployment:
1. **Model risk management.** Agentic systems fall under existing MRM frameworks: documented model behavior, evaluation suites, periodic revalidation, and challenger processes.
2. **Change management compatibility.** Agent actions must map onto existing ITIL/change-advisory processes — pre/post validation, approvals, and rollback evidence — rather than bypassing them. Gartner's 2026 outlook is blunt: as autonomy increases, governance becomes non-negotiable.
3. **Regulatory trajectory.** AI governance is moving from voluntary best practice to enforced requirement — the EU AI Act leads, and Asia-Pacific regulators are legislating fast, as Vietnam's 2025–2026 wave shows. Early investment in governance infrastructure is becoming a competitive advantage, not a tax.
## 6.5 The agent-layer threat model
Every control in this chapter governs what an agent is allowed to do. This section addresses a different question: what happens when the agent layer itself is attacked. An operations agent is, by construction, a privileged actor that reads telemetry and takes action — which makes it a target, and introduces failure modes that classical tooling does not have. A security team must threat-model the agent the way it would threat-model any new privileged service, and a platform that asks a bank to trust autonomous action must show that it has done so. Five attack surfaces recur; each has a concrete mitigation that should be a procurement requirement, not an aspiration.
1. **Telemetry poisoning.** An attacker who can write to a log, emit a metric, or forge an event can manufacture a false incident specifically to trigger an agent action — turning the agent's own responsiveness into an attack vector. **Mitigation:** authenticate and validate signal sources; gate actions on input provenance, not just input content; and treat a spike that would trigger a high-impact action as itself requiring corroboration from an independent signal.
2. **Prompt injection via logs.** Telemetry is untrusted input. Attacker-controlled text in a log line, an error message, or a resource name can attempt to hijack the agent's reasoning — the operations-layer form of prompt injection. **Mitigation:** treat all telemetry as untrusted data, never as instructions; strip or escape control content; and enforce that the agent can never execute an action that originates from the data plane rather than from policy.
3. **Agent privilege abuse.** A compromised or malfunctioning agent will use exactly the credentials it holds. The blast radius of a captured agent is the union of its permissions. **Mitigation:** least-privilege per agent, short-lived scoped credentials, and a per-action policy so that even a fully compromised agent cannot exceed the action classes its domain allows — the guardrail stack of §6.2, read as a containment boundary.
4. **Agent-to-agent trust under A2A.** As agents negotiate across organisational boundaries, a rogue or spoofed peer can issue malicious delegations. Cross-organisation autonomy is only as safe as the identity layer beneath it. **Mitigation:** cryptographically signed Agent Cards, verified peer identity, and explicitly scoped cross-organisation delegations — never an implicit trust of any agent that presents itself as one.
5. **An agent gaming its own KPIs.** An agent optimised against a metric will optimise the metric, not the goal — an agent rewarded for low MTTR can learn to suppress or auto-close alerts. This is the operations-layer form of reward hacking, and self-reported success is exactly where it hides. **Mitigation:** a guardian agent that audits outcomes independently, plus a human review of the metrics themselves — the system's own numbers are never the sole evidence that it is working.
> *Figure 11 — The agent layer is itself an attack surface: five recurring threats and the mitigation each requires before autonomy is granted.*
None of these is a reason not to deploy. They are the reason to deploy with the guardrail stack, the identity model, and the audit trail this book has argued for from the first chapter — a confidently-wrong agent and a maliciously-steered agent fail in the same place, and the same controls catch both. For a regulated bank, the FSI-specific extension of this model — mapped to a CISO's control framework — is the subject of a separate regulated-bank edition still in development.
**DESIGN PRINCIPLE**
Build the audit trail first and the autonomy second. A platform that can prove what it did and why — to an engineer, an auditor, or a regulator — will be allowed to do more. A platform that cannot will be confined to read-only advice forever.
# Chapter 7 · Humans on the Loop: The New Operating Model
Source: https://docs.cloudthinker.io/learn/aio/chapter-7-humans-on-the-loop
Agentic operations is a people transformation wearing a technology costume. From executor to supervisor, how roles change, and the trust ladder.
*Agentic operations is a people transformation wearing a technology costume.*
## 7.1 From executor to supervisor
The phrase "human in the loop" describes a person inside the execution path: every step waits for them. "Human on the loop" describes a supervisor: they set intent and policy, approve consequential actions, audit outcomes, and intervene by exception. The distinction is the difference between a pilot hand-flying an aircraft and a pilot managing an autopilot — the second pilot is not less important; they are responsible for more aircraft state with less manual workload, and they take the controls precisely when judgment matters most.
> *Figure 7 — In the loop, the human is the actuator; on the loop, the human is the supervisor.*
Gartner's I\&O predictions anticipate enterprises rapidly reducing human-in-the-loop involvement as agent autonomy increases through the late 2020s. The teams that thrive will be those that redesign roles deliberately rather than letting erosion happen to them.
## 7.2 How roles change
| Role today | What shrinks | What grows |
| :------------------------- | :------------------------------------------------------------- | :------------------------------------------------------------------------------------ |
| SRE / on-call engineer | Manual triage, log archaeology, 3 a.m. mechanical remediations | Policy design, agent supervision, novel-failure engineering, reliability architecture |
| DevOps / platform engineer | Ticket-driven provisioning, repetitive pipeline fixes | Agent enablement: tool integrations, context curation, golden paths, evaluation |
| Ops manager / I\&O leader | Headcount-based capacity planning, war-room coordination | Autonomy governance, agent portfolio management, outcome-based vendor management |
| Security engineer | Manual misconfiguration hunts, compliance screenshot gathering | Guardrail engineering, agent permission design, continuous-compliance automation |
Two genuinely new functions emerge. The agent operations engineer owns the health of the agent fleet itself — prompts, tools, memory, evaluations, cost. The autonomy policy owner — often a senior SRE or engineering manager — decides which action classes graduate up the autonomy ladder and adjudicates when agents and humans disagree. Both are career paths, not side duties.
## 7.3 Trust is built in increments
Engineer trust follows a predictable arc, and skipping stages backfires:
1. **Watch it investigate.** Agents run in advise-only mode; engineers compare agent root-cause analyses against their own. Accuracy earns the next step.
2. **Approve its actions.** Agents propose complete remediations; humans one-click approve. Every approval is a labeled data point on agent judgment.
3. **Pre-approve the boring.** Action classes with consistent approval records and clean rollbacks graduate to act-with-notification.
4. **Delegate domains.** Bounded domains — idle-resource cleanup, cache management, certificate rotation — are handed over end-to-end, with audits replacing approvals.
Industry experience is consistent on the failure mode: skip steps, and you ship an autonomous agent that is confidently wrong at 3 a.m. — and one such incident can set an agentic program back a year. Sequence the trust ladder, and autonomy compounds.
The ladder is also exactly how the vendors tell you to deploy their own products: AWS, Azure, and Google all ship their operations agents investigation-first, with action gated behind staged customer governance — the adoption path §9.2 sets out in detail. When the sellers of autonomy insist you start without it, take the hint.
## 7.4 The talent dividend
Framed correctly, agentic operations is the answer to the skills shortage, not a threat to the workforce. With roughly two-thirds of organizations unable to hire the AI-era operations engineers they need, the realistic choice is not "agents versus engineers" but "engineers with agents versus engineers without." Teams that adopt the supervisor model report the scarce senior engineers finally doing the work they were hired for — architecture, prevention, performance — instead of being consumed by interrupt-driven toil. Retention follows. Nobody's career goal is restarting pods at 3 a.m.
# Chapter 8 · Measuring What Matters
Source: https://docs.cloudthinker.io/learn/aio/chapter-8-measuring
Agentic operations succeeds or fails on evidence. The eight-KPI dashboard, ROI math, unit economics, and the evaluation harness.
*Agentic operations succeeds or fails on evidence. Instrument the program like the production system it is.*
## 8.1 The headline outcomes
Across published deployments and industry research, four outcome ranges recur for teams that implement agentic incident response with discipline:
| Metric | Documented range | Driver |
| :--------------------- | :-------------------------------------- | :---------------------------------------------------------------------------------------------------------------------- |
| MTTR reduction | 40–70% (vendor pilots report up to 75%) | Automated investigation and pre-approved remediation collapse the diagnosis phase, which consumes most of incident time |
| Alert volume reduction | 80–90% | Correlation, deduplication, and symptom suppression before any human sees a page |
| Toil reduction | 30–50% of L1/L2 operational work | Triage, routine remediation, evidence gathering, and reporting absorbed by agents |
| Cloud cost savings | 10–30% of optimizable spend | Continuous rightsizing, idle cleanup, and commitment management instead of quarterly reviews |
Treat these as benchmarks to verify, not promises to assume. MTTR gains in particular vary with implementation maturity and data quality — the documented pattern is that noise reduction lands first and most consistently, root-cause acceleration second, and autonomous remediation last.
> *Figure 8 — Documented outcome ranges across disciplined deployments, 2025–2026. Verify against your own baseline.*
The named data points behind the ranges: AWS reports preview customers of DevOps Agent seeing up to 75% lower MTTR, 80% faster investigations, and 94% root-cause accuracy, with WGU publicly describing a two-hour investigation cut to 28 minutes; Microsoft reports 35,000+ incidents mitigated and 20,000+ engineering hours saved running 1,300+ SRE agents on its own services. These are vendor and first-party numbers — the strongest currently published, and the right ones to pressure-test in your own pilot rather than accept on faith.
## 8.2 The operating dashboard
A production agentic program runs on roughly eight KPIs, reviewed monthly with the on-call rotation:
1. **MTTD and MTTR** — detection and recovery time, trended by severity, with agent-handled and human-handled incidents separated.
2. **Autonomous resolution rate** — share of incidents resolved with no human action. The single best maturity indicator.
3. **Recommendation acceptance rate** — share of agent proposals approved unmodified. Below \~70%, agent judgment or policy needs tuning; above \~95%, the approval gate is theater and the action class should graduate.
4. **Rollback / intervention rate** — agent actions that had to be reversed or overridden. The safety counterweight to autonomy growth.
5. **Repeat-incident rate** — whether the system is learning. Falling repeats mean memory and problem management are working.
6. **Runbook / coverage ratio** — share of incident classes the agent team can handle at L2 or above.
7. **On-call load** — pages per engineer per week, and after-hours pages specifically. The human-experience metric leadership feels.
8. **Agent unit cost** — model and platform spend per incident resolved and per service covered. Agent economics are a first-class architectural concern; track them from day one.
## 8.3 Building the business case
The ROI model has three lines. First, downtime avoided: multiply your incident frequency by your cost per minute of downtime by the MTTR reduction you validate in pilot. For the cost input, use your own finance team's number if you have one; if you don't, use the Splunk/Oxford Economics 2026 downtime benchmark cited in Chapter 1 (materially higher for payment and trading systems) as the most defensible external anchor. Second, toil converted: hours of L1/L2 work absorbed by agents, valued at loaded engineering cost — typically the largest line in talent-constrained markets. Third, cloud waste recovered: continuous optimization against the 20–30% of spend most organizations privately acknowledge is wasted. Against these, count platform subscription, model usage, and the engineering time to integrate and govern — honestly, including the autonomy-policy owner's time. Disciplined deployments typically reach payback within two to three quarters, with the toil line alone often covering the platform cost. If your model needs the downtime line to work, your pilot domain is wrong; pick one where toil savings carry the case and downtime is upside.
## 8.4 The unit economics of an agent
Chapter 4 argued that two-tier sensing is what makes 24/7 agentic coverage economically viable. This section makes that concrete, because "it pays for itself" is exactly the kind of hand-wave this book tells buyers to reject. The cost of an agentic program is dominated by model inference, and inference cost is dominated by where the expensive reasoning runs. The discipline is to keep cheap perception always-on and reserve frontier reasoning for the few signals that warrant it.
Read a single incident as a cost object. An always-on "pulse" performs lightweight perception across every signal at low cost; a heavyweight resolver performs multi-step reasoning only when the pulse finds something worth investigating. In a representative split, sensing accounts for a small share of per-incident spend, triage a little more, and deep resolution the majority — but deep resolution only fires on the minority of signals that survive triage. A worked illustration: an environment emitting on the order of fifty signals a day, where most are dispatched by pulse-only perception and only a handful escalate to a full resolver run, keeps the expensive tier's duty cycle low while still covering every signal. The exact ratios are yours to measure; the architecture is what makes the ratio favourable.
> *Figure 12 — Where the tokens go in one incident: cheap perception runs on every signal; expensive reasoning runs only on the few that survive triage. Track cost per resolved incident from day one.*
This is why "agent cost per resolved incident" and "agent cost per covered service" are first-class KPIs in the Chapter 8 dashboard, not afterthoughts: an architecture that runs frontier reasoning on every noisy signal erases its own ROI before the first renewal, and you will not see it happen unless you are tracking the per-incident number from day one.
The commercial structure can be aligned to this reality rather than fighting it. An outcome-based option — a fee set as a share of verified savings, with nothing owed if no savings are found — ties the vendor's revenue to the customer's realised benefit and removes the incentive to run up inference cost for its own sake. As a concrete instance: a cost-optimisation engagement priced at 50% of verified annual savings, billing zero if it finds nothing, makes the interest alignment total. Whatever the structure, the figure that belongs in your business case is the one you measure in your own environment against the 90-day baseline of Chapter 9 — not a headline from anyone's slide, including this book's.
## 8.5 Evaluating an operations agent before you trust it
The book has called agentic operations "self-verifying" since Chapter 3 without showing what a verification step concretely checks, or how you would catch a confidently-wrong remediation before it reaches production. This section closes that gap, because the engineers the Foreword promises to serve are precisely the ones who will have to build and run this harness.
An agent evaluation harness has four parts. First, a scenario library: recorded real incidents plus deliberately injected faults, each with a known correct outcome — the right root cause, the safe action, the clean rollback. Second, a shadow run: the agent acts against a sandbox mirror of production, never production itself, so its proposed actions can be observed without consequence. Third, scoring against ground truth: did it reach the correct root cause, did it choose a safe action, would its rollback have worked? Fourth, a regression gate: a behaviour change versus the previous agent version must pass the library before it ships, the same discipline a team applies to any other production code.
> *Figure 13 — An agent evaluation harness: a scenario library, a shadow run against a sandbox mirror, scoring against ground truth, and a regression gate before any version ships.*
Non-determinism is the part that surprises teams coming from deterministic automation: the same scenario can yield different agent behaviour on different runs. The harness handles this by running each scenario many times and scoring the distribution of outcomes, not a single pass — a fix that is correct 70% of the time is a different risk profile from one that is correct 99% of the time, and only a distribution reveals which you have. A concrete verification check makes this tangible: for the database-connection-exhaustion scenario, the harness asserts that the agent identifies connection-pool exhaustion as the root cause, that its action restores the correct connection limit rather than merely restarting the service, and that post-action the connection count returns to baseline and stays there — the same DARV "Validate" step the agent runs in production, run here against a known answer. An agent that cannot pass its own verification check in the harness has no business running it unattended in production.
**MEASUREMENT PRINCIPLE**
Baseline before you deploy. The single most common business-case failure is having no credible "before": capture 90 days of MTTR, alert volume, page counts, and toil hours before the first agent touches production, or you will be arguing from anecdotes forever.
# Chapter 9 · The Implementation Roadmap
Source: https://docs.cloudthinker.io/learn/aio/chapter-9-roadmap
A staged, evidence-driven adoption path: ninety days to first value, twelve months to a new operating model. Five readiness foundations, the 90-day pilot, scaling, and the five ways canceled projects die.
*A staged, evidence-driven adoption path: ninety days to first value, twelve months to a new operating model.*
## 9.1 Readiness: what agents need from you
Agents amplify the environment they inherit. Before the first deployment, honestly assess five foundations:
1. **Observability.** Centralized logs, metrics, and traces with reasonable coverage. Agents cannot reason over signals that don't exist.
2. **Access architecture.** The ability to mint scoped, short-lived credentials per agent. If everything runs on one admin key today, fix that first.
3. **Source of truth.** Infrastructure as code for the surfaces agents will touch, even partially. IaC gives agents a safe change mechanism and you a diffable audit trail.
4. **Documented intent.** SLOs, runbooks, architecture notes — imperfect is fine; absent is not. This becomes the agent's context layer.
5. **An accountable owner.** A named senior engineer with the mandate to set autonomy policy and the credibility to bring the on-call rotation along.
## 9.2 The 90-day pilot
| Phase | Weeks | Focus | Exit criteria |
| :--------------- | :---- | :--------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------- |
| Baseline & scope | 1–2 | Capture MTTR, alert volume, page counts, toil hours. Pick one bounded domain (one product's incident response, or cloud cost for one account). | Signed baseline; scoped domain; success metrics agreed |
| Observe (L0–L1) | 3–6 | Connect telemetry and tools read-only. Agents investigate every incident in parallel with humans; engineers grade the analyses. | ≥70% of agent root-cause analyses rated correct or useful by on-call |
| Approve (L2) | 7–10 | Agents propose complete remediations with evidence; humans one-click approve. Track acceptance and rollback rates. | ≥80% acceptance; zero harmful actions; MTTR visibly improving |
| Graduate (L3) | 11–13 | Pre-approve the 5–10 safest, most-repeated action classes. Agents act and notify. Review every action weekly. | First autonomous resolutions in production; documented MTTR delta vs. baseline |
Resist the urge to start with the hardest problem. The pilot's job is to produce evidence and trust, not heroics. A boring domain with frequent, repetitive incidents — Kubernetes restarts, disk pressure, certificate expiry, cost anomalies — generates statistical confidence fastest.
> *Figure 9 — The 90-day pilot: four phases, each with signed exit criteria before autonomy graduates.*
This pilot shape is now vendor-validated practice, not just prudence: AWS's published adoption guidance for DevOps Agent — one region, one service, recommendation-only for weeks, then measure MTTR before expanding — is this roadmap's Observe and Approve phases in different words, and Azure's staged governance controls assume the same progression. If the hyperscalers gate their own agents this way on their own clouds, a bank should not be talked into skipping it.
## 9.3 Scaling: months 4–12
1. **Expand domains, not just autonomy.** Add specialists — database, security, cost — one at a time, each through the same observe → approve → graduate ladder.
2. **Industrialize governance.** Move autonomy policy from a document to enforced configuration; stand up the guardian/oversight layer; integrate agent actions into change management with automated evidence.
3. **Build the memory moat.** Curate the context layer deliberately: topology, conventions, past incidents, tribal knowledge. This is where your deployment becomes unreasonably effective and un-copyable.
4. **Restructure on-call.** As autonomous resolution rates climb, consolidate rotations, redirect reclaimed senior time to prevention engineering, and formalize the agent-operations and autonomy-policy roles.
5. **Report relentlessly.** Publish the dashboard monthly — MTTR trend, autonomous resolution rate, pages avoided, dollars saved — to engineering and to the business. Funded programs are measured programs.
## 9.4 How the canceled 40% die
Gartner predicts more than 40% of agentic AI projects will be canceled by the end of 2027, naming three killers: escalating costs, unclear business value, and inadequate risk controls. In operations specifically, those abstractions take five concrete forms. Each has a known antidote:
1. **The pilot that never graduates.** (Unclear value.) Advise-only forever feels safe and proves nothing — then the renewal arrives with no MTTR delta to show. *Antidote:* graduation criteria signed on day one, honored on schedule.
2. **Autonomy before evidence.** (Inadequate risk controls.) One confidently-wrong autonomous action at 3 a.m. costs more trust than a hundred good ones earn. *Antidote:* never skip ladder steps, and let acceptance and rollback rates — not enthusiasm — set the pace.
3. **Tool sprawl without orchestration.** (Escalating costs and unclear value.) Five disconnected point agents recreate the swivel-chair problem with extra licenses (the same coordination tax, one layer up, that §10.3 traces across single-cloud agents). *Antidote:* one orchestrator, one audit trail, one dashboard.
4. **Unbounded model spend.** (Escalating costs.) Frontier reasoning on every noisy signal erases the ROI before the first renewal. *Antidote:* two-tier sensing and per-incident cost tracking from day one.
5. **Treating it as a tool purchase.** (All three.) The experiment-to-production gap from Chapter 1 is an operating-model gap, not a technology gap. *Antidote:* budget for the role changes, the policy work, and the trust ladder — not just the license.
## 9.5 When not to deploy: the honest disqualifiers
Before deploying, you put a readiness baseline in place; this section is its harder companion: the cases where the honest answer is to wait. A book that tells buyers to distrust anyone who cannot say no to them should be able to say it about its own category. Each disqualifier below is a reason to fix something first, not a permanent verdict — but deploying through any of them buys an expensive disappointment.
1. **You have no signal to reason over.** If observability is sparse or fragmented — no centralised logs, metrics, or traces across the target domain — the agent has nothing to reason from, and will confidently reason from noise. Fix observability first; an agent amplifies the environment it inherits, and amplifying a blind spot produces a confident blind spot.
2. **Everything runs on one shared admin credential.** If you cannot issue scoped, short-lived credentials per agent, you cannot bound an agent's blast radius or contain a compromised one. Until least-privilege access is real, autonomous action is an unacceptable risk regardless of how good the agent is.
3. **No one owns the autonomy policy.** If there is no named senior engineer with the authority to set autonomy policy and the standing to carry the on-call team, the program will stall at advisory or lurch into ungoverned action. The owner is a prerequisite, not a role to fill later.
4. **Change management cannot accommodate machine-initiated change.** If your change process has no path for a machine-initiated, human-approved change with an audit trail, agent actions will either bypass governance — unacceptable in a regulated environment — or be blocked entirely. Resolve the process question before, not during, deployment.
5. **The first target is your most critical, least reversible system.** Starting on the core path with irreversible actions inverts the trust ladder. If the only available pilot domain is the one where a wrong action is catastrophic and unrecoverable, wait until a bounded, reversible domain is available — or carve one out deliberately. The pilot's job is evidence, not heroism.
There is also a timing disqualifier that has nothing to do with readiness: if the organisation cannot fund the operating-model change — the role redesign, the policy work, the trust ladder — and is buying only a license, it will land in the canceled 40% of §9.4 no matter how ready its infrastructure is. The technology is not the gating factor. The willingness to run the program as a transformation rather than a tool purchase is.
# Introduction
Source: https://docs.cloudthinker.io/learn/aio/introduction
The CloudThinker Field Guide to AgenticOps — why this book exists, the whole argument in one page, and the reading map for the ten chapters that follow.
This is the full text of **Agentic Infrastructure Operations - Leadership Edition**, the CloudThinker Field Guide (June 2026 edition). It is preserved here chapter by chapter — written for SREs, DevOps and platform engineers, infrastructure leaders, CTOs, and CIOs, with explicit attention to regulated industries.
> *Engineers on the loop, not in the loop.*
>
> — The operating principle of agentic operations
***
*Why we wrote this book, and who it is for.*
Every decade, infrastructure operations reinvents itself. Bare metal gave way to virtualization. Virtualization gave way to cloud. Cloud gave way to containers, microservices, and serverless. Each wave promised simplicity and delivered capability — along with an order of magnitude more moving parts to operate.
We are now past the point where humans, however skilled, can hold a modern production environment in their heads. A mid-sized digital business today runs hundreds of services, thousands of containers, and tens of thousands of configuration parameters across multiple clouds. The telemetry those systems emit — logs, metrics, traces, events, alerts — grows faster than any operations team can hire.
Agentic AI changes the equation. For the first time, we can deploy software that does not merely alert a human or execute a pre-scripted runbook, but perceives, reasons, plans, acts, and verifies — the full loop of operational work. This book is a field guide to that shift: what agentic infrastructure operations actually is, how it differs from the automation and AIOps generations that preceded it, how to architect it, how to govern it, and how to adopt it without betting your uptime on hype.
It is written for the people who carry the pager and the people who set the budget: SREs, DevOps and platform engineers, infrastructure leaders, CTOs, and CIOs — especially those in regulated industries where autonomy must be earned, evidenced, and audited. A note on evidence: every figure in this book is attributed, vendor claims are labeled as vendor claims, and each class of number is presented so a reader can weigh it — because a book about earning trust in autonomous systems should hold itself to the same standard.
Our conviction is simple: operational complexity now compounds at machine speed, so operations must scale at machine capacity. Humans should move from being in the loop — executing every step — to being on the loop: setting intent, approving consequential actions, and supervising outcomes. The teams that make this transition deliberately will run faster, safer, and cheaper than those that do not.
One disclosure belongs up front, not in a closing chapter. This book is published by CloudThinker, which builds a platform in the category it describes. We have worked to keep that interest from bending the evidence: every benchmark is sourced, vendor figures — including the hyperscalers' and our own — are labelled as vendor figures, and the framework chapters are written to stand on their own whatever platform you choose. Where the book describes how CloudThinker specifically implements an idea, it is marked as such — chiefly in the clearly-labelled section of Chapter 10 — so that "what the field is converging on" and "how one vendor builds it" never blur together. Judge the category by the evidence; then judge us by the five-question vendor test and the eight data-control questions in these pages, which we wrote knowing we would have to pass them.
***
## Executive Summary
Modern infrastructure has crossed a threshold: microservices, multi-cloud, and AI workloads now generate operational complexity faster than any human team can absorb. Operations teams field 500–1,200 alerts a day; Splunk and Oxford Economics put unplanned downtime at \$600 billion a year across the Global 2000 — roughly \$15,000 per minute for a large enterprise — and around two-thirds of organizations cannot hire the operations skills they need. Hiring, scripting, and dashboards — the three traditional responses — all keep humans in the execution path, and human attention is the bottleneck.
Agentic infrastructure operations is the structural answer: autonomous AI agents that close the full operational loop — Detect → Analyze → Resolve → Validate — under explicit policy, with humans supervising on the loop instead of executing in it. It is the fourth generation of operations, absorbing infrastructure-as-code and AIOps rather than replacing them, and it became practical between 2023 and 2026 through frontier reasoning models, reliable tool use, and the MCP interoperability standard.
The evidence is real, and so is the failure rate — this book takes both seriously. Gartner predicts AI agents will reshape I\&O teams, roles, and operating models over the next five years, expects task-specific agents in 40% of enterprise applications by the end of 2026, and recorded a 1,445% surge in multi-agent system inquiries; AWS and Azure shipped GA reliability agents in early 2026; disciplined adopters report 40–70% MTTR reductions and 80–90% alert-noise elimination. Set against that promise is a hard failure rate — a large share of agentic projects are forecast to be canceled, and most experiments never reach production (Chapters 1, 6, and 9). The difference between the two populations is not the technology. It is execution discipline, and teaching it is this book's entire purpose.
The playbook runs in ten chapters: the complexity crisis and why old answers failed; precise definitions, the L0–L4 autonomy spectrum, and a five-question test for "agent washing"; the reference architecture — one orchestrator, specialist agents, a closed DARV loop, two-tier sensing, and PII tokenization for regulated industries; the guardrail stack and FSI-grade governance; the human operating model and trust ladder; the eight-KPI measurement framework and ROI math; a 90-day pilot and 12-month scaling roadmap, with the five ways canceled projects die and their antidotes; and the road ahead.
**IF YOU READ NOTHING ELSE**
1. **Autonomy is a dial, not a switch:** graduate action classes through observe → approve → act-with-notification → delegate, on evidence.
2. **Architecture matters:** one orchestrator, least-privilege specialists, verification built into the loop, and audit trails before autonomy.
3. **Baseline before you deploy,** measure eight KPIs monthly, and let your own data set the pace.
***
## How to Read This Book
### Part 01 · From Crisis to Agentic Operations
Modern infrastructure has outgrown human cognitive capacity
Four generations of operations, and what changed
The working definition, the L0–L4 autonomy spectrum, and the five-question vendor test
### Part 02 · Architecture & the Agent Team
Multi-agent systems for operations: orchestrator, specialists, DARV loop, two-tier sensing
Roster, work flow, agent-to-human interfaces, four production case studies
### Part 03 · Trust, Governance & People
The five-level guardrail stack, data residency, threat model, FSI lens
The new operating model: from executor to supervisor, the trust ladder
### Part 04 · Proof & Playbook
Eight KPIs, ROI math, unit economics, and the evaluation harness
The 90-day pilot, scaling to 12 months, and the canceled-40% failure modes
### Part 05 · The Road Ahead
Five near-term trajectories, the strategic window, the hyperscaler-vs-unified question
The platform behind the field guide
***
*Copyright © 2026 CloudThinker JSC. All rights reserved.*
*Written by the CloudThinker Product Team. Designed by CloudThinker Design. Published by CloudThinker · [www.cloudthinker.io](http://www.cloudthinker.io). First Edition, June 2026.*
# FAQ
Source: https://docs.cloudthinker.io/learn/faq
Find answers to common questions about CloudThinker, AI agents, pricing, security, and cloud operations
Get answers to common questions about CloudThinker, our AI agents, and how Agentic AI transforms cloud operations.
***
## Getting started
CloudThinker is an autonomous AI cloud operations platform that uses specialized AI agents to automate cloud operations. Our five expert agents - [Alex](/guide/agents/alex) (Cloud Engineer), [Oliver](/guide/agents/oliver) (Security Engineer), [Tony](/guide/agents/tony) (Database Engineer), [Kai](/guide/agents/kai) (Kubernetes Engineer), and [Anna](/guide/agents/anna) (General Manager) - work together to optimize, secure, and manage your cloud infrastructure autonomously.
You interact with agents using natural language commands like `@alex #recommend analyze EC2 instances with low utilization` or `@oliver #report perform security audit`, and they execute hundreds of pre-defined operations or custom tasks.
**Traditional Tools:**
* Manual configuration and monitoring
* Reactive alerts and dashboards
* Human-dependent analysis and decision-making
* Siloed tools for different functions
**CloudThinker:**
* Autonomous AI agents that understand context
* Proactive optimization and issue prevention
* Intelligent analysis with actionable recommendations
* Unified multi-agent collaboration across all cloud operations
CloudThinker supports all major cloud providers:
* **AWS**: Complete service coverage including EC2, RDS, EKS, Lambda, and 100+ services
* **Microsoft Azure**: Full Azure service integration including VMs, AKS, databases, and security services
* **Google Cloud Platform**: Native GCP support for compute, storage, databases, and Kubernetes
* **Multi-Cloud**: Unified management and optimization across multiple cloud providers simultaneously
**Week 1:** Complete infrastructure discovery and baseline establishment
**Week 2:** First optimization recommendations and quick wins (typically 15–25% immediate cost savings)
**Month 1:** Full operational automation and 30–50% cost optimization
**Month 2-3:** Advanced automation, predictive analytics, and strategic optimizations
Most customers see their first measurable results within 48 hours of deployment.
No. CloudThinker is designed to work with your existing cloud architecture without requiring any changes. Our agents analyze and optimize your current setup while respecting your architectural decisions and business requirements.
CloudThinker integrates through:
* Read-only analysis for most operations
* Secure API connections to your cloud providers
* Recommendations that you can review before implementation
* Gradual automation that you control
***
## AI agents & operations
Our AI agents are powered by advanced language models trained on cloud operations expertise. Each agent has:
**Specialized Knowledge:** Deep expertise in their domain (cloud engineering, security, databases, etc.)
**Contextual Understanding:** Ability to understand your infrastructure, requirements, and business context
**Tool Integration:** Access to hundreds of pre-built operations and cloud APIs
**Learning Capabilities:** Continuous improvement based on your environment and feedback
**Safety Controls:** Built-in safeguards and [approval workflows](/guide/approval) for sensitive operations
They operate autonomously within defined parameters but can escalate complex decisions to human operators when needed.
CloudThinker implements multiple safety layers:
**Autonomy modes:** Choose [Manual or Auto](/guide/auto-mode) — Manual requires approval for each action, Auto lets agents act within defined guardrails
**[Approval Workflows](/guide/approval):** Configure required approvals for sensitive operations
**Change Windows:** Restrict automated changes to maintenance windows
**Rollback Capabilities:** Automatic rollback for operations that don't meet success criteria
**Audit Trail:** Complete logging of all agent actions and decisions
**Risk Assessment:** Agents evaluate risk before suggesting or implementing changes
Most customers start with Manual mode and switch to Auto as confidence grows.
CloudThinker agents understand natural language, making communication intuitive:
**Agent Syntax:** Use @agent-name to direct requests to specific agents
**Tool Syntax:** Use #tool-name to specify desired outputs
**Natural Language:** Agents understand context, follow-up questions, and conversational requests
Examples:
* "Show me last month's cost trends"
* "What security issues need immediate attention?"
* "Can you optimize our database performance?"
Yes! Multi-agent collaboration is one of CloudThinker's key strengths:
**Coordinated Analysis:** Multiple agents can investigate issues together
**Knowledge Sharing:** Agents share context and findings automatically
**Specialized Expertise:** Each agent contributes their domain expertise to complex problems
**Unified Reporting:** Collaborative findings presented in coherent recommendations
**Example Scenarios:**
* **Deep Response Engine:** All agents coordinate to analyze different aspects of an outage
* **Security Review:** [Oliver](/guide/agents/oliver) leads with [Alex](/guide/agents/alex) providing infrastructure context
* **Performance Optimization:** [Tony](/guide/agents/tony) (database) + [Kai](/guide/agents/kai) (Kubernetes) + Alex (infrastructure) collaboration
***
## Security & compliance
CloudThinker implements enterprise-grade security:
**Data Protection:**
* End-to-end encryption for all data in transit and at rest
* Zero-trust architecture with minimal required permissions
* No persistent storage of sensitive cloud data
* GDPR, SOC 2, and HIPAA compliant infrastructure
**Access Control:**
* Role-based access control (RBAC) for all users
* Multi-factor authentication (MFA) required
* [Single sign-on (SSO)](/guide/security/sso) integration with enterprise identity providers
* Granular permissions for agent actions
**Cloud Permissions:**
* Principle of least privilege for all cloud API access
* Read-only access for most operations
* Configurable write permissions with [approval workflows](/guide/approval)
* Regular audit and rotation of cloud access credentials
CloudThinker follows the principle of least privilege:
**Minimum Required:**
* **Read access** to cloud resources for analysis and recommendations
* **Cost and billing data** access for optimization analysis
* **CloudTrail/audit logs** access for security and compliance analysis
**Optional (for automation):**
* **Limited write access** for specific approved operations (e.g., tagging, right-sizing)
* **IAM permissions** for security policy enforcement (with approval workflows)
* **Backup and snapshot** permissions for database operations
**Enterprise Options:**
* **Custom permission policies** tailored to your security requirements
* **Temporary access elevation** for specific operations
* **Cross-account roles** for multi-account management
You maintain full control over what permissions to grant and can revoke them at any time.
Yes, CloudThinker maintains compliance with major industry standards:
**Compliance Certifications:**
* **SOC 2 Type II** - Security, availability, processing integrity
* **GDPR** - European data protection regulation
* **HIPAA** - Healthcare data protection (BAA available)
* **PCI DSS** - Payment card industry standards
* **ISO 27001** - Information security management
**Industry-Specific Features:**
* **Healthcare:** HIPAA-compliant operations and audit trails
* **Financial Services:** SOX compliance and financial data protection
* **Government:** FedRAMP moderate authorization in progress
* **Enterprise:** Custom compliance reporting and attestation
[Oliver](/guide/agents/oliver) agent can help maintain compliance in your environment by monitoring configurations and generating compliance reports.
***
## Pricing & plans
CloudThinker uses **per-seat pricing** — each seat comes with a monthly credit allocation and full access to AI agents. Plans range from **Free (free forever)** to **Enterprise (custom pricing)**, with annual billing saving \~20%.
All plans include 100 free daily credits (Light model and chat only) that refresh every day. Most customers save 3-5x their subscription cost through cloud optimizations alone.
See the full plan comparison, credit system, and PAYG details on our [Pricing & Plans](/guide/billing/pricing) page.
A **seat** is a paid slot on your subscription. Each purchased seat immediately grants its full credit allocation — only active members consume seats, and pending invitations don't count.
You can pre-purchase extra seats to boost your credit pool, and the system auto-adds seats when your team grows mid-cycle with prorated billing.
For full details on seat mechanics and minimum requirements, see our [Pricing & Plans](/guide/billing/pricing) page.
Yes! We offer multiple ways to experience CloudThinker before committing:
**14-Day Free Trial (Team Plan):**
* Full access to all Team features, AI agents, and operations
* A credit card is required to start so your service continues seamlessly — but **you won't be charged until day 15**
* Cancel anytime during the trial with one click, no cost and no questions asked
* Real analysis of your cloud environment from day one
**Free Plan:**
* No credit card needed — free forever
* 100 free daily credits (Light model and chat only) that refresh every day
* Great for exploring the platform at your own pace
**Interactive Demo:**
* Live demonstration with sample environment
* See agents in action with real scenarios
* Q\&A with CloudThinker experts
**Proof of Concept (Enterprise):**
* 30-60 day pilot program
* Custom deployment in your environment
* Dedicated success team
* Measurable ROI validation
**Money-Back Guarantee:** If you don't see measurable value within 30 days, we'll refund your subscription.
For more details, see our [Pricing & Plans](/guide/billing/pricing) page.
Based on data from CloudThinker customers, typical ROI spans multiple dimensions:
**Cost Savings (Primary ROI):**
* **30–50% cloud cost reduction** on average
* **\$500K–\$2M annual savings** for mid-to-large organizations
* **3–5x subscription cost recovery** through optimizations alone
**Operational Efficiency:**
* **75% reduction** in time spent on manual cloud operations tasks
* **60–80% faster incident resolution** through automated [RCA](/guide/incident/root-cause-analysis)
**Risk Reduction:**
* Significant improvement in security posture scores through continuous monitoring
* Automated compliance evidence generation reduces audit preparation time
**Time to Value:**
* **Week 1:** Initial cost savings identified
* **Month 1:** 15–25% cost reduction realized
* **Month 3:** Full ROI typically achieved
* **Month 6+:** Compound benefits and strategic value
**Typical Payback Period:** 2–4 months for most customers
Results vary based on cloud spend, team size, and the number of optimization opportunities in your environment. Contact us for a personalized savings estimate.
***
## Technical questions
CloudThinker is designed to work with minimal technical requirements:
**Cloud Requirements:**
* **AWS, Azure, or GCP account** with appropriate permissions
* **API access** enabled for your cloud services
* **Network connectivity** to CloudThinker services (outbound HTTPS)
**Optional Integrations:**
* **Slack workspace** for ChatOps integration
* **[SSO](/guide/security/sso) provider** (Okta, Azure AD, Google Workspace)
* **ITSM tools** (ServiceNow, Jira) for ticketing integration
* **Monitoring tools** (DataDog, New Relic) for enhanced insights
**No Infrastructure Required:**
* No agents or software to install in your environment
* No VPN or complex networking setup
* No database or storage requirements
* Works with existing cloud configurations
Yes! CloudThinker is designed for seamless integration:
**Communication Platforms:**
* **Slack** - Full ChatOps integration with agent interactions
* **Microsoft Teams** - Native bot and notification support
* **Email** - Report delivery and alert notifications
**ITSM & Ticketing:**
* **ServiceNow** - Automatic ticket creation and updates
* **Jira** - Issue tracking and workflow integration
* **PagerDuty** - Alert escalation and incident management
**Monitoring & Observability:**
* **DataDog** - Metrics and dashboard integration
* **New Relic** - Performance monitoring correlation
* **[Grafana connection](/guide/connections/grafana)** - Custom dashboard and alerting
* **Splunk** - Log analysis and security correlation
**DevOps & CI/CD:**
* **Jenkins** - Build and deployment automation
* **GitHub Actions** - Workflow integration
* **GitLab CI** - Pipeline integration
* **Terraform** - Infrastructure as Code integration
***
## Support & training
CloudThinker offers comprehensive support tailored to your plan:
**Free:**
* **Community support** via Discord
* **[Knowledge base](/guide/knowledge)** with detailed documentation and tutorials
* **Video tutorials** and getting started guides
* **AI features:** [Code Review](/guide/code-review/overview), [Deep Response Engine](/guide/incident/overview) (Pulse + RCA), Prompt Hub Query
**Team:**
* **Email support**
* **AI features:** All Free features + CloudKeeper Action + Slack Integration
* All Free resources included
**Scale:**
* **99.9% SLA · email support**
* **AI features:** All Team features + Cloud Assessment, Jira Integration, SSO & Audit Log, BYOK
* All Team features included
**Scale +:**
* **99.9% SLA · priority support**
* **AI features:** Same as Scale
* All Scale features included
**Enterprise:**
* **24×7 support** with 1-hour response SLA and a dedicated TAM
* **AI features:** All features included, plus SCIM provisioning
* **Custom training** and onboarding programs
* **Direct access** to engineering team for complex issues
* **Quarterly business reviews** with executive team
CloudThinker provides multiple channels for getting help quickly:
**Immediate Help:**
* **In-app help** and contextual guidance
* **Knowledge base** with searchable articles and FAQs
* **Community Discord** for peer support
**Expert Support:**
* **Email support** with guaranteed response times (Team+)
* **Priority support** with SLA for tracking issues (Scale+)
* **Dedicated team** for complex issues (Enterprise)
**Contact Methods:**
* Email: [support@cloudthinker.io](mailto:support@cloudthinker.io)
* Discord: Join our user community
* Enterprise: Dedicated support channels
***
## Related
Get help from our expert support team
See CloudThinker in action with your environment
Connect with other CloudThinker users
Suggest new features or improvements