Elasticsearch

Connect Elasticsearch to CloudThinker

Elasticsearch

Connect your Elasticsearch clusters to enable CloudThinker agents to search logs, analyze observability data, and investigate incidents.

---

Supported Platforms

Platform	Support
Elasticsearch	7.x, 8.x
OpenSearch	1.x, 2.x
AWS OpenSearch Service	All versions
Elastic Cloud	All deployments

---

Setup

1. Open Kibana

   Navigate to your Kibana instance and sign in with admin access.

2. Navigate to API Keys

   Go to Stack Management → Security → API Keys.

3. Create API Key

   Click Create API key and enter:

   • Name: cloudthinker-readonly
4. Set Permissions

   For read-only access, add role descriptors:

   {
     "cluster": ["monitor"],
     "indices": [
       {
         "names": ["*"],
         "privileges": ["read", "view_index_metadata"]
       }
     ]
   }
   
5. Generate and Copy

   Click Create API key and immediately copy the encoded key (it won’t be shown again).

6. Note Cluster Endpoint

   Get your Elasticsearch endpoint:

   • Self-hosted: https://elasticsearch.your-domain.com:9200
   • Elastic Cloud: Found in deployment settings
   • AWS OpenSearch: Domain endpoint from AWS console
7. Add Connection in CloudThinker

   Navigate to Connections → Elasticsearch and enter:

   • Cluster URL: Your Elasticsearch endpoint
   • API Key: The encoded key you copied
   • Index patterns: Patterns to access (e.g., logs-*)

Copy the API key immediately after creation. Elasticsearch will not show it again, and you’ll need to create a new key if lost.

---

Connection Details

Field	Description	Example
ES_URL	Elasticsearch cluster URL	https://your-cluster.es.region.cloud.es.io:9243
ES_API_KEY	Base64 encoded API key	VnVhQk...

---

Required Permissions

Minimum

{
  "cluster": ["monitor"],
  "indices": [
    {
      "names": ["logs-*", "metrics-*"],
      "privileges": ["read"]
    }
  ]
}

Recommended

{
  "cluster": ["monitor", "read_ilm"],
  "indices": [
    {
      "names": ["*"],
      "privileges": ["read", "view_index_metadata"]
    }
  ]
}

---

Agent Capabilities

Once connected, agents can:

Capability	Description
Log Search	Search application and infrastructure logs
Error Analysis	Find error patterns and anomalies
Incident Investigation	Correlate events across services
Metrics Query	Query stored metrics data

Example Prompts

@oliver search logs for authentication failures in the last 24 hours
@alex find error patterns in application logs for service-api
@kai search for OOMKilled events in Kubernetes logs
@tony analyze database error logs for connection issues

---

Common Index Patterns

Pattern	Use Case
logs-*	Application logs
filebeat-*	Filebeat-collected logs
metricbeat-*	Infrastructure metrics
apm-*	Application performance data
kubernetes-*	Kubernetes cluster logs

---

Troubleshooting

Connection refused

• Verify cluster URL is correct and accessible
• Check firewall allows CloudThinker IPs
• For AWS OpenSearch: Enable public access or VPC endpoint
• Confirm Elasticsearch is running

Authentication failed

• Verify API key is correct
• Check API key hasn’t expired
• Ensure user/role has required permissions
• For AWS: Verify IAM role or access policy

API key expired

• Navigate to Stack Management → Security → API Keys
• Create a new API key with the same permissions
• Update the key in CloudThinker connection settings

No data returned

• Verify index patterns match existing indices
• Check user has read permission on indices
• Ensure indices contain data in the time range
• Confirm field mappings are correct

---

Security Best Practices

• Read-only access - Grant only read privileges, never write
• Limited indices - Restrict access to necessary index patterns only
• API key rotation - Rotate API keys every 90 days
• Expiration - Set appropriate expiration on API keys
• Audit logging - Enable Elasticsearch audit logs

---

Related

Oliver Agent

Security log analysis

Grafana Connection

Visualize Elasticsearch data