Building SentinelOps AI: Automating Site Reliability with Multi-Agent Orchestration on Qwen Cloud
Cloud downtime is expensive, and for Site Reliability Engineers (SREs), the clock starts ticking the second a 504 Gateway Timeout alert fires. Investigating these incidents usually involves manually correlating logs, checking Kubernetes pod states, and analysing recent deployments.
Introduction Cloud downtime is expensive, and for Site Reliability Engineers (SREs), the clock starts ticking the second a 504 Gateway Timeout alert fires. Investigating these incidents usually involves manually correlating logs, checking Kubernetes pod states, and analysing recent deployments.
For the Global AI Hackathon Series with Qwen Cloud, I decided to tackle this exact pain point. I built SentinelOps AI an Autonomous Site Reliability Engineering Agent (submitted under Track 4: Autopilot Agent). The goal was simple: reduce Mean Time to Resolution (MTTR) by allowing an AI to autonomously triage, diagnose, and propose fixes for cloud infrastructure incidents, while strictly maintaining human oversight.
Why Qwen Cloud? Architecting AI-integrated infrastructure platforms requires more than just wrapping a prompt around a standard API. It demands reliability, structured reasoning, and the ability to process noisy telemetry data. Qwen Cloud’s flagship models offered the advanced reasoning required to parse dense error logs and execute complex, multi-step tool calls without losing context. It is purpose-built for production AI, which made it the ideal engine for an SRE tool.
The Architecture: A Multi-Agent Approach Rather than relying on a single, monolithic prompt, SentinelOps relies on a multi-agent state machine orchestrated via LangGraph. The backend is powered by FastAPI, while Gradio serves the frontend interface.
The workload is distributed across three specialised agents:
The Triage Agent: Ingests the initial alert and searches our vector database (ChromaDB) for semantic matches to past resolved incidents.
The Diagnostic Agent: Dynamically selects which tools to run. It can pull logs, check specific Kubernetes pod health, or evaluate deployment states.
The Resolution Agent: Synthesises the telemetry gathered by the diagnostic tools, identifies the root cause, and formulates a remediation plan.
Engineering for the Real World: Human-in-the-Loop In enterprise environments, no one wants an autonomous agent independently restarting live production databases. To bridge the gap between automation and safety, I implemented a strict Human-in-the-Loop Approval Gate.
Before executing any remediation (like a restart_pod command), the system halts and presents the proposed action, target service, and root cause analysis to the SRE. The engineer can either reject the proposal or click "Approve & Execute," at which point the agent fires the command to the cluster.
To make the platform truly testable and production-ready, the system features a split architecture: a Mock Mode using simulated data for safe demonstrations, and a Real Mode (real_tools.py) that connects directly to a live Kubernetes cluster.
Deployment The entire stack is containerised using Docker and deployed on an Alibaba Cloud Elastic Compute Service (ECS) instance, ensuring the backend runs securely on enterprise-grade infrastructure.
Conclusion Building SentinelOps AI was an incredible deep dive into the practical applications of agentic workflows. Qwen Cloud provided the reasoning horsepower needed to make this autonomous SRE a reality.
https://github.com/Emjaay20/sentinelops-ai