# Comprehensive Final Report: SRE Sandbox — OpenEnv Complying Evaluation System

## Executive Summary
The **SRE Sandbox** is an automated environment designed to evaluate AI models on real-world Site Reliability Engineering tasks. By deliberately simulating catastrophic configuration failures on Linux machines, LLMs are tested on their execution, reasoning, and standard bash administration skills. 
The system has been comprehensively upgraded to firmly align with both the **OpenEnv Specification Standard** and **Hugging Face Spaces**.

### Problem Statement Fulfillment
1. **Real-world task**: Fixes actual `nginx` configuration corruption, `resolv.conf` DNS wipeouts, firewall ports being dropped, permissions locked down, and runaway CPU processes filling disk spaces.
2. **OpenEnv Spec Compliance**: Typed Observations (`SREObservation`), Actions (`SREAction`), and States (`SREState`) correctly interface with the standard `openenv.core` interface.
3. **Tasks with Graders**: 8 progressive scenarios ranging strictly from `easy` (process killed) -> `medium` -> `hard` -> `cascade` (multiple services failing at once).
4. **Baseline Inference**: Ships with a flawless `inference.py` outputting `[START]`, `[STEP]`, and `[END]` syntax specifically matched with the OpenAI standard specification APIs.
5. **HF Spaces Delivery**: Employs a robust single-container native-subprocess integration to allow standard HF deployability free from nested-Docker restrictions. 

## Architectural Refinements

### 1. Removing Nested-Docker for Hugging Face
**The Problem**: Hugging Face Spaces provides an execution container, but actively locks out the internal Docker SDK.
**The Solution**: We eliminated all nested container invocations. Instead, `SREEnvironment` operates locally inside the HF Space Container.
- `/reset` logic seamlessly uses file copy backups (e.g. `cp /etc/nginx.bak /etc/nginx`) to restore clean state.
- Scripts run directly via Python's `subprocess` to sabotage.
- The `CommandSanitizer` dynamically restricts commands like `rm -rf /` ensuring the HF Space cannot be deliberately deleted locally. 

### 2. Dense Normalised Rewards 
Instead of sparse `1` for pass or `0` for fail, the `RewardCalculator` scans via regex to identify strong SRE practices:
- Standard diagnostic commands (`tail`, `cat`, `systemctl status`) give `+0.2`.
- Phase alignment (e.g. diagnosing during the *research* phase) yields `+0.1`.
- Clean resolutions yield full execution awards capped cleanly between `0` and `1.0` via `get_normalized_score()`.

### 3. Pluggable BYOA Architecture
We removed all LLM parsing logic and static prompt engineering from `inference.py`. It is natively extracted into `custom_agent.py`. This guarantees any external user evaluating this environment can Bring Their Own Agent by natively overriding the LLM prompt wrapper without breaking the core validator script API constraints.

## Validation Metrics
1. **Validation Tool**: Successfully passes `openenv validate` verification scripts checking lockfiles and webservers cleanly.
2. **Testing coverage**: Over 90 Python unit tests testing schemas, safety overrides, reward triggers, schema validations, passing instantly.
3. **HTTP Web Server Integration**: Automatically bundled using `openenv.core.env_server.http_server.HTTPEnvServer` enabling standard REST interactions externally on Port 7860.

This SRE Evaluation platform proves itself to be a rigorous, technically robust, highly realistic standard to measure SRE AI agent performance in real-world scenarios.



---
title: SRE Sandbox
emoji: 🐳
colorFrom: blue
colorTo: green
sdk: docker
pinned: false
---

# SRE Sandbox — OpenEnv Competition Track

An **OpenEnv-compliant** sandbox that evaluates AI SRE agents by deliberately breaking Linux infrastructure and measuring how effectively they diagnose and fix the issues via native bash commands.

This environment perfectly follows the **OpenEnv** specifications, running directly on Hugging Face Spaces and testing large language models realistically.

## 🚀 Hugging Face Space & OpenEnv Spec
- **Subprocess native**: The environment breaks its *own* container natively via `subprocess`. No nested Docker API requirements, making it 100% compliant with standard HF Spaces.
- **Strict Spec Definitions**: Employs `openenv.core.env_server.interfaces.Environment`. Uses `SREAction`, `SREObservation`, and `SREState` derived cleanly from standard `openenv` types.
- **Automatic Server**: Uses `HTTPEnvServer` wrapper running on port `7860` natively, ensuring `/reset`, `/state`, `/step`, `/metadata`, and `/schema` work perfectly for external REST access.
- **Inference Ready**: Implements all strict competition standard logs: `[START]`, `[STEP]`, `[END]`.

## 🛠️ Quick Start (Clone & Test)

**1. Clone the environment and configure Python**
```bash
git clone https://github.com/MohammedMutee/my-openenv-sre-sandbox.git
cd my-openenv-sre-sandbox
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt
```

**2. Verify compliance using OpenEnv tool**
```bash
openenv validate
```

**3. Test the baseline inference**
```bash
# Will use API_BASE_URL, MODEL_NAME, and HF_TOKEN. 
# Safe mock defaults kick in if keys are missing to guarantee format tests pass.
python inference.py
```

**4. 🧠 Bring Your Own Agent (BYOA)**
The environment is decoupled entirely from prompt engineering. To test your own AI agent:
1. Open [`custom_agent.py`](custom_agent.py)
2. Modify `SYSTEM_PROMPT` or the `get_agent_action` logic to connect your own framework (LangChain, AutoGen, pure OpenAI, etc.)
3. Run `python inference.py` to evaluate your agent's performance against the leaderboard!

**5. Test Hugging Face deployment locally (Docker)**
```bash
docker build -t sre-sandbox .
docker run -p 7860:7860 sre-sandbox
```
*(You can now send POST requests to `http://localhost:7860/reset`)*

## 🧱 Scenarios (8 Total)

| Scenario | Sabotage | Difficulty |
|---|---|---|
| **easy** | Nginx process killed | ⭐ |
| **medium** | Nginx config syntax error | ⭐⭐ |
| **hard** | 500MB disk fill + Postgres port corrupted | ⭐⭐⭐ |
| **dns** | resolv.conf wiped | ⭐⭐ |
| **firewall** | iptables blocking ports 80/443 | ⭐⭐ |
| **permissions** | Web root + config chmod 000 | ⭐⭐ |
| **resource** | /tmp filled + CPU consumers spawned | ⭐⭐⭐ |
| **cascade** | Nginx upstream broken + Postgres auth reject + log flood | ⭐⭐⭐⭐ |

## 📈 Dense Reward System
Instead of simple binary grades, this custom environment parses agent bash execution with regex to reward strong methodologies. 

| Signal | Reward | Examples |
|---|---|---|
| Diagnostic | +0.2 | `cat`, `tail`, `grep` on logs; `df`, `ss`, `systemctl status` |
| Phase bonus | +0.1 | Research phase + diagnostic, or Execute phase + remediation |
| Remediation | +0.3 | `service restart`, `sed -i`, `chmod`, `iptables -F` |
| Environment fixed | +0.5 | HTTP 200 + DB OK |
| Blocked command | −0.5 | `rm -rf /`, `mkfs`, `shutdown` (Blocked securely without execution) |

*(Normalised gracefully between 0.0 - 1.0 per task constraint via `get_normalized_score()`)*

## 🔒 Security Gate
`CommandSanitizer` stops the LLM from destroying the hugging face space securely.
- **17 blocked patterns**: `rm -rf /`, `mkfs`, `dd /dev/`, fork bombs.
- Safe overrides: Log file removal `/var/log/*` allowed. 

## 🧪 Tests
Over 90+ tests guarantee scenario loading and rewards functionality perfectly.

```bash
pytest tests/ -v
```

## 🗂️ Project Structure

```
my-openenv/
├── custom_agent.py        # Pluggable BYOA (Bring Your Own Agent) architecture
├── env.py                 # Core SREEnvironment engine + grader (Subprocess Native)
├── inference.py           # CLI evaluation runner + OpenEnv client 
├── models.py              # Pydantic Action + Observation schemas based on openenv objects
├── security.py            # Command sanitizer (blocklist + overrides) protects HF Space
├── rewards.py             # Regex-based reward calculator
├── openenv.yaml           # Spec configurations and model metadata
├── server/
│   └── app.py             # FastAPI OpenEnv wrapper bound to port 7860
├── examples/
│   ├── client_http.py     # HTTP API client example
│   └── client_python.py   # Python Generic Client directly
├── scripts/
│   ├── scenario_easy.sh   ... scenario_cascade.sh (8 scripts defining SABOTAGE)
├── tests/
│   ├── test_models.py     ... test_env.py (94 rigorous tests)
├── .github/workflows/
│   └── ci.yml             # Lint → Test → Docker validation
├── Dockerfile             # Ubuntu 22.04 unified container natively hosting APIs + Bash
├── CONTRIBUTING.md        # Guide: pinging the environment via OpenEnv clients
├── LICENSE                # MIT License
├── pyproject.toml         # v2.0.0 with openenv-core
└── REPORT.md              # Detailed final deliverables report
```

## License
MIT



github link : https://github.com/MohammedMutee/my-openenv-sre-sandbox