diff --git a/master_plan.md b/master_plan.md deleted file mode 100644 index 6657160..0000000 --- a/master_plan.md +++ /dev/null @@ -1,417 +0,0 @@ -# AUTARCH Evolution Plan -## From CLI Framework to Full-Stack Security Platform - -### Context - -AUTARCH is currently a CLI-only Python security framework at `/home/snake/dh_framework/` with 20+ modules across 6 categories (Defense, Offense, Counter, Analyze, OSINT, Simulate), LLM integration (llama.cpp/transformers/Claude), Metasploit RPC integration, and an OSINT sites database with 7,200+ sites. - -There is also a UPnP port forwarding script at `/home/snake/wg_setec/upnp-renew.sh` (using miniupnpc) that forwards ports 443/TCP, 51820/UDP, 8080/TCP but has **no cron job installed** despite being intended to run every 12 hours. - -The goal is to evolve AUTARCH into a portable, cross-platform security platform with a web dashboard, advanced OSINT search engine, Wireshark integration, UPnP management, and Docker packaging -- while preserving the existing CLI interface. - ---- - -## Phase 0: Backup & New Working Directory - -**What:** Copy the codebase to `~/autarch`, excluding models (9GB). - -**Steps:** -1. `mkdir ~/autarch` -2. `rsync -av --exclude='models/' --exclude='snoop/' --exclude='__pycache__/' /home/snake/dh_framework/ ~/autarch/` -3. Verify the copy: `ls ~/autarch/` and `du -sh ~/autarch/` -4. Switch working directory to `~/autarch` -5. `git init` in the new directory for version control going forward - -**Files:** No code changes. Pure copy operation. - ---- - -## Phase 1: UPnP Manager Integration - -**What:** Integrate UPnP port forwarding management into AUTARCH with a config-driven approach and cron job management. - -**New files:** -- `core/upnp.py` -- UPnP manager class wrapping `upnpc` (miniupnpc CLI) -- `modules/upnp_manager.py` -- CLI menu module (CATEGORY: defense) - -**Changes to existing files:** -- `core/config.py` -- Add `[upnp]` config section with port mappings, internal IP, refresh interval -- `core/menu.py` -- No changes needed (module auto-discovered) - -**`core/upnp.py` design:** -```python -class UPnPManager: - def list_mappings() # upnpc -l - def add_mapping(internal_ip, internal_port, external_port, protocol, description) - def remove_mapping(external_port, protocol) - def refresh_all() # Re-add all configured mappings - def get_external_ip() # upnpc -e - def install_cron(interval_hours) # Write crontab entry - def uninstall_cron() # Remove crontab entry - def get_cron_status() # Check if cron is active - def load_mappings_from_config() - def save_mappings_to_config() -``` - -**Config section:** -```ini -[upnp] -enabled = true -internal_ip = 10.0.0.26 -refresh_hours = 12 -mappings = 443:TCP,51820:UDP,8080:TCP -``` - -**CLI module menu:** -``` -UPnP Port Manager - [1] Show Current Mappings - [2] Add Port Mapping - [3] Remove Port Mapping - [4] Refresh All Mappings - [5] Show External IP - [6] Cron Job Settings (currently: every 12h / not installed) - [7] Edit Internal IP - [0] Back -``` - -**Cron management:** Uses `subprocess` to read/write crontab via `crontab -l` and `crontab -` pipe. Adds entry like: -``` -0 */12 * * * /usr/bin/python3 /home/snake/autarch/autarch.py --upnp-refresh > /dev/null 2>&1 -``` - -**Also add to autarch.py:** `--upnp-refresh` CLI flag that calls `UPnPManager.refresh_all()` and exits (for cron use). - ---- - -## Phase 2: Web UI Dashboard (Flask) - -**What:** Add a Flask-based web dashboard running alongside the CLI. Reuse patterns from wg_setec (dark theme, session auth, template inheritance). - -**New directory structure:** -``` -~/autarch/ -├── web/ -│ ├── __init__.py -│ ├── app.py # Flask app factory -│ ├── auth.py # Login/session auth (from wg_setec pattern) -│ ├── routes/ -│ │ ├── __init__.py -│ │ ├── dashboard.py # Main dashboard -│ │ ├── defense.py # Defense tools page -│ │ ├── offense.py # Offense tools page -│ │ ├── counter.py # Counter-intel page -│ │ ├── analyze.py # Analysis page -│ │ ├── osint.py # OSINT search engine -│ │ ├── simulate.py # Simulation page -│ │ ├── settings.py # Settings management -│ │ ├── upnp.py # UPnP management page -│ │ └── wireshark.py # Wireshark page (Phase 4) -│ ├── templates/ -│ │ ├── base.html # Master template (dark theme, sidebar nav) -│ │ ├── login.html -│ │ ├── dashboard.html # Overview with status cards -│ │ ├── defense.html -│ │ ├── offense.html -│ │ ├── counter.html -│ │ ├── analyze.html -│ │ ├── osint.html # Advanced search UI -│ │ ├── simulate.html -│ │ ├── settings.html -│ │ ├── upnp.html -│ │ └── wireshark.html -│ └── static/ -│ ├── css/style.css # Dark theme (reuse wg_setec CSS variables) -│ └── js/app.js # Vanilla JS for interactions -``` - -**Dashboard landing page shows:** -- System status cards (LLM loaded, MSF connected, UPnP active, services running) -- Quick action buttons for each category -- Recent activity/scan results -- Server info (hostname, IP, uptime) - -**Each category page provides:** -- Module listing with descriptions -- Quick-launch buttons that trigger module functions via API -- Results display area -- Settings specific to that category - -**Flask app startup:** -- Add `--web` flag to `autarch.py` to start web server -- Add `--web-port PORT` (default 8080) -- Web server runs in background thread or separate process -- Can run CLI and web simultaneously - -**Authentication:** -- Session-based with bcrypt (same as wg_setec) -- Default admin/admin, forced password change on first login -- Config section `[web]` for port, host, secret_key - -**Key reusable patterns from wg_setec:** -- CSS variables dark theme (`/home/snake/wg_setec/static/css/style.css`) -- Template inheritance (`/home/snake/wg_setec/templates/base.html`) -- Auth decorator (`/home/snake/wg_setec/auth.py`) -- Flash messages, status dots, data tables - ---- - -## Phase 3: OSINT Search Engine (Web) - -**What:** Build an advanced OSINT search interface in the web UI that surpasses the CLI capabilities. - -**Route:** `web/routes/osint.py` -**Template:** `web/templates/osint.html` - -**Search interface features:** -- Search box with type selector (username / email / phone / domain / IP) -- Category multi-select checkboxes (filter site categories) -- NSFW toggle -- Scan depth selector (Quick 100 / Standard 500 / Full 7000+) -- Advanced options panel: - - Thread count slider - - Timeout setting - - User-agent selection - - Proxy support field - - Export format (JSON/HTML/CSV) - -**Real-time results (Server-Sent Events):** -- `GET /osint/search/stream?username=X&...` -- SSE endpoint -- Progressive results as sites are checked -- Live progress bar (sites checked / total) -- Results appear in cards grouped by status (good/maybe/bad) -- Filter/sort results client-side - -**Results display:** -- Card layout per found profile: site name, URL (clickable), confidence %, category badge -- Expandable detail: detection method, page title, response info -- Bulk actions: save to dossier, export, open all in new tabs -- Summary statistics bar chart - -**Backend integration:** -- Reuses existing `modules/recon.py` scanning functions -- Reuses `core/sites_db.py` for site queries -- New `web/tasks.py` for async scan management (threading, not Celery -- keep it simple) - -**Dossier management page:** -- List existing dossiers -- Create new dossier -- Link search results to dossier -- View/edit dossier details -- Export dossier as report - ---- - -## Phase 4: Wireshark Module (tshark + pyshark) - -**What:** Add packet capture and analysis capabilities using tshark CLI and pyshark Python library. - -**New files:** -- `core/wireshark.py` -- tshark/pyshark wrapper class -- `modules/wireshark.py` -- CLI menu module (CATEGORY: analyze) -- `web/routes/wireshark.py` -- Web dashboard page - -**`core/wireshark.py` design:** -```python -class WiresharkManager: - def list_interfaces() # tshark -D - def start_capture(interface, filter, duration, output_file) - def stop_capture() - def get_capture_stats() - def read_pcap(filepath) # pyshark FileCapture - def live_capture(interface, filter, callback) # pyshark LiveCapture - def get_protocol_hierarchy() # Protocol distribution - def extract_conversations() # IP conversations - def extract_dns_queries() # DNS analysis - def extract_http_requests() # HTTP analysis - def extract_credentials() # Plain-text credential detection - def apply_display_filter(capture, filter_string) - def export_packets(capture, format) # JSON/CSV export -``` - -**CLI module menu:** -``` -Wireshark / Packet Analysis - [1] List Interfaces - [2] Start Live Capture - [3] Open PCAP File - [4] Protocol Analysis - [5] Conversation Analysis - [6] DNS Query Analysis - [7] HTTP Traffic Analysis - [8] Credential Detection - [9] Export Results - [0] Back -``` - -**Web UI features:** -- Interface selector dropdown -- Capture filter input (BPF syntax) -- Start/stop capture buttons -- Live packet stream (SSE) -- Protocol distribution pie chart -- Conversation table -- Packet detail view (expandable tree) -- PCAP file upload and analysis - -**Dependencies:** `pyshark`, system `tshark` (part of Wireshark package) - ---- - -## Phase 5: Path Portability & Windows Support - -**What:** Make all paths relative/configurable, remove hardcoded paths, add Windows compatibility. - -**Key changes:** - -1. **Path resolution system** (`core/paths.py`): -```python -import platform, os -from pathlib import Path - -def get_app_dir(): - """Returns the application root directory""" - return Path(__file__).parent.parent - -def get_data_dir(): - return get_app_dir() / 'data' - -def get_config_path(): - return get_app_dir() / 'autarch_settings.conf' - -def get_results_dir(): - return get_app_dir() / 'results' - -def is_windows(): - return platform.system() == 'Windows' - -def get_platform(): - return platform.system().lower() # 'linux', 'windows', 'darwin' -``` - -2. **Audit all files for hardcoded paths:** - - `core/config.py` -- WG paths, model paths - - `core/msf.py` -- msfrpcd path - - `core/upnp.py` -- upnpc path - - `modules/*.py` -- any `/home/snake/` references - - Replace all with `paths.get_*()` calls or config-driven paths - -3. **Platform-specific tool detection:** - - Check for `tshark`, `nmap`, `upnpc`, `wg`, `msfrpcd` availability - - Graceful degradation when tools missing (disable features, show install instructions) - - Windows: use `where` instead of `which` - -4. **Cross-platform subprocess calls:** - - Use `shutil.which()` for tool discovery - - Use `pathlib.Path` everywhere instead of string concatenation - - Handle Windows vs Unix shell differences - -5. **Windows-specific considerations:** - - `nmap` available on Windows (nmap.org installer) - - `tshark` available on Windows (Wireshark installer) - - `upnpc` available on Windows (miniupnpc builds) - - Metasploit available via WSL or native Windows build - - WireGuard available on Windows - - No `sudo` -- use `ctypes.windll.shell32.IsUserAnAdmin()` check - ---- - -## Phase 6: Docker Packaging - -**What:** Create Docker configuration for portable deployment with all dependencies bundled. - -**New files:** -``` -~/autarch/ -├── Dockerfile -├── docker-compose.yml -├── .dockerignore -└── scripts/ - ├── entrypoint.sh # Container startup script - └── install-tools.sh # Install nmap, tshark, upnpc, etc. -``` - -**Dockerfile approach:** -- Base: `python:3.11-slim` -- Install system tools: `nmap`, `tshark`, `miniupnpc`, `wireguard-tools` -- Install Python deps from `requirements.txt` -- Copy application code -- Expose ports: 8080 (web UI), 55553 (MSF RPC passthrough) -- Volume mounts: `/data` (persistent), `/models` (LLM models) -- Entrypoint: can start CLI, web, or both - -**docker-compose.yml:** -```yaml -services: - autarch: - build: . - ports: - - "8080:8080" - volumes: - - ./data:/app/data - - ./models:/app/models - - ./results:/app/results - environment: - - AUTARCH_MODE=web # or 'cli' or 'both' - network_mode: host # Needed for network scanning tools -``` - -**requirements.txt** (consolidated): -``` -flask -bcrypt -pyshark -qrcode -Pillow -requests -llama-cpp-python # optional -transformers # optional -torch # optional -anthropic # optional -msgpack -``` - ---- - -## Execution Order - -| Step | Phase | Effort | Description | -|------|-------|--------|-------------| -| 1 | Phase 0 | 5 min | Backup & create ~/autarch | -| 2 | Phase 1 | 1 session | UPnP manager + cron job management | -| 3 | Phase 2 | 2-3 sessions | Flask web dashboard skeleton + all pages | -| 4 | Phase 3 | 1-2 sessions | OSINT search engine with SSE streaming | -| 5 | Phase 4 | 1-2 sessions | Wireshark/tshark integration | -| 6 | Phase 5 | 1-2 sessions | Path portability + Windows support | -| 7 | Phase 6 | 1 session | Docker packaging | - -**Each phase is self-contained and testable before moving to the next.** - ---- - -## Verification Plan - -After each phase: -1. **Phase 0:** `ls ~/autarch/` matches expected structure, `python3 ~/autarch/autarch.py --help` works -2. **Phase 1:** UPnP menu accessible, `upnpc -l` shows mappings, cron installs/uninstalls correctly -3. **Phase 2:** `python3 autarch.py --web` starts Flask on :8080, login works, all pages render, CLI still works independently -4. **Phase 3:** OSINT search returns results in web UI, SSE streaming shows live progress, results match CLI output -5. **Phase 4:** `tshark -D` lists interfaces from web UI, capture start/stop works, PCAP analysis shows results -6. **Phase 5:** No hardcoded paths remain (`grep -r '/home/snake' .` returns nothing), app runs from any directory -7. **Phase 6:** `docker-compose up` starts the app, web UI accessible, tools available inside container - ---- - -## Critical Files Reference - -| Existing File | Role | Reuse For | -|---|---|---| -| `/home/snake/wg_setec/static/css/style.css` | Dark theme CSS | Web UI base theme | -| `/home/snake/wg_setec/templates/base.html` | Template inheritance | Web UI base template | -| `/home/snake/wg_setec/auth.py` | bcrypt session auth | Web UI authentication | -| `/home/snake/wg_setec/upnp-renew.sh` | UPnP port script | Reference for port mappings | -| `core/config.py` | Config handler | Add [upnp], [web] sections | -| `core/sites_db.py` | Sites DB (7200+ sites) | OSINT search backend | -| `modules/recon.py` | OSINT scanner (2193 lines) | OSINT search engine backend | -| `core/menu.py` | Module discovery | Auto-discover new modules | -| `autarch.py` | Entry point | Add --web, --upnp-refresh flags |