Delete master_plan.md
This commit is contained in:
Digi J
GitHub
parent
1792c871b3
commit
e7f68031f2
417
master_plan.md
417
master_plan.md
@ -1,417 +0,0 @@
|
|||||||
# AUTARCH Evolution Plan
|
|
||||||
## From CLI Framework to Full-Stack Security Platform
|
|
||||||
|
|
||||||
### Context
|
|
||||||
|
|
||||||
AUTARCH is currently a CLI-only Python security framework at `/home/snake/dh_framework/` with 20+ modules across 6 categories (Defense, Offense, Counter, Analyze, OSINT, Simulate), LLM integration (llama.cpp/transformers/Claude), Metasploit RPC integration, and an OSINT sites database with 7,200+ sites.
|
|
||||||
|
|
||||||
There is also a UPnP port forwarding script at `/home/snake/wg_setec/upnp-renew.sh` (using miniupnpc) that forwards ports 443/TCP, 51820/UDP, 8080/TCP but has **no cron job installed** despite being intended to run every 12 hours.
|
|
||||||
|
|
||||||
The goal is to evolve AUTARCH into a portable, cross-platform security platform with a web dashboard, advanced OSINT search engine, Wireshark integration, UPnP management, and Docker packaging -- while preserving the existing CLI interface.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 0: Backup & New Working Directory
|
|
||||||
|
|
||||||
**What:** Copy the codebase to `~/autarch`, excluding models (9GB).
|
|
||||||
|
|
||||||
**Steps:**
|
|
||||||
1. `mkdir ~/autarch`
|
|
||||||
2. `rsync -av --exclude='models/' --exclude='snoop/' --exclude='__pycache__/' /home/snake/dh_framework/ ~/autarch/`
|
|
||||||
3. Verify the copy: `ls ~/autarch/` and `du -sh ~/autarch/`
|
|
||||||
4. Switch working directory to `~/autarch`
|
|
||||||
5. `git init` in the new directory for version control going forward
|
|
||||||
|
|
||||||
**Files:** No code changes. Pure copy operation.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 1: UPnP Manager Integration
|
|
||||||
|
|
||||||
**What:** Integrate UPnP port forwarding management into AUTARCH with a config-driven approach and cron job management.
|
|
||||||
|
|
||||||
**New files:**
|
|
||||||
- `core/upnp.py` -- UPnP manager class wrapping `upnpc` (miniupnpc CLI)
|
|
||||||
- `modules/upnp_manager.py` -- CLI menu module (CATEGORY: defense)
|
|
||||||
|
|
||||||
**Changes to existing files:**
|
|
||||||
- `core/config.py` -- Add `[upnp]` config section with port mappings, internal IP, refresh interval
|
|
||||||
- `core/menu.py` -- No changes needed (module auto-discovered)
|
|
||||||
|
|
||||||
**`core/upnp.py` design:**
|
|
||||||
```python
|
|
||||||
class UPnPManager:
|
|
||||||
def list_mappings() # upnpc -l
|
|
||||||
def add_mapping(internal_ip, internal_port, external_port, protocol, description)
|
|
||||||
def remove_mapping(external_port, protocol)
|
|
||||||
def refresh_all() # Re-add all configured mappings
|
|
||||||
def get_external_ip() # upnpc -e
|
|
||||||
def install_cron(interval_hours) # Write crontab entry
|
|
||||||
def uninstall_cron() # Remove crontab entry
|
|
||||||
def get_cron_status() # Check if cron is active
|
|
||||||
def load_mappings_from_config()
|
|
||||||
def save_mappings_to_config()
|
|
||||||
```
|
|
||||||
|
|
||||||
**Config section:**
|
|
||||||
```ini
|
|
||||||
[upnp]
|
|
||||||
enabled = true
|
|
||||||
internal_ip = 10.0.0.26
|
|
||||||
refresh_hours = 12
|
|
||||||
mappings = 443:TCP,51820:UDP,8080:TCP
|
|
||||||
```
|
|
||||||
|
|
||||||
**CLI module menu:**
|
|
||||||
```
|
|
||||||
UPnP Port Manager
|
|
||||||
[1] Show Current Mappings
|
|
||||||
[2] Add Port Mapping
|
|
||||||
[3] Remove Port Mapping
|
|
||||||
[4] Refresh All Mappings
|
|
||||||
[5] Show External IP
|
|
||||||
[6] Cron Job Settings (currently: every 12h / not installed)
|
|
||||||
[7] Edit Internal IP
|
|
||||||
[0] Back
|
|
||||||
```
|
|
||||||
|
|
||||||
**Cron management:** Uses `subprocess` to read/write crontab via `crontab -l` and `crontab -` pipe. Adds entry like:
|
|
||||||
```
|
|
||||||
0 */12 * * * /usr/bin/python3 /home/snake/autarch/autarch.py --upnp-refresh > /dev/null 2>&1
|
|
||||||
```
|
|
||||||
|
|
||||||
**Also add to autarch.py:** `--upnp-refresh` CLI flag that calls `UPnPManager.refresh_all()` and exits (for cron use).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 2: Web UI Dashboard (Flask)
|
|
||||||
|
|
||||||
**What:** Add a Flask-based web dashboard running alongside the CLI. Reuse patterns from wg_setec (dark theme, session auth, template inheritance).
|
|
||||||
|
|
||||||
**New directory structure:**
|
|
||||||
```
|
|
||||||
~/autarch/
|
|
||||||
├── web/
|
|
||||||
│ ├── __init__.py
|
|
||||||
│ ├── app.py # Flask app factory
|
|
||||||
│ ├── auth.py # Login/session auth (from wg_setec pattern)
|
|
||||||
│ ├── routes/
|
|
||||||
│ │ ├── __init__.py
|
|
||||||
│ │ ├── dashboard.py # Main dashboard
|
|
||||||
│ │ ├── defense.py # Defense tools page
|
|
||||||
│ │ ├── offense.py # Offense tools page
|
|
||||||
│ │ ├── counter.py # Counter-intel page
|
|
||||||
│ │ ├── analyze.py # Analysis page
|
|
||||||
│ │ ├── osint.py # OSINT search engine
|
|
||||||
│ │ ├── simulate.py # Simulation page
|
|
||||||
│ │ ├── settings.py # Settings management
|
|
||||||
│ │ ├── upnp.py # UPnP management page
|
|
||||||
│ │ └── wireshark.py # Wireshark page (Phase 4)
|
|
||||||
│ ├── templates/
|
|
||||||
│ │ ├── base.html # Master template (dark theme, sidebar nav)
|
|
||||||
│ │ ├── login.html
|
|
||||||
│ │ ├── dashboard.html # Overview with status cards
|
|
||||||
│ │ ├── defense.html
|
|
||||||
│ │ ├── offense.html
|
|
||||||
│ │ ├── counter.html
|
|
||||||
│ │ ├── analyze.html
|
|
||||||
│ │ ├── osint.html # Advanced search UI
|
|
||||||
│ │ ├── simulate.html
|
|
||||||
│ │ ├── settings.html
|
|
||||||
│ │ ├── upnp.html
|
|
||||||
│ │ └── wireshark.html
|
|
||||||
│ └── static/
|
|
||||||
│ ├── css/style.css # Dark theme (reuse wg_setec CSS variables)
|
|
||||||
│ └── js/app.js # Vanilla JS for interactions
|
|
||||||
```
|
|
||||||
|
|
||||||
**Dashboard landing page shows:**
|
|
||||||
- System status cards (LLM loaded, MSF connected, UPnP active, services running)
|
|
||||||
- Quick action buttons for each category
|
|
||||||
- Recent activity/scan results
|
|
||||||
- Server info (hostname, IP, uptime)
|
|
||||||
|
|
||||||
**Each category page provides:**
|
|
||||||
- Module listing with descriptions
|
|
||||||
- Quick-launch buttons that trigger module functions via API
|
|
||||||
- Results display area
|
|
||||||
- Settings specific to that category
|
|
||||||
|
|
||||||
**Flask app startup:**
|
|
||||||
- Add `--web` flag to `autarch.py` to start web server
|
|
||||||
- Add `--web-port PORT` (default 8080)
|
|
||||||
- Web server runs in background thread or separate process
|
|
||||||
- Can run CLI and web simultaneously
|
|
||||||
|
|
||||||
**Authentication:**
|
|
||||||
- Session-based with bcrypt (same as wg_setec)
|
|
||||||
- Default admin/admin, forced password change on first login
|
|
||||||
- Config section `[web]` for port, host, secret_key
|
|
||||||
|
|
||||||
**Key reusable patterns from wg_setec:**
|
|
||||||
- CSS variables dark theme (`/home/snake/wg_setec/static/css/style.css`)
|
|
||||||
- Template inheritance (`/home/snake/wg_setec/templates/base.html`)
|
|
||||||
- Auth decorator (`/home/snake/wg_setec/auth.py`)
|
|
||||||
- Flash messages, status dots, data tables
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 3: OSINT Search Engine (Web)
|
|
||||||
|
|
||||||
**What:** Build an advanced OSINT search interface in the web UI that surpasses the CLI capabilities.
|
|
||||||
|
|
||||||
**Route:** `web/routes/osint.py`
|
|
||||||
**Template:** `web/templates/osint.html`
|
|
||||||
|
|
||||||
**Search interface features:**
|
|
||||||
- Search box with type selector (username / email / phone / domain / IP)
|
|
||||||
- Category multi-select checkboxes (filter site categories)
|
|
||||||
- NSFW toggle
|
|
||||||
- Scan depth selector (Quick 100 / Standard 500 / Full 7000+)
|
|
||||||
- Advanced options panel:
|
|
||||||
- Thread count slider
|
|
||||||
- Timeout setting
|
|
||||||
- User-agent selection
|
|
||||||
- Proxy support field
|
|
||||||
- Export format (JSON/HTML/CSV)
|
|
||||||
|
|
||||||
**Real-time results (Server-Sent Events):**
|
|
||||||
- `GET /osint/search/stream?username=X&...` -- SSE endpoint
|
|
||||||
- Progressive results as sites are checked
|
|
||||||
- Live progress bar (sites checked / total)
|
|
||||||
- Results appear in cards grouped by status (good/maybe/bad)
|
|
||||||
- Filter/sort results client-side
|
|
||||||
|
|
||||||
**Results display:**
|
|
||||||
- Card layout per found profile: site name, URL (clickable), confidence %, category badge
|
|
||||||
- Expandable detail: detection method, page title, response info
|
|
||||||
- Bulk actions: save to dossier, export, open all in new tabs
|
|
||||||
- Summary statistics bar chart
|
|
||||||
|
|
||||||
**Backend integration:**
|
|
||||||
- Reuses existing `modules/recon.py` scanning functions
|
|
||||||
- Reuses `core/sites_db.py` for site queries
|
|
||||||
- New `web/tasks.py` for async scan management (threading, not Celery -- keep it simple)
|
|
||||||
|
|
||||||
**Dossier management page:**
|
|
||||||
- List existing dossiers
|
|
||||||
- Create new dossier
|
|
||||||
- Link search results to dossier
|
|
||||||
- View/edit dossier details
|
|
||||||
- Export dossier as report
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 4: Wireshark Module (tshark + pyshark)
|
|
||||||
|
|
||||||
**What:** Add packet capture and analysis capabilities using tshark CLI and pyshark Python library.
|
|
||||||
|
|
||||||
**New files:**
|
|
||||||
- `core/wireshark.py` -- tshark/pyshark wrapper class
|
|
||||||
- `modules/wireshark.py` -- CLI menu module (CATEGORY: analyze)
|
|
||||||
- `web/routes/wireshark.py` -- Web dashboard page
|
|
||||||
|
|
||||||
**`core/wireshark.py` design:**
|
|
||||||
```python
|
|
||||||
class WiresharkManager:
|
|
||||||
def list_interfaces() # tshark -D
|
|
||||||
def start_capture(interface, filter, duration, output_file)
|
|
||||||
def stop_capture()
|
|
||||||
def get_capture_stats()
|
|
||||||
def read_pcap(filepath) # pyshark FileCapture
|
|
||||||
def live_capture(interface, filter, callback) # pyshark LiveCapture
|
|
||||||
def get_protocol_hierarchy() # Protocol distribution
|
|
||||||
def extract_conversations() # IP conversations
|
|
||||||
def extract_dns_queries() # DNS analysis
|
|
||||||
def extract_http_requests() # HTTP analysis
|
|
||||||
def extract_credentials() # Plain-text credential detection
|
|
||||||
def apply_display_filter(capture, filter_string)
|
|
||||||
def export_packets(capture, format) # JSON/CSV export
|
|
||||||
```
|
|
||||||
|
|
||||||
**CLI module menu:**
|
|
||||||
```
|
|
||||||
Wireshark / Packet Analysis
|
|
||||||
[1] List Interfaces
|
|
||||||
[2] Start Live Capture
|
|
||||||
[3] Open PCAP File
|
|
||||||
[4] Protocol Analysis
|
|
||||||
[5] Conversation Analysis
|
|
||||||
[6] DNS Query Analysis
|
|
||||||
[7] HTTP Traffic Analysis
|
|
||||||
[8] Credential Detection
|
|
||||||
[9] Export Results
|
|
||||||
[0] Back
|
|
||||||
```
|
|
||||||
|
|
||||||
**Web UI features:**
|
|
||||||
- Interface selector dropdown
|
|
||||||
- Capture filter input (BPF syntax)
|
|
||||||
- Start/stop capture buttons
|
|
||||||
- Live packet stream (SSE)
|
|
||||||
- Protocol distribution pie chart
|
|
||||||
- Conversation table
|
|
||||||
- Packet detail view (expandable tree)
|
|
||||||
- PCAP file upload and analysis
|
|
||||||
|
|
||||||
**Dependencies:** `pyshark`, system `tshark` (part of Wireshark package)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 5: Path Portability & Windows Support
|
|
||||||
|
|
||||||
**What:** Make all paths relative/configurable, remove hardcoded paths, add Windows compatibility.
|
|
||||||
|
|
||||||
**Key changes:**
|
|
||||||
|
|
||||||
1. **Path resolution system** (`core/paths.py`):
|
|
||||||
```python
|
|
||||||
import platform, os
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
def get_app_dir():
|
|
||||||
"""Returns the application root directory"""
|
|
||||||
return Path(__file__).parent.parent
|
|
||||||
|
|
||||||
def get_data_dir():
|
|
||||||
return get_app_dir() / 'data'
|
|
||||||
|
|
||||||
def get_config_path():
|
|
||||||
return get_app_dir() / 'autarch_settings.conf'
|
|
||||||
|
|
||||||
def get_results_dir():
|
|
||||||
return get_app_dir() / 'results'
|
|
||||||
|
|
||||||
def is_windows():
|
|
||||||
return platform.system() == 'Windows'
|
|
||||||
|
|
||||||
def get_platform():
|
|
||||||
return platform.system().lower() # 'linux', 'windows', 'darwin'
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Audit all files for hardcoded paths:**
|
|
||||||
- `core/config.py` -- WG paths, model paths
|
|
||||||
- `core/msf.py` -- msfrpcd path
|
|
||||||
- `core/upnp.py` -- upnpc path
|
|
||||||
- `modules/*.py` -- any `/home/snake/` references
|
|
||||||
- Replace all with `paths.get_*()` calls or config-driven paths
|
|
||||||
|
|
||||||
3. **Platform-specific tool detection:**
|
|
||||||
- Check for `tshark`, `nmap`, `upnpc`, `wg`, `msfrpcd` availability
|
|
||||||
- Graceful degradation when tools missing (disable features, show install instructions)
|
|
||||||
- Windows: use `where` instead of `which`
|
|
||||||
|
|
||||||
4. **Cross-platform subprocess calls:**
|
|
||||||
- Use `shutil.which()` for tool discovery
|
|
||||||
- Use `pathlib.Path` everywhere instead of string concatenation
|
|
||||||
- Handle Windows vs Unix shell differences
|
|
||||||
|
|
||||||
5. **Windows-specific considerations:**
|
|
||||||
- `nmap` available on Windows (nmap.org installer)
|
|
||||||
- `tshark` available on Windows (Wireshark installer)
|
|
||||||
- `upnpc` available on Windows (miniupnpc builds)
|
|
||||||
- Metasploit available via WSL or native Windows build
|
|
||||||
- WireGuard available on Windows
|
|
||||||
- No `sudo` -- use `ctypes.windll.shell32.IsUserAnAdmin()` check
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 6: Docker Packaging
|
|
||||||
|
|
||||||
**What:** Create Docker configuration for portable deployment with all dependencies bundled.
|
|
||||||
|
|
||||||
**New files:**
|
|
||||||
```
|
|
||||||
~/autarch/
|
|
||||||
├── Dockerfile
|
|
||||||
├── docker-compose.yml
|
|
||||||
├── .dockerignore
|
|
||||||
└── scripts/
|
|
||||||
├── entrypoint.sh # Container startup script
|
|
||||||
└── install-tools.sh # Install nmap, tshark, upnpc, etc.
|
|
||||||
```
|
|
||||||
|
|
||||||
**Dockerfile approach:**
|
|
||||||
- Base: `python:3.11-slim`
|
|
||||||
- Install system tools: `nmap`, `tshark`, `miniupnpc`, `wireguard-tools`
|
|
||||||
- Install Python deps from `requirements.txt`
|
|
||||||
- Copy application code
|
|
||||||
- Expose ports: 8080 (web UI), 55553 (MSF RPC passthrough)
|
|
||||||
- Volume mounts: `/data` (persistent), `/models` (LLM models)
|
|
||||||
- Entrypoint: can start CLI, web, or both
|
|
||||||
|
|
||||||
**docker-compose.yml:**
|
|
||||||
```yaml
|
|
||||||
services:
|
|
||||||
autarch:
|
|
||||||
build: .
|
|
||||||
ports:
|
|
||||||
- "8080:8080"
|
|
||||||
volumes:
|
|
||||||
- ./data:/app/data
|
|
||||||
- ./models:/app/models
|
|
||||||
- ./results:/app/results
|
|
||||||
environment:
|
|
||||||
- AUTARCH_MODE=web # or 'cli' or 'both'
|
|
||||||
network_mode: host # Needed for network scanning tools
|
|
||||||
```
|
|
||||||
|
|
||||||
**requirements.txt** (consolidated):
|
|
||||||
```
|
|
||||||
flask
|
|
||||||
bcrypt
|
|
||||||
pyshark
|
|
||||||
qrcode
|
|
||||||
Pillow
|
|
||||||
requests
|
|
||||||
llama-cpp-python # optional
|
|
||||||
transformers # optional
|
|
||||||
torch # optional
|
|
||||||
anthropic # optional
|
|
||||||
msgpack
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Execution Order
|
|
||||||
|
|
||||||
| Step | Phase | Effort | Description |
|
|
||||||
|------|-------|--------|-------------|
|
|
||||||
| 1 | Phase 0 | 5 min | Backup & create ~/autarch |
|
|
||||||
| 2 | Phase 1 | 1 session | UPnP manager + cron job management |
|
|
||||||
| 3 | Phase 2 | 2-3 sessions | Flask web dashboard skeleton + all pages |
|
|
||||||
| 4 | Phase 3 | 1-2 sessions | OSINT search engine with SSE streaming |
|
|
||||||
| 5 | Phase 4 | 1-2 sessions | Wireshark/tshark integration |
|
|
||||||
| 6 | Phase 5 | 1-2 sessions | Path portability + Windows support |
|
|
||||||
| 7 | Phase 6 | 1 session | Docker packaging |
|
|
||||||
|
|
||||||
**Each phase is self-contained and testable before moving to the next.**
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Verification Plan
|
|
||||||
|
|
||||||
After each phase:
|
|
||||||
1. **Phase 0:** `ls ~/autarch/` matches expected structure, `python3 ~/autarch/autarch.py --help` works
|
|
||||||
2. **Phase 1:** UPnP menu accessible, `upnpc -l` shows mappings, cron installs/uninstalls correctly
|
|
||||||
3. **Phase 2:** `python3 autarch.py --web` starts Flask on :8080, login works, all pages render, CLI still works independently
|
|
||||||
4. **Phase 3:** OSINT search returns results in web UI, SSE streaming shows live progress, results match CLI output
|
|
||||||
5. **Phase 4:** `tshark -D` lists interfaces from web UI, capture start/stop works, PCAP analysis shows results
|
|
||||||
6. **Phase 5:** No hardcoded paths remain (`grep -r '/home/snake' .` returns nothing), app runs from any directory
|
|
||||||
7. **Phase 6:** `docker-compose up` starts the app, web UI accessible, tools available inside container
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Critical Files Reference
|
|
||||||
|
|
||||||
| Existing File | Role | Reuse For |
|
|
||||||
|---|---|---|
|
|
||||||
| `/home/snake/wg_setec/static/css/style.css` | Dark theme CSS | Web UI base theme |
|
|
||||||
| `/home/snake/wg_setec/templates/base.html` | Template inheritance | Web UI base template |
|
|
||||||
| `/home/snake/wg_setec/auth.py` | bcrypt session auth | Web UI authentication |
|
|
||||||
| `/home/snake/wg_setec/upnp-renew.sh` | UPnP port script | Reference for port mappings |
|
|
||||||
| `core/config.py` | Config handler | Add [upnp], [web] sections |
|
|
||||||
| `core/sites_db.py` | Sites DB (7200+ sites) | OSINT search backend |
|
|
||||||
| `modules/recon.py` | OSINT scanner (2193 lines) | OSINT search engine backend |
|
|
||||||
| `core/menu.py` | Module discovery | Auto-discover new modules |
|
|
||||||
| `autarch.py` | Entry point | Add --web, --upnp-refresh flags |
|
|
||||||
Loading…
x
Reference in New Issue
Block a user