Private Remote AI Infrastructure – Security-Focused Design
Remote-BackyardAI is a security-oriented infrastructure experiment designed to explore:
- Private remote access architecture
- SSH hardening strategies
- Attack surface reduction
- Container isolation boundaries
- Local-first AI inference
- Zero public exposure design
This project demonstrates defensive infrastructure thinking rather than feature development.
All model inference runs locally.
Remote access is achieved without exposing public ports.
Tested stable on Apple Silicon Mac (M4).
This is a practical template you can pilot quickly, without building a full web app or exposing a public web endpoint.
It’s built to communicate well to a mixed audience:
- non-technical stakeholders (clear and simple access story)
- security-minded reviewers (explicit boundaries and controls)
- Enable remote access without public internet exposure
- Eliminate password-based authentication
- Prevent interactive shell access
- Restrict service binding scope
- Minimize container responsibility
- Clearly define trust boundaries
Security was considered first, convenience second.
User Device
↓
SSH over Tailscale (Private Mesh Network)
↓
Gateway Container (Forced Command Only)
↓
Docker Internal Network
↓
Ollama Container
↓
Local LLM Inference
[Phone / Laptop]
|
SSH over Tailscale (private mesh)
|
[Gateway Container] (forced command, no shell)
|
Docker internal network
|
[Ollama Container] (local inference)
|
[Local Models]
There is:
- No public HTTP endpoint
- No router port forwarding (recommended)
- No password authentication
- No interactive shell access
ARCHITECTURE.md(system structure and boundaries)THREAT_MODEL.md(assets, threats, mitigations, residual risk)SECURITY.md(security policy and reporting)
- Mature key-based authentication
- Smaller exposed surface
- Avoids common web server attack classes
- Simple revocation model (remove a key, access is gone)
SSH is configured with a forced command (command="..." in authorized_keys):
- Prevents shell access
- Prevents arbitrary command execution
- Limits capability to the chat program only
Principle applied: Least Privilege.
Binding to 0.0.0.0 increases exposure.
Instead, bind the SSH port to your host’s Tailscale IP (100.x):
- Service is reachable only inside the private mesh
- No router port forwarding required
- Reduces scanning and remote exploitation risk
Assets considered:
- Host machine
- Docker daemon
- Gateway container
- Ollama container
- SSH keys
- Tailscale identity
Threats considered:
- Unauthorized remote access
- SSH key compromise
- Container escape / privilege escalation
- Misconfiguration exposure
- Supply chain vulnerabilities
See THREAT_MODEL.md for full analysis.
- SSH key-only authentication (no passwords)
- Forced-command execution (no interactive shell)
- Docker internal networking (gateway can reach Ollama; Ollama not exposed externally)
- Tailscale-only exposure (recommended binding)
- Minimal container responsibility (single-purpose design)
This is not a production-hardened enterprise system.
Residual risks include:
- Compromised user device (stolen SSH key)
- Docker zero-day vulnerability
- Host OS compromise
- Tailscale account compromise
Security is risk reduction, not risk elimination.
If deploying in production, consider:
- Rootless Docker
- Rate limiting / fail2ban-like controls
- Key rotation policy
- Mandatory MFA on Tailscale
- Signed container images
- Vulnerability scanning pipeline
- Centralized logging / SIEM integration
- Intrusion detection monitoring
- Create config:
cp .env.example .env
# edit .env- Start the stack:
docker compose up -d --build- Download a few lightweight models:
docker exec -it ollama ollama pull phi
docker exec -it ollama ollama pull dolphin-phi
docker exec -it ollama ollama listModel catalog: https://ollama.com/library
This is invite-only by design:
- Invite the user into your Tailscale network.
- The user generates an SSH key (Termius recommended).
- Add their public key to a gateway instance.
- They connect to your Tailscale IP and land directly in the chat program.
They never get a Linux shell.
Detailed onboarding:
docs/FRIENDS.md
Create a copy of an existing model with your own system prompt:
docker exec -it ollama sh -lc 'cat > /tmp/Modelfile.custom << "EOF"
FROM dolphin-phi:latest
SYSTEM """You are the company assistant. Be concise and professional."""
EOF'
docker exec -it ollama ollama create company-assistant -f /tmp/Modelfile.custom
docker exec -it ollama ollama list | grep company-assistantDesigned behavior:
- Inference runs locally (Ollama in Docker).
- The gateway can be configured to avoid persisting chat logs.
- Optional log auto-purge (24h retention) can be enabled.
Caveats:
- Tailscale uses a cloud coordination plane (traffic is end-to-end encrypted).
- Termius may sync configuration data if enabled.
- Mobile devices may store scrollback locally (and backups may capture app data).
Stop this stack:
docker compose downStop all containers:
docker stop $(docker ps -q)This repository demonstrates practical experience with:
- Docker networking and isolation
- SSH hardening configuration
- Key-based authentication systems
- Forced command restrictions
- Network exposure control
- Threat modeling and risk analysis
- Secure architecture documentation
- Least privilege design
Pick a license before publishing (MIT / Apache-2.0 / etc.). This template includes an MIT license stub.