Secure self-hosted infrastructure

Context

This environment is not intended to imitate a multi-region enterprise cloud.

It is a single-host personal and development platform used for:

• Software development
• Internal applications
• Source control
• Container images
• Storage
• Monitoring
• PKI
• Security engineering
• Infrastructure experimentation

The design problem is therefore constrained.

It must remain understandable and operable by a small number of people, while avoiding the common default of exposing every administrative interface directly on a home or public network.

The core architectural goal is:

The public network should expose only the access mechanism required to enter the private environment. Individual internal services should not each become separate public ingress points.

This leads to a routed VPN design in which certificate possession establishes network reachability, while individual services continue to enforce their own authentication and authorization.

High-level architecture

The architecture separates access into three contexts:

1. Public network
2. VPN client network
3. Internal service network

These are not merely different address ranges. They represent different trust assumptions.

Decision 1: Expose one management ingress

The OpenVPN endpoint is the intended public entry point for internal administration.

Internal dashboards, source-control interfaces, registries, storage systems, and monitoring services are not intentionally exposed directly to the internet.

This reduces the externally reachable surface.

Instead of separately maintaining internet-facing authentication and TLS configuration for every internal service, remote clients first establish a mutually authenticated VPN session.

The trade-off is concentration.

The VPN endpoint becomes critical infrastructure. If it is unavailable, remote administrative access is unavailable. If its certificate or server configuration is compromised, the attacker may gain network reachability to internal services.

The design therefore depends on:

• Strong certificate lifecycle management
• Revocation
• Limited VPN server exposure
• Service-level authentication after entry
• Monitoring of VPN access
• Recovery access when the VPN fails

The VPN is not a replacement for every other security control.

It is the outer access boundary.

Decision 2: Use mutual TLS for client access

VPN access requires a client certificate issued by the operational intermediate CA.

This provides stronger device or credential identity than a shared VPN password.

Certificate-based access supports:

• Individual revocation
• Certificate expiry
• Per-client identification
• Controlled reissuance
• Separation between server and client profiles
• Audit of certificate lifecycle events

The trade-off is operational overhead.

Certificates must be issued, distributed, renewed, revoked, and tracked. Lost devices and departing users require a reliable revocation workflow. CRL publication becomes part of access control.

This is why the VPN and vpn-certd projects are closely related. Network design and certificate lifecycle cannot be treated as independent concerns.

Decision 3: Use routed access instead of hiding clients behind NAT

VPN clients receive a route to the internal service network. Internal systems maintain an explicit return route to the VPN client network through the VPN gateway.

This preserves end-to-end addressing.

An internal service can see the VPN client’s tunnel address rather than every connection appearing to originate from the gateway.

Advantages include:

• Better attribution
• Clearer firewall rules
• Easier service logging
• Direct bidirectional communication
• Fewer NAT-specific debugging problems

The trade-off is configuration responsibility.

Every internal system or network path must know how to return traffic to the VPN network. A missing return route produces asymmetric connectivity: the request reaches the service, but the response follows the wrong path.

The environment therefore documents the return-route requirement as part of VM onboarding rather than treating it as implicit knowledge.

Decision 4: Use split tunnelling

VPN clients send traffic for internal networks through the tunnel. General internet traffic continues to use the client’s normal internet connection.

Advantages:

• The VPN host does not become a general internet gateway
• Lower bandwidth and processing requirements
• Less impact on ordinary browsing
• Internal access remains focused on its actual purpose
• Failure of the VPN does not remove general internet access

Trade-offs:

• Public internet traffic does not benefit from the VPN’s network path
• Client DNS behaviour must be designed carefully
• A compromised client simultaneously has internet access and internal reachability
• The VPN cannot centrally inspect all client traffic
• Route and DNS configuration must avoid leakage or ambiguity

For this environment, the VPN exists to reach private services, not to provide anonymous or centralized internet egress.

Split tunnelling is therefore appropriate to the stated purpose.

Decision 5: Use private DNS for internal services

VPN clients receive an internal DNS resolver that maps private service names to internal addresses.

The same names may not resolve publicly or may resolve differently outside the VPN.

This creates a split-DNS model.

Advantages:

• Stable service names
• No need to expose private addresses in public DNS
• Easier movement of internal services
• Familiar HTTPS hostnames inside the private environment
• Reduced reliance on memorized addresses

Trade-offs:

• DNS becomes part of the VPN access path
• Resolver failure disrupts service discovery
• Clients may cache internal answers
• Misconfiguration can expose names or create confusing resolution behaviour
• The same domain namespace may represent different views inside and outside the VPN

The design should therefore distinguish between:

• Public records required for external services
• Internal records available only to VPN clients
• Certificates appropriate for each trust context

Management boundary

Access to internal services requires several independent conditions:

1. The client possesses a valid VPN certificate
2. Mutual TLS succeeds
3. The client receives the required route
4. Internal DNS resolves the target where needed
5. The service is reachable on the internal network
6. The service itself authenticates and authorizes the user

The last condition is important.

A VPN certificate establishes that a device or identity may enter the network.

It does not establish that the user may:

• Administer source control
• Read monitoring data
• Manage storage
• Access an internal application
• Modify infrastructure
• Operate the certificate authority

Network access and application authorization are different layers.

The environment deliberately retains service-level authentication rather than treating the VPN as universal authorization.

Internal-only and dual-connected machines

The onboarding model distinguishes two categories of VM.

Internal-only machines

These systems attach only to the internal service network.

They are appropriate for services that:

• Need to be reachable from VPN clients
• Communicate with other internal services
• Do not require direct general internet access
• Should not appear on the home LAN or public network

Advantages:

• Smaller network exposure
• Simpler routing
• Clear trust placement
• Reduced accidental egress

Trade-offs:

• Package updates and external integrations require another controlled path
• Troubleshooting external dependencies is more complex
• Complete isolation may be impractical for some applications

Dual-connected machines

Some workloads require both:

• Direct internet access
• Private service-network access

These machines receive two interfaces.

The external-facing interface owns the default route. The internal interface carries only explicit routes for private networks.

This is a crucial routing rule.

Adding a second default route to the internal interface can produce:

• Ambiguous egress
• Asymmetric routing
• Intermittent connectivity
• DNS confusion
• Traffic leaving through the wrong gateway
• Difficult-to-reproduce failures

The VM-onboarding documentation records this distinction and requires route verification after configuration.

The trade-off of dual attachment is increased attack surface.

A dual-connected workload can potentially bridge an external compromise into the internal service network. Such systems should therefore be treated as higher risk than internal-only VMs.

PKI boundary

The VPN PKI is not embedded directly into the VPN administration interface.

vpn-certd holds the operational intermediate key behind a local Unix socket, while the root remains offline.

This separates:

• Public VPN ingress
• Operator workflows
• Certificate-policy enforcement
• Operational signing
• Root trust

A compromise of an internal web interface does not directly reveal the intermediate key unless the attacker also crosses the daemon or host boundary.

The limitation is that the certificate daemon and VPN gateway remain within the same broader infrastructure environment.

A complete host compromise can cross many local process boundaries.

The architecture therefore improves containment against application-level compromise, not against total hypervisor or host control.

Monitoring design

Internal systems provide metrics to Prometheus and logs to Loki. Grafana acts as the operational interface, while alerts are forwarded to an external notification channel.

The monitoring stack answers several distinct questions:

• Is the host reachable?
• Is resource use abnormal?
• Which containers are running?
• Did a service restart?
• Are logs arriving?
• Did an alert condition occur?
• Which system is degrading before users notice?

Components include:

• Node Exporter for host metrics
• cAdvisor for container metrics
• Promtail for log forwarding
• Prometheus for time-series collection
• Loki for log aggregation
• Grafana for dashboards and alerting

Centralization improves visibility, but it creates another trust concentration.

The monitoring system may reveal:

• Hostnames
• Service topology
• Container names
• Image references
• Log contents
• Resource pressure
• Internal failures

It therefore belongs on the private service network and should not be treated as a public dashboard by default.

Monitoring also does not equal incident response.

Metrics and logs are useful only when:

• Alerts are meaningful
• Someone receives them
• Evidence is retained
• The compromised service cannot erase every record
• Operators know what action to take

A future improvement is stronger off-host or separately controlled security logging.

Repeatable onboarding as a security control

Manual infrastructure often drifts because every new system is configured slightly differently.

The onboarding process documents:

• Hypervisor network attachment
• Address allocation
• Required routes
• DNS registration
• Connectivity tests
• Monitoring agents
• Prometheus registration
• Internal-only versus dual-connected topology

This is more than convenience documentation.

It reduces the likelihood of:

• Missing return routes
• Duplicate addresses
• Undocumented DNS
• Accidental default routes
• Unmonitored systems
• Inconsistent network exposure
• Operator-specific knowledge

The process remains manual, which creates human-error risk.

A future configuration-as-code model could make the intended state enforceable rather than merely documented.

For the current scale, a clear procedure is still materially better than undocumented ad hoc provisioning.

Trust model

The environment does not treat everything behind the VPN as equally trusted.

A more accurate trust model is:

Public attacker
    ↓ must compromise or authenticate to VPN
VPN-authorized client
    ↓ has network reachability, not universal application authority
Internal application user
    ↓ has service-specific permissions
Infrastructure operator
    ↓ may modify systems and routes
Certificate operator
    ↓ may request PKI operations
Certificate daemon
    ↓ controls operational signing
Offline root operator
    ↓ controls hierarchy replacement

Each step represents additional authority.

This matters because “internal” is not a security role.

A compromised VPN client is not equivalent to a trusted infrastructure administrator. The design should preserve that distinction through service authentication, limited credentials, and auditable operations.

Failure modes

VPN gateway unavailable

Remote access to internal services fails.

General client internet access continues because the tunnel is split.

Recovery requires console or local network access to the gateway or hypervisor.

Internal DNS unavailable

VPN routing may still work by address, but named services become difficult to reach.

This makes DNS availability part of the access path even though it is not the security boundary itself.

Proxmox host failure

Most or all internal services become unavailable because the environment is concentrated on one physical host.

This is the largest infrastructure failure domain.

Monitoring node unavailable

Systems may continue operating, but centralized visibility and alerting degrade.

This creates the risk of silent failure rather than immediate service loss.

VPN client compromised

The attacker gains the network reachability associated with that client certificate.

Service-level authentication, certificate revocation, and internal segmentation determine the remaining blast radius.

Dual-connected VM compromised

The attacker may gain a foothold with both internet and internal network access.

These machines require greater scrutiny than internal-only workloads.

Certificate daemon unavailable

New issuance and revocation stop, but existing VPN sessions and certificates may continue functioning.

Hypervisor compromised

Most local separation can be bypassed.

Process boundaries, VM boundaries, and private networks do not protect against an attacker controlling the underlying host.

Security decisions and trade-offs

Decision	Benefit	Trade-off
VPN-only management access	Reduces public administrative surface	Makes the VPN gateway a critical dependency
Mutual-TLS clients	Individual identity, expiry and revocation	Requires certificate lifecycle operations
Routed VPN	Preserves client identity and clear paths	Requires explicit return routes
Split tunnelling	Avoids centralizing all internet traffic	VPN clients retain simultaneous internet access
Private DNS	Stable internal service discovery	Resolver becomes part of the access path
Single Proxmox host	Low cost and operational simplicity	Large hardware and hypervisor failure domain
Central monitoring	Unified operational visibility	Monitoring data becomes sensitive infrastructure information
Manual onboarding procedure	Reduces undocumented configuration drift	Still depends on human execution
Unix-socket CA boundary	Avoids network-facing signing API	Host compromise can cross the boundary
Broad VPN network access	Simple administration	Limited east-west segmentation

What the environment demonstrates

The environment demonstrates:

• Certificate-gated management access
• Mutual-TLS VPN authentication
• Explicit routed connectivity
• Split tunnelling
• Private DNS
• Internal-only service placement
• Separate treatment of dual-connected systems
• Offline-root and operational-intermediate PKI
• Local-only CA control
• Central monitoring and logs
• Repeatable VM onboarding
• Documented network and certificate responsibilities

More importantly, it demonstrates that the architecture was built around stated assumptions rather than a list of technologies.

The design assumes:

• A small number of operators
• A single primary virtualization host
• Limited public services
• Manual but documented operations
• Service-level authentication after VPN access
• No requirement for multi-region availability
• A willingness to trade some automation for understandable control

Those assumptions are reasonable for a personal reference environment.

They would not be sufficient without modification for a high-availability commercial production platform.

Limitations

Single physical failure domain

A complete hypervisor or hardware failure affects most services.

VM separation does not create hardware redundancy.

Broad reachability after VPN access

VPN clients can reach much of the internal service network.

The design relies on service-level authorization rather than full microsegmentation.

Manual configuration

Routes, DNS records, and monitoring registration are documented but not fully enforced through configuration management.

Shared infrastructure trust

Some services share the same host and network environment. A complete host compromise has a broad impact.

Recovery maturity

Automated off-box backups and measured restore exercises remain planned work rather than completed controls.

Artifact provenance

Digest-pinned deployment and complete build provenance are planned improvements, not current guaranteed properties.

Planned evolution

• Kubernetes-based development environments
• Digest-pinned and attributable deployments
• Automated off-box backups
• Measured restore exercises
• Stronger service-to-service identities
• More complete configuration as code
• Formal incident-response exercises
• Greater east-west segmentation
• Reduced dependency on one physical host
• Off-host security logging
• More explicit operator and service credential separation

Status

This is a personal operating environment and reference build.

It is actively used for:

• Software development
• Internal services
• Infrastructure experimentation
• PKI operations
• Monitoring
• Security engineering

The public case study intentionally omits exact addresses, internal hostnames, personal identities, certificate locations, route values, operational credentials, and detailed recovery material.

The value of the project is not that it represents an idealized enterprise platform. It shows the reasoning behind a practical small-scale architecture, the boundaries it establishes, the risks it accepts, and the areas where further engineering would materially improve the system.

← back to work