Enterprise Architecture

This guide covers architectural patterns and best practices for designing and managing large-scale Kubernetes deployments across AWS, Azure, and GCP.

---

Multi-Cluster Architecture Models

Hub and Spoke Model

The hub cluster centrally manages configuration, security policies, and observability for multiple spoke clusters.

                     ┌──────────────┐
                     │  Hub Cluster │
                     │  (Admin/Mgmt)│
                     └───────┬──────┘
                             │
           ┌─────────────────┼─────────────────┐
           │                 │                 │
    ┌──────▼──────┐   ┌──────▼──────┐   ┌──────▼──────┐
    │ Spoke       │   │ Spoke       │   │ Spoke       │
    │ (Workload)  │   │ (Workload)  │   │ (Workload)  │
    └─────────────┘   └─────────────┘   └─────────────┘

Real-life example: Financial services organization with regulated workloads in separate clusters but unified governance.

Multi-Regional Architecture

Independent cluster instances deployed across regions for data sovereignty and resilience.

┌──────────────────────┐  ┌──────────────────────┐  ┌──────────────────────┐
│  US-East Region      │  │  Europe Region       │  │  APAC Region         │
│                      │  │                      │  │                      │
│ ┌────────┐ ┌────────┐│  │ ┌────────┐ ┌────────┐│  │ ┌────────┐ ┌────────┐│
│ │Prod    │ │Staging ││  │ │Prod    │ │Staging ││  │ │Prod    │ │Staging ││
│ │Cluster │ │Cluster ││  │ │Cluster │ │Cluster ││  │ │Cluster │ │Cluster ││
│ └────────┘ └────────┘│  │ └────────┘ └────────┘│  │ └────────┘ └────────┘│
└──────────────────────┘  └──────────────────────┘  └──────────────────────┘

Real-life example: Global SaaS provider maintaining regional data residency while providing uniform service.

---

Cloud-Specific Implementation Patterns

AWS EKS Architecture

┌─────────────────────────────────────────────────────────┐
│                      Region                             │
│ ┌─────────────────┐      ┌─────────────────┐            │
│ │   AZ-1          │      │   AZ-2          │            │
│ │                 │      │                 │            │
│ │ ┌─────────────┐ │      │ ┌─────────────┐ │            │
│ │ │Worker Nodes │ │      │ │Worker Nodes │ │            │
│ │ └─────────────┘ │      │ └─────────────┘ │            │
│ │                 │      │                 │            │
│ └─────────────────┘      └─────────────────┘            │
│                                                         │
│ ┌─────────────────────────────────────────────────────┐ │
│ │                  EKS Control Plane                  │ │
│ └─────────────────────────────────────────────────────┘ │
│                                                         │
│ ┌───────────┐┌──────────┐┌─────────┐┌─────────────────┐ │
│ │ AWS ALB   ││  AWS ECR ││Route 53 ││ CloudWatch Logs │ │
│ └───────────┘└──────────┘└─────────┘└─────────────────┘ │
└─────────────────────────────────────────────────────────┘

Best Practices:

• Use EKS add-ons for CNI, CoreDNS, and kube-proxy

• Leverage AWS Load Balancer Controller for ALB/NLB integration

• Use Node Groups with Auto Scaling Groups

• Implement dedicated VPC endpoints for ECR, S3, and other AWS services

• Configure AWS IAM for Kubernetes RBAC integration

Real-life considerations:

• ALB for ingress offers native integrations with AWS WAF and Shield

• Use Cluster Autoscaler with multiple node groups for cost optimization

• Auto-scaling with Karpenter provides faster node provisioning

Azure AKS Architecture

┌─────────────────────────────────────────────────────────┐
│                      Region                             │
│ ┌─────────────────┐      ┌─────────────────┐            │
│ │   AZ-1          │      │   AZ-2          │            │
│ │                 │      │                 │            │
│ │ ┌─────────────┐ │      │ ┌─────────────┐ │            │
│ │ │ Node Pools  │ │      │ │ Node Pools  │ │            │
│ │ └─────────────┘ │      │ └─────────────┘ │            │
│ │                 │      │                 │            │
│ └─────────────────┘      └─────────────────┘            │
│                                                         │
│ ┌─────────────────────────────────────────────────────┐ │
│ │                  AKS Control Plane                  │ │
│ └─────────────────────────────────────────────────────┘ │
│                                                         │
│ ┌───────────────┐┌─────────────┐┌──────────────────────┐│
│ │Azure App Gtwy ││ Azure Cont. ││ Azure Monitor        ││
│ │               ││ Registry    ││ Container Insights   ││
│ └───────────────┘└─────────────┘└──────────────────────┘│
└─────────────────────────────────────────────────────────┘

Best Practices:

• Enable managed identity and RBAC integration

• Implement Azure CNI networking for enterprise-scale deployments

• Use separate node pools for system and application workloads

• Configure CSI drivers for Azure Disk and File storage

• Leverage Azure Policy for AKS

Real-life considerations:

• Application Gateway Ingress Controller for WAF capabilities

• Azure Container Registry with geo-replication for multi-region deployments

• Use Virtual Node (with Azure Container Instances) for burst workloads

GCP GKE Architecture

┌─────────────────────────────────────────────────────────┐
│                      Region                             │
│ ┌─────────────────┐      ┌─────────────────┐            │
│ │   Zone A        │      │   Zone B        │            │
│ │                 │      │                 │            │
│ │ ┌─────────────┐ │      │ ┌─────────────┐ │            │
│ │ │ Node Pools  │ │      │ │ Node Pools  │ │            │
│ │ └─────────────┘ │      │ └─────────────┘ │            │
│ │                 │      │                 │            │
│ └─────────────────┘      └─────────────────┘            │
│                                                         │
│ ┌─────────────────────────────────────────────────────┐ │
│ │                  GKE Control Plane                  │ │
│ └─────────────────────────────────────────────────────┘ │
│                                                         │
│ ┌─────────────┐┌─────────────┐┌───────────────────────┐ │
│ │Cloud Load   ││ Container   ││ Cloud Monitoring      │ │
│ │Balancing    ││ Registry    ││ & Logging             │ │
│ └─────────────┘└─────────────┘└───────────────────────┘ │
└─────────────────────────────────────────────────────────┘

Best Practices:

• Use GKE Autopilot for simplified operations

• Enable GKE Standard clusters with node auto-provisioning

• Implement Workload Identity for secure GCP API access

• Configure Cloud NAT for private GKE clusters

• Use Binary Authorization for supply chain security

Real-life considerations:

• Multi-cluster ingress and service mesh with Cloud Service Mesh

• GKE Enterprise for enhanced multi-cluster management

• Container-Optimized OS for improved security posture

---

Multi-Cloud Kubernetes Architecture

For organizations operating across multiple clouds, these patterns enable consistent management:

Fleet Management Approach

┌─────────────────────────────────────────────────────────────┐
│                  Central Management Plane                   │
│                                                             │
│  ┌───────────────┐  ┌───────────────┐  ┌────────────────┐   │
│  │GitOps System  │  │Fleet Manager  │  │Centralized     │   │
│  │(Flux/ArgoCD)  │  │(e.g., Rancher)│  │Observability   │   │
│  └───────┬───────┘  └───────┬───────┘  └────────┬───────┘   │
└─────────────────────────────────────────────────────────────┘
           │                  │                    │
           ▼                  ▼                    ▼
┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
│  AWS EKS        │  │  Azure AKS      │  │  Google GKE     │
│  Clusters       │  │  Clusters       │  │  Clusters       │
└─────────────────┘  └─────────────────┘  └─────────────────┘

Implementation strategies:

• Unified configuration repository with environment-specific overlays

• Federation layer for cross-cluster service discovery

• Standardized CRDs across all clusters

• Central identity management with federation to cloud IAM systems

• Common observability and alerting platform

---

Network Architecture Models

Multi-Tier Network Security Model

┌────────────────────────────────────────────┐
│ Ingress Tier                               │
│ ┌──────────────────────────────────────┐   │
│ │WAF & DDoS Protection                 │   │
│ └────────────────┬─────────────────────┘   │
│                  │                         │
│ ┌────────────────▼─────────────────────┐   │
│ │API Gateway / Ingress Controller      │   │
│ └────────────────┬─────────────────────┘   │
└──────────────────┼─────────────────────────┘
                   │
┌──────────────────▼─────────────────────────┐
│ Service Mesh                               │
│ ┌──────────────────────────────────────┐   │
│ │mTLS Encryption                       │   │
│ │Traffic Management                    │   │
│ │Service-to-Service Authorization      │   │
│ └────────────────┬─────────────────────┘   │
└──────────────────┼─────────────────────────┘
                   │
┌──────────────────▼─────────────────────────┐
│ Pod Security                               │
│ ┌──────────────────────────────────────┐   │
│ │Pod Security Policies/Standards       │   │
│ │Network Policies                      │   │
│ │Runtime Security                      │   │
│ └──────────────────────────────────────┘   │
└────────────────────────────────────────────┘

Implementation components:

• AWS: ALB + AWS Shield + WAF + AppMesh/Istio + Calico

• Azure: App Gateway + Azure Firewall + Istio/Linkerd + Azure CNI + Calico

• GCP: Cloud Load Balancer + Cloud Armor + Anthos Service Mesh + Calico

---

Storage Architecture Best Practices

Data-Intensive Workload Architecture

┌────────────────────────────────────────────────────────────┐
│ Stateful Application Deployment                            │
│                                                            │
│ ┌──────────────────────────┐  ┌─────────────────────────┐  │
│ │ StatefulSet              │  │ Operator-managed DB     │  │
│ │                          │  │                         │  │
│ │ ┌────────────┐           │  │ ┌─────────────────────┐ │  │
│ │ │PVC Templates│           │  │ │Custom Resource Def.│ │  │
│ │ └─────┬──────┘           │  │ └──────────┬──────────┘ │  │
│ └───────┼─────────────────┘   └────────────┼────────────┘  │
│         │                                  │               │
│ ┌───────▼──────────────────────────────────▼──────────────┐│
│ │             Storage Class Abstraction                   ││
│ └─────────────────────────┬────────────────────────────────┘
│                           │                                │
│ ┌─────────────────────────▼────────────────────────────────┐
│ │          Cloud Provider Storage Integration              │
│ │                                                          │
│ │ AWS: EBS, EFS, FSx      Azure: Disk, Files    GCP: PD   │
│ └──────────────────────────────────────────────────────────┘
└────────────────────────────────────────────────────────────┘

Cloud-specific recommendations:

• AWS: Use gp3 volumes for general workloads, io2 for high-performance databases

• Azure: Use Premium SSD v2 for dynamic scaling of performance

• GCP: Use Regional Persistent Disks for high-availability storage

---

Multi-Tenancy Models

Hard Multi-tenancy

Separate clusters for each tenant ensure complete isolation.

┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
│Tenant A Cluster │  │Tenant B Cluster │  │Tenant C Cluster │
└─────────────────┘  └─────────────────┘  └─────────────────┘
        │                   │                    │
┌───────▼───────────────────▼────────────────────▼───────┐
│               Central Management Plane                 │
│ (Configuration, Monitoring, Security Policy, Billing)  │
└─────────────────────────────────────────────────────────┘

Soft Multi-tenancy

Namespace-based isolation within a shared cluster.

┌───────────────────────────────────────────────────────┐
│                   Shared Cluster                      │
│                                                       │
│ ┌─────────────┐   ┌─────────────┐   ┌─────────────┐   │
│ │Namespace A  │   │Namespace B  │   │Namespace C  │   │
│ │(Tenant A)   │   │(Tenant B)   │   │(Tenant C)   │   │
│ │             │   │             │   │             │   │
│ │ ┌─────────┐ │   │ ┌─────────┐ │   │ ┌─────────┐ │   │
│ │ │Resource │ │   │ │Resource │ │   │ │Resource │ │   │
│ │ │Quotas   │ │   │ │Quotas   │ │   │ │Quotas   │ │   │
│ │ └─────────┘ │   │ └─────────┘ │   │ └─────────┘ │   │
│ │             │   │             │   │             │   │
│ │ ┌─────────┐ │   │ ┌─────────┐ │   │ ┌─────────┐ │   │
│ │ │Network  │ │   │ │Network  │ │   │ │Network  │ │   │
│ │ │Policies │ │   │ │Policies │ │   │ │Policies │ │   │
│ │ └─────────┘ │   │ └─────────┘ │   │ └─────────┘ │   │
│ └─────────────┘   └─────────────┘   └─────────────┘   │
└───────────────────────────────────────────────────────┘

Implementation tools:

• Hierarchical namespace controller

• Network policies with advanced CNI implementations

• OPA Gatekeeper or Kyverno for policy enforcement

• ResourceQuotas and LimitRanges

• Pod Security Standards

---

Control Plane Scaling Considerations

API Server Scaling

Maximum number of clusters:

• AWS EKS: 100 clusters per region per account (soft limit)

• Azure AKS: 1000 clusters per subscription (soft limit)

• GCP GKE: 50 clusters per project (soft limit)

Maximum nodes per cluster:

• AWS EKS: 5,000 nodes

• Azure AKS: 5,000 nodes

• GCP GKE: 15,000 nodes

API server recommendations:

• Implement efficient watch caches

• Use server-side filtering of list requests

• Optimize etcd for large clusters

• Consider specialized control plane scaling for >5000 nodes

---

Disaster Recovery Architecture

Multi-Region Active-Passive Pattern

┌─────────────────────────┐         ┌─────────────────────────┐
│   Primary Region        │         │   Secondary Region      │
│                         │         │                         │
│ ┌─────────────────────┐ │         │ ┌─────────────────────┐ │
│ │ Active K8s Cluster  │ │         │ │ Passive K8s Cluster │ │
│ └─────────┬───────────┘ │         │ └─────────┬───────────┘ │
│           │             │         │           │             │
│ ┌─────────▼───────────┐ │         │ ┌─────────▼───────────┐ │
│ │Database Primary     │◄├─Sync/Async─┤Database Replica    │ │
│ └─────────────────────┘ │         │ └─────────────────────┘ │
└─────────────────────────┘         └─────────────────────────┘
            │                                   ▲
            │                                   │
            │         ┌───────────────┐         │
            └────────►│ Global Load   │◄────────┘
                      │ Balancer      │
                      └───────────────┘

Recovery strategies:

• Regular etcd snapshots with cross-region backup

• GitOps-driven configuration ensures consistent redeployment

• Stateful data replication with appropriate consistency models

• DNS or global load balancer for traffic redirection

---

Cost Optimization Architecture

Cost-Efficient Node Design

┌───────────────────────────────────────────────────┐
│                  Kubernetes Cluster                │
│                                                   │
│  ┌────────────────────┐    ┌────────────────────┐ │
│  │  System Node Pool  │    │  General Workload  │ │
│  │  (On-demand)       │    │  (Spot/Preemptible)│ │
│  └────────────────────┘    └────────────────────┘ │
│                                                   │
│  ┌────────────────────┐    ┌────────────────────┐ │
│  │  Memory-Optimized  │    │  Compute-Optimized │ │
│  │  (Critical DBs)    │    │  (Batch Processing)│ │
│  └────────────────────┘    └────────────────────┘ │
└───────────────────────────────────────────────────┘

Cloud-specific recommendations:

• AWS: Mix Spot Instances with On-Demand and Savings Plans

• Azure: Use Spot VMs with AKS and Azure Reservations

• GCP: Combine Spot VMs with Committed Use Discounts

Optimization techniques:

• Cluster autoscaler with scale-down rules

• Pod Priority and Preemption for critical workloads

• Right-sizing deployments with VPA

• Implement node auto-provisioning

• Schedule non-critical batch jobs during off-peak hours

---

References

• Kubernetes Patterns

• AWS EKS Best Practices Guide

• Azure AKS Production Baseline

• Google GKE Enterprise Best Practices

• CNCF Cloud Native Trail Map

• Kubernetes SIG Multi-Cluster