Kubernetes Service Mesh | Crusader Two-One

Overview

A service mesh is a dedicated infrastructure layer for handling service-to-service communication within a distributed application architecture. It provides capabilities like traffic management, security, and observability without requiring changes to application code.

Service Mesh Fundamentals

What is a Service Mesh?

A service mesh consists of:

Data Plane - Network proxies (sidecars) handling service communication
Control Plane - Management layer configuring proxies and collecting telemetry
Service Discovery - Automatic detection and registration of services
Load Balancing - Intelligent traffic distribution
Security - mTLS, authentication, and authorization
Observability - Metrics, logs, and tracing

Benefits of Service Mesh

Key advantages:

Traffic Management - Advanced routing, load balancing, circuit breaking
Security - Zero-trust networking with mTLS by default
Observability - Comprehensive metrics and tracing
Policy Enforcement - Consistent security and compliance policies
Resilience - Fault injection, timeouts, and retries
Multi-cluster Support - Service communication across clusters

Istio Service Mesh

Istio Architecture

Istio is the most popular service mesh implementation:

# istio-installation.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-control-plane
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster1
      network: network1
  components:
    pilot:
      k8s:
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
    proxy:
      k8s:
        resources:
          requests:
            cpu: 10m
            memory: 128Mi
    gateways:
      istio-ingressgateway:
        k8s:
          service:
            type: LoadBalancer

Installing Istio

Step-by-step installation:

# Download and install Istio
curl -L https://istio.io/downloadIstio | sh -
export PATH=$PWD/istio-1.20.0/bin:$PATH

# Install Istio with demo profile
istioctl install --set values.defaultRevision=default -y

# Enable automatic sidecar injection
kubectl label namespace default istio-injection=enabled

# Verify installation
kubectl get pods -n istio-system
istioctl verify-install

Sidecar Injection

Enable automatic sidecar proxy injection:

# sidecar-injection.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    istio-injection: enabled
---
# Manual sidecar injection for specific deployments
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
      annotations:
        sidecar.istio.io/inject: "true"
    spec:
      containers:
      - name: myapp
        image: myapp:v1.0
        ports:
        - containerPort: 8080

Traffic Management

Virtual Services

Control request routing and traffic splitting:

# virtual-service.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews-route
  namespace: bookinfo
spec:
  hosts:
  - reviews
  http:
  - match:
    - headers:
        end-user:
          exact: jason
    route:
    - destination:
        host: reviews
        subset: v2
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 80
    - destination:
        host: reviews
        subset: v3
      weight: 20

Destination Rules

Configure service subsets and load balancing:

# destination-rules.yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: reviews-destination
  namespace: bookinfo
spec:
  host: reviews
  trafficPolicy:
    loadBalancer:
      simple: LEAST_CONN
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 10
        maxRequestsPerConnection: 2
    circuitBreaker:
      consecutiveErrors: 3
      interval: 30s
      baseEjectionTime: 30s
  subsets:
  - name: v1
    labels:
      version: v1
    trafficPolicy:
      portLevelSettings:
      - port:
          number: 9080
        loadBalancer:
          simple: ROUND_ROBIN
  - name: v2
    labels:
      version: v2
  - name: v3
    labels:
      version: v3

Gateways

Configure ingress and egress traffic:

# istio-gateway.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: bookinfo-gateway
  namespace: bookinfo
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - bookinfo.example.com
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: bookinfo-tls
    hosts:
    - bookinfo.example.com
---
# Virtual Service for Gateway
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: bookinfo-vs
  namespace: bookinfo
spec:
  hosts:
  - bookinfo.example.com
  gateways:
  - bookinfo-gateway
  http:
  - match:
    - uri:
        exact: /productpage
    - uri:
        prefix: /static
    - uri:
        exact: /login
    - uri:
        exact: /logout
    - uri:
        prefix: /api/v1/products
    route:
    - destination:
        host: productpage
        port:
          number: 9080

Security Policies

Mutual TLS (mTLS)

Enforce encrypted communication between services:

# mtls-policy.yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT
---
# Mesh-wide mTLS policy
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

Authorization Policies

Control access between services:

# authorization-policy.yaml
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: productpage-authorization
  namespace: bookinfo
spec:
  selector:
    matchLabels:
      app: productpage
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/bookinfo/sa/default"]
  - to:
    - operation:
        methods: ["GET", "POST"]
        paths: ["/productpage", "/api/*"]
  - when:
    - key: source.ip
      values: ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
---
# Deny-all policy (default deny)
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: production
spec: {}

JWT Authentication

Validate JSON Web Tokens:

# jwt-authentication.yaml
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: jwt-authentication
  namespace: bookinfo
spec:
  selector:
    matchLabels:
      app: productpage
  jwtRules:
  - issuer: "https://accounts.google.com"
    jwksUri: "https://www.googleapis.com/oauth2/v3/certs"
    audiences:
    - "bookinfo-app"
---
# Authorization policy requiring JWT
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: require-jwt
  namespace: bookinfo
spec:
  selector:
    matchLabels:
      app: productpage
  action: ALLOW
  rules:
  - when:
    - key: request.auth.claims[iss]
      values: ["https://accounts.google.com"]

Observability

Telemetry Configuration

Configure metrics, logs, and traces:

# telemetry-config.yaml
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: default
  namespace: istio-system
spec:
  metrics:
  - providers:
    - name: prometheus
  - overrides:
    - match:
        metric: ALL_METRICS
      tagOverrides:
        destination_service_name:
          value: "{{.destination_service_name | default \"unknown\"}}"
  accessLogging:
  - providers:
    - name: otel
  tracing:
  - providers:
    - name: jaeger

Distributed Tracing

Configure Jaeger for distributed tracing:

# jaeger-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
  namespace: istio-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
    spec:
      containers:
      - name: jaeger
        image: jaegertracing/all-in-one:1.35
        env:
        - name: COLLECTOR_ZIPKIN_HOST_PORT
          value: ":9411"
        ports:
        - containerPort: 16686
          name: http-query
        - containerPort: 14268
          name: http-collector
        - containerPort: 9411
          name: zipkin
        readinessProbe:
          httpGet:
            path: /
            port: 14269
          initialDelaySeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: jaeger-query
  namespace: istio-system
spec:
  ports:
  - name: http-query
    port: 16686
    targetPort: 16686
  selector:
    app: jaeger

Prometheus Metrics

Configure Prometheus for service mesh metrics:

# prometheus-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: istio-system
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
    - job_name: 'istio-mesh'
      kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names:
          - istio-system
          - production
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: istio-proxy;http-monitoring
    - job_name: 'istio-policy'
      kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names:
          - istio-system
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: istio-policy;http-monitoring

Advanced Traffic Management

Canary Deployments

Gradual rollout of new service versions:

# canary-deployment.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews-canary
  namespace: bookinfo
spec:
  hosts:
  - reviews
  http:
  - match:
    - headers:
        canary:
          exact: "true"
    route:
    - destination:
        host: reviews
        subset: v2
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 95
    - destination:
        host: reviews
        subset: v2
      weight: 5
---
# Canary destination rule
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: reviews-canary-dr
  namespace: bookinfo
spec:
  host: reviews
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
    trafficPolicy:
      connectionPool:
        tcp:
          maxConnections: 10
        http:
          http1MaxPendingRequests: 5
          maxRequestsPerConnection: 1
      circuitBreaker:
        consecutiveErrors: 2
        interval: 10s
        baseEjectionTime: 10s

Fault Injection

Test service resilience:

# fault-injection.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: ratings-fault
  namespace: bookinfo
spec:
  hosts:
  - ratings
  http:
  - match:
    - headers:
        end-user:
          exact: jason
    fault:
      delay:
        percentage:
          value: 100.0
        fixedDelay: 7s
    route:
    - destination:
        host: ratings
        subset: v1
  - fault:
      abort:
        percentage:
          value: 10.0
        httpStatus: 500
    route:
    - destination:
        host: ratings
        subset: v1

Traffic Mirroring

Shadow traffic for testing:

# traffic-mirroring.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin-mirror
  namespace: test
spec:
  hosts:
  - httpbin
  http:
  - route:
    - destination:
        host: httpbin
        subset: v1
      weight: 100
    mirror:
      host: httpbin
      subset: v2
    mirrorPercentage:
      value: 100.0

Multi-Cluster Service Mesh

Cross-Cluster Communication

Configure service mesh across multiple clusters:

# cross-cluster-config.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: primary-cluster
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: primary
      network: network1
  components:
    pilot:
      env:
        EXTERNAL_ISTIOD: true
---
# Secondary cluster configuration
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: secondary-cluster
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: secondary
      network: network2
      remotePilotAddress: ${DISCOVERY_ADDRESS}
  components:
    pilot:
      env:
        EXTERNAL_ISTIOD: true

Service Endpoints Discovery

Configure cross-cluster service discovery:

# Create secret for cluster access
kubectl create secret generic cacerts -n istio-system \
  --from-file=root-cert.pem \
  --from-file=cert-chain.pem \
  --from-file=ca-cert.pem \
  --from-file=ca-key.pem

# Label secret for cross-cluster discovery
kubectl label secret cacerts -n istio-system istio/multiCluster=prepare

# Install cross-cluster discovery
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: cross-network-gateway
  namespace: istio-system
spec:
  selector:
    istio: eastwestgateway
  servers:
  - port:
      number: 15443
      name: tls
      protocol: TLS
    tls:
      mode: ISTIO_MUTUAL
    hosts:
    - "*.local"
EOF

Performance Optimization

Resource Management

Optimize sidecar proxy resources:

# sidecar-resource-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: istio-sidecar-injector
  namespace: istio-system
data:
  config: |
    policy: enabled
    alwaysInjectSelector:
      []
    neverInjectSelector:
      []
    template: |
      spec:
        containers:
        - name: istio-proxy
          image: docker.io/istio/proxyv2:1.20.0
          resources:
            requests:
              cpu: 10m
              memory: 40Mi
            limits:
              cpu: 100m
              memory: 128Mi
          env:
          - name: ISTIO_META_INTERCEPTION_MODE
            value: "REDIRECT"
          - name: PILOT_CERT_PROVIDER
            value: istiod

Connection Pooling

Optimize connection management:

# connection-pool-config.yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: circuit-breaker
  namespace: production
spec:
  host: productpage
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
        connectTimeout: 30s
        keepAlive:
          time: 7200s
          interval: 75s
      http:
        http1MaxPendingRequests: 64
        http2MaxRequests: 1000
        maxRequestsPerConnection: 10
        maxRetries: 3
        consecutiveGatewayErrors: 5
        interval: 30s
        baseEjectionTime: 30s
        maxEjectionPercent: 50
        minHealthPercent: 30

Troubleshooting

Common Issues and Solutions

# Check Istio installation
istioctl verify-install

# Analyze proxy configuration
istioctl proxy-config cluster <pod-name> -n <namespace>
istioctl proxy-config listener <pod-name> -n <namespace>
istioctl proxy-config route <pod-name> -n <namespace>

# Debug sidecar injection
kubectl get pods -o jsonpath='{.items[*].spec.containers[*].name}'
istioctl analyze -n <namespace>

# Check mTLS status
istioctl authn tls-check <pod-name>.<namespace>.svc.cluster.local

# Proxy logs
kubectl logs <pod-name> -c istio-proxy -n <namespace>

# Configuration validation
istioctl validate -f virtual-service.yaml

Performance Debugging

# Check proxy metrics
kubectl exec <pod-name> -c istio-proxy -- curl localhost:15000/stats

# Analyze traffic
istioctl proxy-config endpoints <pod-name> -n <namespace>

# Check certificates
istioctl proxy-config secret <pod-name> -n <namespace>

# Pilot debug
kubectl logs -n istio-system deployment/istiod

Best Practices

Security Recommendations

Enable mTLS by default - Use STRICT mode for all services
Implement least privilege - Use specific authorization policies
Regular certificate rotation - Automate certificate management
Network segmentation - Use namespaces and network policies
Audit access patterns - Monitor and log all service communications

Performance Guidelines

Resource sizing - Right-size sidecar proxy resources
Connection pooling - Configure appropriate connection limits
Circuit breaking - Implement failure handling mechanisms
Observability overhead - Balance monitoring detail with performance
Gradual rollouts - Use canary deployments for safety

Operational Excellence

GitOps workflows - Version control all Istio configurations
Automated testing - Test traffic policies in staging environments
Monitoring setup - Implement comprehensive observability
Backup strategies - Maintain configuration backups
Disaster recovery - Plan for control plane failures

Kubernetes Networking - Network fundamentals
Ingress - HTTP/HTTPS routing
Certificate Management - TLS certificate automation
Container Security - Security best practices
Monitoring - Observability and monitoring

This guide covers service mesh architecture and Istio implementation from basic concepts to advanced multi-cluster configurations. Service mesh provides powerful capabilities for managing microservices communication, security, and observability.

Table of Contents