Designing and migrating to OpenShift is not about installing a cluster. It is about controlling failure domains, aligning schedulers, and avoiding hidden infrastructure bottlenecks that only surface under load or during outages.

This post walks through concrete implementation patterns using Terraform and Ansible, explains why they exist, and highlights what will break if you get them wrong.

---

Migration Strategy: Phased Approach

Every failed migration I have seen skipped or compressed one of these phases. The pressure to “just move it” creates technical debt that surfaces as production incidents.

Phase 1: Discovery and Assessment

Before touching infrastructure, you need a complete inventory of what exists and how it behaves.

# VMware dependency discovery script
# Export VM metadata, network connections, storage mappings

$vms = Get-VM | Select-Object Name, PowerState, NumCpu, MemoryGB,
  @{N='Datastore';E={(Get-Datastore -VM $_).Name}},
  @{N='Network';E={(Get-NetworkAdapter -VM $_).NetworkName}},
  @{N='VMHost';E={$_.VMHost.Name}}

$vms | Export-Csv -Path "vm-inventory.csv" -NoTypeInformation

# Capture network flows for dependency mapping
# Run for minimum 2 weeks to capture batch jobs and monthly processes

What you are looking for:

• Hard-coded IPs in application configs
• NFS mounts and shared storage dependencies
• Inter-VM communication patterns (what talks to what)
• Authentication integrations (LDAP, AD, service accounts)
• Scheduled jobs and their timing dependencies

Assessment deliverables:

• Application dependency map
• Containerization readiness score per workload
• Risk register with mitigation strategies
• Estimated effort per application (T-shirt sizing)

Phase 2: Target Architecture Design

Design the OpenShift environment before migration begins. This includes cluster topology, namespace strategy, and resource quotas.

# Namespace strategy example
# Environments separated by namespace, not cluster

apiVersion: v1
kind: Namespace
metadata:
  name: app-prod
  labels:
    environment: production
    cost-center: "12345"
    data-classification: confidential
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: app-prod-quota
  namespace: app-prod
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 80Gi
    limits.cpu: "80"
    limits.memory: 160Gi
    persistentvolumeclaims: "20"

Phase 3: Pilot Migration

Select two to three non-critical applications that exercise different patterns:

• One stateless web application
• One application with persistent storage
• One application with external integrations

The pilot validates your tooling, processes, and assumptions before you scale.

Phase 4: Wave Migration

Group applications into waves based on dependencies and risk. Each wave should be independently deployable and rollback-capable.

# Wave planning structure
wave_1:
  applications:
    - name: static-website
      risk: low
      dependencies: none
      estimated_downtime: 0
  success_criteria:
    - all pods healthy for 24 hours
    - response times within 10% of baseline
    - zero error rate increase

wave_2:
  applications:
    - name: api-gateway
      risk: medium
      dependencies:
        - static-website
      estimated_downtime: 5 minutes
  gate:
    - wave_1 success criteria met
    - stakeholder sign-off

Phase 5: Cutover and Decommission

Final traffic switch and legacy teardown. This is where DNS TTL planning matters.

Common cutover failures:

• DNS TTLs not reduced in advance (reduce to 60 seconds, 48 hours before cutover)
• Client-side caching ignoring TTL
• Hardcoded IPs in partner systems
• Certificate mismatches after DNS change

---

VMware to OpenShift: Migration Patterns

Not every VM becomes a container. The migration pattern depends on application architecture, not convenience.

Pattern 1: Lift and Containerize

For applications that are already 12-factor compliant or close to it. Package existing binaries into containers with minimal modification.

# Dockerfile for legacy Java application
FROM registry.access.redhat.com/ubi8/openjdk-11-runtime

COPY target/app.jar /deployments/app.jar

ENV JAVA_OPTS="-Xms512m -Xmx2048m"

EXPOSE 8080
CMD ["java", "-jar", "/deployments/app.jar"]

When to use: Application reads config from environment variables, logs to stdout, and has no local state.

Pattern 2: Replatform with Refactoring

Application requires changes to run in containers but core logic remains. Typical changes include externalizing configuration and adding health endpoints.

# Spring Boot health endpoint addition
management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus
  endpoint:
    health:
      probes:
        enabled: true
      show-details: always

When to use: Application has some container-unfriendly patterns (file-based config, local logging) but is otherwise sound.

Pattern 3: Retain on VM

Some workloads should not be containerized:

• Legacy applications with kernel dependencies
• Workloads requiring specific hardware (GPU passthrough, SR-IOV)
• Applications with licensing tied to VM or physical host
• Databases with extreme I/O requirements (evaluate case by case)

OpenShift Virtualization (KubeVirt) can run VMs alongside containers when needed.

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: legacy-app-vm
spec:
  running: true
  template:
    spec:
      domain:
        cpu:
          cores: 4
        memory:
          guest: 8Gi
        devices:
          disks:
            - name: rootdisk
              disk:
                bus: virtio
      volumes:
        - name: rootdisk
          persistentVolumeClaim:
            claimName: legacy-app-pvc

Pattern 4: Rebuild or Replace

Application is fundamentally incompatible and would require complete rewrite. Evaluate whether a commercial off-the-shelf replacement makes more sense.

Decision matrix:

Factor	Containerize	Keep on VM	Replace
Strategic value	High	Low/Legacy	Medium
Maintenance cost	Acceptable	High but stable	Unsustainable
12-factor compliance	Partial or full	None	N/A
Vendor support	Available	Legacy only	N/A

---

Infrastructure Provisioning with Terraform

Why Terraform First

OpenShift installation assumes the underlying infrastructure is deterministic. If VM placement, CPU topology, or networking varies between environments, the cluster will behave differently under identical workloads. Terraform is used to lock infrastructure intent before OpenShift ever runs.

---

Example: vSphere Control Plane and Worker Provisioning

provider "vsphere" {
  user           = var.vsphere_user
  password       = var.vsphere_password
  vsphere_server = var.vsphere_server
  allow_unverified_ssl = true
}

data "vsphere_datacenter" "dc" {
  name = var.datacenter
}

data "vsphere_compute_cluster" "cluster" {
  name          = var.cluster
  datacenter_id = data.vsphere_datacenter.dc.id
}

data "vsphere_datastore" "datastore" {
  name          = var.datastore
  datacenter_id = data.vsphere_datacenter.dc.id
}

data "vsphere_network" "network" {
  name          = var.network
  datacenter_id = data.vsphere_datacenter.dc.id
}

resource "vsphere_virtual_machine" "control_plane" {
  count            = 3
  name             = "ocp-master-${count.index}"
  resource_pool_id = data.vsphere_compute_cluster.cluster.resource_pool_id
  datastore_id     = data.vsphere_datastore.datastore.id
  folder           = var.vm_folder

  num_cpus = 8
  memory   = 32768
  guest_id = "rhel8_64Guest"

  # Critical: Reservations prevent resource contention
  cpu_reservation    = 8000
  memory_reservation = 32768

  # Anti-affinity rule reference
  depends_on = [vsphere_compute_cluster_vm_anti_affinity_rule.control_plane_anti_affinity]

  network_interface {
    network_id = data.vsphere_network.network.id
  }

  disk {
    label            = "root"
    size             = 120
    thin_provisioned = false  # Thick provisioning for control plane
  }

  disk {
    label            = "etcd"
    size             = 100
    thin_provisioned = false
    unit_number      = 1
  }
}

# Anti-affinity ensures control plane nodes run on different hosts
resource "vsphere_compute_cluster_vm_anti_affinity_rule" "control_plane_anti_affinity" {
  name               = "ocp-control-plane-anti-affinity"
  compute_cluster_id = data.vsphere_compute_cluster.cluster.id
  virtual_machine_ids = [for vm in vsphere_virtual_machine.control_plane : vm.id]
}

CPU and memory reservations are not optional. Without them, vSphere ballooning and scheduling delays will surface as random etcd latency and API instability.

What usually breaks:

• etcd timeouts under load (etcd requires consistent sub-10ms disk latency)
• API server flapping during node pressure
• Unexplained cluster degradation after vMotion events
• Split-brain scenarios when anti-affinity is not enforced

---

Worker Node Pools by Workload Type

resource "vsphere_virtual_machine" "workers_general" {
  count = 6
  name  = "ocp-worker-general-${count.index}"

  num_cpus = 8
  memory   = 32768

  # General workers can use thin provisioning
  disk {
    label            = "root"
    size             = 120
    thin_provisioned = true
  }
}

resource "vsphere_virtual_machine" "workers_stateful" {
  count = 3
  name  = "ocp-worker-stateful-${count.index}"

  num_cpus = 16
  memory   = 65536

  # Stateful workers need guaranteed resources
  cpu_reservation    = 16000
  memory_reservation = 65536

  disk {
    label            = "root"
    size             = 120
    thin_provisioned = false
  }
}

resource "vsphere_virtual_machine" "workers_infra" {
  count = 3
  name  = "ocp-worker-infra-${count.index}"

  num_cpus = 8
  memory   = 32768

  # Infrastructure nodes for routers, monitoring, logging
  disk {
    label            = "root"
    size             = 200
    thin_provisioned = false
  }
}

Different workloads require different failure and performance characteristics. Trying to “let Kubernetes figure it out” leads to noisy neighbors and unpredictable latency.

---

Post-Provision Configuration with Ansible

Why Ansible Is Still Required

Terraform stops at infrastructure. OpenShift nodes require OS-level hardening, kernel tuning, and configuration consistency before installation. Ignoring this step leads to subtle instability that manifests weeks later.

---

Example: Node OS Hardening

---
- name: Prepare OpenShift nodes
  hosts: openshift_nodes
  become: true
  tasks:

    - name: Disable swap
      command: swapoff -a
      changed_when: false

    - name: Remove swap from fstab
      replace:
        path: /etc/fstab
        regexp: '^([^#].*swap.*)$'
        replace: '# \1'

    - name: Set kernel parameters for OpenShift
      sysctl:
        name: "{{ item.key }}"
        value: "{{ item.value }}"
        state: present
        sysctl_file: /etc/sysctl.d/99-openshift.conf
      loop:
        - { key: net.ipv4.ip_forward, value: 1 }
        - { key: net.bridge.bridge-nf-call-iptables, value: 1 }
        - { key: net.bridge.bridge-nf-call-ip6tables, value: 1 }
        - { key: vm.max_map_count, value: 262144 }
        - { key: fs.inotify.max_user_watches, value: 1048576 }
        - { key: fs.inotify.max_user_instances, value: 8192 }
        - { key: net.core.somaxconn, value: 32768 }
        - { key: net.ipv4.tcp_max_syn_backlog, value: 32768 }

    - name: Load required kernel modules
      modprobe:
        name: "{{ item }}"
        state: present
      loop:
        - br_netfilter
        - overlay
        - ip_vs
        - ip_vs_rr
        - ip_vs_wrr
        - ip_vs_sh

    - name: Ensure kernel modules load on boot
      copy:
        dest: /etc/modules-load.d/openshift.conf
        content: |
          br_netfilter
          overlay
          ip_vs
          ip_vs_rr
          ip_vs_wrr
          ip_vs_sh

These values are not arbitrary. OpenShift components and container runtimes will fail silently or degrade under load if kernel defaults are used.

---

Container Runtime Configuration

- name: Configure CRI-O
  copy:
    dest: /etc/crio/crio.conf.d/99-custom.conf
    content: |

[crio.runtime]

default_ulimits = [ “nofile=1048576:1048576” ] pids_limit = 4096

[crio.image]

pause_image = “registry.redhat.io/openshift4/ose-pod:latest” – name: Configure container storage copy: dest: /etc/containers/storage.conf content: |

[storage]

driver = “overlay” runroot = “/run/containers/storage” graphroot = “/var/lib/containers/storage”

[storage.options.overlay]

mountopt = “nodev,metacopy=on”

Default ulimits are insufficient for high-density clusters. You will hit file descriptor exhaustion before CPU or memory limits.

---

Security Architecture

RBAC Design Principles

Role-based access control should follow least privilege. Avoid cluster-admin grants; use namespace-scoped roles.

# Developer role - namespace scoped
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer
  namespace: app-dev
rules:
  - apiGroups: ["", "apps", "batch"]
    resources: ["pods", "deployments", "services", "configmaps", "secrets", "jobs", "cronjobs"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["pods/log", "pods/exec"]
    verbs: ["get", "create"]
  # Explicitly deny access to cluster resources
  - apiGroups: [""]
    resources: ["nodes", "persistentvolumes"]
    verbs: []
---
# Operations role - read-only cluster wide, write in specific namespaces
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: operations-readonly
rules:
  - apiGroups: [""]
    resources: ["nodes", "namespaces", "persistentvolumes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods", "services", "endpoints"]
    verbs: ["get", "list", "watch"]

Network Policies

Default deny with explicit allows. Every namespace should have a baseline policy.

# Default deny all ingress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: app-prod
spec:
  podSelector: {}
  policyTypes:
    - Ingress
---
# Allow ingress from same namespace only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: app-prod
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector: {}
---
# Allow ingress from OpenShift router
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-router
  namespace: app-prod
spec:
  podSelector:
    matchLabels:
      app: web-frontend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              network.openshift.io/policy-group: ingress

Image Security and Supply Chain

# Image policy to restrict registries
apiVersion: config.openshift.io/v1
kind: Image
metadata:
  name: cluster
spec:
  registrySources:
    allowedRegistries:
      - registry.redhat.io
      - registry.access.redhat.com
      - quay.io
      - ghcr.io
      - registry.internal.example.com
    blockedRegistries:
      - docker.io  # Block Docker Hub for compliance
---
# Require signed images in production
apiVersion: policy.sigstore.dev/v1alpha1
kind: ClusterImagePolicy
metadata:
  name: require-signatures
spec:
  images:
    - glob: "registry.internal.example.com/prod/**"
  authorities:
    - keyless:
        url: https://fulcio.sigstore.dev
        identities:
          - issuer: https://accounts.google.com
            subject: release-team@example.com

Pod Security Standards

# Enforce restricted security context
apiVersion: v1
kind: Namespace
metadata:
  name: app-prod
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
---
# Security Context Constraints for OpenShift
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: app-restricted
allowPrivilegedContainer: false
allowPrivilegeEscalation: false
requiredDropCapabilities:
  - ALL
runAsUser:
  type: MustRunAsNonRoot
seLinuxContext:
  type: MustRunAs
fsGroup:
  type: MustRunAs
volumes:
  - configMap
  - emptyDir
  - projected
  - secret
  - downwardAPI
  - persistentVolumeClaim

---

Networking: Where Most Migrations Fail

Ingress and Load Balancer Alignment

External load balancers must align with OpenShift router expectations. Health checks should target readiness endpoints, not TCP ports.

# HAProxy configuration for OpenShift routers
frontend openshift_router_https
    bind *:443
    mode tcp
    option tcplog
    default_backend openshift_router_https_backend

backend openshift_router_https_backend
    mode tcp
    balance source
    option httpchk GET /healthz/ready HTTP/1.1\r\nHost:\ router-health
    http-check expect status 200
    server router-0 192.168.1.10:443 check port 1936 inter 5s fall 3 rise 2
    server router-1 192.168.1.11:443 check port 1936 inter 5s fall 3 rise 2
    server router-2 192.168.1.12:443 check port 1936 inter 5s fall 3 rise 2

Common failure: Load balancer marks routers healthy while the application is unavailable. TCP health checks pass even when the router pod is terminating.

---

MTU and Overlay Networking

MTU mismatches between underlay, NSX, and OpenShift overlays cause:

• Intermittent pod-to-pod packet loss
• gRPC failures (large payloads fragment incorrectly)
• Random CI/CD pipeline timeouts
• TLS handshake failures

# Verify MTU across the path
# Physical network: 9000 (jumbo frames)
# NSX overlay: 8900 (100 byte overhead)
# OpenShift OVN: 8800 (additional 100 byte overhead)

# Test from inside a pod
kubectl exec -it debug-pod -- ping -M do -s 8772 target-service

# If this fails, reduce MTU until it works
# Then configure cluster network appropriately

# OpenShift cluster network configuration
apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
spec:
  clusterNetwork:
    - cidr: 10.128.0.0/14
      hostPrefix: 23
  serviceNetwork:
    - 172.30.0.0/16
  defaultNetwork:
    type: OVNKubernetes
    ovnKubernetesConfig:
      mtu: 8800
      genevePort: 6081

This is almost never diagnosed correctly on first pass. Symptoms look like application bugs.

---

DNS Configuration for Migration

# CoreDNS custom configuration for migration
apiVersion: v1
kind: ConfigMap
metadata:
  name: dns-custom
  namespace: openshift-dns
data:
  legacy.server: |
    legacy.example.com:53 {
      forward . 10.0.0.53 10.0.0.54
      cache 30
    }

During migration, pods may need to resolve legacy DNS names. Configure forwarding rules before cutting over applications.

---

Storage: Persistent Volumes and CSI Reality

StorageClass Design

# Pure Storage FlashArray - Fast tier
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: pure-fast
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: pure-csi
parameters:
  backend: flasharray
  csi.storage.k8s.io/fstype: xfs
  createoptions: -q
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
# Pure Storage FlashBlade - Shared/NFS tier
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: pure-shared
provisioner: pure-csi
parameters:
  backend: flashblade
  exportRules: "*(rw,no_root_squash)"
reclaimPolicy: Retain
volumeBindingMode: Immediate
---
# Standard tier for non-critical workloads
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/vsphere-volume
parameters:
  diskformat: thin
  datastore: vsanDatastore
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

WaitForFirstConsumer is critical for block storage. Without it, volumes are bound before pod placement, breaking topology-aware scheduling.

What breaks if ignored:

• Pods stuck in Pending state
• Volumes attached to unreachable nodes
• Zone-aware deployments fail silently

Stateful Application Migration

# Database migration pattern using PVC cloning
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: db-data-migrated
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: pure-fast
  resources:
    requests:
      storage: 500Gi
  dataSource:
    kind: PersistentVolumeClaim
    name: db-data-legacy

---

Observability and Migration Validation

Baseline Metrics Before Migration

You cannot validate a migration without knowing what normal looks like. Capture baselines for at least two weeks before migration.

# Key metrics to baseline
# Application metrics
- request_duration_seconds (p50, p95, p99)
- request_total (rate)
- error_total (rate)
- active_connections

# Infrastructure metrics
- cpu_usage_percent
- memory_usage_bytes
- disk_io_seconds
- network_bytes_transmitted
- network_bytes_received

# Business metrics
- transactions_per_second
- successful_checkouts
- user_sessions_active

Prometheus Rules for Migration

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: migration-validation
  namespace: openshift-monitoring
spec:
  groups:
    - name: migration.rules
      rules:
        # Alert if latency increases more than 20% post-migration
        - alert: MigrationLatencyRegression
          expr: |
            (
              histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{migrated="true"}[5m])) by (le, service))
              /
              histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{migrated="false"}[5m])) by (le, service))
            ) > 1.2
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Latency regression detected post-migration"
            description: "Service {{ $labels.service }} p95 latency increased by more than 20%"

        # Alert on error rate increase
        - alert: MigrationErrorRateIncrease
          expr: |
            (
              sum(rate(http_requests_total{status=~"5..", migrated="true"}[5m])) by (service)
              /
              sum(rate(http_requests_total{migrated="true"}[5m])) by (service)
            ) > 0.01
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Error rate exceeded 1% post-migration"

Grafana Dashboard for Migration

# Dashboard JSON snippet for migration comparison
{
  "panels": [
    {
      "title": "Request Latency Comparison",
      "type": "timeseries",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{env='legacy'}[5m])) by (le))",
          "legendFormat": "Legacy p95"
        },
        {
          "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{env='openshift'}[5m])) by (le))",
          "legendFormat": "OpenShift p95"
        }
      ]
    },
    {
      "title": "Error Rate Comparison",
      "type": "stat",
      "targets": [
        {
          "expr": "sum(rate(http_requests_total{status=~'5..', env='openshift'}[5m])) / sum(rate(http_requests_total{env='openshift'}[5m]))",
          "legendFormat": "OpenShift Error Rate"
        }
      ]
    }
  ]
}

Log Aggregation for Troubleshooting

# Loki configuration for migration logs
apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki
  namespace: openshift-logging
spec:
  size: 1x.small
  storage:
    schemas:
      - version: v12
        effectiveDate: "2024-01-01"
    secret:
      name: logging-loki-s3
      type: s3
  storageClassName: pure-fast
  tenants:
    mode: openshift-logging

---

CI/CD and GitOps: What Actually Works

Immutable Image Promotion

Do not rebuild images per environment. Build once, scan once, promote through environments.

# Tekton pipeline for build-once promotion
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: build-and-promote
spec:
  params:
    - name: git-revision
      type: string
  tasks:
    - name: build
      taskRef:
        name: buildah
      params:
        - name: IMAGE
          value: "registry.internal/app:$(params.git-revision)"

    - name: scan
      taskRef:
        name: trivy-scan
      runAfter:
        - build

    - name: sign
      taskRef:
        name: cosign-sign
      runAfter:
        - scan

    - name: promote-to-dev
      taskRef:
        name: skopeo-copy
      runAfter:
        - sign
      params:
        - name: srcImage
          value: "registry.internal/app:$(params.git-revision)"
        - name: destImage
          value: "registry.internal/app:dev"

If you rebuild per environment:

• Debugging becomes impossible (which build has the bug?)
• Security attestation is meaningless
• Promotion is not promotion, it is a new deployment

---

ArgoCD Application Example

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-prod
  namespace: openshift-gitops
spec:
  project: production
  destination:
    namespace: app-prod
    server: https://kubernetes.default.svc
  source:
    repoURL: https://github.com/org/app-config
    targetRevision: main
    path: overlays/prod
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=false
      - PrunePropagationPolicy=foreground
      - PruneLast=true
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas  # Allow HPA to control replicas

Self-heal is not optional in regulated or audited environments. Manual drift is operational debt that compounds.

Environment Promotion with Kustomize

# Base kustomization
# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
  - configmap.yaml

# Production overlay
# overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patches:
  - patch: |
      - op: replace
        path: /spec/replicas
        value: 5
    target:
      kind: Deployment
      name: app
images:
  - name: app
    newName: registry.internal/app
    newTag: v1.2.3  # Pinned version, updated by CI

---

Rollback Strategy

Application-Level Rollback

# ArgoCD rollback to previous version
argocd app history app-prod
argocd app rollback app-prod <revision>

# Or using kubectl
kubectl rollout undo deployment/app -n app-prod
kubectl rollout status deployment/app -n app-prod

Traffic-Based Rollback

# OpenShift route for blue-green deployment
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: app
  namespace: app-prod
spec:
  to:
    kind: Service
    name: app-green
    weight: 100
  alternateBackends:
    - kind: Service
      name: app-blue
      weight: 0
---
# To rollback, shift traffic back to blue
# oc patch route app -p '{"spec":{"to":{"weight":0},"alternateBackends":[{"kind":"Service","name":"app-blue","weight":100}]}}'

Full Migration Rollback

For critical systems, maintain the ability to roll back the entire migration for a defined period.

# Rollback checklist
rollback_criteria:
  - error_rate > 5% for 15 minutes
  - p99_latency > 2x baseline for 30 minutes
  - data_integrity_check_failed
  - critical_integration_broken

rollback_procedure:
  1. Announce rollback decision
  2. Stop writes to new system (if applicable)
  3. Verify data sync to legacy is current
  4. Switch DNS/load balancer to legacy
  5. Verify legacy system health
  6. Communicate rollback complete
  7. Schedule post-mortem

rollback_window: 14 days  # Maintain legacy systems for 2 weeks post-migration

Data Rollback Considerations

# Continuous data sync for rollback capability
apiVersion: batch/v1
kind: CronJob
metadata:
  name: data-sync-to-legacy
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: sync
              image: registry.internal/data-sync:latest
              env:
                - name: SOURCE_DB
                  value: "postgresql://new-db:5432/app"
                - name: TARGET_DB
                  value: "postgresql://legacy-db:5432/app"
                - name: SYNC_MODE
                  value: "incremental"
          restartPolicy: OnFailure

Key principle: Never decommission legacy systems until the rollback window has passed and stakeholders have signed off.

---

Migration Execution: What People Underestimate

State and Cutover

Databases and stateful services require parallel runs and controlled traffic switching. DNS TTLs must be reduced days in advance, not minutes.

Most outages during migration are caused by:

• Hidden hard-coded IPs in application configs, scripts, and cron jobs
• Legacy authentication dependencies (service accounts with IP-based trust)
• Assumed local storage paths that do not exist in containers
• Timezone differences between legacy VMs and containers (UTC default)
• Environment variables that were set manually and never documented

Communication Plan

# Migration communication template
stakeholders:
  - business_owners
  - development_teams
  - operations
  - security
  - support

communications:
  - timing: T-14 days
    message: "Migration scheduled, review runbook"
    audience: all

  - timing: T-2 days
    message: "DNS TTL reduced, final validation"
    audience: operations, development

  - timing: T-0 (cutover)
    message: "Migration in progress, reduced SLA"
    audience: all

  - timing: T+1 hour
    message: "Initial validation complete"
    audience: all

  - timing: T+24 hours
    message: "Migration successful, monitoring continues"
    audience: all

---

Operational Testing (Non-Negotiable)

Before production:

• Kill a control plane node and verify automatic recovery
• Force etcd leader re-election during load
• Simulate storage controller failure
• Drain workers during peak load
• Test certificate rotation
• Verify backup and restore procedures
• Run security scan and penetration test

# Chaos testing example with Litmus
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: control-plane-chaos
  namespace: litmus
spec:
  engineState: active
  appinfo:
    appns: openshift-etcd
    applabel: app=etcd
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"
            - name: CHAOS_INTERVAL
              value: "10"
            - name: FORCE
              value: "true"

If the platform team is afraid to do this, the cluster is not ready.

---

In short…

OpenShift migration is not a technology project. It is an operational transformation that happens to involve technology.

The patterns in this post exist because I have seen the alternatives fail. Every shortcut skipping reservations, ignoring kernel tuning, compressing testing phases creates debt that surfaces as production incidents.

Key principles:

• Infrastructure must be deterministic before OpenShift installation
• Security is architecture, not an afterthought
• Migration strategy matters more than migration speed
• Observability validates success; without baselines, you are guessing
• Rollback capability is not optional for production systems
• Test failure modes before they test you

The goal is not to move workloads. The goal is to move workloads without moving your problems with them and without creating new ones.