Nutanix NKP Backup With Velero

ทำการ enable velero ที่ nkp dashboard เนื่องจาก velero ต้องการ s3 storage เพื่อเก็บข้อมูล backup จึงต้อง enable rookceph และ rookceph cluster โดยจะต้องรอจนกว่าจะ running สมบูรณ์ก่อนจะ enable velero

ขั้นตอน enable จะต้อง set ค่า configuration เพิ่มเติม โดยต้องสร้าง secret สำหรับให้ velero ใช้งาน S3 Storage โดยสร้าง secret key ได้ตามตัวอย่างดังนี้

apiVersion: v1
kind: Secret
metadata:
  name: pongsak-velero-ceph
  namespace: kommander-default-workspace
type: Opaque
stringData:
  aws: |
    [default]
    aws_access_key_id = HUD5Z1JPRDS8IQOA5U5G
    aws_secret_access_key = ASvtTAAUPtcbEEpCnCuvczQgF3N4hiTIn9qUruzA

ทำการ enable Velero โดยใช้ configuration ตามตัวอย่างนี้

เพิ่ม custom config เพื่อให้ Velero สามารถ backup persistent volume ได้ 

configuration:
  backupStorageLocation:
    - bucket: backup
      config:
        region: dkp-object-store
        s3Url: https://10.38.37.19:8085
        s3ForcePathStyle: "true"
        insecureSkipTLSVerify: true
        profile: default
      provider: aws
      credential:
        key: aws
        name: pongsak-velero-ceph
  features: EnableCSI
  uploaderType: kopia
  volumeSnapshotLocation:
    - bucket: backup
      config:
        region: dkp-object-store
        s3Url: https://10.38.37.19:8085
      provider: aws
deployNodeAgent: true
initContainers:
  - image: velero/velero-plugin-for-aws:v1.13.2
    imagePullPolicy: IfNotPresent
    name: velero-plugin-for-aws
    volumeMounts:
      - mountPath: /target
        name: plugins    
metrics:
  enabled: true
  serviceMonitor:
    enabled: true        
nodeAgent:
  annotations:
    secret.reloader.stakater.com/reload: pongsak-velero-ceph
  priorityClassName: dkp-critical-priority
  resources:
    limits: null

หลังจาก enable แล้วตรวจสอบว่า velero pod ทำงานได้ปกติ รวมถึงตรวจสอบ backup location และ volume snapshot location

[nutanix@harbor ~]$ kubectl get pod -A | grep velero
kommander-default-workspace       object-bucket-claims-check-dkp-velero-xmxsx                       0/1     Completed   0            3h1m
kommander-default-workspace       velero-858485b79-b9rdc                                            1/1     Running     0            72s
kommander-default-workspace       velero-backup-storage-location-updater-d4d54cf6f-rhccm            1/1     Running     0            14s
kommander-default-workspace       velero-pre-install-cr4rf                                          0/1     Completed   0            105s
[nutanix@harbor ~]$ kubectl get backupstoragelocations -A
NAMESPACE                     NAME      PHASE       LAST VALIDATED   AGE   DEFAULT
kommander-default-workspace   default   Available   42s              13m   true
[nutanix@harbor ~]$ kubectl get volumesnapshotlocations -A
NAMESPACE                     NAME      AGE
kommander-default-workspace   default   14m

ติดตั้ง Velero cli สำหรับการใช้งาน

# For Apple Silicon Macs (M1/M2/M3) Download ARM64 version
curl -LO https://github.com/vmware-tanzu/velero/releases/latest/download/velero-darwin-arm64.tar.gz
# Extract and install
tar -xzf velero-darwin-arm64.tar.gz
sudo mv velero-*/velero /usr/local/bin/
chmod +x /usr/local/bin/velero
rm -rf velero-* velero-darwin-arm64.tar.gz
#Ubuntu/Debian (APT)
# Add Velero repository
curl -s https://packagecloud.io/install/repositories/vmware/velero/script.deb.sh | sudo bash
# Install Velero
sudo apt-get install velero
#CentOS/RHEL/Fedora (YUM/DNF)
# Add Velero repository
curl -s https://packagecloud.io/install/repositories/vmware/velero/script.rpm.sh | sudo bash
# Install Velero (CentOS/RHEL)
sudo yum install velero
# Or for Fedora
sudo dnf install velero

ตรวจสอบ version ของ Velero หลังจากติดตั้ง

[nutanix@harbor ~]$ velero version
Client:
	Version: v1.17.2
	Git commit: 7013a4097f17a97a0201c0ad3c9dd3f810d32bf6

Velero จะใช้ kubeconfig file เพื่อ connect ไปยัง kubernetes จึงต้อง export kubeconfig environment ก่อน จากนั้นตรวจสอบว่า velero เห็น backup location หรือไม่

[nutanix@harbor ~]$ export KUBECONFIG=/home/nutanix/dev2-kubeconfig.conf
[nutanix@harbor ~]$ velero get backup-location -n kommander-default-workspace
NAME      PROVIDER   BUCKET/PREFIX   PHASE       LAST VALIDATED                  ACCESS MODE   DEFAULT
default   aws        backup          Available   2026-01-30 11:25:11 +0000 UTC   ReadWrite     true
[nutanix@harbor ~]$ velero get snapshot-location -n kommander-default-workspace
NAME      PROVIDER
default   aws

ติดตั้ง wordpress เพื่อใช้ในการทดสอบ backup และ restore โดยติดตั้งที่ namespace application ตามตัวอย่าง yaml ดังนี้

apiVersion: v1
kind: Secret
metadata:
  name: mysql-pass
type: Opaque
stringData:
  password: nutanix
---
apiVersion: v1
kind: Service
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  ports:
    - port: 3306
  selector:
    app: wordpress
    tier: mysql
  clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  labels:
    app: wordpress
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: mysql
    spec:
      containers:
        - image: mysql:8.0
          name: mysql
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-pass
                  key: password
            - name: MYSQL_DATABASE
              value: wordpress
            - name: MYSQL_USER
              value: wordpress
            - name: MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-pass
                  key: password
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - name: mysql-persistent-storage
              mountPath: /var/lib/mysql
      volumes:
        - name: mysql-persistent-storage
          persistentVolumeClaim:
            claimName: mysql-pv-claim
---
apiVersion: v1
kind: Service
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  ports:
    - port: 80
  selector:
    app: wordpress
    tier: frontend
  type: ClusterIP
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wp-pv-claim
  labels:
    app: wordpress
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: frontend
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: frontend
    spec:
      containers:
        - image: wordpress:6.4-apache
          name: wordpress
          env:
            - name: WORDPRESS_DB_HOST
              value: wordpress-mysql
            - name: WORDPRESS_DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-pass
                  key: password
            - name: WORDPRESS_DB_USER
              value: wordpress
            - name: WORDPRESS_DB_NAME
              value: wordpress
            - name: WORDPRESS_DEBUG
              value: "1"  
          ports:
            - containerPort: 80
              name: wordpress
          volumeMounts:
            - name: wordpress-persistent-storage
              mountPath: /var/www/html
      volumes:
        - name: wordpress-persistent-storage
          persistentVolumeClaim:
            claimName: wp-pv-claim
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: wordpress-ingress
spec:
  ingressClassName: kommander-traefik
  rules:
  - host: wordpress.10.38.37.19.sslip.io
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: wordpress
            port:
              number: 80

หมายเหตุ – “10.38.37.19” คือ ingress ip

ตรวจสอบว่า wordpress ทำงานได้ปกติ และเพิ่ม page เข้าไปใน wordpress เพื่อใช้ในการตรวจสอบการ restore

~/repos/lab  k get pod -n application
NAME                               READY   STATUS    RESTARTS   AGE
wordpress-57ff4b4485-6tl9g         1/1     Running   0          10h
wordpress-mysql-6dd978bc8b-jr2gq   1/1     Running   0          10h

เข้า wordpress ที่ ur https://wordpress.10.38.37.19.sslip.io/ และทำการสร้าง page เพื่อตรวจอสอบผลการ restore

ทำการ backup wordpress ด้วย velero cli โดย backup ทั้ง default namespace

[nutanix@harbor ~]$ velero backup create wordpress-backup-1 --include-namespaces default --snapshot-volumes=true -n kommander-default-workspace --wait
Backup request "wordpress-backup-1" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background.
...........
Backup completed with status: Completed. You may check for more information using the commands `velero backup describe wordpress-backup-1` and `velero backup logs wordpress-backup-1`.

ตรวจสอบข้อมูลที่ velero ทำการ backup ได้จาก cli ดังนี้

[nutanix@harbor ~]$ velero backup describe wordpress-backup-1 --insecure-skip-tls-verify -n kommander-default-workspace --details
Name:         wordpress-backup-1
Namespace:    kommander-default-workspace
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.34.1
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=34
Phase:  Completed
Namespaces:
  Included:  default
  Excluded:  <none>
Resources:
  Included cluster-scoped:    <none>
  Excluded cluster-scoped:    volumesnapshotcontents.snapshot.storage.k8s.io
  Included namespace-scoped:  *
  Excluded namespace-scoped:  volumesnapshots.snapshot.storage.k8s.io
Label selector:  <none>
Or label selector:  <none>
Storage Location:  default
Velero-Native Snapshot PVs:    true
File System Backup (Default):  false
Snapshot Move Data:            false
Data Mover:                    velero
TTL:  720h0m0s
CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s
Hooks:  <none>
Backup Format Version:  1.1.0
Started:    2026-01-30 11:35:35 +0000 UTC
Completed:  2026-01-30 11:35:47 +0000 UTC
Expiration:  2026-03-01 11:35:35 +0000 UTC
Total items to be backed up:  56
Items backed up:              56
Backup Item Operations:
  Operation for volumesnapshots.snapshot.storage.k8s.io default/velero-wp-pv-claim-skrl8:
    Backup Item Action Plugin:  velero.io/csi-volumesnapshot-backupper
    Operation ID:               default/velero-wp-pv-claim-skrl8/2026-01-30T11:35:41Z
    Items to Update:
              volumesnapshots.snapshot.storage.k8s.io default/velero-wp-pv-claim-skrl8
              volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-305912d8-4ec2-484e-9273-c2ea65992c55
    Phase:    Completed
    Created:  2026-01-30 11:35:41 +0000 UTC
    Started:  2026-01-30 11:35:41 +0000 UTC
    Updated:  2026-01-30 11:35:46 +0000 UTC
  Operation for volumesnapshots.snapshot.storage.k8s.io default/velero-mysql-pv-claim-rbqkn:
    Backup Item Action Plugin:  velero.io/csi-volumesnapshot-backupper
    Operation ID:               default/velero-mysql-pv-claim-rbqkn/2026-01-30T11:35:46Z
    Items to Update:
              volumesnapshots.snapshot.storage.k8s.io default/velero-mysql-pv-claim-rbqkn
              volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-9681ba15-76fd-4b98-8411-9c083793a14e
    Phase:    Completed
    Created:  2026-01-30 11:35:46 +0000 UTC
    Started:  2026-01-30 11:35:46 +0000 UTC
    Updated:  2026-01-30 11:35:46 +0000 UTC
Resource List:
  apiextensions.k8s.io/v1/CustomResourceDefinition:
    - ciliumendpoints.cilium.io
  apps/v1/Deployment:
    - default/wordpress
    - default/wordpress-mysql
  apps/v1/ReplicaSet:
    - default/wordpress-645b586f68
    - default/wordpress-mysql-5ccb49cfb
  cilium.io/v2/CiliumEndpoint:
    - default/wordpress-645b586f68-dbwgz
    - default/wordpress-mysql-5ccb49cfb-rxwtt
  discovery.k8s.io/v1/EndpointSlice:
    - default/kubernetes
    - default/wordpress-dfcfz
    - default/wordpress-mysql-jvhnt
  networking.k8s.io/v1/Ingress:
    - default/wordpress-ingress
  snapshot.storage.k8s.io/v1/VolumeSnapshot:
    - default/velero-mysql-pv-claim-rbqkn
    - default/velero-wp-pv-claim-skrl8
  snapshot.storage.k8s.io/v1/VolumeSnapshotClass:
    - nutanix-snapshot-class
  snapshot.storage.k8s.io/v1/VolumeSnapshotContent:
    - snapcontent-305912d8-4ec2-484e-9273-c2ea65992c55
    - snapcontent-9681ba15-76fd-4b98-8411-9c083793a14e
  v1/ConfigMap:
    - default/kube-root-ca.crt
  v1/Endpoints:
    - default/kubernetes
    - default/wordpress
    - default/wordpress-mysql
  v1/Event:
    - default/mysql-pv-claim.188f7ebeb41953b0
    - default/mysql-pv-claim.188f7ebeb69531d4
    - default/mysql-pv-claim.188f7ebeb6b8fdd8
    - default/mysql-pv-claim.188f7ebfa2a36c2e
    - default/wordpress-645b586f68-dbwgz.188f7ebfa7ff0ffd
    - default/wordpress-645b586f68-dbwgz.188f7ec107862336
    - default/wordpress-645b586f68-dbwgz.188f7ec1dd681605
    - default/wordpress-645b586f68-dbwgz.188f7ec50eaead7b
    - default/wordpress-645b586f68-dbwgz.188f7ec5112993aa
    - default/wordpress-645b586f68-dbwgz.188f7ec5164380f9
    - default/wordpress-645b586f68.188f7ebeb8bd04b2
    - default/wordpress-mysql-5ccb49cfb-rxwtt.188f7ebfa5454bde
    - default/wordpress-mysql-5ccb49cfb-rxwtt.188f7ec1009f740f
    - default/wordpress-mysql-5ccb49cfb-rxwtt.188f7ec1c720235a
    - default/wordpress-mysql-5ccb49cfb-rxwtt.188f7ec53b9f0b43
    - default/wordpress-mysql-5ccb49cfb-rxwtt.188f7ec53d9cdaf2
    - default/wordpress-mysql-5ccb49cfb-rxwtt.188f7ec542ec7a5e
    - default/wordpress-mysql-5ccb49cfb.188f7ebeb5c2a7f3
    - default/wordpress-mysql.188f7ebeb540f2c9
    - default/wordpress.188f7ebeb782a76e
    - default/wp-pv-claim.188f7ebeb60eb52f
    - default/wp-pv-claim.188f7ebeb9ead9ae
    - default/wp-pv-claim.188f7ebeba1102f5
    - default/wp-pv-claim.188f7ebfa2bccfc7
  v1/Namespace:
    - default
  v1/PersistentVolume:
    - pvc-cdcf92fb-8b3e-4d6a-8751-4d6b7dc31cf3
    - pvc-e3188791-445f-4789-8f6c-5bd7c7f82134
  v1/PersistentVolumeClaim:
    - default/mysql-pv-claim
    - default/wp-pv-claim
  v1/Pod:
    - default/wordpress-645b586f68-dbwgz
    - default/wordpress-mysql-5ccb49cfb-rxwtt
  v1/Secret:
    - default/mysql-pass
  v1/Service:
    - default/kubernetes
    - default/wordpress
    - default/wordpress-mysql
  v1/ServiceAccount:
    - default/default
Backup Volumes:
  Velero-Native Snapshots: <none included>
  CSI Snapshots:
    default/mysql-pv-claim:
      Snapshot:
        Operation ID: default/velero-mysql-pv-claim-rbqkn/2026-01-30T11:35:46Z
        Snapshot Content Name: snapcontent-9681ba15-76fd-4b98-8411-9c083793a14e
        Storage Snapshot ID: NutanixVolumes-685b4989-23b6-4d8e-be83-8784ad5e7992:8e517250-1f33-4ee1-a9e1-0d523e8c442c
        Snapshot Size (bytes): 21474836480
        CSI Driver: csi.nutanix.com
        Result: succeeded
    default/wp-pv-claim:
      Snapshot:
        Operation ID: default/velero-wp-pv-claim-skrl8/2026-01-30T11:35:41Z
        Snapshot Content Name: snapcontent-305912d8-4ec2-484e-9273-c2ea65992c55
        Storage Snapshot ID: NutanixVolumes-86c441ab-529c-4966-bceb-c2856f18a94c:2456bb0f-8792-46f2-8366-8067d5305cde
        Snapshot Size (bytes): 21474836480
        CSI Driver: csi.nutanix.com
        Result: succeeded
  Pod Volume Backups: <none included>
HooksAttempted:  0
HooksFailed:     0

ตรวจสอบสถานะการ backup

[nutanix@harbor ~]$ velero get backup -n kommander-default-workspace
NAME                 STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
wordpress-backup-1   Completed   0        0          2026-01-30 11:35:35 +0000 UTC   29d       default            <none>

ทำการ ลบ wordpress application

[nutanix@harbor ~]$ kubectl delete -f wordpress.yaml
secret "mysql-pass" deleted
service "wordpress-mysql" deleted
persistentvolumeclaim "mysql-pv-claim" deleted
deployment.apps "wordpress-mysql" deleted
service "wordpress" deleted
persistentvolumeclaim "wp-pv-claim" deleted
deployment.apps "wordpress" deleted
ingress.networking.k8s.io "wordpress-ingress" deleted

ทำการ restore wordpress application ทั้งนี้สามารถ restore จากอีก cluster ได้โดยทั้งสอง cluster จะต้องชี้ไปยัง S3 bucket เดียวกัน

[nutanix@harbor ~]$ velero restore create wordpress-restore-1 --from-backup wordpress-backup-1 --restore-volumes=true  -n kommander-default-workspace --wait
Restore request "wordpress-restore-1" submitted successfully.
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.
..
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe wordpress-restore-1` and `velero restore logs wordpress-restore-1`.

เรียกใช้ wordpress เพื่อตรวจสอบว่าข้อมูลมีการ restore กลับมาครบถ้วน