ทำการ enable velero ที่ nkp dashboard เนื่องจาก velero ต้องการ s3 storage เพื่อเก็บข้อมูล backup จึงต้อง enable rookceph และ rookceph cluster โดยจะต้องรอจนกว่าจะ running สมบูรณ์ก่อนจะ enable velero
เพิ่ม custom config เพื่อให้ Velero สามารถ backup persistent volume ได้
configuration:
backupStorageLocation:
- bucket: dkp-velero
config:
region: dkp-object-store
s3Url: https://10.38.16.132:8085
s3ForcePathStyle: "true"
insecureSkipTLSVerify: true
profile: default
provider: aws
credential:
key: aws
name: dkp-velero-ceph
features: EnableCSI
uploaderType: kopia
volumeSnapshotLocation:
- bucket: dkp-velero
config:
region: dkp-object-store
s3Url: https://10.38.16.132:8085
provider: aws
deployNodeAgent: true
initContainers:
- image: velero/velero-plugin-for-aws:v1.11.1
imagePullPolicy: IfNotPresent
name: velero-plugin-for-aws
volumeMounts:
- mountPath: /target
name: plugins
nodeAgent:
annotations:
secret.reloader.stakater.com/reload: dkp-velero-ceph
priorityClassName: dkp-critical-priority
resources:
limits: null
ข้อมูล custom configuration ต้องการ s3 bucket และ secret key เพื่อ connect ไปยัง s3 bucket โดยสร้าง secret key ได้ตามตัวอย่างดังนี้
apiVersion: v1
kind: Secret
metadata:
name: dkp-velero-ceph
namespace: kommander-default-workspace
type: Opaque
stringData:
aws: |
[default]
aws_access_key_id = 2N8IL5CIUIWHR2MC287L
aws_secret_access_key = bNzQDPy23GzdWKDuokZOtlT1fO7oR01YcOTicBCMหลังจาก enable แล้วตรวจสอบว่า velero pod ทำงานได้ปกติ รวมถึงตรวจสอบ backup location และ volume snapshot location
~/repos/note k get pod -A | grep velero
kommander object-bucket-claims-check-dkp-velero-229gs 0/1 Completed 0 10d
kommander velero-54db8ff6f8-vcvfw 1/1 Running 0 12m
kommander velero-backup-storage-location-updater-5d849446c9-kt4h8 1/1 Running 0 11m
kommander velero-pre-install-wt6nv 0/1 Completed 0 13m
~/repos/lab k get backupstoragelocations -A
NAMESPACE NAME PHASE LAST VALIDATED AGE DEFAULT
kommander default Available 1s 25m true
~/repos/lab k get volumesnapshotlocations -A
NAMESPACE NAME AGE
kommander default 26mติดตั้ง Velero cli สำหรับการใช้งาน
# For Apple Silicon Macs (M1/M2/M3) Download ARM64 version
curl -LO https://github.com/vmware-tanzu/velero/releases/latest/download/velero-darwin-arm64.tar.gz
# Extract and install
tar -xzf velero-darwin-arm64.tar.gz
sudo mv velero-*/velero /usr/local/bin/
chmod +x /usr/local/bin/velero
rm -rf velero-* velero-darwin-arm64.tar.gz
#Ubuntu/Debian (APT)
# Add Velero repository
curl -s https://packagecloud.io/install/repositories/vmware/velero/script.deb.sh | sudo bash
# Install Velero
sudo apt-get install velero
#CentOS/RHEL/Fedora (YUM/DNF)
# Add Velero repository
curl -s https://packagecloud.io/install/repositories/vmware/velero/script.rpm.sh | sudo bash
# Install Velero (CentOS/RHEL)
sudo yum install velero
# Or for Fedora
sudo dnf install veleroตรวจสอบ version ของ Velero หลังจากติดตั้ง
~/repos/note velero version
Client:
Version: v1.15.2
Git commit: 804d73c4f2349f1ca9bd3d6c751956e1d2021c01Velero จะใช้ kubeconfig file เพื่อ connect ไปยัง kubernetes จึงต้อง export kubeconfig environment ก่อน จากนั้นตรวจสอบว่า velero เห็น backup location หรือไม่
~/repos/lab export KUBECONFIG=/Users/pongsak.khamdee/repos/lab/cluster2-kubeconfig.yaml
~/repos/lab velero get backup-location -n kommander
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE DEFAULT
default aws dkp-velero Available 2025-07-03 21:54:38 +0700 +07 ReadWrite true
~/repos/lab velero get snapshot-location -n kommander
NAME PROVIDER
default awsติดตั้ง wordpress เพื่อใช้ในการทดสอบ backup และ restore โดยติดตั้งที่ namespace application ตามตัวอย่าง yaml ดังนี้
apiVersion: v1
kind: Secret
metadata:
name: mysql-pass
type: Opaque
stringData:
password: nutanix
---
apiVersion: v1
kind: Service
metadata:
name: wordpress-mysql
labels:
app: wordpress
spec:
ports:
- port: 3306
selector:
app: wordpress
tier: mysql
clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
labels:
app: wordpress
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: wordpress-mysql
labels:
app: wordpress
spec:
selector:
matchLabels:
app: wordpress
tier: mysql
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: mysql
spec:
containers:
- image: mysql:8.0
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-pass
key: password
- name: MYSQL_DATABASE
value: wordpress
- name: MYSQL_USER
value: wordpress
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-pass
key: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claim
---
apiVersion: v1
kind: Service
metadata:
name: wordpress
labels:
app: wordpress
spec:
ports:
- port: 80
selector:
app: wordpress
tier: frontend
type: ClusterIP
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: wp-pv-claim
labels:
app: wordpress
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: wordpress
labels:
app: wordpress
spec:
selector:
matchLabels:
app: wordpress
tier: frontend
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: frontend
spec:
containers:
- image: wordpress:6.4-apache
name: wordpress
env:
- name: WORDPRESS_DB_HOST
value: wordpress-mysql
- name: WORDPRESS_DB_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-pass
key: password
- name: WORDPRESS_DB_USER
value: wordpress
- name: WORDPRESS_DB_NAME
value: wordpress
- name: WORDPRESS_DEBUG
value: "1"
ports:
- containerPort: 80
name: wordpress
volumeMounts:
- name: wordpress-persistent-storage
mountPath: /var/www/html
volumes:
- name: wordpress-persistent-storage
persistentVolumeClaim:
claimName: wp-pv-claim
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: wordpress-ingress
spec:
ingressClassName: kommander-traefik
rules:
- host: wordpress.10.38.204.16.sslip.io
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: wordpress
port:
number: 80 ตรวจสอบว่า wordpress ทำงานได้ปกติ และเพิ่ม page เข้าไปใน wordpress เพื่อใช้ในการตรวจสอบการ restore
~/repos/lab k get pod -n application
NAME READY STATUS RESTARTS AGE
wordpress-57ff4b4485-6tl9g 1/1 Running 0 10h
wordpress-mysql-6dd978bc8b-jr2gq 1/1 Running 0 10hทำการ backup wordpress ด้วย velero cli
[nutanix@harbor ~]$ velero backup create wordpress10 --include-namespaces application --snapshot-volumes=true -n kommander --wait
Backup request "wordpress10" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background.
.............
Backup completed with status: Completed. You may check for more information using the commands `velero backup describe wordpress10` and `velero backup logs wordpress10`.ตรวจสอบข้อมูลที่ velero ทำการ backup ได้จาก cli ดังนี้
[nutanix@harbor ~]$ velero backup describe wordpress10 --insecure-skip-tls-verify -n kommander --details
Name: wordpress10
Namespace: kommander
Labels: velero.io/storage-location=default
Annotations: velero.io/resource-timeout=10m0s
velero.io/source-cluster-k8s-gitversion=v1.31.4
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=31
Phase: Completed
Namespaces:
Included: application
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: auto
Label selector: <none>
Or label selector: <none>
Storage Location: default
Velero-Native Snapshot PVs: true
Snapshot Move Data: false
Data Mover: velero
TTL: 720h0m0s
CSISnapshotTimeout: 10m0s
ItemOperationTimeout: 4h0m0s
Hooks: <none>
Backup Format Version: 1.1.0
Started: 2025-07-06 08:13:24 +0000 UTC
Completed: 2025-07-06 08:13:37 +0000 UTC
Expiration: 2025-08-05 08:13:24 +0000 UTC
Total items to be backed up: 29
Items backed up: 29
Backup Item Operations:
Operation for volumesnapshots.snapshot.storage.k8s.io application/velero-wp-pv-claim-hwnp8:
Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper
Operation ID: application/velero-wp-pv-claim-hwnp8/2025-07-06T08:13:32Z
Items to Update:
volumesnapshots.snapshot.storage.k8s.io application/velero-wp-pv-claim-hwnp8
volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-7e8eaf1f-5c59-46ec-9356-887e65b15769
Phase: Completed
Created: 2025-07-06 08:13:32 +0000 UTC
Started: 2025-07-06 08:13:32 +0000 UTC
Updated: 2025-07-06 08:13:37 +0000 UTC
Operation for volumesnapshots.snapshot.storage.k8s.io application/velero-mysql-pv-claim-s6q97:
Backup Item Action Plugin: velero.io/csi-volumesnapshot-backupper
Operation ID: application/velero-mysql-pv-claim-s6q97/2025-07-06T08:13:37Z
Items to Update:
volumesnapshots.snapshot.storage.k8s.io application/velero-mysql-pv-claim-s6q97
volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-e8de1f6e-4bce-43e7-9894-c7a180c9dd2d
Phase: Completed
Created: 2025-07-06 08:13:37 +0000 UTC
Started: 2025-07-06 08:13:37 +0000 UTC
Updated: 2025-07-06 08:13:37 +0000 UTC
Resource List:
apiextensions.k8s.io/v1/CustomResourceDefinition:
- ciliumendpoints.cilium.io
apps/v1/Deployment:
- application/wordpress
- application/wordpress-mysql
apps/v1/ReplicaSet:
- application/wordpress-7456589d7f
- application/wordpress-mysql-7cf779d5dd
cilium.io/v2/CiliumEndpoint:
- application/wordpress-7456589d7f-8s8fk
- application/wordpress-mysql-7cf779d5dd-c6mpk
discovery.k8s.io/v1/EndpointSlice:
- application/wordpress-mysql-hcrbc
- application/wordpress-zpdjc
networking.k8s.io/v1/Ingress:
- application/wordpress-ingress
snapshot.storage.k8s.io/v1/VolumeSnapshot:
- application/velero-mysql-pv-claim-s6q97
- application/velero-wp-pv-claim-hwnp8
snapshot.storage.k8s.io/v1/VolumeSnapshotClass:
- nutanix-snapshot-class
snapshot.storage.k8s.io/v1/VolumeSnapshotContent:
- snapcontent-7e8eaf1f-5c59-46ec-9356-887e65b15769
- snapcontent-e8de1f6e-4bce-43e7-9894-c7a180c9dd2d
v1/ConfigMap:
- application/kube-root-ca.crt
v1/Endpoints:
- application/wordpress
- application/wordpress-mysql
v1/Namespace:
- application
v1/PersistentVolume:
- pvc-32abc23d-9637-4343-af61-86e24dbbab5d
- pvc-65ca02e4-4396-45fc-b91d-f7f915db0095
v1/PersistentVolumeClaim:
- application/mysql-pv-claim
- application/wp-pv-claim
v1/Pod:
- application/wordpress-7456589d7f-8s8fk
- application/wordpress-mysql-7cf779d5dd-c6mpk
v1/Secret:
- application/mysql-pass
v1/Service:
- application/wordpress
- application/wordpress-mysql
v1/ServiceAccount:
- application/default
Backup Volumes:
Velero-Native Snapshots: <none included>
CSI Snapshots:
application/mysql-pv-claim:
Snapshot:
Operation ID: application/velero-mysql-pv-claim-s6q97/2025-07-06T08:13:37Z
Snapshot Content Name: snapcontent-e8de1f6e-4bce-43e7-9894-c7a180c9dd2d
Storage Snapshot ID: NutanixVolumes-079a8061-ad99-451e-aace-195ba16a72dd:88d833af-04d3-4f7e-b1d5-671cf7ec4656
Snapshot Size (bytes): 21474836480
CSI Driver: csi.nutanix.com
Result: succeeded
application/wp-pv-claim:
Snapshot:
Operation ID: application/velero-wp-pv-claim-hwnp8/2025-07-06T08:13:32Z
Snapshot Content Name: snapcontent-7e8eaf1f-5c59-46ec-9356-887e65b15769
Storage Snapshot ID: NutanixVolumes-29fd7391-322f-4fa3-bf35-a857011e0991:8d300ddc-de02-4650-9b88-baa4afb35376
Snapshot Size (bytes): 21474836480
CSI Driver: csi.nutanix.com
Result: succeeded
Pod Volume Backups: <none included>
HooksAttempted: 0
HooksFailed: 0ตรวจสอบสถานะการ backup
~/repos/lab velero get backup -n kommander-default-workspace
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
23:39:26 +0700 +07 29d default <none>
wordpress10 Completed 0 0 2025-07-06 ทำการ ลบ wordpress application
~/repos/lab k delete ns application
namespace "application" deletedทำการ restore wordpress application ซึ่งตัวอย่างนี้เป็นการ restore จากอีก cluster โดยให้ทั้งสอง cluster ใช้ S3 bucket เดียวกัน
~/repos/lab velero restore create wordpress-restore-10 --from-backup wordpress10 --restore-volumes=true -n kommander-default-workspace
Restore request "wordpress-restore-10" submitted successfully.
Run `velero restore describe wordpress-restore-10` or `velero restore logs wordpress-restore-10` for more details.เรียกใช้ wordpress เพื่อตรวจสอบว่าข้อมูลมีการ restore กลับมาครบถ้วน
Tech Notes 