Centralized ArgoCD Workload Cluster Deployment

A step by step guide to deploy a sample workload to a workload cluster, using a centralized GitOps approach with ArgoCD.

Prerequisites

A Nephio Management cluster. See the installation guides for detailed environment options.
An ArgoCD installation, such as the ArgoCD Full package.
Optional:
- Access to Gitea, which is used in the demo environment as the Git provider.
- Access to the ArgoCD UI is optional, but highly recommended.

Note

If using a sandbox demo environment, most of the above prerequisites are already satisfied.

This exercise will take us from a system with only the Nephio Management cluster setup, to a deployment with:

A ArgoCD workload cluster.
A repository for said cluster, registered with Nephio Porch.
A sample workload deployed to said cluster via an ArgoCD Config Management Plugin (CMP).

Step 1: Deploy the ArgoCD Workload Cluster

Note

After fresh docker install, verify the docker supplementary group is loaded by executing id | grep docker. If not, logout and login to the VM or execute the newgrp docker to ensure the docker supplementary group is loaded.

ArgoCD support in Nephio relies upon our custom ArgoCD KPT Config Management Plugins (CMP). To add the KPT Repo and KPT Render CMPs to the ArgoCD instance we can use one of the following options.

Either, a. apply the patch directly, changing the VERSION of the plugins:

kubectl patch deployment argocd-repo-server -n argocd --type json -p='[
   {
    "op": "add",
    "path": "/spec/template/spec/containers/-",
    "value": {
      "name": "kpt-repo-argo-cmp",
      "image": "docker.io/nephio/kpt-repo-argo-cmp:VERSION",
      "command": ["/var/run/argocd/argocd-cmp-server"],
      "securityContext": {
          "runAsNonRoot": true,
          "runAsUser": 999
      },
      "volumeMounts": [
        {
           "name": "var-files",
           "mountPath": "/var/run/argocd"
        },
        {
           "name": "cmp-tmp",
           "mountPath": "/tmp"
        },
        {
           "name": "plugins",
           "mountPath": "/home/argocd/cmp-server/plugins"
        }
      ]
    }
  },
  {
    "op": "add",
    "path": "/spec/template/spec/containers/-",
    "value": {
      "name": "kpt-render-argo-cmp",
      "image": "docker.io/nephio/kpt-render-argo-cmp:VERSION",
      "command": ["/var/run/argocd/argocd-cmp-server"],
      "securityContext": {
          "runAsNonRoot": true,
          "runAsUser": 999
      },
      "volumeMounts": [
        {
           "name": "var-files",
           "mountPath": "/var/run/argocd"
        },
        {
           "name": "cmp-tmp",
           "mountPath": "/tmp"
        },
        {
           "name": "plugins",
           "mountPath": "/home/argocd/cmp-server/plugins"
        }
      ]
    }
  },
  {
    "op": "add",
    "path": "/spec/template/spec/volumes/-",
    "value": {
       "name": "cmp-tmp",
       "emptyDir": {}
    }
  },
  {
    "op": "add",
    "path": "/spec/template/spec/volumes/-",
    "value": {
       "name": "var-run-argocd",
       "emptyDir": {}
    }
  }
]'

or, b. apply the patch via our convenience shell script:

Note

The is will deploy the latest image versions.

curl -s https://raw.githubusercontent.com/nephio-project/nephio/refs/heads/main/gitops-tools/kpt-argocd-cmp/patch.sh > /tmp/patch.sh
/bin/bash /tmp/patch.sh

Verify that the patch has been applied, and that the 2 new CMP containers have been added to the argocd-repo-server:

kubectl get pods -n argocd | grep argocd-repo-server

Sample output:

argocd-repo-server-6d887d9cfc-9jvds                3/3     Running   0          3h29m

Verify that the catalog blueprint repositories are registered and Ready:

kubectl get repository

Sample output:

NAME                        TYPE   CONTENT   DEPLOYMENT   READY   ADDRESS
catalog-distros-sandbox     git    Package   false        True    https://github.com/nephio-project/catalog.git
catalog-infra-capi          git    Package   false        True    https://github.com/nephio-project/catalog.git
catalog-nephio-core         git    Package   false        True    https://github.com/nephio-project/catalog.git
catalog-nephio-optional     git    Package   false        True    https://github.com/nephio-project/catalog.git
catalog-workloads-free5gc   git    Package   false        True    https://github.com/nephio-project/catalog.git
catalog-workloads-oai-ran   git    Package   false        True    https://github.com/nephio-project/catalog.git
catalog-workloads-tools     git    Package   false        True    https://github.com/nephio-project/catalog.git
mgmt                        git    Package   true         True    http://172.18.0.200:3000/nephio/mgmt.git
mgmt-staging                git    Package   false        True    http://172.18.0.200:3000/nephio/mgmt-staging.git
oai-core-packages           git    Package   false        True    https://github.com/OPENAIRINTERFACE/oai-packages.git

Once Ready, we can utilize blueprint packages from these upstream repositories.

In this example, we will use the Porch package variant controller to deploy the new Workload Cluster.

This fully automates the onboarding process, including the auto approval and publishing of the new package.

Create a new PackageVariant CR for the Workload Cluster:

cat << EOF | kubectl apply -f - 
apiVersion: config.porch.kpt.dev/v1alpha1
kind: PackageVariant
metadata:
  name: argo-regional-cluster
spec:
  upstream:
    repo: catalog-infra-capi
    package: nephio-workload-cluster-argo
    workspaceName: main
  downstream:
    repo: mgmt
    package: regional
  annotations:
    approval.nephio.org/policy: initial
  pipeline:
    mutators:
    - image: gcr.io/kpt-fn/set-labels:v0.2.0
      configMap:
        nephio.org/site-type: regional
        nephio.org/region: us-west1
EOF

Sample output:

packagevariant.config.porch.kpt.dev/argo-regional-cluster created

Config Sync running in the Management cluster will create all the resources necessary to provision a KinD cluster, and register it with Nephio. This can take some time.

Step 2: Check the cluster installation

You can check if the cluster has been added to the management cluster:

kubectl get cl

kubectl get clusters.cluster.x-k8s.io

Sample output:

NAME       CLUSTERCLASS   PHASE         AGE    VERSION
regional   docker         Provisioned   179m   v1.31.0

To access the API server of that cluster, you can retrieve the kubeconfig file by pulling it from the Kubernetes Secret and decode the base64 encoding:

kubectl get secret regional-kubeconfig -o jsonpath='{.data.value}' | base64 -d > $HOME/.kube/regional-kubeconfig
export KUBECONFIG=$HOME/.kube/config:$HOME/.kube/regional-kubeconfig

You can then use it to access the Workload cluster directly:

kubectl get ns --context regional-admin@regional

Sample output:

NAME                 STATUS   AGE
default              Active   3h
kube-node-lease      Active   3h
kube-public          Active   3h
kube-system          Active   3h
local-path-storage   Active   179m
metallb-system       Active   179m

You should also check that the KinD cluster has come up fully by checking the machinesets. You should see READY and AVAILABLE replicas.

kubectl get machinesets

Sample output:

NAME                        CLUSTER    REPLICAS   READY   AVAILABLE   AGE    VERSION
regional-md-0-lmsqz-7nzzc   regional   1          1       1           3h1m   v1.31.0

Step 3: Investigate the ArgoCD specific CRs

Verify that the regional-argo-gitrepo PackageRevision has been created for the ArgoCD specific CRs. We want the Published v1 revision.

kubectl get packagerevision | grep "regional-argo-gitrepo"

Sample output:

mgmt.regional-argo-gitrepo.main                                 regional-argo-gitrepo                main               -1         false    Published   mgmt
mgmt.regional-argo-gitrepo.packagevariant-1                     regional-argo-gitrepo                packagevariant-1   1          true     Published   mgmt

Once verified:

Check the Ready state of the regional Git Repository:

kubectl get repository regional

Sample output:

NAME       TYPE   CONTENT   DEPLOYMENT   READY   ADDRESS
regional   git    Package   true         True    http://172.18.0.200:3000/nephio/regional.git

Now, we can get more details of the GitRepository CR:

kubectl get repository regional -o jsonpath='{.spec}'

Sample output:

{
  "content":"Package",
  "deployment":true,
  "git":
  {
    "branch":"main",
    "directory":"/",
    "repo":"http://172.18.0.200:3000/nephio/regional.git",
    "secretRef":
    {
      "name":"regional-access-token-porch"
    }
  },
  "type":"git"
}

The default git repo points to the internal Gitea repository created as part of the Workload Cluster installation, along with an associated secretRef to access it. It also defaults to reference the main branch of the repository and the root path.

Check that the jobs used to create the ArgoCD repo and cluster secrets are complete, and that the secrets were created in the argocd namespace:

kubectl get jobs
kubectl get secrets -n argocd | grep regional

Sample output:

NAME                                             STATUS     COMPLETIONS   DURATION   AGE
regional-create-argocd-cluster-from-kubeconfig   Complete   1/1           4s         8s
regional-create-argocd-repo-secret-from-porch    Complete   1/1           3s         73s
cluster-regional              Opaque   3      4h22m
regional-repo                 Opaque   4      4h22m

Next check the Ready state of the associated ArgoCD App, ensuring that it is Synced and Healthy. This App will point at the kpt-repo CMP, acting as an app-of-apps for the regional cluster. The apps created subsequently through the workloads will target the kpt-render CMP:

kubectl get apps -n argocd

Sample output:

NAME                    SYNC STATUS   HEALTH STATUS
regional                Synced        Healthy

Step 4: Deploy a sample workload to the Cluster

Create a new PackageVariant CR for the sample workload: For demo purposes, we are deploying the Free5GC Control Plane blueprint pkg, again, using a Porch PackageVariant.

cat << EOF | kubectl apply -f - 
apiVersion: config.porch.kpt.dev/v1alpha1
kind: PackageVariant
metadata:
  name: free5gc-control-plane
spec:
  upstream:
    repo: catalog-workloads-free5gc
    package: free5gc-cp
    workspaceName: main
  downstream:
    repo: regional
    package: free5gc-cp
  annotations:
    approval.nephio.org/policy: initial
EOF

Sample output:

packagevariant.config.porch.kpt.dev/free5gc-control-plane created

Once the Porch and Nephio controllers have completed their tasks, the mutated spec.upstream.package resources are pushed to the spec.downstream.repo git repository.

We can then check the state of the Workload Cluster deployment:

kubectl get po -n free5gc-cp --context regional-admin@regional

Sample output:

NAME                            READY   STATUS    RESTARTS   AGE
free5gc-ausf-5c44c54dbc-cr4n6   1/1     Running   0          11m
free5gc-nrf-b65d7f6-mn2qt       1/1     Running   0          11m
free5gc-nssf-857dd8d4b5-pqln2   1/1     Running   0          11m
free5gc-pcf-85c58bc6-5kcsj      1/1     Running   0          11m
free5gc-udm-784cc65c78-pwlcm    1/1     Running   0          11m
free5gc-udr-57549dcdf8-hl9cp    1/1     Running   0          11m
free5gc-webui-f54cb7d6f-89xcl   1/1     Running   0          11m
mongodb-0                       1/1     Running   0          11m

We can also check the ArgoCD apps to see our free5gc sub-app created:

kubectl get apps -n argocd

Sample output:

NAME                  SYNC STATUS   HEALTH STATUS
regional              Synced        Healthy
regional-free5gc-cp   Synced        Healthy