AWX on Kubernetes/OpenShift

Ansible AWX provides a RESTful API, web interface and a task engine that is built on top of Ansible. It is the upstream project for Red Hat Ansible Automation Platform.

Persistent Volume

We need storage to persist AWX data. There is no limitation on the type of storage that has already been integrated with the OpenShift or K8 cluster.

Storage class

A K8 StorageClass describes and classifies storage that can be requested, as well as provides a means for passing parameters for dynamically provisioned storage on demand – it abstract the details of underlying storage in a simple fashion.

Query the current storage classes from your cluster:

oc get sc
NAME                                  PROVISIONER                                 RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
cephfs                                ceph.com/cephfs                             Delete          Immediate              false                  712d
nfs                                   cluster.local/wso2-nfs-server-provisioner   Delete          Immediate              true                   49d
ocs-storagecluster-cephfs (default)   openshift-storage.cephfs.csi.ceph.com       Delete          Immediate              true                   299d

In our case, we shall use ocs-storagecluster-cephfs (default) StorageClass.

Tools

Make Tools and Git

Ensure that you have installed Git and Make Tools on your bastion host that you’re using to interact with your OpenShift/K8 cluster.

RHEL based systems

#RHEL based systems #

sudo yum install curl jq


Updated:
  git.x86_64 0:1.8.3.1-25.el7_9                                                                                                                                      

Dependency Updated:
  perl-Git.noarch 0:1.8.3.1-25.el7_9                                                                                                                                 

Complete!

Dev Tools

sudo yum group install "Development Tools"


Installing for group install "Development Tools":
 bison                                         x86_64                          3.0.4-2.el7                                    base                             674 k
 byacc                                         x86_64                          1.9.20130304-3.el7                             base                              65 k
 cscope                                        x86_64                          15.8-10.el7                                    base                             203 k
 ctags                                         x86_64                          5.8-13.el7                                     base                             155 k
 diffstat                                      x86_64                          1.57-4.el7                                     base                              35 k
 doxygen                                       x86_64                          1:1.8.5-4.el7                                  base                             3.6 M

...

Installed:
  bison.x86_64 0:3.0.4-2.el7            byacc.x86_64 0:1.9.20130304-3.el7                 cscope.x86_64 0:15.8-10.el7           ctags.x86_64 0:5.8-13.el7           
  diffstat.x86_64 0:1.57-4.el7          doxygen.x86_64 1:1.8.5-4.el7                      elfutils.x86_64 0:0.176-5.el7         flex.x86_64 0:2.5.37-6.el7          
  gcc-gfortran.x86_64 0:4.8.5-44.el7    indent.x86_64 0:2.2.11-13.el7                     intltool.noarch 0:0.50.2-7.el7        patchutils.x86_64 0:0.3.3-5.el7_9   
  rcs.x86_64 0:5.9.0-7.el7              redhat-rpm-config.noarch 0:9.1.0-88.el7.centos    rpm-build.x86_64 0:4.11.3-48.el7_9    rpm-sign.x86_64 0:4.11.3-48.el7_9   
  subversion.x86_64 0:1.7.14-16.el7     swig.x86_64 0:2.0.10-5.el7                        systemtap.x86_64 0:4.0-13.el7        

Dependency Installed:
  boost-date-time.x86_64 0:1.53.0-28.el7                    bzip2.x86_64 0:1.0.6-13.el7                           dwz.x86_64 0:0.11-3.el7                           
  dyninst.x86_64 0:9.3.1-3.el7                              efivar-libs.x86_64 0:36-12.el7                        emacs-filesystem.noarch 1:24.3-23.el7_9.1         
  gdb.x86_64 0:7.6.1-120.el7                                gettext-common-devel.noarch 0:0.19.8.1-3.el7          gettext-devel.x86_64 0:0.19.8.1-3.el7             
  kernel-debug-devel.x86_64 0:3.10.0-1160.95.1.el7          libdwarf.x86_64 0:20130207-4.el7                      libgfortran.x86_64 0:4.8.5-44.el7                 
  libmodman.x86_64 0:2.0.1-8.el7                            libproxy.x86_64 0:0.4.11-11.el7                       libquadmath.x86_64 0:4.8.5-44.el7                 
  libquadmath-devel.x86_64 0:4.8.5-44.el7                   mokutil.x86_64 0:15-8.el7                             neon.x86_64 0:0.30.0-4.el7                        
  pakchois.x86_64 0:0.4-10.el7                              perl-XML-Parser.x86_64 0:2.41-10.el7                  perl-srpm-macros.noarch 0:1-8.el7                 
  python-srpm-macros.noarch 0:3-34.el7                      subversion-libs.x86_64 0:1.7.14-16.el7                systemtap-client.x86_64 0:4.0-13.el7              
  systemtap-devel.x86_64 0:4.0-13.el7                       systemtap-runtime.x86_64 0:4.0-13.el7                

Complete!

Debian

### Debian / Ubuntu ###
sudo apt update
sudo apt install git build-essential curl jq  -y

AWX Installation on OpenShift/K8

Download the AWX Operator

git clone to download the operator to your host.

[root@svdt8bastion ~]# git clone https://github.com/ansible/awx-operator.git

Cloning into 'awx-operator'...
remote: Enumerating objects: 9223, done.
remote: Counting objects: 100% (1938/1938), done.
remote: Compressing objects: 100% (287/287), done.
remote: Total 9223 (delta 1745), reused 1702 (delta 1647), pack-reused 7285
Receiving objects: 100% (9223/9223), 2.47 MiB | 0 bytes/s, done.
Resolving deltas: 100% (5331/5331), done.

Create AWX namespace

OpenShift

export NAMESPACE=ansible-awx
oc create ns ${NAMESPACE}

K8s

export NAMESPACE=ansible-awx
kubectl create ns ${NAMESPACE}

Switch to the created namespace

# oc project ${NAMESPACE}
Now using project "ansible-awx" on server "https://api.ocp.technnix.net:6443".

Release tag

AWX_RELEASE_TAG=curl -s https://api.github.com/repos/ansible/awx-operator/releases/latest | grep tag_name | cut -d '"' -f 4 ;
echo $AWX_RELEASE_TAG
git checkout $AWX_RELEASE_TAG

Expected output

# git checkout $AWX_RELEASE_TAG
Note: checking out '2.5.2'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 7012a6a... Modify how pg password is set in postgres pod (#1540)

Deploy AWS operator to the OpenShift/K8s cluster

export NAMESPACE=ansible-awx
make deploy

Expected output

usage: git ls-remote [--heads] [--tags]  [-u <exec> | --upload-pack <exec>]
                     [-q|--quiet] [--exit-code] [--get-url] [<repository> [<refs>...]]
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
namespace/ansible-awx configured
customresourcedefinition.apiextensions.k8s.io/awxbackups.awx.ansible.com configured
customresourcedefinition.apiextensions.k8s.io/awxrestores.awx.ansible.com configured
customresourcedefinition.apiextensions.k8s.io/awxs.awx.ansible.com configured
serviceaccount/awx-operator-controller-manager created
role.rbac.authorization.k8s.io/awx-operator-awx-manager-role created
role.rbac.authorization.k8s.io/awx-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/awx-operator-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/awx-operator-proxy-role unchanged
rolebinding.rbac.authorization.k8s.io/awx-operator-awx-manager-rolebinding created
rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/awx-operator-proxy-rolebinding configured
configmap/awx-operator-awx-manager-config created
service/awx-operator-controller-manager-metrics-service created
deployment.apps/awx-operator-controller-manager created

After successful deployment of the operator, check the running pods

oc get po
NAME                                               READY   STATUS                   RESTARTS   AGE
awx-operator-controller-manager-6544864fcd-rdpzr   2/2     Running                  0          2m12s
awx-operator-controller-manager-6544864fcd-sb6sm   0/2     ContainerStatusUnknown   2          3m48s

Install AWX on OpenShift

PVC

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: awx-data-pvc
  namespace: ansible-awx
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ocs-storagecluster-cephfs
  resources:
    requests:
      storage: 10Gi
EOF

Expected output

persistentvolumeclaim/awx-data-pvc created

PVC created successfully

oc get pvc

NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                AGE
awx-data-pvc   Bound    pvc-8ea4b8c1-535e-4bcc-b129-95e6b6e2268e   10Gi       RWX            ocs-storagecluster-cephfs   94s

Create AWX deployment File

Lets deploy AWX instance by creating an ansible-awx.yaml file with following content.

vim ansible-awx-deployment.yaml

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: ansible-awx
spec:
  service_type: ClusterIP
  projects_persistence: true
  projects_storage_access_mode: ReadWriteMany
  web_extra_volume_mounts: |
    - name: static-data
      mountPath: /var/lib/projects
  extra_volumes: |
    - name: static-data
      persistentVolumeClaim:
        claimName:  awx-data-pvc

Deploy

#  oc apply -f ansible-awx-deployment.yaml

awx.awx.ansible.com/ansible-awx created

Check pods

oc get po

NAME                                               READY   STATUS    RESTARTS   AGE
ansible-awx-postgres-13-0                          1/1     Running   0          14h
ansible-awx-task-675bb6846d-7lhgj                  4/4     Running   0          13h
ansible-awx-web-66d967d896-7h7tk                   3/3     Running   0          12h
awx-operator-controller-manager-6544864fcd-rdpzr   2/2     Running   0          21h

Access AWX Container Shell

Get the deployment

oc get deployment

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
ansible-awx-task                  1/1     1            1           16h
ansible-awx-web                   1/1     1            1           16h
awx-operator-controller-manager   1/1     1            1           21h

Access container shell

oc exec -ti deploy/ansible-awx-task -c  ansible-awx-task -- /bin/bash
oc exec -ti deploy/ansible-awx-web -c  ansible-awx-web -- /bin/bash

Expose the AWX service

A K8 service is an abstraction layer that defines a logical set of pods and enables external traffic exposure, service discovery and load balancing for those pods.

Check the services in the ansible-awx namespace. Take a keen interest in the ansible-awx-service through the ClusterIP.

oc get svc

NAME                                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
ansible-awx-postgres-13                           ClusterIP   None             <none>        5432/TCP   2d18h
ansible-awx-service                               ClusterIP   172.30.251.48    <none>        80/TCP     2d18h
awx-operator-controller-manager-metrics-service   ClusterIP   172.30.227.128   <none>        8443/TCP   2d22h

At this stage, the assumption is that you already have an OpenShift/K8 cluster with an ingress controller. We can epose the servie through a route using the ingress .

Check the DNS records for the OpenShift ingress.

# nslookup *.apps.devocp.technnix.net

Server:		xx.xx.xx.xx
Address:	xx.xx.xx.xx#53

Name:	*.apps.devocp.technnix.net
Address: 10.196.101.31

# oc get ingresses.config/cluster -o jsonpath={.spec.domain}

apps.devocp.technnix.net

Create route configuration manifest for AWX. A route allows us to host our application(AWX) at a public URL.

Please note the following elements in the manifest:

spec:
  host: ansible-awx.apps.devocp.technnix.net
  to:
    kind: Service
    name: ansible-awx-service
    weight: 100
  port:
    targetPort: http

YAML AWX Route Manifest Definition:

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: ansible-awx
  managedFields:
    - manager: OpenAPI-Generator
      operation: Update
      apiVersion: route.openshift.io/v1
      fieldsType: FieldsV1

    - manager: openshift-router
      operation: Update
      apiVersion: route.openshift.io/v1
  namespace: ansible-awx
  labels:
    app.kubernetes.io/component: awx
    app.kubernetes.io/managed-by: awx-operator
    app.kubernetes.io/operator-version: 2.2.1
    app.kubernetes.io/part-of: ansible-awx
spec:
  host: ansible-awx.apps.devocp.technnix.net
  to:
    kind: Service
    name: ansible-awx-service
    weight: 100
  port:
    targetPort: http
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect
  wildcardPolicy: None
status:
  ingress:
    - host: ansible-awx.apps.devocp.technnix.net
      routerName: default
      wildcardPolicy: None
      routerCanonicalHostname: router-default.apps.devocp.technnix.net

Deploy the route yaml file

oc apply -f awx-route.yaml

route.route.openshift.io/ansible-awx created

Validate the created route

oc get routes

NAME                  HOST/PORT                                                   PATH   SERVICES              PORT   TERMINATION     WILDCARD
ansible-awx           ansible-awx.apps.devocp.technnix.net                              ansible-awx-service   http   edge/Redirect   None
ansible-awx-service   ansible-awx-service-ansible-awx.apps.devocp.technnix.net          ansible-awx-service   http                   None

Access the AWX Web console

Access the web console from the URL above:  http://ansible-awx.apps.devocp.technnix.net .

Extract the admin password

From your namespace, lets extract the admin password

# oc get secrets | grep -i admin-password

ansible-awx-admin-password                        Opaque                                1      18h

# oc get secret ansible-awx-admin-password -o jsonpath="{.data.password}" -n awx | base64 --decode ; echo

ru0Wq6LP9dAJrbZZTI4tX7IH4WVBGaSk

Use the password displayed to login to the Dashboard

Username : admin

Password : <password>