OpenDJ Pets on Kubernetes





Stateless "12-factor" applications are all the rage, but there are some kinds of services that are inherently stateful. Good examples are things like relational databases (Postgres, MySQL) and NoSQL databases (Cassandra, etc).

These services are difficult to containerize, because the default docker model favours ephemeral containers where the data disappears when the container is destroyed.

These services also have a strong need for identity. A database "primary" server is different than the "slave". In Cassandra, certain nodes are designated as seed nodes, and so on.

OpenDJ is an open source LDAP directory server from ForgeRock. LDAP servers are inherently "pet like" insomuch as the directory data must persist beyond the container lifetime. OpenDJ nodes also replicate data between themselves to provide high-availability and therefore need some kind of stable network identity.

Kubernetes 1.3  introduces a feature called "Pet Sets" that is designed specifically for these kinds of stateful applications.   A Kubernetes PetSet provides applications with:
  • Permanent hostnames that persist across restarts
  • Automatically provisioned persistent disks per container that live beyond the life of a container
  • Unique identities in a group to allow for clustering and leader election
  • Initialization containers which are critical for starting up clustered applications

These features are exactly what we need to deploy OpenDJ instances.  If you want to give this a try, read on...

You will need access to a Kubernetes 1.3 environment. Using minikube is the recommended way to get started on the desktop. 

You will need to fork and clone the ForgeRock Docker repository to build the OpenDJ base image. The repository is on our stash server: 


To build the OpenDJ image, you will do something like:

cd opendj
docker build -t forgerock/opendj:latest . 

If you are using minikube,  you should connect your docker client to the docker daemon running in your minikube cluster (use minikube docker-env).  Kubernetes will not need to "pull" the image from a registry - it will already be loaded.  For development this approach will speed things up considerably.


Take a look at the README for the OpenDJ image. There are a few environment variables that the container uses to determine how it is bootstrapped and configured.  The most important ones:
  • BOOTSTRAP: Path to a shell script that will initialize OpenDJ. This is only executed if the data/config directory is empty. Defaults to /opt/opendj/boostrap/setup.sh
  • BASE_DN: The base DN to create. Used in setup and replication
  • DJ_MASTER_SERVER: If set, run.sh will call bootstrap/replicate.sh to enable replication to this master. This only happens if the data/config directory does not exist

There are sample bootstrap setup.sh scripts provided as part of the container, but you can override these and provide your own script.  

Next,  fork and clone the ForgeRock Kubernetes project here:

The opendj directory contains the Pet Set example.  You must edit the files to suit your needs, but as provided, the artifacts do the following:

  • Configures two OpenDJ servers (opendj-0 and opendj-1) in a pet set. 
  • Runs the  cts/setup.sh script provided as part of the docker image to configure OpenDJ as an OpenAM CTS server.
  • Optionally assigns persistent volumes to each pet, so the data will live across restarts
  • Assigns "opendj-0" as the master.  The replicate.sh script provided as part of the Docker image will replicate each node to this master.  The script ignores any attempt by the master to replicate to itself.  As each pet is added (Kubernetes creates them in order) replication will be configured between that pet and the opendj-0 master. 
  • Creates a Kubernetes service to access the OpenDJ instances. Instances can be addressed by their unique name (opendj-1), or by a service name (opendj) which will go through a primitive load balancing function (at this time round robin).  Applications can also perform DNS lookup on the opendj SRV record to obtain a list of all the OpenDJ instances in the cluster.

The replication topology is quite simple. We simply replicate each OpenDJ instance to opendj-0. This is going to work fine for small OpenDJ clusters. For more complex installations you will need to enhance this example.


To create thet petset:

kubectl create -f opendj/


If you bring up the minikube dashboard:

minikube dashboard 

You should see the two pets being created (be patient, this takes a while). 




Take a look at the pod logs using the dashboard or:

kubectl logs opendj-0 -f 

Now try scaling up your PetSet. In the dashboard, edit the Pet Set object, and change the number of replicas from 2 to 3:



















You should see a new OpenDJ instance being created. If you examine the logs for that instance, you will see it has joined the replication topology. 

Note: Scaling down the Pet Set is not implemented at this time. Kubernetes will remove the pod, but the OpenDJ instances will still think the scaled down node is in the replication topology. 




Comments

Post a Comment

Popular posts from this blog

Introducing ds-operator, the ForgeRock Directory Services Operator for Kubernetes

Image
ForgeRock Directory Services 7.0 was a big achievement for the Grenoble Directory team.  It is the only "Kubernetes native" directory where you can add a new replica using kubectl: kubectl scale sts/ds-idrepo --replicas=3 The 7.0 deployment is assembled using standard Kubernetes primitives such as StatefulSets , Persistent Volume Claims, and Services.  This is all built and orchestrated using Skaffold and Kustomize .  An emerging pattern in the Kubernetes world is the use of Custom Resources and Operators .  Broadly speaking, a custom resource is the declaration of the desired system state, and the operator's job is to observe the current state and bring the system into alignment with the declared state: source: https://blog.container-solutions.com/kubernetes-operators-explained The Kubernetes API server (the thing that responds to your kubectl commands) can be extended to handle new custom types.  A custom resource definition (CRD) describes to the API server the syntax
Read more

Automating OpenDJ backups on Kubernetes

Image
Kubernetes   StatefulSets  are designed to run "pet" like services such as databases.   ForgeRock's OpenDJ LDAP server is an excellent fit for StatefulSets as it requires stable network identity and persistent storage. The ForgeOps project contains a Kubernetes Helm chart to deploy DJ to a Kubernetes cluster. Using a StatefulSet, the cluster will auto-provision persistent storage for our pod. We configure OpenDJ to place its backend database on this storage volume. This gives us persistence that survives container restarts, or even restarts of the cluster. As long as we don't delete the underlying persistent volume, our data is safe. Persistent storage is quite reliable, but we typically want additional offline backups. The high level approach to accomplish this is as follows: Configure the OpenDJ container to support scheduled backups to a volume. Configure a Kubernetes volume to store the backups. Create a sidecar container that archives the backups
Read more

Deploying the ForgeRock platform on Kubernetes using Skaffold and Kustomize

Image
If you are following along with the  ForgeOps repository, you will see some significant changes in the way we deploy the ForgeRock IAM platform to Kubernetes.  These changes are aimed at dramatically simplifying the workflow to configure, test and deploy ForgeRock Access Manager, Identity Manager, Directory Services and the Identity Gateway. To understand the motivation for the change, let's recap the current deployment approach: The Kubernetes manifests are maintained in one git repository ( forgeops ), while the product configuration is another ( forgeops-init ). At runtime ,  Kubernetes init containers clone the configuration from git and make it  available to the component using a shared volume. The advantage of this approach is that the docker container for a product can be (relatively) stable. Usually it is the configuration that is changing, not the product binary. This approach seemed like a good idea at the time, but in retrospect it created a lot of c
Read more