High availability and large scale configuration

From ControlTier

As of 3.2, the default ControlTier installation uses a client-server configuration wherein the clients stay synchronized with the server via a pull based mechanism. During the execution of a workflow command, the necessary resource model data, module code and package artifacts are downloaded from the centralized ControlTier server via HTTP/GET.

This arrangement works well in development and QA environments because in that stage, it's preferable to stay in sync with the latest changes pushed to the server. This also facilitates the development iteration cycle.

On the other hand, for large scale and isolated production environments the default configuration poses these potential drawbacks:

• Performance bottleneck: Server must respond to a mixture of requests: file downloads, object data lookups, data transactions. This mixture of workload may interfere with optimal management operations.
• Single point of failure: If the server is down, some commands on the client nodes may stop working or have unexpected behavior.
• Uncontrolled change: Changes to the object data, module code or package artifacts may be uploaded and released immediately, outside of change control windows.

This document describes a configuration and update model where:

• all necessary data and artifacts are produced in a build process that results in packaged artifacts
• the centralized server bottleneck is avoided by disconnecting the clients from live data synchronization and using packaged data instead
• updates to object data, modules and deployments are done during approved change windows.

Contents 1 Resource model data 1.1 Process 1.2 Delivery 2 Module code 2.1 Process 2.2 Delivery 3 Package artifacts 3.1 Process 3.2 Delivery

Resource model data

CTL commands are data driven. This data is provided as a data representation of a CTL resource, normally in a file called resource.properties. During the course of a command execution, a step exists to synchronize the data representation with the state on the server (i.e., via Get-Properties).

When a completely off-line mode is desirable, object data can be archived and delivered out-of-band to avoid the server request. Two facilities exist in the framework to support this notion:

1. ProjectBuilder's archive-resources command: This command queries the server and produces an archive of object data that can be delivered to the client machines in packaged form.
2. CTL's ctl-archive command: This command extracts the archives produced by archive-resources

Process

This approach uses a traditional software life cycle wherein,

• the object data is maintained as a set of text files in a source code repository (i.e., in project.xml),
• a build process reads the files, loads them into workbench and then generates a releasable artifact
• the artifact is released at a change control governed time
• the archive is extracted to the appropriate Nodes

Delivery

The diagram below describes the overall flow.

The diagram shows a process flow in roughly 6 steps

1. A ProjectBuilder query specifies a set of objects it wants to archive data for and,
2. then retrieves a representation for each one in the query result
3. An archive of the object data is produced.
4. A distribution tool (which could be ctl-exec or a package facility like RPM) copies it to each needed CTL node
5. and the relevant object data is extracted from the archive.
6. A framework configuration disables the Get-Properties step from connecting to the server.

Here's a diagram that shows a subset of the process but using RPM and Yum as the distribution tool:

It shows Yum is the distribution mechanism and the postinstall step in the RPM invokes the ctl-archive command.

Module code

A module is a software artifact produced from a type.xml and a set of module source files during the ProjectBuilder build process. This build process includes archiving the generated artifacts into a JAR file which is loaded into the ControlTier server where the type definitions are merged into the project model. CTL has a built in mechanism to download new versions of these modules automatically or on demand so the clients stay synchronized with the latest versions on the server.

When a completely off-line mode is desirable, modules can be archived and delivered out-of-band to avoid these server requests. Two facilities exist in the framework to support this notion:

1. ProjectBuilder's build-library command: This command reads module sources and produces a library that can be delivered to the client machines in packaged form. This form is called a CTL "extension".
2. CTL's ctl-extension command: This command extracts the archives produced by build-library and installs them to the client.

Process

This approach uses a traditional software life cycle wherein,

• the type data and module sources are maintained as a set of text files in a source code repository (i.e., in type.xml),
• the build-library process reads the files, generates a releasable artifact and loads them into the server.
• the extension artifact is released at a change control governed time
• the extension archive is extracted to the appropriate Nodes

Delivery

The diagram below describes the overall flow.

The diagram shows a process flow in roughly 4 steps

1. A ProjectBuilder reads a set of module source directories and generates an extension JAR file which can be uploaded to the server,
2. An extension jar can be staged as a releasable artifact .
3. A distribution tool (which could be ctl-exec or a package facility like RPM) copies it to each needed CTL node
4. and the relevant module data is extracted from the extension archive.

Package artifacts

Package artifacts are delivered via "Deploy", a foundational workflow that includes a step to install package dependencies for a given software deployment. This particular step is handled by Packages-Install, a command which reads the object's package dependencies and for each one carries out its package install cycle. This cycle is also a workflow and the step named "get", downloads the package from the server's WebDAV component.

For a large scale, high availability configuration, serving all file download requests from a centralized server may be impeded by various bottlenecks (network, disk, cpu). A centralized server will also open itself to various availability risks (system, subnet).

An alternative configuration would be to use traditional web scaling techniques: caching proxies and VIPs.

• caching servers can be read only copies of the authoritative copies maintained on the WebDAV. Caching servers can be deployed in a fanned out arrangement, to best match up with subnet routing and bandwidth constraints.
• Redundant proxies can be advertised as VIP addresses. ControlTier client machines can be configured to use this VIP addresses to avoid availability outages.

Process

The process to support this model is roughly the same as the default configuration. Depending on the proxy server software employed, there may be additional cache synchronization procedures.

Delivery

The diagram below describes a possible deployment configuration: