Model-driven command dispatching

From ControlTier

This document describes the important concepts and mechanisms that CTL Command Dispatcher uses to execute Commands based on the logical representation stored in the ControlTier resource model.

This diagram indicates the benefit of using a model-driven command dispatcher, versus an approach that uses rote scripts or procedures:

Description

The Control Tier is a way of modeling your Business Service so that software programs and artifacts (Services and Packages), groups of related Services (Sites), and coordination points that affect the model (Builders, Updaters) can be identified and grouped by their logical purposes. You will also model each Virtual Machine or physical computer as a Node, which represents an end-point for deployment and dispatching in the real-world. (See: Understanding the abstracted administration paradigm)

Once you have created a representative model of your system, those Resources can be mapped to the real-world as effective Business Service components by modeling the connection between each deployable Resource and any set of Nodes to which it should be deployed.

Sounds great, but how does that work?

Diagrammed below are the steps that happen right after installation of the ControlTier software, up to the point of dispatching a Command to a remote Node based on Model configuration.

How it works

The mechanism for dispatching a specific command to the correct real-world location requires that we create a useful Model of the real-world.

The ControlTier Server holds a database which contains the Model and all of the data about the resources' relationships and properties. Each ControlTier Client installation on a virtual machine or physical computer is represented in the Model as a Node Resource when it is registered.

The manager can then specify a relationship between a Node and a specific deployable Resource, such as a software Service or a set of related Resources called a Site.

The Client can then query the server to retrieve all necessary control modules and metadata for the Services that are related to the Node.

Once installed, the Client can execute Commands for the Service without needing to to know all of the detail about specific configuration. The user only needs to know the name of the Resource.

Sites are used for managing a group of Resources which may all be deployed on different Nodes.

The user can then execute a command for the logical Site Resource, and the command will be "node-dispatched" to the appropriate Nodes.

Basics of Modeling

The modeling system of ControlTier is complex, and it needs to be in order to accurately represent complex systems and also to be functionally useful.

However, at the basic level there are only three important concepts in the modeling system:

1. Resources
2. Relationships
3. Properties

Resources 

every resource that we want to represent in the system is called a Resource.

Relationships

every Resource can have multiple other Resources as "parents". Some Resources can have multiple other Resources as "children".

Properties

every Resource can have literal values attached to it which are identified by a name

These basics allow you to model things in your enironment, as well as relationships between them and some of the pertinent data about them. In addition to these concepts, the only other important concept to add is that of Types.

Type 

a class of Resources, which confers on its members the ability to have certain Properties, Relationships, and other specific traits.

You can see that for example, there is a need to differentiate between the concept of a Node and a software Service, and that Resources of each of those Types will have different needs: different Properties will be necessary, as well as the ability to have only certain other Types of Resources as children.

This is the basics of the modeling system, covered in more depth in the Type modeling section.

Basics of Dispatching

The ControlTier Client installation has two components:

1. A set of commandline tools for executing commands and libraries (stored in CTL_HOME directory)
2. A local directory for storing control modules, metadata, and configuration (stored in CTL_BASE directory)

Once you have registered the Client installation as a Node to the Server, you will be able to "deploy" Services, Sites, etc. onto the Node. This is simply a matter of adding the Resource as a child resource of the Node:

Adding a Service "deploys" it to the Client inside the Model:

Now that the Model reflects the configuration we want, we can effect the change in the ControlTier Client that the Node represents in two ways:

1. Remotely, using a pull-based mechanism
2. (Locally on the server, automatically)
   • When you deploy to the Node that represents the Server itself, the deployment steps below happen automatically

This shows an example of running the "install" action to do a pull-based install:

In either case, a set of Metadata for the deployed Resource, and any necessary Control Modules are installed into the Client's CTL_BASE directory.

The Metadata includes all properties and relationships of the Resource (scope is configurable).

Each deployed Resource also gets an installation directory inside CTL_BASE. This directory holds the metadata file, and serves as a location where the Resource can install templates, files, etc.

The Dispatcher then dynamically scans the available Resources for use, and the Control Module used by each Resource, providing the commandline user with a list of the commands available for execution with that Resource.

How Metadata is Deployed

When a ControlTier Client wants to retrieve all of the Resources deployed to its Node, it runs the "install" action using the "ctl-project" command:

$ ctl-project -a install

defaulting to project: demo
"Install" command running for resource: (Service) myService

This action does a few things:

1. Retrieves a resources.properties file from the Server (WebDAV GET)
2. Retrieves a nodes.properties file from the Server (WebDAV GET)
3. Retrieves any control Modules that are directly deployed to the Client's Node, based on resources.properties (WebDAV GET)
4. Creates the necessary resource.instance.dir for any deployed Resources, based on resources.properties
5. Retrieves any resource.properties files for Resources that are deployed to the Client's Node, based on resources.properties (via Server API using HTTP GET)
6. Runs the built-in "Install" Command for those Resources, which allows custom installation code to occur if necessary.

When an Resource is installed in the CTL_BASE directory, it lives inside the appropriate "depot" directory, at a path using the Resource's name and Type name:

$CTL_BASE/projects/project/resources/type/Resource

This installation directory is referred to as the "resource.instance.dir" location for the Resource. Inside the resource.instance.dir, is:

${resource.instance.dir}/var/resource.properties

The "resource.properties" file contains the Metadata, serialized as Java Properties format. This is a simple "key=value" text file, containing all of the metadata that is available for use by the Resource inside a Command execution context.

For example here is some of the content for the "myService" Resource deployed earlier:

...
resource.classname=Service
resource.controller-module=Service
resource.deployment-basedir=/Users/greg/ctierc/ctl/projects/demo/resources/Service/myService/base
resource.deployment-install-root=/Users/greg/ctierc/ctl/projects/demo/resources/Service/myService/install
resource.deployment-manages=false
resource.deployment-startup-rank=1
resource.depot=demo
resource.description=my service
resource.dirlist=bin,logs,var,var/transforms
resource.install.auto=true
resource.install.nomodule=false
resource.instance.dir=/Users/greg/ctierc/ctl/projects/demo/resources/Service/myService
resource.mapref-uri=http\://www.opendepo.org/ct/demo/Resources\#obj_2
resource.mapref-uri.encoded=http%3A%2F%2Fwww.opendepo.org%2Fct%2Fdemo%2FResources%23obj_2
resource.name=myService
resource.order=Service
...

This Metadata file provides the contextual data that the Dispatcher uses when it executes a Command.

How Modules are Deployed

After the "install" action is run (see above), the Client downloads any Modules that it needs as described in the resources.properties file.

The file contains a line for each deployed Resource like this:

Resource-deployment.project.type.Resource=node[,node...]

For example, this is the content produced in the above example:

Resource-deployment.demo.Service.myService=Gozer.local

After seeing that a certain Resource is deployed to this Node, the Client will retrieve the resource.properties for that Resource. Based on this information, it determines which Control Module is needed.

Definition:

Control Module 

An archive file which contains Type and Command definitions and metadata. Every deployable Type in the Model has a corresponding Control Module.

Thus, the Control Module that is needed is the one for the Type of the deployed Resource.

If the necessary Control Module is not already installed, it is downloaded from the Server, using the WebDAV directory:

http://server-host:port/webdav-path/project/publish/modules/Type-head.jar

Since a Module corresponds to a Type in the Type Modeling system, it may be necessary to also download the Module for any Supertype that the Type is extended from. All of those Module dependencies are downloaded at this point.

These Modules are Java Jar archives, and their contents are now expanded into the "modules" directory for the particular project "depot" directory:

$CTL_BASE/projects/project/modules

$CTL_BASE/projects/project/modules/Service

$CTL_BASE/projects/project/modules/CustomService

At this point the Resource can be used to execute Commands.

Advanced Command Dispatching

The ControlTier Client tool "ctl" has several commandline options that allow you to easily dispatch Commands to more than one Node.

For example, suppose you had configured two Nodes to use the myService Resource:

Normally, you could execute a command on the current Node by running it directly:

ctl -t Service -r myService -c Start

However, you can use the "node filtering" options of the ctl tool to send this command to all Nodes where the myService Resource is installed:

ctl -I '.*' -t Service -r myService -c Start

begin workflow command (1/1) -> "assertServiceIsUp " ...
end workflow command (1/1) -> "assertServiceIsUp "
Connecting to Gozer.local:22
cmd : ctl -p demo -t Service -r myService -c Start 
[greg@localhost demo.Service.myService Start][INFO] begin workflow command (1/1) -> "assertServiceIsUp " ...
[greg@localhost demo.Service.myService Start][INFO] end workflow command (1/1) -> "assertServiceIsUp "

You can see that first the command is executed locally, and then subsequently it was dispatched via a remote SSH connection to the "Gozer.local" Node.

The "ctl" tool has these options for dispatching remotely:

Both of the -I/-X filter options allow these filter specifiers in the value:

For example, you could dispatch to all Unix nodes:

ctl -I osfamily=unix -t Service -r myService -c Start

Or you could filter based on the Node tags:

ctl -I tag=dev -t Service -r myService -c Start

All of the metadata needed to perform these filters comes from the nodes.properties file which is retrieved during the "install" action as described above.