Running ad hoc commands

From ControlTier

ctl-exec

An ad hoc command is any system command or shell script can executed via the command dispatcher. Ad hoc commands can be executed via a command line utility named ctl-exec or as a Job run from a graphical console called CTL Center.

A screencast showing ctl-exec is available.

Ad hoc commands can be executed to registered nodes. All that needs to be running is SSH on the remote hosts and the server's public key in the destination account's allowed_keys file.

Command line usage

The ctl-exec tool has a variety of options that can be dispalyed via -help. Typing ctl-exec -help shows a complete listing of all the ctl-exec options.

$ ctl-exec -help
usage: ctl-exec [-h] [-v] [-V] [-q] [-p project] [-I nodes] [-X xnodes]
                [--threadcount <1>] [-keepgoing] [[-S] | [-s <>] | [-- command-args]]
 -Z,--filter-exclude-precedence    true/false. if true, exclusion filters have
                                   precedence over inclusion filters
 -C,--threadcount                  number of threads
 -F,--failednodes                  Filepath to store failed nodes (Since 3.4.7)
 -I,--nodes                        include node list
 -K,--keepgoing                    keep going if there are errors
 -S,--stdin                        read script from stdin
 -X,--xnodes                       exclude node list
 -h,--help                         print usage
 -p,--depot                        depot name
 -q,--quiet                        quiet mode
 -s,--scriptfile                   scriptfile script file
 -v,--verbose                      verbose mode

The general option pattern follows this form:

ctl-exec [options] [-- [command-options]]

These options control nodedispatch:

• -I,--nodes: Run the command on all nodes matching the regular expression.
• -X,--xnodes: Do not run the command on any node matching the regular expression. If no -I flag is specified, all nodes are looked up and then those matching the -X expression are filtered out.

Two other options of interest for are:

• -C,--threadcount: For all the matching nodedispatch results, execute the commands in the specified number of threads.
• -K: If specified, ctl-exec will continue if any errors occur.
• -F, --failednodes: specify a file to store the list of any nodes that failed. Specify again on retry to read the nodes list and execute only on the failed nodes.

See the document Dispatcher options for further discussion of node filtering.

Node Dispatch

The ctl-exec shell command includes a feature called node dispatch that allows you to call commands across a set of hosts without having to specify specific host names. Internally, CTL looks up the nodes that have the module or object, and then dispatches that command to those nodes. Options provide a way to filter the lookup results.

The ctl-exec command supports a listing feature that lists all the nodes to which commands can be dispatched. Run the ctl-exec command without arguments:

$ ctl-exec   
strongbad dev@development centos

By adding the "-v" option you can see more information about each node:

$ ctl-exec -v
strongbad:
   hostname: strongbad
   os-arch: i386
   os-family: unix
   os-name: Mac OS X
   os-version: 10.5.5
   tags: []
development:
   hostname: demo@development
   os-arch: i386
   os-family: unix
   os-name: Linux
   os-version: 2.5.5
   tags: []
centos:
   hostname: centos
   os-arch: i386
   os-family: unix
   os-name: Linux
   os-version: 2.5.5
   tags: []

Command line execution

Execute a command

The following executes the "ps" command across all registered nodes:

ctl-exec -- ps

This says execute the "ps" but limited to those nodes tagged with the name "tomcats":

ctl-exec -I tags=tomcats -- ps

Continue executing to next node if an error occurs:

ctl-exec -K -I tags=tomcats -- ps

• Check if the web servers are listening on port 80:

ctl-exec --nodes "web.*" -- netstat -an |grep 80

• Run who command on all the dev machines except don't run it on localhost:

ctl-exec --nodes ".*.dev.acme.com" --xnode localhost -- who

• Run the pgrep command across all the machines in dev using 10 threads:

ctl-exec -C 10 -I ".*.dev.acme.com" -X localhost -- pgrep httpd

• Run a ctl defined command:

$ ctl-exec -- ctl -p demo -m netutil -c listening -- -port 8080
number of nodes to dispatch to: 1, (threadcount=1)
true

Execute a script

• Execute the /tmp/bounce.sh script across all the servers tagged "tomcats"

ctl-exec -K -I tags=tomcats -s /tmp/bounce.sh

The ctl-exec command copies the "/tmp/bounce.sh" script to each of the matching hosts and then executes it:

Execute from stdin

• Execute the script provided via STDIN:

echo "uname -a" | ctl-exec --stdin 
number of nodes to dispatch to: 1, (threadcount=1)
Darwin strongbad.local 9.6.0 Darwin Kernel Version 9.6.0: Mon Nov 24 17:37:00 PST 2008; root:xnu-1228.9.59~1/RELEASE_I386 i386

• Execute the scripts using here doc form:

$ ctl-exec --stdin <<EOS
> uname -a;
> whoami;
> EOS
number of nodes to dispatch to: 1, (threadcount=1)
Darwin strongbad.local 9.6.0 Darwin Kernel Version 9.6.0: Mon Nov 24 17:37:00 PST 2008; root:xnu-1228.9.59~1/RELEASE_I386 i386
alexh

Job execution

You can also run an ad hoc command or a script via the GUI by exposing it as a "job". Jobs can be defined in an XML file called "job.xml" and loaded to the server by a utility module called ProjectBuilder. The sample below illustrates job definition. The first definition specifies a shell pipeline script expression inside the script tag. The second job definition describes a script file specified via the scriptfile:

<joblist>
  <job>
    <name>check port</name>
    <description></description>
    <loglevel>WARN</loglevel>
    <additional/>
    <context>
      <depot>demo</depot>
    </context>
    <script>netstat -an | grep 8080 | grep LISTENING</script>
  </job>
 
  <job>
    <name>listening.sh</name>
    <description/>
    <additional/>
    <loglevel>WARN</loglevel>
    <context>
      <depot>demo</depot>
    </context>
    <scriptfile>/path/to/listening.sh</scriptfile>
  </job>
</joblist>

Job listing

Job.xml files can be loaded using ProjectBuilder's load-jobs command which will read the job definitions and store them on the server:

ctl -m ProjectBuilder -c load-jobs -- -file job.xml
Succeeded creating/updating 2 Jobs:
	1: check port [1] </jobcenter/scheduledExecution/show/1>
	2: listening.sh [2] </jobcenter/scheduledExecution/show/2>

Job running

With a browser you can view the defined jobs and execute them by accessing the GUI's home page where they are listed or go directly to a job URL. To access the "check port" job, reference it by its ID, "1": http://strongbad:8080/jobcenter/scheduledExecution/show/1

ProjectBuilder also includes the find-jobs command useful to list the jobs already loaded:

$ ctl -p demo -m ProjectBuilder -c find-jobs 
No filters specified, finding all jobs
Jobs xml content written to: /Users/alexh/ctier/ctl/var/tmp/find-jobs.1386687012.xml
Found 2 Jobs:
	1: check port [1]
	2: listening.sh [2]
Deleting: /Users/alexh/ctier/ctl/var/tmp/find-jobs.1386687012.xml