Scripters cookbook

From ControlTier

Revision as of 19:08, 11 November 2010 by Ahonor (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Jump to: navigation, search

This page is under development but the intention is to offer a set of short useful examples that cover common scripting solutions. The cookbook would be useful to those using the Script-driven approach. The first goal is to outline the kinds of tips that would be useful and then to flesh each of them out as a separate wiki page all linked (possibly transcluded) into this page.

Strategies

Use tags to define logical node groups

Use tags to describe the roles your machines play to define logical groups. This lets you target command execution by functional group rather than having to specify individual hosts. Both ctl-exec and ctl shell commands as well as Jobcenter allow commands to be executed against tagged nodes.

Here's a ctl-exec command example that executes a unix command pipeline.

First you can list what nodes are tagged "simpleTomcat"

$ ctl-exec -I tags=simpleTomcat
centos2 ubuntu

The "-S" flag causes ctl-exec to dispatch an inline script read from STDIN.

$ ctl-exec -I tags=simpleTomcat -S
ps auxww|grep http
^D

You can add and remove tags anytime using ProjectBuilder's add-tag and remove-tag commands.

$ ctl -p demo -m ProjectBuilder -c add-tag -- -type Node -name centos2 -tag test
$ ctl-exec -v -I centos2
centos2:
  hostname: centos2
  os-arch: i386
  os-family: unix
  os-name: Linux
  os-version: 2.6.9-34.EL
  tags: [simpleTomcat, test]

Use remove-tag to remove the tag

$ ctl -p demo -m ProjectBuilder -c remove-tag -- -type Node -name centos2 -tag test
$ ctl-exec -v -I centos2
centos2:
  hostname: centos2
  os-arch: i386
  os-family: unix
  os-name: Linux
  os-version: 2.6.9-34.EL
  tags: [simpleTomcat]

See Bulk Node registration for information on using project.xml to register multiple nodes from a single definition file.
See modelutil#add-tag for a command line tool.

Eliminate SSH loops from your scripts

Here's a typical script executing commands across a set of nodes:

for node in $SOMENODES; do
  ssh $node command1
  [ $? != 0 ] do error-mgt statement
  ...
  ssh $node another command
  [ $? != 0 ] do error-mgt statement
  ... and so on
done

A few down sides to this style of scripting include:

Remote command execution (a form of dispatching) and individual deployment tasks are intermingled
Due to #1, this approach is harder to debug because you can't run the procedure locally on just one node
Evaluating conditions is tricky when using SSH. Relies on non-zero SSH result code processing
Must be conscious of quoting when executing multipart command strings over SSH (eg, when having to do things in relative directories)

An alternative would be to eliminate the looping and ssh calls from that script and use ctl-exec.

Here's the script with the for loop and ssh calls removed:

command1
[ $? != 0 ] do error-mgt statement
  ...
another command
[ $? != 0 ] do error-mgt statement
  ... and so on

Rather than manage explicit lists of nodes in a shell variable, use node tags to loggically group the hosts in "$SOMENODES". Then call the script via ctl-exec :

ctl-exec -I tags=SOMENODES -s scriptfile

Benefits to this approach include:

Separate the SSH logic from the functional steps by introducing ctl-exec
Make all the steps work more atomically
Enable easier debugging

Refactor scripts into reusable building blocks

Often times a script implements a cycle of steps. The implicit design of the script is done in two layers

task specific steps
coordination layer that calls the task specific steps that comprise the procedure

In pseudo code the script structure might look like this

# script: do_procedure
 
# do step1 stuff
step1_part1
step1_part2
...
step1_partN
 
# do step2 stuff
step2_part1
step2_part2
...
step2_partN
 
...and so on

If such a script becomes a normal part of operation, the script would be easier maintained if it was decomposed into a set of scripts. Each logical step in the cycle would become its own script (eg, scripts named: "step1", "step2", "stepN") while the original script would call out to these external scripts. Here's the pseudo code refactored.

# script: do_cycle
 
# do step1 stuff
call script step1
 
# do step2 stuff
call script step2

The task specific steps would be isolated in their own scripts. Eg:

# script: step1
step1_part1
step1_part2
...
step1_partN

This style of design is preferable because it facilitates better reuse, flexibility and testing and supports modeling life cycles.

This design does incur the requirement for distributing multiple script files rather than just the one. To offset this, there are a couple of ControlTier facilities useful for distributing the scripts:

Stage the scripts to the WebDAV repository. They can be executed via shellutil: Execute a script file stored in the WebDAV
Deploy the scripts inside a utility module: See First CTL module.

Use WebDAV for pull-based file distribution

A common solution for file distribution is to copy the files to a central server and then distribute them out by remotely copying them to the target hosts. For example a preparatory step might stage the files to repository through remote copy:

# file: stage.sh
scp file1 server:/path/to/staging/dir/file1
scp file2 server:/path/to/staging/dir/file2
...
scp fileN server:/path/to/staging/dir/fileN

A second script then distributes the files to the target hosts.

FILES="file1 file2 ... fileN"
NODES="host1 host2 ... hostN"
for node in $NODES
do
  for file in $FILES
  do
     scp /path/to/staging/dir/$file $node:/path/to/target/dir/$file
  done
done

This example uses the SCP command but RSYNC is a popular choice for remote file copying, too.

There are a few of scaling disadvantages to the push-model of file distribution in a distributed environment.

Puts all processing on the server making it a single bottleneck
In order to support a fan out, need to introduce intermediate staging servers and custom logic to relay file copies
If a remote host is down during the push and comes up later the client will not pull the file on its own

The ControlTier server includes a WebDAV repository that facilitates a pull-based file distribution model. There are a number of advantages to using WebDAV as a repository.

Openness of HTTP. Most networks allow HTTP access across subnets.
Offers the PUT method allowing remote to storage of files on the repository.
Any web client can retrieve files from the WebDAV. Uses normal HTTP "GET" method.
Access to the WebDAV can be scaled up using normal web scaling practices including proxies and caching servers.
Files in the WebDAV are referenced as URLs

Here's an example script that uses davutil#put to store files onto the repository. (Note: the dav:// prefix is a shorthand.)

# file: stage.sh
ctl -m davutil -c put -- -file file1 -url dav://file1
ctl -m davutil -c put -- -file file2 -url dav://file2
...
ctl -m davutil -c put -- -file fileN -url dav://fileN

A second script then distributes the files to the target hosts. Here a ctl-exec script is used to call the davutil#copy command for all the files:

ctl-exec -I tags=targets --stdin <<EOS
> FILES="file1 file2 ... fileN"
> for file in $FILES
> do
>   ctl -m davutil -c copy -- -dest $file -src dav://$file
> done
EOS

Script with utility modules

When scripting procedures for deployment management one often faces similar scenarios requiring testing a condition, manipulating files and their content, executing commands and other scripts, etc. These individual tasks are often just a means to an end and are part of the drudgery along the way to implementing the actual business logic purpose for the procedure at hand.

For example, here's a common situation faced during for startup of a long running process.

A startup command or script is invoked to bring up the long running process
The startup time is not immediate since it takes time for the process to boot up the application and acquire system resources
To check successful startup, it therefore requires some wait time before the status check should be performed
Need to implement retry logic in case startup is longer than anticipated

The script code below shows the startup of Tomcat along with the status check with that logic in mind:

# run the startup script
$CATALINA_HOME/bin/startup.sh || exit $?; # exit because startup script failed for some reason
#
# give 10 seconds time to start up the webapp's listening process on port 28080
sleep 10;
#
# Check if it binds to the 28080 port 
netstat -an|grep 28080|grep -q LISTEN 
if [ $? = 0 ] 
then
    echo "Tomcat listening (port=28080). Startup successful"
else 
     sleep 10; # give it another 10 seconds
     netstat -an|grep 28080|grep -q LISTEN 
     if [ $? = 0 ] 
     then
        echo "Tomcat listening (port=28080). Startup successful"
     else 
         echo "Waited 20 seconds and the server did not listen on 28080" ; exit 1 ; # exit with an error. time exceeded
     fi
fi

There are a few issues to point out in this scripting approach:

Verbose logic to check and then recheck the port listening status. Could have also implemented a loop that checked every one second but that would be verbose, too.
A more rigorous check to for the listening process would be to connect to the port and see if the webapp actually responds.
Have to wait full wait period to find out if check should be retried again.

This example is representative of the myriad conditions the deployment management scripter faces. A preferable solution would be to implement these kinds of solutions in a utility module that can be shared by all scripts. The utility modules act like a tool kit your scripts build on letting you focus on the main logic of the procedure at hand. The dirty work can be isolated in the utilities.

Here's the same scenario but using the netutil listening command:

$CATALINA_HOME/bin/startup.sh || exit $?; # startup script failed for some reason
# give 20 seconds time to start up the webapp's listening process on port 28080 but check every half second
ctl -m netutil -c listening -- -port 28080 -checkevery  500 -maxwait  20000 -maxwaitunit  millisecond

The netutil listening command checks every half second up to 20 seconds. That means if the webapp begins to respond to the 28080 port in 1 second the check will complete in that much time. The script became much less verbose since that part of the code is avoided and taken care of by the utility command. Utility modules are beneficial since they provide a standard scripting API to common solutions.

ControlTier includes a number of utility types but you can create your own, too.

Model life cycles

Often times a script implements a cycle of steps. The implicit design of the script is done in two layers

task specific steps
coordination logic that calls the task specific steps in the order that reflects the normal cycle

In pseudo code the script structure might look like this

# script: do_cycle
 
# do step1 stuff
step1_part1
step1_part2
...
step1_partN
 
# do step2 stuff
step2_part1
step2_part2
...
step2_partN
 
...and so on

If such a script becomes a normal part of operation, the script would be more useful if the cycle and the individual steps were explicitly defined. Here's the pseudo code refactored.

# script: do_cycle <action>
 
Step1() {
# do step1 stuff
step1_part1
step1_part2
...
step1_partN
}
 
Step2() {
# do step2 stuff
step2_part1
step2_part2
...
step2_partN
}
 
Cycle() {
  Step1;
  Step2;
}
 
# Do the user specified action
case $1 in
   Step1) Step1;
    ;;
   Step2) Step2;
    ;;
   Cycle) Cycle;
    ;;
esac

Exiting the script

ControlTier makes a simple but familiar assumption about whether your script was successful. If your script was successful it should exit with exit code "0" otherwise it exits with a non-zero value. [1]

# No error and successfully achieved objective
exitCode=0
 
# An error occurred or objective was not met (non-zero code)
exitCode=1
 
# Exit the script
exit $exitCode

Commands your script call also complete with exit codes and you can simply return their status. For example, the following script runs a unix shell pipeline and then exits with the status of the final call to the "grep" command. Grep returns "0" if the pattern was found in the output.

netstat -an | grep 8080 | grep -q LISTEN; exit $?

Define a Job for your script

Main article: Creating jobs

You can allow others to run your script by defining it as a job in Jobcenter. You can define a job to call a script defined in the job itself or refer to a script that you maintain on the server.

You can define the job graphically via the Jobcenter UI or you can define the job using an XML file and upload it to Jobcenter.

Maintain script files in an SCM

Any script used to perform management procedures in the operational environment should be stored in a source code repository. This practice is beneficial for a number of reasons including, auditing, change management, collaboration, etc.

Scripts can be checked out and staged to a distribution server for later use in the distributed environment.

Stage script files in the WebDAV

Scripts don't always start off with the intention of becoming implementations to perform routine procedures. Sometimes they begin as a short term time saver to avoid manual repetition. Many of these scripts must be distributed to remote nodes where they need to run. Due to the uncertain permanent nature of these scripts they are often distributed on an ad-hoc basis using commands like SCP or RSYNC. Over time it is even difficult to know what copy is the authoritative one. Of course, scripts should be versioned in a source code repository.

Once scripts do become part of the operation, it is best to treat them like any other kind of release artifact and store them in an operational repository. The WebDAV is a suitable and convenient facility to stage and distribute script files. You can create an structure in the WebDAV, creating a directory namespace to organize the scripts by their role.

Here's an example usage of the davutil#put that uploads a directory of scripts into the WebDAV repository:

ctl -z -p demo -m davutil -c put --\
   -file /home/demo/simpleTomcat/scripts \
    -url dav://simpleTomcat/scripts

Create a utility type to formalize routine scripted procedures

Routine scripted procedures can be formalized into a set of defined commands in a utility type. Executing scripts inside a type has several benefits:

User interface: Defined commands have a standard syntax. This avoids teaching users script-specific usage rules
Programming interface: All your scripts are callable using the CTL dispatcher
Distribution: Defined commands are packaged into a module Jar file and distributed using CTL administrative tools

Create a utility type by sub typing Managed-Entity. A sub type can be defined using either ProjectBuilder or the Graphical type editor.

Imagine a directory named "scripts" containing several bourne shell scripts:

scripts
|-- a.sh
|-- b.sh
`-- c.sh

Using ProjectBuilder#create-type a new utility type named "myapputil" is created with three commands defined, one for each of the scripts:

ctl -m ProjectBuilder  -c create-type --\
      -supertype Managed-Entity -type myapputil -initialize -scriptdir /path/to/scripts

The next step is to use ProjectBuilder#build-type to create the module:

ctl  -m ProjectBuilder  -c build-type --\
      -type myapputil -upload -deploy

Each script in /path/to/scripts will be used to create a new command in the module with the same name as the script.

List the commands in the now deployed myapputil:

ctl -m myapputil

Files

Most application management processes involve interacting with files. This section describes a number of common place scenarios a scripter is likely to face.

Check if a file exists

In unix shell typically you check the existence of a file using the test operator (Windows does not have a built-in function):

if [ -f file ] 
then   
   echo true
else
   echo false 
fi

Doing that check across a set of hosts via ssh might look like this:

for node in $NODES
do
 ssh node "if [ -f /etc/motd ]; then echo true; else echo false; fi"
done

Here's an alternative using fileutil#available:

ctl -m fileutil -c available -- -file /etc/motd

Will return "true" or "false". You can cause the command to exit non-zero with the -failonerror flag.

ctl -m fileutil -c available -- -file /etc/bogus -failonerror

Copy a file

Copy a file with fileutil#copy:

ctl -m fileutil -c copy -- -file file1  -target file2

Copy a local file but backup pre-existing

Here is unix bourne shell script that will conditionally backup a file before a copy is created:

# create a backup if necessary
if [ -f file2 ]
then
   cp file2 file2.backup
fi
 
# copy the file
cp file1 file2

Doing those same steps remotely would be a bit inconvenient.

Here's an example using fileutil#copy:

ctl -m fileutil -c copy -- -file file1  -target file2 -backup

Check if a file is executable

In unix shell typically you check if a file is set executable using the test operator (Windows does not have a built-in function):

if [ -x file ] 
then   
   echo true
else
   echo false 
fi

Doing that check across a set of hosts via ssh might look like this:

for node in $NODES
do
 ssh node "if [ -x file ]; then echo true; else echo false; fi"
done

Here's an alternative using fileutil#executable:

ctl -m fileutil -c executable -- -file /bin/sh

Use the "-failonerror" option to have the command exit with a non-zero code if it is not executable.

ctl -m fileutil -c executable -- -file /etc/motd -failonerror

Create a symbolic link

Use fileutil#link. Similar to: ln -s file1 file

ctl -m fileutil -c link -- -file file1 -target file2

Only works on Unix OS family hosts.

List directory contents

Imagine a directory structure like below:

/tmp/test
|-- file1
|-- file2
|-- file3
`-- subdir
    |-- file1
    `-- file2

To list the contents on Unix you could use the ls or find commands. Windows has the dir command.

You can use the fileutil#ls command. To list just files inside the first dir run:

ctl -p demo -m fileutil -c ls -- -dir /tmp/test
/tmp/test/file1
/tmp/test/file2
/tmp/test/file3

Or to do a recursive list run:

ctl -p demo -m fileutil -c ls -- -dir /tmp/test -recursive
/tmp/test/file1
/tmp/test/file2
/tmp/test/file3
/tmp/test/subdir/file1
/tmp/test/subdir/file2

Create a directory

Use fileutil#mkdir:

ctl -p demo -m fileutil -c mkdir -- -dir /tmp/newdir

Move a file

Move a file with fileutil#move:

ctl -p demo -m fileutil -c move -- -file file1 -target file2

Check if one file is newer than another

Checking if one file is newer than another is a bit cumbersome and can be one in a number of ways. One could use long directory listings and parse the output for the last-modification time and compare the values. On unix, one might also use the find command.

For example, assume there are two files with different modification times (file1 is older than file2)

demo@ubuntu:/tmp/test/subdir$ ls -l
total 0
-rw-r--r-- 1 demo demo 0 2009-07-12 09:42 file1
-rw-r--r-- 1 demo demo 0 2009-07-12 09:52 file2

The find command could be used to check if one is newer than the other. If file2 is newer then file2 will be written to stdout:

$ cd /tmp/test/subdir
$ find . -name file2 -newer file1
./file2

If the opposite is tried,

$ find . -name file1 -newer file2

... then no output will be written to stdout.

Doing that kind of check across a set of hosts would be cumbersome.

Alternatively, one can use the fileutil#newer command like so:

ctl -m fileutil -c newer -- -file file2 -target file1

The string "true" will be written to stdout if file2 is newer, "false" otherwise.

The "-failonerror" flag can be used to cause the command to exit non-zero if the file is not newer.

Check if one file is older than another

Use fileutil#older:

ctl -m fileutil -c older -- -file file1 -target file2

Check if a file is readable

In unix shell typically you check if a file is set readable using the test operator (Windows does not have a built-in function):

if [ -r file ] 
then   
   echo true
else
   echo false 
fi

Doing that check across a set of hosts via ssh might look like this:

for node in $NODES
do
 ssh node "if [ -r file ]; then echo true; else echo false; fi"
done

Here's an alternative using fileutil#readable:

ctl -m fileutil -c readable -- -file file

Use the "-failonerror" option to have the command exit with a non-zero code if it is not readable.

ctl -m fileutil -c readable -- -file /etc/motd -failonerror

Check if a file is writeable

In unix shell typically you check if a file is set writeable using the test operator (Windows does not have a built-in function):

if [ -w file ] 
then   
   echo true
else
   echo false 
fi

Doing that check across a set of hosts via ssh might look like this:

for node in $NODES
do
 ssh node "if [ -w file ]; then echo true; else echo false; fi"
done

Here's an alternative using fileutil#writeable:

ctl -m fileutil -c writeable -- -file file

Use the "-failonerror" option to have the command exit with a non-zero code if it is not writeable.

ctl -m fileutil -c writeable -- -file /etc/motd -failonerror

Remove a file

Remove a file with fileutil#remove:

ctl -m fileutil -c remove -- -file file1

Remove a directory

Remove a directory with fileutil#rmdir:

ctl -m fileutil -c rmdir -- -dir mydir

Touch a file

Create a file and set its last modified time using fileutil#touch:

ctl -m fileutil -c touch -- -datetime "'06/28/2000 2:02 pm'" -file file1

Network

Check if a port is listening

There are a number of methods useful to check if a port is open and listening. On unix hosts one might use the netstat command and grep:

netstat -an | grep 8080 | grep LISTENING

On windows, one might use the telnet command and attempt to connect to it

telnet host 8080

An alternative would be to use the netutil#listening command:

ctl -m netutil -c listening  -- -port 8080

Check if a port is listening up to a max wait time

Sometimes you are checking if a process that is starting has bound to the port but it takes some time. Do this using netutil#listening.

This example checks every half second up to 60 seconds:

ctl -m netutil -c listening  -- -port 8080 -maxwait 60 -maxwaitunit second -checkevery 500

Check if a web server is up

To check if a web server is reachable, one might connect to it's listening port and attempt a page request using tools like wget, curl or even a simple network port check. Using one of these methods, the scripter will then need to parse and intepret the results to determine if the web server is up.

An alternative, is to use netutill#reachable. This command returns "true" or "false" depending on the web server being reachable.

ctl -m netutil -c reachable -- -timeout 10 -url http://someaddress:80

The "-failonerror" flag can be added to cause the command to exit non-zero.

Shell

Get environment information

Depending on the login shell, there is a method to obtain environment variable values. Unix hosts also include the env command that will print all currently exported environment variables. You can combine it with the grep command to lookup just one value:

env | grep VAR

One can also use the shellutil#env command to access one or more environment variable settings:

ctl -m shellutil -c env -- -key CTL_BASE
CTL_BASE=/Users/alexh/ctier/ctl

The "key" parameter can use a regular expression to look up multiple variables:

ctl -p demo -m shellutil -c env -- -key 'CTL_.*'
CTL_HOME=/Users/alexh/ctier/pkgs/ctl-1.4.6
CTL_CLI_TERSE=true
CTL_BASE=/Users/alexh/ctier/ctl

Execute a command

Execute an arbitrary command by invoking the executable and passing it arguments using shellutil#exec:

ctl -m shellutil -c exec -- -executable /bin/echo -argline hello

Execute a script expression

Call an executable and pass it a script expression using shellutil#exec:

$ ctl -m shellutil -c exec -- -executable /bin/sh -script 'echo hello me'

Execute a script file

Execute script file using shellutil#exec:

ctl -m shellutil -c exec -- -executable /bin/sh -scriptfile scriptfile.sh

The exec command will default the executable to several interpreters depending on the file's extension. For example files that end in ".sh" are assumed to be bourne shell scripts so the "/bin/sh" is the default value used for the "-executable" option.

ctl -m shellutil -c exec -- -scriptfile scriptfile.sh

Other default mappings include: .bat → cmd.exe, .xml → Ant.

Execute a script file stored in the WebDAV

Maintaining script files in the WebDAV provides a centralized location to get them and use them wherever needed in the distributed environment. The shellutil#exec command supports this model.

This example shows a trivial script that is first placed in the WebDAV and then executed.

Script file: hi.sh

#!/bin/sh
echo hi there $@

Put it into the repository

ctl -m davutil -c put -- -file hi.sh -url dav://hi.sh

Execute the script.

ctl -m shellutil -c exec -- -executable /bin/sh -scripturl dav://hi.sh

The file is downloaded and then executed.

Execute a command or script with a timeout

Use the shellutil#exec command and the "-timeout" option.

Give the script 300 seconds to finish. If it exceeds the allowed time limit then terminate it.

ctl -m shellutil -c exec -- -executable /bin/sh -scriptfile longrunning.sh -timeout 300

Kill system processes by name

Kill all the processes named java using shellutil#pkill:

ctl -m shellutil -c pkill -- -pname java -sig TERM

Text

Checksum a file

Use textutil#checksum to generate a checksum for a file:

ctl -m textutil -c checksum -- -file file1
d41d8cd98f00b204e9800998ecf8427e

This time save the checksum to file:

$ ctl -m textutil -c checksum -- -file file1 -createfile
$ ls
file1  file1.MD5  file2
$ cat file1.MD5 
d41d8cd98f00b204e9800998ecf8427e

Check if a file contains a pattern

It is often necessary to check the content of a file to see if it contains a particular string. On unix hosts, the grep command is the immediate tool to consider for the task. Windows users might use QGREP.

grep PATTERN file

The textutil#contains command can also be used:

ctl -m textutil -c contains -- -file <> -pattern <>

Check if a file contains a pattern and retry up to maximum wait time

Sometimes it is necessary to retry checking a file to see if it contains a particular string. This might be due to a process that is starting and has not reached the point where it has printed that line.

An example unix shell script that accomplishes the objective is shown below. It loops up until a maximum wait time (20s) and retries every 1 second. If the pattern is not found then the script exits with a non-zero exit code:

elapsed=0
while [ $elapsed -lt 20 ]
do
  grep Started example.log && exit 0
  sleep 1
  elapsed=`expr $elapsed + 1`
done
 
exit 1

The textutil#contains command provides a "-maxwait" flag that can be used to continuously retry the check:

ctl -m textutil -c contains -file example.log  -pattern Started -maxwait 20

Show file content with line numbers

Use the textutil#nl command.

$ ctl -m textutil -c nl -- -file /etc/motd
1 Linux ubuntu 2.6.27-7-generic #1 SMP Fri Oct 24 06:42:44 UTC 2008 i686
2 
3 The programs included with the Ubuntu system are free software;
4 the exact distribution terms for each program are described in the
5 individual files in /usr/share/doc/*/copyright.
6 
7 Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
8 applicable law.
9 
10 To access official Ubuntu documentation, please visit:
11 http://help.ubuntu.com/

Replace a string in a file with a different string

Use the textutil#replace command

ctl -m textutil -c replace -- -basedir /path/to/conf/dir -exts .conf -filebase * -token ORIG -value NEW

Print the last lines of a file

Use the textutil#tail command

ctl -m textutil -c tail [-count <10>] -file <>

WebDAV

dav:// prefix

As of 3.4.5, the root of the WebDAV repository that is configured in the CTL framework.properties (i.e., framework.webdav.uri) file can be addressed as "dav://". This is a shorthand provided as a convenience letting users specify a shorter URL string. (ie, dav:// versus http://strongbad:8080/jackrabbit/repository/controltier/projects). The Davutil commands support this addressing and overtime the rest of ControlTier commands that reference the WebDAV resources will eventually.

For example this URL:

http://strongbad:8080/jackrabbit/repository/controltier/projects/junk/fun

Can be shortened to:

dav://junk/fun

The "dav://" prefix is translated to the value of framework.webdav.uri defined in framework.properties.

Copy WebDAV resource to a local file

Copy the resource accessible as the URL http://strongbad:8080/jackrabbit/repository/controltier/projects/junk/fun to a local file named /tmp/fun/file1 :

ctl -m davutil -c copy -- \
	  -src http://strongbad:8080/jackrabbit/repository/controltier/projects/junk/fun \
	  -dest /tmp/fun/file1

Copy WebDAV resource to a local file but backup pre-existing

You can use the "-backup" option to make a backup of the existing file.

ctl -m davutil -c copy -- \
	  -src http://strongbad:8080/jackrabbit/repository/controltier/projects/junk/fun \
	  -dest /tmp/fun/file1 -backup

By default the backup file name will be suffixed with ".backup". Eg file1.backup

The suffix can be overridden using the "suffix" option.Eg. file1.orig

ctl -m davutil -c copy -- \
	  -src http://strongbad:8080/jackrabbit/repository/controltier/projects/junk/fun \
	  -dest /tmp/fun/file1 -backup -suffix ".orig"

Copy local file to the the WebDAV

ctl -m davutil -c copy -- \
	  -src /tmp/fun/file1 \
	  -dest http://strongbad:8080/jackrabbit/repository/controltier/projects/junk/fun

OR

ctl -m davutil -c copy -- \
	  -src /tmp/fun/file1 \
	  -dest dav://junk/fun

Retrieve an archived WebDAV resource and extract it

Retrieving and extracting an archived resource from the WebDAV is usually done in two main step:

using a web client like curl or wget, providing to it URL output file name and possibly user/pass info
tool suitable to extract the archive depending on its type

Here's an example using curl and unzip:

# get the file
curl -s --user default:default http://strongbad:8080/jackrabbit/repository/controltier/projects/pkgs/demo/zip/zips/apache-tomcat-5.5.27.zip \
    -r /usr/local/apache-tomcat-5.5.27.zip 
if [ $? != 0 ]
then
   echo "Download failed: apache-tomcat-5.5.27.zip"; exit 1
fi
 
# extract the archive
unzip -q /usr/local/apache-tomcat-5.5.27.zip 
if [ $? != 0 ]
then
   echo "Extract failed: apache-tomcat-5.5.27.zip"; exit 1;
fi

Alternatively, the davutil#extract command can be used to do this in a single step.

ctl -m davutil -c extract -- -dir /usr/local -url dav://pkgs/demo/zip/zips/apache-tomcat-5.5.27.zip  -type zip

Note: The dav:// prefix is a convenient shorthand to the URL to the root of the WebDAV.

Create a WebDAV resource copy

You can use davutil#copy to copy one resoure in the dav to another:

ctl -m davutil -c copy -- \
	  -src http://strongbad:8080/jackrabbit/repository/controltier/projects/junk/fun \
	  -dest http://strongbad:8080/jackrabbit/repository/controltier/projects/junk/fun_copy

Delete a WebDAV resource

Remove a resource from the WebDAV using davutil#remove:

ctl -m davutil -c remove -- \
      -url http://strongbad:8080/jackrabbit/repository/controltier/projects/junk/fun_copy

Create a WebDAV collection

Create a new collection (similar in concept to a directory) using davutil#mkcol:

ctl -m davutil -c mkcol -- \
      -url http://strongbad:8080/jackrabbit/repository/controltier/projects/junk/new_collection

Rename a WebDAV resource

Use davutil#move to rename a resource in the WebDAV:

ctl -m davutil -c move -- \
      -url http://strongbad:8080/jackrabbit/repository/controltier/projects/junk/old_name
      -dest http://strongbad:8080/jackrabbit/repository/controltier/projects/junk/new_name

Put a file in the WebDAV

Upload a file to the repository using davutil#put:

ctl -m davutil -c put -- \
      -file /etc/motd
      -dest http://strongbad:8080/jackrabbit/repository/controltier/projects/junk/motd

Put a directory in the WebDAV

Use davutil#put a directory of files into the WebDAV:

ctl -m davutil -c put -- \
      -file /etc
      -dest http://strongbad:8080/jackrabbit/repository/controltier/projects/junk/etc

Scripters cookbook

From ControlTier

Contents

Strategies

Use tags to define logical node groups

Eliminate SSH loops from your scripts

Refactor scripts into reusable building blocks

Use WebDAV for pull-based file distribution

Script with utility modules

Model life cycles

Exiting the script

Define a Job for your script

Maintain script files in an SCM

Stage script files in the WebDAV

Create a utility type to formalize routine scripted procedures

Files

Check if a file exists

Copy a file

Copy a local file but backup pre-existing

Check if a file is executable

Create a symbolic link

List directory contents

Create a directory

Move a file

Check if one file is newer than another

Check if one file is older than another

Check if a file is readable

Check if a file is writeable

Remove a file

Remove a directory

Touch a file

Network

Check if a port is listening

Check if a port is listening up to a max wait time

Check if a web server is up

Shell

Get environment information

Execute a command

Execute a script expression

Execute a script file

Execute a script file stored in the WebDAV

Execute a command or script with a timeout

Kill system processes by name

Text

Checksum a file

Check if a file contains a pattern

Check if a file contains a pattern and retry up to maximum wait time

Show file content with line numbers

Replace a string in a file with a different string

Print the last lines of a file

WebDAV

dav:// prefix

Copy WebDAV resource to a local file

Copy WebDAV resource to a local file but backup pre-existing

Copy local file to the the WebDAV

Retrieve an archived WebDAV resource and extract it

Create a WebDAV resource copy

Delete a WebDAV resource

Create a WebDAV collection

Rename a WebDAV resource

Put a file in the WebDAV

Put a directory in the WebDAV

Personal tools

Namespaces

Variants

Views

Actions

Search

Navigation

Communication

Development

Toolbox

Print/export