Creating jobs

From ControlTier

Jobs can be created using the GUI interface and a web browser, or using job.xml loaded with ctl-jobs. Jobs can also be exported into XML format from CTL Center.

For information on Executing Jobs and Viewing their results, see: Running jobs

Description

CTL Center is a tool for running Commands in the ControlTier system without the need to use the commandline. It is a web application that you can access from your web browser.

It provides a simplified yet still powerful way of executing commands and ad-hoc scripts.

What you can do with CTL Center

CTL Center allows you to pre-configure all of the options you would provide when executing a command on the commandline: named arguments, node dispatch configuration, context and command name; and store that configuration as a "Job". You can organize Jobs, and you can configure Jobs to run automatically on a certain schedule.

Since CTL Center respects the ACLs definition for the ControlTier Client installation, you can use role-based authorization to restrict some users to only a subset of Jobs. This provides a Self-Service type interface, where some users can have access to a limited set of Jobs to execute.

Kinds of Jobs

Jobs in CTL Center are stored in one of three ways:

1. Stored - These jobs cab be manually executed at will or scheduled to run at certain intervals

   Stored Jobs can be executed at any time. Scheduled Jobs have a certain schedule, and run automatically at certain times. They also can be manually executed.

2. Temporary - These jobs are executed immediately and not stored.

   Temporary jobs are executed as soon as they are created, and their configuration is not stored for later use.

Jobs in CTL Center run a Workflow, a sequence of one or more commands. There are several types of Workflow items:

• Command - Calls a system command
• Script - An line defined script
• Script file - A script local to the server copied to each Node and executed
• Defined Command - Executes a Defined Command for a particular context
• Job Reference - Calls another job

A Temporary Job that runs a single Command or Script workflow item is the same as using the commandline tool ctl-exec, which can use the Node Dispatching mechanism to execute a script on a set of remote Nodes.

A Job that runs a single Defined Command is the same as using the commandline tool ctl to execute a command.

Grouping Jobs

Jobs can be organized into Groups. Each Job can specify a "group path", which is a "/"-delimited string. When viewed in the GUI, the groups will be shown with a Folder icon and a number indicating how many Jobs are in that Group. The Groups can be expanded or collapsed to simplify the interface.

Using Jobs

Information on how to use Jobs once they are defined can be found here:

• Running jobs

The Jobs List and Job View pages

The main view for the CTL Center GUI is the Jobs List Page. This is the first page shown after you log in, and shows you the list of all available Jobs organized by Group. Many actions can be performed on a Job from this list view. Each Job can be viewed in more detail by clicking on the Job name in the list, which will bring you to the Job View Page. From the Job View Page, more detail is shown.

Using the GUI

The GUI provides a simple interface for managing Jobs as well as executing them and observing them as they run (see Running jobs).

Creating Jobs

To create a new Job, click the "New Job…" link on the right-side:

You will see the Create New Job form:

First select the project that you are going to use. (If there is only one project then it will be selected for you.)

Fundamentally, a Job defines a Workflow that contains one or more items. Press the "Add an item" link to add an item.

You will next choose the type of Workflow item. There are several types: Command, Script, Script file, Defined Command, Job Reference.

Each workflow item type is described in the next few sections.

Command workflow item

Calls a shell command in a single line.

This is similar to calling ctl-exec -- <command>.

Script workflow item

Executes the supplied shell script content. Optionally, can pass an argument to the script specified in the lower text field.

This is similar to calling ctl-exec --stdin -- args <<EOF script code here EOF.

Create a Script file workflow item

Executes the script file local to the sever to the filtered Node set. Arguments can be passed to the script by specifying them in the lower text field.

This is similar to calling ctl-exec -s scriptfile.

Defined Command workflow item

Calls a command defined in a Module.

You can now specify the Context for the Defined Command. This is the same as setting the flags -m <module> [-t type -r <resource>] -c <command> for the commandline tool ctl. (Note that you do not need to select an Resource if you want to execute a Static command.)

Select the Module or Resource for the context:

Once you have selected the context, the Commands list will be populated with the Commands available for that context.

Select the command you wish to run:

Then any commandline options that are available for the command will populate the Options section. If any are available you may fill them in:

Each field in the options list can have several indicators next to it. The purple diamond indicates that an Attribute supplies a default value. (See: Attributes). Gray text indicates that the option has a default value. A Red warning sign indicates that the option is required (must have a value to execute successfully).

Note: You can set any option when Running job, so even though the option may be required to execute, you don not necessarily need to set it in the Job definition.

Some options with indicators:

Job Reference workflow item

To call another saved Job, create a Job Reference item. Enter the name of the Job and its group. You might also navigate and select a Job from the "Choose A Job..." link. Finally, if the Job defines Options, you can specify them in the commandline arguments text field.

Reordering the Workflow sequence

The order of the Workflow items can be modified by hovering over any item and then clicking and dragging the double arrow icon to the desired position.

Dispatching to Nodes

By default, jobs execute on the CTL Center server. To dispatch the commands to client Nodes check the "Dispatch to Nodes" option.

This reveals a "Filtering Options" control where include and exclude filters can be configured.

These options correspond to the -I/--nodes -X/--xnodes -Z,--filter-exclude-precedence commandline options for both the ctl and ctl-exec commandline tools.

You can choose to either use the Inclusion filters, or the Exclusion filters. (See detail in the Use node filtering options section.)

To choose a filter and set its value, click one of the Filter names:

Set the value or regular expression for the filter. You can remove the filter by clicking the red "x" next to the filter field.

You can use both Include and Exclude filters to customize the Node dispatch.

Optionally, set the precedence of the filters (exclude or include) with the "Exclude Filters have precedence?" option. If you choose "No" then the Include filters will have precedence.

The "Matched nodes" area will display Nodes that match the filtering criteria.

If no match is found then the "Matched nodes" area will say "None".

The "Dispatch Options" section lets you specify how many parallel threads should be used in the dispatch, and whether the entire Job should fail if one of the dispatched executions fails.

Enter the number of parallel threads (default: 1), and select either "Yes" or "No" for the "Keep going on error?" option:

Choose the Logging level

Select a value from the "Log level" pop-up menu to set the logging level.

By default the value is "4. Warning". This will have moderate logging output. If you need to see more output during execution, you can select a lower level. If you want to see less output, you can select "5. Error" which is the highest level.

Choosing the storage type

When creating the job you will see these options:

Run and forget

If you want to create a Temporary Job that is executed immediately, choose "No". After defining a Workflow, click the "Run And forget" button to run the job immediately.

Saved

If you want to create a Stored Job, select "Yes" for "Save this job?".

• Job Name: Enter a Name for the Job.
• Group: Enter a /-separated path for the Group. You can also click the "Choose Group…" option to select an existing group name.
• Job Description: Enter a Description to describe what the Job is used for.
• Additional Information: Enter any Additional Information. This might include instructions for Operators or users about the use or purpose of the Job.

Once the job information the Workflow have been set, click the "Create" button to create the Job. You will see the Job definition:

If the Job was placed in a group, navigate through the triangular disclosure icons to reveal the detail and control bar.

Scheduled

If you want to create a Scheduled Job, select Yes under "Schedule to run repeatedly?"

The schedule can be defined in a simple graphical chooser or Unix crontab format.

To use the simple chooser, choose an hour and minute. You can then choose "Every Day" (default), or uncheck that option and select individual days of the week. You can select "Every Month" (default) or unselect that option and choose specific months of the year:

If the crontab time and date format is preferred, enter a cron expression.

The exact crontab syntax used is referenced here: CronExpression

Job Options

Jobs that require user input before running, can declare named options. Once chosen from the run job page, the selected option values can be referenced from the commands in the Workflow.

To create options, click the "add options" link.

To edit existing options press the "edit" link:

Option definition

The option edit form includes a number of parameters described below:

• Option Name: the option name
• Description: description of the option (optional)
• Default Value: the default option value (optional)

Allowed Values (optional)

• List: comma separated list of values
• Remote URL: URL to a list of JSON values. See #Remote option values

Restrictions

• None: Any value can be used
• Enforced from Allowed Values: Must choose from a value defined by "Allowed Values"
• Match Regular Expression: Value must match the specified regular expression

Requirement

• No: Specifying the option is not required
• Yes: Specifying the option is required

Edit options

The Job edit page contains a link to edit each of the options. It provides a brief summary of how each option is defined.

Move the mouse over the row for the option to delete or modify it. Click the edit link to open the option editor.

Remote option values

If the "valuesUrl" is specified for an Option, then the list of allowed values is retrieved from this URL. File URL schemes are acceptable (e.g, file:///path/to/job/options/optA.json).

The format of the response is expected to be JSON data.

Two styles of return data are supported:

value list: A list of values.

Example JSON:

 ["x value for test","y value for test"]

This will populate the select menu with the given values.

label/value map: a map of values with label text.

Example JSON:

 
 {"X Label":"x value","Y Label":"y value", "A Label":"a value"}

This will show only the labels, but the corresponding value will be used in the option.

Variable expansion in remote URLs

The URL used in the "valuesUrl" can embed variables which will be filled with certain context items when making the remote request.

Two types of expansions are available, Job context, and Option context.

To include job information in the URL, specify a variable of the form ${job.property}:

Properties available for Job context:

To include Option information in the URL, specify a variable of the form ${option.property}:

Properties available for Option context:

Examples:

valuesUrl="http://server.com/test?name=${option.name}"

Passes the option name as the "name" query parameter to the URL.

valuesUrl="http://server.com/test?jobname=${job.name}&jobgroup=${job.group}"

Passes the job name and group as query parameters.

Remote request failures

If the request for the remote option values fails, then the GUI form will display a warning message, and the option will be allowed to use a textfield to set the value.

Updating and Copying Jobs

All of the data you set when creating a job can be modified. To edit a Job, you can either click the Pencil icon in the Job list:

Similarly, to Copy a Job definition to a new Job, choose the Copy icon or the Copy button.

Searching for Jobs

The Job list view lets you search for Jobs using the Filter option.

Click the "Filter" link to show the filter options:

This will show the Filter fields:

Enter a value in any of the filter fields:

• Job Name - the name of the job
• Group - the name of the job group
• Description - Job description text
• Additional Desc: Additional description text
• Loglevel: Output log level

The Job list will be filtered to include only the matching jobs:

To refine the filter, click on the blue-outlined Filter description, and change the filter fields.

To reset the filter and go back to the full job list, either click the "CTL Center" link in the header bar, or click the "Clear" button in the Filter fields.

Deleting Jobs

In the Job list, click the red "X" icon for the Job you want to delete:

Click "Yes" when it says "Really delete this Job?"

Importing Jobs as XML

If you have a "jobs.xml" file (See below) and want to upload it via the GUI web interface, you can do so.

Click on the "Create a new Job" button in the Job list.

In the "Create New Job" form, click on the button that says "Uplaod Job XML File..." on the right side:

You will see the "Upload XML" form:

Click the Choose File button and choose your jobs.xml file to upload.

Choose an option where it says "When a job with the same name already exists:":

• Update - this means that a job defined in the xml will overwrite any existing job with the same name
• Skip - this means that a job defined in the xml will be skipped over if there is an existing job with the same name
• Create - this means that the job defined in the xml will be used to create a new job if there is an existing job with the same name.

Click the Upload button. If there are any errors with the Job definitions in the XML file, they will show up on the page.

Note: see the #Troubleshooting jobs.xml upload if you have problems upload the jobs.xml file.

Using the CLI

CTL Center has a service interface that can be used by the ctl-jobs command to perform many of the actions that can be performed by the GUI.

ctl-jobs has two actions that let you interact with CTL Center from the commandline:

• list - search for jobs matching a filter, and downlod the XML definitions of matching jobs.
• load - upload a jobs.xml file to define new jobs and update existing jobs.

Note: The XML format for the jobs is defined in the section: job-v20.xml.

Creating Jobs

To define new Jobs, first create a new xml file to hold your job definitions. Typically this is called "jobs.xml".

It may be useful for you to have a templatized version of this file that you can use to generate your job definitions.

Here is an example jobs.xml:

<?xml version="1.0" encoding="UTF-8"?>

<joblist>
  <job>
    <name>dev1-myserver - Deploy Job</name>
    <description>Deploy the dev1 server</description>
    <additional/>
    <loglevel>WARN</loglevel>
    <group>DEV/Deployment</group>
    <context>
      <depot>test2</depot>
      <type>MyService</type>
      <object>dev1-myserver</object>
      <command>Deploy</command>
    </context>
    <dispatch>
      <threadcount>1</threadcount>
      <keepgoing>false</keepgoing>
    </dispatch>
  </job>
</joblist>

This defines a job that runs the "Deploy" command for MyService resource. The <context> section defines the contextual info for the Job.

Store your jobs.xml file, and use the load-jobs command to upload it to the server:

ctl-jobs load -file jobs.xml

This should produce output like this:

Succeeded creating/updating 1 Jobs:
	1: dev1-myserver - Deploy Job [2] </ctlcenter/scheduledExecution/show/2>

Troubleshooting jobs.xml upload

If there is an error with your jobs.xml definition, you will see a message in the output.

• "Error performing request. Jobs XML Document was not valid: JobXMLException: XXX element not found"
  • This indicates that the element named XXX is required and was not found in your jobs.xml. Refer to the documentation for the particular element: job-v20.xml
• "Error performing request. Unable to parse file: org.xml.sax.SAXParseException: ..."
  • Your XML file was malformed. Make sure your tags are closed correctly.

If your XML file parses correctly, but there is an error storing a particular Job, you will see a message like this:

Failed to add 1 Jobs:
	1: dev1-myserver - Deploy Job [2] </ctlcenter/scheduledExecution/show/2> : Project was not found: test2
Error retrieving command from the context: Framework resource not found: 'test2'. basedir: /Users/greg/ctier3/ctl/projects

Each Job that had an error will be listed, along with the error message(s) that cause it to not be stored.

Some error messages you might see:

• "Project was not found: PROJECT, Error retrieving command from the context: Framework resource not found: PROJECT. basedir: CTL_BASE/projects"
  • There is no project named "PROJECT" in the CTL_BASE shown. Make sure that the <context><depot> section has the correct project name, and that you have created the depot in your CTL_BASE with the ctl-project command.
• "Error retrieving command from the context: module not found: Module"
  • The <type> context element indicates a Type that is not deployed in the CTL_BASE. Make sure the Object/Module is deployed.
• "Error retrieving command from the context: Framework resource not found: COMMAND. basedir: /path/to/Module"
  • The command named was not found in the Module. Make sure you are specifying the command name correctly.

About Job IDs

Each job definition on the Server has an internal unique ID number assigned to it. If you are simply interacting with the GUI this number is not important to understand. However, when using the ctl-jobs load and list commands it can be useful.

The output of the load command shown earlier was:

Succeeded creating/updating 1 Jobs:
	1: dev1-myserver - Deploy Job [2] </ctlcenter/scheduledExecution/show/2>

The text "[2]" shows that the job that was created or modified has ID "2". The text </ctlcenter/scheduledExecution/show/2> shows the URL path to the job view page for ID 2.

You can use the ID number to identify jobs uniquely, which is useful especially when Deleting Jobs.

Updating Jobs

By default, running the load command uses the --duplicate with a default of "update". This means that any job definitions in your XML file that have the same name as an existing job in the server will overwrite those existing jobs.

This is typically what you want to do, as you might modify your jobs.xml file and the re-run the load command to update the jobs on the server.

You can also specify these values for the --duplicate option:

• "update" - update existing jobs with the new definitions
• "create" - create new jobs with the same name
• "skip" - skip existing jobs

(These correspond to the "When a job with the same name already exists" options in the #Importing Jobs as XML section.)

The value of "skip" would allow you to add new job definitions in jobs.xml to the server without modifying existing jobs.

Searching for Jobs

You can search for Jobs using the same set of filters as the GUI allows, with the ctl-jobs list command.

$ ctl-jobs list --name deploy

This specifies the "Job Name" filter with a value of "deploy".

The output would look like:

Found 1 Jobs:
 - info [7] <http://strongbad:8080/ctlcenter/scheduledExecution/show/7>

You can find all Jobs by simply not using a filter option:

$ ctl-jobs list
Found 2 Jobs:
	- Update MyServer [11] <http://strongbad:8080/ctlcenter/scheduledExecution/show/11>
	- dev1-myserver - Deploy Job [12] <http://strongbad:8080/ctlcenter/scheduledExecution/show/12>

The commandline options for filtering correspond to the Filter options in the section #Searching for Jobs:

• --name - Job name filter
• --project - Project name filter

In addition, these options provide further filtering:

• --group - Specify a group name filter
• --idlist - find Jobs with the specified IDs. A comma-separated list, e.g. "1,2,3".

Exporting Jobs as XML

If you specify the -f option for the ctl-jobs list command, then the Jobs found by the filters will be stored in a file in XML format:

$ ctl-jobs list --name deploy -f temp.xml

This specifies the "Job Name" filter with a value of "deploy", and stores the jobs.xml definitions of the found jobs in a file called "temp.xml".

The output would look like:

Wrote XML file to: /Users/greg/ctier3/temp.xml

You can use the output XML file directly with ctl-jobs load if desired.

Deleting Jobs

You can delete jobs from CTL Center by using the -purge option of the ProjectBuilder find-jobs command.

All jobs matching the specified set of filters will be deleted.

It is recommended that the -out option be used in conjunction with this to minimize lost data. These two options can be combined to both store the Jobs XML of the matched jobs and delete them on the server.

When deleting jobs, it is also useful to use the -idlist option to specifically delete a list of Jobs by ID. This way you will not inadvertently delete any Jobs that someone else might have created in between your last find-jobs execution and your -purge execution.

$ ctl -m ProjectBuilder -c find-jobs -- -idlist 1,2 -out deleted.xml -purge
Jobs xml content written to: /Users/greg/ctier3/deleted.xml
Delete succeeded
	Deleted 2 Jobs: [1,2]
	Job 'Update MyServer' was successfully deleted.
	Job 'dev1-myserver - Deploy Job' was successfully deleted.

You can later use the "deleted.xml" file to reload the jobs that you have deleted if desired.