FAQ
From ControlTier
General
Source
- Anti Rayap & Pest Control
- Perlengkapan Bayi
- Hipnoterapi Surabaya
- Toko Bunga Online
- Roller Blind
- Jual Locker
What is ControlTier?
ControlTier is a software system used to orchestrate service management activities across multiple nodes and application tiers (code, data, configuration, and content). The project is fully open source and many of the project contributions come from ongoing consulting work for large scale e-commerce, software-as-a-service, and financial services operations. Project development as ControlTier began in 2003.
Licensing and Pricing
ControlTier software is free and completely open source. See the License Information here. There are no hidden "gotchas" or some other "enterprise" version that you must pay for, everything we do is free open source software. Download it. Try it. Use it. If you find value in it and would like to actively participate in the project, introduce yourself on the mailing list.
Where do I download ControlTier builds?
Information for getting binaries and source code is available in the download documentation.
What can I do with ControlTier?
Think about all of the activity that has to happen to get from source code to a running business service in your production environment. ControlTier is the automation framework, tools, and best practices to make that happen as reliably and efficiently as possible.
Common ControlTier uses include:
- Coordinated deployment of code, data, content, and configuration into any multi-node development, QA, or production environment. Targets can include physical nodes, virtual machines, or cloud infrastructure.
- Deployment and configuration of application infrastructure such as app servers, databases, web servers, and middleware. This includes configuring them to be aware of each other and be ready to function as an integrated application service.
- Coordinated and intelligent starting and stopping of specific application tiers or of an entire business service.
- Building self-service control panels (using the JobCenter tool) that enable the safe delegation of deployment and service management tasks to other teams (based on permissions). This is useful for extending control of Development, QA, and Production environments to lesser skilled staff and freeing up your senior resources from the daily operations grind.
- Rolling back an application service to a previously known state with a few simple commands. If trouble strikes, this gets you back as quickly as possible to where you know your service will function properly.
- Coordinating the various build processes that assemble your build artifacts. This is an optional step that gives you a prebuilt way for your build or continuous integration tools to register application packages and other deployment artifacts with the ControlTier system. If you are following a continuous integration methodology, you can make ControlTier part of your CI loop and have fully automated deployments to smoke test environments (with push button promotion of those builds to other QA or Production environments).
- Automatically maintaining a system of record for applications deployed within your various environments. Your team will have a centralized place to go for information on what versions of what artifacts are deployed where, how they are configured, and what deployment or service management actions got them into that state. This is a critical resource for both avoiding mistakes and recovering from outages.
- Centralized logging and reporting. For either management, troubleshooting, or auditing reasons, know exactly who did what and when they did it.
For more specific examples of what you can do with ControlTier, see the Examples
What does ControlTier do out of the box?
At its core, ControlTier is a framework that gives you a leg up in designing and implementing your own service management automation (not unlike how an application framework gives you a head start in building applications). But ControlTier is also more than just a framework, ControlTier also provides you with extensive built-in best practices and pre-built automation modules (contained within the ControlTier automation library). If you are using common technologies and want to follow common service management processes, a lot of the work is already done for you.
ControlTier framework:
- Command dispatching with built in features like node abstraction, logical naming, and parallelization (dispatches ad-hoc or pre-defined commands)
- Security
- Logging of distributed actions (uses log4j)
- Management and organization of configuration data and environment details
- Object oriented design makes it easy to maintain and extend your automation
- Command line interface with interactive capabilities
Pre-built automation:
- Pre-built automation for managing the lifecycle of specific technologies
- Pre-built process automation that has best practices for common deployment and service management procedures
- Object oriented design so you can inherit or override/extend only where you need to
ControlTier tools:
- JobCenter provides web-based command execution, job scheduling via the built in scheduler, and allow a group to follow the progress of complex automation as it executes.
- Workbench provides a deployment repository for staging both deployment artifacts and automation modules as well as a web-based design tool for defining and reviewing automation commands and configuration data. All data in Workbench is versioned.
- Report Center is a web-based reporting tool for viewing or querying activity that tool place within the ControlTier system.
To see examples of how you combine these components into actual service management solutions, check out the current Solution Matrix
How does ControlTier compare to Tivoli, Clearcase, SVN, Puppet, Remedy, Nagios, etc?
Because ControlTier is a service management solution that has broad applicability, the most common introductory questions are always around how does it compares to other well-known development and operations management tools.
The best way to do this is to point out what ControlTier is not:
- ControlTier is not a version control system. It is not meant to replace your favorite SCM tool (e.g., CVS, SVN, P4, etc.). ControlTier versions model data and helps you maintain a repository of versioned release artifacts but ControlTier is not a place where you would want to maintain your source code. You can also use a file-based approach to managing your ControlTier data and store those files in your organization's SCM tool alongside your application code.
- ControlTier is not a bare metal provisioning tool (e.g., Kickstart, Jumpstart, etc.). Bare metal provisioning tools typically support net booting, manage OS profiles and are good for "stamping" out server operating system configurations. ControlTier can be used to manage the configuration or kick off the execution of a bare metal provisioning tool but is not meant to take its place. After a provisioning tools has installed a base OS it might also install the ControlTier agent so the new machine can receive release changes.
- ControlTier is not an approval or business process workflow tool (e.g., rt, kintana, remedy, bugzilla, etc.). ControlTier is not meant to replace an approval-based change control system that manages people's actions and business processes they work within. The role of these "people management" tools are still central to coordinating the release activity. ControlTier would come into play to actually execute the change once an approval decision has been made in these other tools.
- ControlTier is not made to replace software build tools. (e.g., ant, maven, cruisecontrol, buildforge, etc.). ControlTier is often used to interface build tools so that they can be coordinated by ControlTier to drive an end-to-end build and deployment process. It is also very common to integrate ControlTier with a continuous integration tool like CruiseControl so ControlTier based deployment can be a part of the CI loop.
- ControlTier is not an enterprise management system (e.g., Tivoli, HP OpenView, Opsware, BMC, etc.). These tools are large frameworks for managing many aspects operations groups are concerned with including monitoring, user administration, event correlation, patching, hardware diagnostics, software distribution, etc. ControlTier does overlap with an EMS when it comes to service deployment and management. Our experience has shown the best way to coordinate the two systems is to use an EMS to manage the system infrastructure and distribute base software infrastructure that change more or less independently from the normal release activity in the application tier. Where the exact boundary is drawn depends entirely on your organization. It's also good to point out that ControlTier is entirely open source and very flexible in its integration points, which is rarely the case with an EMS.
- ControlTier is not Puppet, nor does it conflict with Puppet. In fact, many ControlTier users also use Puppet. Puppet is an excellent system configuration management tool that configures a server instance according to a defined profile and Puppet acts as a compliance tool to keep the box adhering to that profile. Read more about Puppet and ControlTier
- ControlTier is not a monitoring tool. You can use ControlTier commands to run status and diagnostic tasks at a regular intervals, but this should only be done in addition to using a regular monitoring tool.
How does ControlTier speed up and improve the quality of an app lifecycle that spans DEV, QA, PROD, etc?
Moving change from development through QA and on into production is generally a time consuming, error-prone, and all around painful experience. All kinds of new artifacts and related knowledge must be passed for both major and minor releases including:
- versioning and dependencies across multiple packages
- data and database schema updates
- node names, file locations, and permissions (often different in each environment)
- configuration files and settings
- control scripts and command line syntax
- sequencing of startup across components
It's difficult enough to get these details right if you are the primary developer of the service, but it's almost impossible to get all of this knowledge handed off to QA and Production staff in a way that is both efficient and reliable.
To further compound the problem, it's not uncommon for each team to have their own tools and processes to accomplish the changes in their respective environments. When you add up all of the miscommunication, manual errors, "wheel reinvention", and process inconsistencies, you get a bad situation where your key senior staff's time gets eaten up by what should be routine deployment and service management activity. This is both costly to a company and a real drag on quality of life for your team.
The ControlTier solution is for all groups to share a common release platform, a common release repository, and a common set of automation modules. (Note: for security reasons you can have different instances of the same system for production and non-production environments). You are then assured of 3 things:
- what is built in development is the exact same thing that is being tested in QA and deployed into production
- the exact same procedures are use to deploy and manage your services across all environments (removing any process variability)
- you have one system of record through which handoffs, troubleshooting, and other collaboration will take place
Using ControlTier the development group typically bootstraps this process, creating an application model of the integrated system as well as developing and testing the automation in their first integration environment. Once new builds come out of the build process, the QA group will use the same automation created by the development to setup their QA environments and deploy the desired release. Once the QA group has signed off on a release, that same automation is used to deploy the same approved packages to the Production environments. Across all of your environments, the high-level commands to deploy or manage a service stay the same even if the implementation varies from release to release. Working in this way you've cut out all of the variability and inefficiency that plagues a normal release cycle. Aside from getting releases from development to production quicker and cutting down on mistakes, you can now safely assign lesser skilled staff to these tasks and free up your senior staff to do more value-adding work.
Can ControlTier provision whole environments?
We fully support the notion of a fully automated environment, but you have to use the right mix of tools to properly achieve that goal. ControlTier is really designed to deploy and manage business services that span multiple nodes. In the wild, you'll often find ControlTier working in conjunction with an OS provisioning solution (as explained in separate faq entry). For example, you might use a combination of tools like Kickstart and Puppet to build and configure your OS image. ControlTier would then install, configure, and startup the application tier across the OS images. Whether you go for deeper integration of just run the tools side-by-side, what you get in the end is a fully automated environment.
How do I get started using ControlTier?
The subject of "getting started" is taken up in the HOWTOs and demos found on this wiki. But, the general methodology is to pick a single process within the app lifecycle, start small and incrementally, develop the configuration and control models. Beginning users often believe they need to account for every resource, configuration setting, dependency and procedure on the outset because they see value in documenting these things. While this is a great long term goal, we have found that taking a pragmatic stance that concentrates on useful automation goals is the most efficient method.
If you are new to ControlTier it is highly recommended that read the Getting Started page.
Can ControlTier be used for managing disaster recovery?
Yes. Disaster recovery is an excellent use case for ControlTier. As long as you have your source code, data, and any other necessary artifacts, ControlTier can reprovision your services or replicate them at a disaster recovery site.
I live and breathe this problem, so why don't I just solve it myself?
In short, you are solving it yourself. But if the ControlTier community has done a good job, you can focus your expertise on the things that make your organization and business unique.
What the ControlTier project provides you with is a framework, a set of tools, and common best-practices that have come from organizations across the ControlTier community. Rather than going it alone, ControlTier aims to take away the grunt work and needless reinventing of the wheel that takes place across so many IT groups, including yours.
It's not unlike a choice a developer makes when starting to write a new application. Choosing to start from scratch and not use one of the available application frameworks would seem crazy to most modern developers. We feel is should be the same way for service management automation development.
By joining the ControlTier community, everyone benefits from each others expertise and all of our lives get better.
Where are the old manuals?
All of the documentation for ControlTier 3.4 and later can be found on this wiki.
Pre-3.4 manuals are archived:
What powers ControlTier?
ControlTier software was not written all from scratch. It leverages a great deal of other well known open source projects. Here are just some of the projects that underly the ControlTier components: Jetty, Grails, Jackrabbit, Ant, ant-contrib, BSF, Groovy, JSCH, Quartz, HSQL, commons, castor, lucene, Jena, Log4j, dom4j, IzPack.
Interoperability
Is ControlTier cross-platform?
Yes. The ControlTier server and agent components are both Java based and hence cross platform in that regard. ControlTier uses CTL as the core of the agent, the generated code is Ant-based, which also provides cross platform scripting.
ControlTier server is essentially a set of Java-based servlets that use a REST style architecture for external integration. The server has been developed and tested on Tomcat but other servlet containers should also work.
ControlTier agent is a standalone java application that bundles Ant. Theoretically, wherever Ant can run, the ControlTier agent can run.
The most common platforms that ControlTier is run on are Linux, Windows, Solaris, and Mac OS X (in that order).
What operating systems does ControlTier run on?
ControlTier server and agent have been tested on Red Hat AS3, EL4, EL5, and the equivalent CentOS versions; Solaris 8, 9, and 10; Mac OS X 10.4 and 10.5; Windows 2000, XP, 2003. No issues have been reported about running ControlTier in cloud or virtualized hosts (AWS, VMWare, Xen, etc.). ControlTier can be reasonably expected to work on most Linux and UNIX systems with a Java 1.5 VM, usually with just minor tweaking. Windows support is solid as well but we don't do as much testing of the demo's and HOWTO's on Windows. Post any questions you may have about compatibility to the mailing list.
Can you integrate with my favorite version control system?
The primary interface to integrating with version control systems is currently via Ant tasks. These include: Clearcase, Continuus/Synergy, CVS, SVN, PVCS, Perforce, StarTeam, and MS SourceSafe.
Can you integrate with JMX?
Depending on your needs, JMX4Ant might meet your requirements.
Security
How is ControlTier secure?
ControlTier security is managed at various levels.
- Authentication: ControlTier server users must authenticate to access information managed in the server. The server tools use a form-based login mechanism that has session expiration control. User information is stored in an LDAP database or flat file.
- Authorization: Authorization is managed via security roles defined in LDAP. A security role is a privilege granted to a group of users allowing a particular Workbench action. Workbench requires users to have a login ID to authenticate to the server, and to determine the user's role, which provides authorization to access different functionality. See the Security Roles section in the Workbench documentation for more info. Whenever a user executes a command via the client, the request passes through an authorization stage, where the user request is matched up against the access control list. Part of the authorization process includes a user role lookup from an LDAP repository. The client uses an authorization configuration file that can grant access based on deployment context, user group, module and command, as well as time of day.
- Auditing: All commands, software failures, application actions and errors are logged centrally by the Reportcenter application and written to a file local to the server. The server and client applications use Log4j API for all logging.
- Communication: Communication between the ControlTier server and clients is done via HTTP. The server can be configured to use SSL. Communication between ControlTier clients is done over SSH.
- Integrity: Workbench can be configured to generate modules that are digitally signed. Each file within the module are digitally signed and users can configure to have these modules verified before installation. Project and pattern archives are also digitally signed and verified before installation.
Can I use SSL?
Yes, ControlTier server and agent can be configured to use SSL. Consult the Security section in the respective manuals.
Can I encrypt sensitive data in the model?
Typically, this question is asked in regards to passwords used to access services, databases, etc. Workbench supports the ability encrypt Setting values. Users that created the object or have ctlmin level access can edit and view the data in clear text. The data is stored as an encrypted string in the RDBMS. When the data is distributed as a view to a ControlTier agent it remains encrypted. An included Ant task must be used to decrypt the data on the target machine.
Technical
What is ControlTier's architecture? How is it deployed?
ControlTier uses a client-server architecture. The ControlTier server is based on several components, a webapp that runs in a servlet container (e.g. Jetty), a RDBMS (e.g. Mysql) or flat file datastore, an LDAP server (optional). ControlTier (passive) agent is installed on each target machine to which releases and deployments are made. Read more in the Architecture page of this wiki.
Go to the Installation page for details on how to use the ControlTier installer.
Does the ControlTier Agent take up resources when not in use?
No, the agent is passive and is invoked only when necessary. The server will connect via SSH and invoke it. Effectively sshd is the long-running daemon process. The agent is the ctl command.
If you are using the Script-driven approach you do not even need the Client agent installed on the remote node. The ctl-exec command is used for this mode of operation.
What is the process for designing and deploying a classic 3-tier app?
The best way to get a feel for this process is to follow the demos and the HOWTO's, starting with the ControlTier Demo.
Can ControlTier handle co-deployment? (multiple application instances or versions on one OS instance)
Firstly, what do we mean by co-deployment? When we say co-deployment we are referring to the scenario where an administrator would like to install several applications of the same kind on one OS instance. Co-deployment makes sense when many applications must be run but there is limited hardware to run them. Problems often occur due to collisions of software packages, configuration settings, file paths, etc.
ControlTier enables and supports the co-deployment of applications. By automatic convention, each software deployment is provided its own workspace where instance specific files and directories can be maintained (e.g., logs, bins, var, etc.). Secondly, because ControlTier provides a configuration database developers and administrators can prescribe ahead of time what resources and settings each instance or class of deployment should use. In other words, co-deployment is possible as a by product of ControlTier's ability to achieve environment abstraction for applications.
What if somebody changes/installs something outside of ControlTier?
For example, ControlTier was used to deploy an instance of Apache and unbeknownst to the team at large, a rogue hand edited the httpd.conf file to change the listening port. We often refer to this scenario as an "uncoordinated change". Changes like these are often the most difficult to diagnose and fix since mis-configurations often don't stop something from running but instead cause it to run differently. If ControlTier was used to manage the deployment of the configuration file, then you can use a provided command to verify it has not been modified.
The Managed-Entity module contains commands called Doc-Generate and Doc-Validate that work with checksums to verify generated files.
ControlTier does not provide a Tripwire IDS feature that baselines a server's filesystem state. You can write ControlTier commands that verify the state of specific files and make those preconditions in a command workflow.
Does ControlTier support undeploy/rollback?
The simple answer is that you want to "roll forward" to a previously known good release. In actual practice it is not always so simple. There are two strategies for managing undeploy or rollback using ControlTier: model-driven vs. module-driven.
- Model-driven reversion is based on completely model-driven commands and ControlTier's versioned model. By rolling the model back to a previous state, and re-running the deployment, you effectively bring the deployment back to the previous configuration state. The model-driven strategy is the favored approach as it assumes the state of the configuration model has the primary affect over the behavior of commands. In essence, this just means ControlTier encourages a data-driven programming style.
- Module-driven reversion relies on rollback logic that is contained in the control module and may or may not be driven from the centrally managed model state. Module-driven reversion is typically only used for custom applications or databases who's internal state must be specially treated. The module-driven rollback strategy typically involves creating versioned directories in the deployments workspace directory, and storing snapshots of the application internal state. The rollback semantics of the module then understand how to roll back to previous snapshots.
How do you input the dependencies? Can I bulk load data into the model?
Once you have installed the ControlTier stack the very first step is to start defining objects in the model. This can be done via the Workbench web GUI, using ProjectBuilder and XML source files, or programmatically via the included Ant tasks (see Object-Create/Object-Update).
How do I manage the LDAP users for ControlTier?
ControlTier authentication and authorization layers both use LDAP to lookup users and roles. Depending on your preference you may wish to manage the LDAP database through graphical tools or manage it as a text file which is versioned and uploaded to the LDAP server as needed.
If you prefer the graphical approach, we have reported success using JXplorer for administering OpenLdap. Binding to an ActiveDirectory server has also been successfully reported.
If you prefer the text file approach, we suggest managing an LDIF file and there are a number of script-based utilities to help with that.
Can ControlTier prompt during a command?
ControlTier has built in support for prompting from inside commands. If a user is prompted while using the shell interface, a message is written to stdout and the response read from stdin. If a user is running the graphical console, a dialog is raised.
ControlTier has a unique network-savvy prompting task that should be used by commands that are executing commands in remote modules. In this mode, any command that calls the prompt Ant task, will pass the prompt request to a network service that relays the request to the user. This is tunnelled back through the SSH connection with which the remote command was invoked
Prompting can be placed within commands that you develop. Prompts can also be raised during a command workflow failure.
Database schema change detection?
ControlTier does not have any special support for checking if a database's schema has changed. This kind of check can be added as a command that might call a sql script and then added to a workflow command sequence as a precondition.
Can ControlTier scan for exception messages in my app server log?
Application servers (i.e., servlet containers, J2EE servers) are notorious for creating the scenario where the server itself starts up fine but the webapp or EJBs that they load and run begin to spew exception messages during their startup. These exception messages might be caused by ports that are not listening, database misconfigurations, etc.
ControlTier provides an Ant task called FileMonitor which should be used in a Start workflow sequence that scans the log file for success and failure criteria. This FileMonitor runs in its own thread and reads the log file for pre-configured regular expressions. When a success or failure point is reached, the configured action is then dispatched. We often also use the WaitFor task in this process.
Can you define package/jar dependencies?
ControlTier provides a base class called Package which supports dependency relationships. Its subtypes, jar, war, rpm, etc., therefore inherit this capability.
Using ControlTier's correlation query service, you can get back a set of packages all related by their composition hierarchy (i.e., the transitive closure).
Can I validate/diff config files ?
Generation and validation of files is supported by the framework. There are two base commands in the Managed-Entity module, Doc-Generate and Doc-Verify that generate and verify a declared document.
Can commands be run in parallel?
Command workflows can be configured to execute their task list in parallel. The Workflow tag has attributes that control the threading and batching. Since commands will execute more or less asynchronously result data must be specially managed. Refer to the PropertyResults and PropertiesQueryTask for information about how to access and respond to result data.
Can I insure the right versions are deployed to the right boxes?
Assuming you use ControlTier to deploy the applications and packages to the boxes, then you can use and/or develop verification commands to determine if the state of the box corresponds to the state of the model. We consider this a natural byproduct of using ControlTier.
How deeply can things be monitored/checked?
The short answer here is, "as deep as you need". Of course, this is a bit misleading since it relies on the tests that you developed and plugged in as commands (usually as part of workflows). Normally, the kinds of tests that you would use or develop include checking if the application services are running (e.g., by checking the process table, accessing a network port, submitting a transaction, etc.).
A best practice that should be followed is to provide a "Status" command for any module that controls a service. Then a top-level module used for centralized logical management can recursively call the Status command effectively checking each object down the hierarchy.
What is the format of the ControlTier model?
ControlTier uses a semantic web approach to managing information about the configuration and control of application environments. Internally, all data is represented in RDF and stored in a relational database. ControlTier uses the excellent Jena API to access and manipulate the model data. ControlTier uses RDFS along with a handful of its own vocabulary to build ontologies that reflect your environment. Day to day use of ControlTier does NOT require you to understand any of these technologies!
Workbench does provide data export and import of the model data in a number of RDF formats (n-triple, RDF/XML, N3). This makes it possible to use other RDF tools to edit and visualize model data. Of course, care must be used when editing model data outside of Workbench. Only advanced users should consider doing this!
Is Java 6 Supported?
See: Java 6 support notes
ControlTier Administration
Server
Server logs
Workbench Logging level
For example to set the log level to debug in workbench, modify log4j.properties and change the top level to DEBUG
log4j.rootCategory=DEBUG, stdout, R
In 3.4.2, you would edit: $CTIER_ROOT/pkgs/jetty-6.1.14/webapps/itnav/WEB-INF/classes/log4j.properties
Too many open files
In the $CTIER_ROOT/pkgs/jetty-6.1.14/logs/YYYY_MM_DD.stderrout.log you notice a message like:
Caused by: java.io.IOException: Too many open files
Linux
Here's a resolution to raise the limits on Linux.
Run the ulimit command and check the "open files" entry to see the current limit. It might be the default (1024):
[deploy@nantahala ~]$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 102400 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 102400 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
To raise the limit:
1) Edit /etc/security/limits.conf and add the lines:
* soft nofile 1024 * hard nofile 65535
NOTE: the asterisk (*) specifies all users, alternatively a specific user can be specified which would be the user the ctier server (jetty) runs as
2) Edit /etc/pam.d/login, adding the line:
session required /lib/security/pam_limits.so
3) The system file descriptor limit is set in /proc/sys/fs/file-max. The following command will increase the limit to 65535:
echo 65535 > /proc/sys/fs/file-max
4) You should then be able to increase the file descriptor limits using:
ulimit -n unlimited
The above command will set the limits to the hard limit specified in /etc/security/limits.conf.
5) Modified $HOME/.bash_profile as follows:
ulimit -Sn unlimited
6) source $HOME/.bash_profile
. $HOME/.bash_profile
7) restart jetty
jetty.sh stop jetty.sh start
Address already in use: connect
This error occurs when deploying the ControlTier server to Windows XP (and other variants), and turns out that to be caused by exhaustion of the pool of "ephemeral" ports available for making new socket connections.
The means of increasing the pool of such ports is documented on the Microsoft support site.
Raising the upper port number limit from 5000 to 65534 eliminates the issue.
Jobcenter threadCount and concurrency
The maximum number of threads used by Jobcenter for concurrent jobs is set in the "quartz.properties" file located at $CTIER_ROOT/pkgs/jetty-6.1.14/webapps/jobcenter/WEB-INF/classes/quartz.properties
To change the maximum threadCount modify this file and alter the line:
org.quartz.threadPool.threadCount = 20
Set the threadCount value to the max number of threads you want to run concurrently.
Jobcenter uses the Quartz library for job scheduling, and it seems the "SimpleThreadPool" implementation that is used by default is pretty basic, and does not grow on demand. Quartz does not apparently ship with a flexible threadpool implementation, so the best solution is to change that number to something slightly higher than you anticipate needing.
Out of memory
You may see errors in the ControlTier server's Jetty log file indicating that Java heap has been exhausted. Depending on how you're starting ControlTier there are a number of places to set Java options:
- If you're using jetty.sh on Unix/Linux from a Zip install you should change the Java memory options in $HOME/.ctierrc and restart ControlTier:
$ fgrep JAVA_OPTIONS ~/.ctierrc export JAVA_OPTIONS="-XX:MaxPermSize=256m -Xmx1024m -Xms256m $CONFIG_PROPS"
- If you're using ControlTier on Redhat/CentOS Linux using the RPM installer, then there's a default configuration file for the init script to update:
[anthony@centos55 ~]$ fgrep JAVA_OPTIONS /etc/default/ctier export JAVA_OPTIONS="-XX:MaxPermSize=256m -Xmx1024m -Xms256m $CONFIG_PROPS"
- If you're using the start.bat script on Windows then you'll need to fix up %HOMEPATH%\ctier.bat:
C:\Documents and Settings\Anthony Shortland>cd %HOMEPATH% C:\Documents and Settings\Anthony Shortland>findstr JAVA_OPTIONS ctier.bat set JAVA_OPTIONS=-XX:MaxPermSize=128m -Xmx768m -Xms256m %CONFIG_PROPS%
- If you're using ControlTier as a Windows service then you can update the Java and additional options in the Java wrapper service configuration file:
$ egrep 'memory|additional' jetty-ctier-service.conf wrapper.java.additional.1=-Djetty.home=../ wrapper.java.additional.2=-Djetty.logs=../logs wrapper.java.additional.3=-Dctlcenter.config.location=%CTIER_ROOT%/ctlcenter/ctlcenter-config.properties wrapper.java.additional.4=-XX:MaxPermSize=128m wrapper.java.initmemory=256 wrapper.java.maxmemory=1024
Client
Out of memory
You may see errors in the ControlTier client's output and/or Ctl log files indicating that Java heap has been exhausted. Unfortunately (as of ControlTier 3.6.0) the Java options to allocate more memory to the client JVM (e.g. "-Xms64m -Xmx128m") are hard-coded in $CTL_HOME/bin/ctl and %CTL_HOME%\bin\ctl.bat and so those scripts will need to be "hacked" to remediate the problem.
SSH
SSH error: com.jcraft.jsch.JSchException: Auth fail
Password inadvertently set to white space
Credit: Eric Black (2009 Jan 14)
I was having a problem contacting my remote server through ssh, but the problem ended up being trivial. The test client (remote server) was working for quite awhile and then all the sudden it stopped working. I kept thinking it was something to do with the firewall since the OPs team had been messing with the server and firewall recently. I was able to ssh to the system with a command on the command line and could also use a test ant sshexec command without a problem. I then added the same node to a different project and it worked ok. Going back to the project in workbench and looking at the node, I somehow had added a password. By editing it back to a blank password it worked ok. This was the exception:
# ctl-exec -v -p demo -- whoami
number of nodes to dispatch to: 2, (threadcount=1)
preparing for sequential execution...
preparing for remote execution ...
dispatching to proxy on node: ctierremote
Connecting to ctierremote:22
com.jcraft.jsch.JSchException: Auth fail
at org.apache.tools.ant.taskdefs.optional.ssh.SSHExec.execute
(SSHExec.java:188)
at com.controltier.ctl.cli.CtlExec$CommandAction.doAction
(CtlExec.java:871)
at com.controltier.ctl.cli.CtlExec.run(CtlExec.java:261)
at com.controltier.ctl.cli.CtlExec.main(CtlExec.java:357)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at launcher.CtlExecLauncher.main(CtlExecLauncher.java:31)
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:454)
at com.jcraft.jsch.Session.connect(Session.java:142)
at org.apache.tools.ant.taskdefs.optional.ssh.SSHBase.openSession
(SSHBase.java:212)
at org.apache.tools.ant.taskdefs.optional.ssh.SSHExec.execute
(SSHExec.java:158)
... 8 more
--- Nested Exception ---
com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:454)
at com.jcraft.jsch.Session.connect(Session.java:142)
at org.apache.tools.ant.taskdefs.optional.ssh.SSHBase.openSession
(SSHBase.java:212)
at org.apache.tools.ant.taskdefs.optional.ssh.SSHExec.execute
(SSHExec.java:158)
at com.controltier.ctl.cli.CtlExec$CommandAction.doAction
(CtlExec.java:871)
at com.controltier.ctl.cli.CtlExec.run(CtlExec.java:261)
at com.controltier.ctl.cli.CtlExec.main(CtlExec.java:357)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at launcher.CtlExecLauncher.main(CtlExecLauncher.java:31)
Mismatched SSH keys
Credit: Chuck Scott (2009 Jan 20)
getting a com.jcraft.jsch.JSchException: Auth cancel from dispatchCmd
I had a source user: ctier on controltier server
A remote user: deploy
in object model the remote hostname has:
- CTL Username: deploy
- Hostname: smc-dev01.(domain excluded)
ssh works:
[deploy@ctier objects]$ ssh deploy@smc-dev01.(domain excluded) id uid=517(deploy) gid=517(deploy) groups=517(deploy)
... dispatchCmd doesn't:
[deploy@ctier objects]$ ctl -p SmartyCard -t TomcatSite -r dev -c dispatchCmd -- -command Deploy dispatching command: "Deploy " to: [(TomcatContext) devBackoffice, (TomcatServer) dev ]... dispatching to resource: devBackoffice [TomcatContext] -> "Deploy " Connecting to smc-dev01.smartycard.com:22 dispatching to resource: dev [TomcatServer] -> "Deploy " Connecting to smc-dev01.smartycard.com:22 Command failed: The following error occurred while executing this line: /home/deploy/ctier/ctl/projects/SmartyCard/modules/Mediator/commands/ dispatchCmd.xml:161: The following error occurred while executing this line: /home/deploy/ctier/ctl/projects/SmartyCard/modules/Mediator/commands/ dispatchCmd.xml:181: The following error occurred while executing this line: /home/deploy/ctier/ctl/projects/SmartyCard/modules/Mediator/commands/ dispatchCmd.xml:61: com.jcraft.jsch.JSchException: Auth cancel
it turned out to be the wrong key (rsa -vs dsa) where ssh was working with rsa but ctl was configured for dsa key.... i added dsa to authorized_keys , problem resolved...
ProjectBuilder
My project xml fails to validate. What could be wrong?
The project XML DTD specifies that the elements must be in a particular order. In other words, all nodes must come first, then settings, in this order:
<!DOCTYPE project PUBLIC
"-//ControlTier Software Inc.//DTD Project Document 1.0//EN" "project.dtd">
<project>
<node .../>
<setting ... />
<package ... />
<deployment ... />
</project>
Another common cause of failed validation is the presence of tags that are not closed properly, such as missing slashes at the end of settings or other non-container tags.
High Availability
Can ControlTier server be configured for Redundancy?
It is possible to configure a ControlTier server for active/warm standby configuration. Since all data and configuration are filesystem friendly, the use of rsync and additional system configuration is described
