Using Workload Management to Optimize Operations Console
In our previous blog, we gave readers an overview of the Operations Console and its components. This post will continue on the topic of Operations Console, but will focus mainly on Workload Management and how using it can optimize overall engine performance.
With the release of v11.3, Workload Management is now by default enabled. IIS Workload Management’s main role is to monitor the machine resource usage, keep tracking workload (job) count, and dynamically dispatch jobs for execution or place jobs in a queue and when resources become available, release the job for execution. Workload management uses user-defined queues to control the starting of both parallel and server jobs.
Using workload management on an engine system involves the following components:
- A server process called WLMServer, which provided information about the workload management queues to the DataStage/QualityStage Operations Console.
- The Operations Console, which displays information about workload management queues for each engine system.
- You must configure the workload management system before you can use the workload management queues to control the starting your jobs
- You will need to create the operations database schema that stores the monitoring data that is displayed by the Operations Console
The UI for configuring and controlling Workload Management has two sections:
- System policies – used for setting systems level policies
- Queued Jobs / Queued Management – used for configuring and managing queues.
System policies are used to determine whether another job should be allowed to start processing.
Thresholds can be set for the following system policies:
Job Count: Maximum number of running jobs
- If the maximum limit has been reached, the request to start a new job is queued
- Selecting the link in the ‘Currently Running’ box displays the same “Job Activity” graph like the one shown in the Dashboard
Job Start: Start no more than N runs within a given period
- If N runs have been activated in the last X seconds, subsequent requests are queued
CPU Usage: Only start new runs when CPU is below X%
- If the CPU usage at the point when a new job run request is submitted is greater than X%, the request is queued
- Selecting the link in the ‘Current CPU(%)’ box displays CPU Usage graph
Memory Usage: Only start new runs when the physical memory is below X%
- If the Memory usage at the point when a new job run request is submitted is greater than X%, the request is queued
- Selecting the link in the ‘Current Memory(%)’ box displays Memory usage graph
The system resource usage is checked against the four system policies to determine if another job can start. If the checks are all successful, workload management checks the queue definitions to determine which job to start. If one of the check fails, the job will not start until more resources are available.
Allows users to:
- Create, edit, and delete queues
- Specify a default queue
- Choose a queue priority rule
Workload Manager comes with 6 pre-defined queues out of the box
Three System queues cannot be deleted:
- Information Services Director (ISD)
- Dedicated to real-time jobs submitted by ISD
- The queued properties are read-only
- Information Analyzer (IA)
- For all runs submitted by IA
- For all runs submitted by DataClick
Three template queues that can be deleted by the user:
- MediumPriorityJobs (default queue)
- If jobs are sent without specifying a queue or with an invalid queue, they are sent to the default queue
The Queued Jobs tab allows users to:
- View general queue including the current running jobs, queued jobs, and average time of a job in a queue
- View the list of jobs in each queue, including total number in that queue
- Move jobs to a different queue (not to or from system queues)
- Move jobs to top of the queues
- Remove jobs from queues
Users can view specific information on jobs running by clicking on the running jobs hyperlink.
How to Assign Queues
- Can specify a queue to be the project default queue in the properties dialog.
- If none is set, the WLM default queue is used as the project default
- Jobs submitted from this project that did not specify a queue name will be sent to this default queue.
- Need DS Admin role to change the default.
- WLM frame is disabled when WLM is off
- Workload Management frame is available to prompt for a queue name.
- The Project queue is the default item
- Only non-system queues are presented
- Users cannot send jobs to ISD/IA/DataClick queues
When a system is underutilized, it causes inefficiency. When the system is overloaded, jobs can run into a variety of different problems. IIS Workload Management’s primary function is to ensure system operation optimization by monitoring and regulating the engine resource usage to maximize system throughput and maintain a stable and predictable runtime environment.