Using Workload Management to Optimize Operations Console

Using Workload Management to Optimize Operations Console

Introduction

In our previous blog, we gave readers an overview of the Operations Console and its components. This post will continue on the topic of Operations Console, but will focus mainly on Workload Management and how using it can optimize overall engine performance.

With the release of v11.3, Workload Management is now by default enabled. IIS Workload Management’s main role is to monitor the machine resource usage, keep tracking workload (job) count, and dynamically dispatch jobs for execution or place jobs in a queue and when resources become available, release the job for execution. Workload management uses user-defined queues to control the starting of both parallel and server jobs.

Using workload management on an engine system involves the following components:

  • A server process called WLMServer, which provided information about the workload management queues to the DataStage/QualityStage Operations Console.
  • The Operations Console, which displays information about workload management queues for each engine system.

Getting Started

  • You must configure the workload management system before you can use the workload management queues to control the starting your jobs
  • You will need to create the operations database schema that stores the monitoring data that is displayed by the Operations Console

Workload Management

The UI for configuring and controlling Workload Management has two sections:

  • System policies – used for setting systems level policies
  • Queued Jobs / Queued Management – used for configuring and managing queues.

WM Overview

System Policies

System policies are used to determine whether another job should be allowed to start processing.

Thresholds can be set for the following system policies:

WM System Policies

Job Count: Maximum number of running jobs

  • If the maximum limit has been reached, the request to start a new job is queued
  • Selecting the link in the ‘Currently Running’ box displays the same “Job Activity” graph like the one shown in the Dashboard

Job Start: Start no more than N runs within a given period

  • If N runs have been activated in the last X seconds, subsequent requests are queued

CPU Usage: Only start new runs when CPU is below X%

  • If the CPU usage at the point when a new job run request is submitted is greater than X%, the request is queued
  • Selecting the link in the ‘Current CPU(%)’ box displays CPU Usage graph

Memory Usage: Only start new runs when the physical memory is below X%

  • If the Memory usage at the point when a new job run request is submitted is greater than X%, the request is queued
  • Selecting the link in the ‘Current Memory(%)’ box displays Memory usage graph

The system resource usage is checked against the four system policies to determine if another job can start. If the checks are all successful, workload management checks the queue definitions to determine which job to start. If one of the check fails, the job will not start until more resources are available.

Queue Management

Allows users to:

  • Create, edit, and delete queues
  • Specify a default queue
  • Choose a queue priority rule

WM Queue Management

Workload Manager comes with 6 pre-defined queues out of the box

Three System queues cannot be deleted:

  • Information Services Director (ISD)
    • Dedicated to real-time jobs submitted by ISD
    • The queued properties are read-only
  • Information Analyzer (IA)
    • For all runs submitted by IA
  • DataClick
    • For all runs submitted by DataClick

Three template queues that can be deleted by the user:

  • HighPriorityJobs
  • MediumPriorityJobs (default queue)
    • If jobs are sent without specifying a queue or with an invalid queue, they are sent to the default queue
  • LowPriorityJobs

Queued Jobs

WM Queued Jobs

The Queued Jobs tab allows users to:

  • View general queue including the current running jobs, queued jobs, and average time of a job in a queue
  • View the list of jobs in each queue, including total number in that queue
  • Move jobs to a different queue (not to or from system queues)
  • Move jobs to top of the queues
  • Remove jobs from queues

Users can view specific information on jobs running by clicking on the running jobs hyperlink.

WM Running Jobs

How to Assign Queues

Administrator Client

  • Can specify a queue to be the project default queue in the properties dialog.
    • If none is set, the WLM default queue is used as the project default
    • Jobs submitted from this project that did not specify a queue name will be sent to this default queue.
  • Need DS Admin role to change the default.
  • WLM frame is disabled when WLM is off

WM Administrator Client

Director/Designer Clients

  • Workload Management frame is available to prompt for a queue name.
    • The Project queue is the default item
    • Only non-system queues are presented
    • Users cannot send jobs to ISD/IA/DataClick queues

WM Director Designer Client - Copy

Review

When a system is underutilized, it causes inefficiency. When the system is overloaded, jobs can run into a variety of different problems. IIS Workload Management’s primary function is to ensure system operation optimization by monitoring and regulating the engine resource usage to maximize system throughput and maintain a stable and predictable runtime environment.