IBM DataStage Operations Console: an Introduction

IBM DataStage Operations Console

Introduction

Are your jobs hanging up or failing? When a large volume of jobs are either submitted or are run simultaneously in a small time frame, the system is put under extreme load to manage all of the processes. IBM DataStage Operations Console can help in several ways!
Operations Console and Workload Management are now part of the base Information Server installation. In this post, we will provide an overview of Operations Console and how it works.

Benefits of Operations Console

  • Can protect the system from overloading by restricting how or when jobs can run
  • Provide a configurable way to limit the number of simultaneous job starts
  • Give users ways to prioritize the starting of job runs.

What is Operations Console?

The Operations Console provides engine-wide information about job runs, system resources, workload management queues, and engine status. You can obtain all of this information yourself by viewing the individual engines and job runs on every system that you own. However, piecing that information together to get an overall view of your environment can be difficult.
The IBM InfoSphere Information Server (IIS) engine status and job run information that is displayed in the console is supplied by the operations database. The operations database is populated with information from all of the engines that you have connected to your database. You can view only one IIS engine at a time in the console, but you can easily switch between the multiple engines that are connected to your operations database and associated with the current services tier machine
Operations Console is a SQL-based operations database (ODB) that:

  • Records information about jobs that are running, or that have run, on the engine system
  • Records system resource usage over time
  • Is open with a published schema

It is a mechanism for monitoring job runs in real-time, thus ensuring the runs part of the database is up-to-date and complete. It is also a mechanism for capturing operating system resource usage statistics and storing the history of that usage in the database. As a services-based Java API, it provides query, write, and maintenance access to that database.

Components of Operations Console

There are 4 main components: Dashboard, Projects, Activity, and Workload Management

Dashboard

The Operations Console Dashboard is a GUI which depicts the current state of the system

Operation Console Dashboard

Projects

A UI which navigates through the projects present on the system. It allows users to:

  • Traverse through individual projects
  • Browse Project properties
  • See job designs
  • Run/Stop/Reset jobs
  • Investigate into job runs
  • Compare job runs
  • Capture process metadata
  • Monitor individual job performance
  • Shares a resolution hyperlink for jobs not finishing with OK status

Operation Console Projects

Activity

A GUI which navigates through the current/past activities on the system

  • Traverse through jobs currently Running/Queued/Completed
  • Go through the details of the jobs Running/Queued/Completed Stop the jobs
  • Graphical charts depict trends overtime on following parameters:
    • CPU usage
    • Memory usage
    • Disk space
    • Job runs
    • Processes run on the system

Operation Console Activity

Workload Management

The UI for configuring and controlling Workload Management has two sections

  • System Policies – used for setting system level policies
  • Queued Jobs / Queue Management – used for configuring and managing queues

Operation Console Workload Management

Review

IBM lists the Operations Console and Workload Management as new features in the 11.3 release documentation, even though these components have already been introduced in previous releases. Both components are now part of the base Information Server installation and Workload Management is now by default enabled.
Users can use the DataStage/QualityStage Operations Console to monitor the job runs, services, system resources, and workload management queues on several IIS engines. Each component offers users the ability to customize reports based on the needs of the business.

For a complete white paper on Operations Console, please submit your email here.

Stay tuned for our next blog, Using Workload Management to optimize Operations Console, where we will give a more in-depth look at the Workload Management component and how using it can optimize the overall performance of Operations Console.