BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation

BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation

by Qiang Ding
Released October 2012
Publisher(s): Packt Publishing
ISBN: 9781849682565

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Master one of the world’s most powerful enterprise workload automation tools – BMC Control-M 7

  • Implement and utilize a world class enterprise batch scheduling and workload automation tool in the best possible ways with this book and e-book
  • Hands-on implementation and administration of a Control-M environment
  • Easily develop Control-M job flows to meet simple and complex business requirements
  • Written in a conversational manner and supported by a lot of practical, easy-to-follow examples accompanied by screenshots

In Detail

Control-M is one of the most widely used enterprise class batch workload automation platform. With a strong knowledge of Control-M, you will be able to use the tool to meet ever growing batch needs. There has been no book that can guide you to implement and manage this powerful tool successfully... until now. With this book you will quickly master Control-M and be able to call yourself "a Control-M" specialist!

“BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation” will lead you into the world of Control-M and guide you to implement and maintain a Control-M environment successfully. By mastering this workload automation tool, you will see new opportunities opening up before you.

With this book you will be able to take away and put into practice knowledge from every aspect of Control-M – implementation, administration, design and management of Control-M job flows, and more importantly how to move into workload automation and let batch processing utilize the cloud.

You will start off with batch processing and workload automation, and then get an understanding of how Control-M meets these needs. Then we will look more in depth at the technical details of Control-M, and finally look at how to work with it to meet critical business needs. Throughout the book, you will learn important concepts and features, as well as learn from the Author’s experience, accumulated over many years. By the end of the book you will be set up to work efficiently with this tool and also understand how to utilize the latest features of Control-M.

Table of contents

  1. BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation
    1. Table of Contents
    2. BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation
    3. Credits
    4. About the Author
    5. Acknowledgement
    6. About the Reviewers
    7. www.PacktPub.com
      1. Support files, eBooks, discount offers and more
        1. Why Subscribe?
        2. Free Access for Packt account holders
        3. Instant Updates on New Packt Books
    8. Preface
      1. What this book covers
      2. Who this book is for
      3. Conventions
      4. Reader feedback
      5. Customer support
        1. Errata
        2. Piracy
        3. Questions
    9. 1. Get to Know the Concept
      1. Introduce batch processing
        1. The history of batch processing
        2. Batch processing versus interactive processing
        3. Time-based batch- and event-driven batch
        4. Is this the end for batch processing?
        5. Running batch processing tasks
      2. Automating batch processing
        1. Basic elements of a job
          1. What to trigger
          2. When to trigger (Job's scheduling criteria)
          3. Dependencies (Job's predecessors and dependents)
          4. More advanced features of scheduling tools
          5. Ability to generate notifications for specified events
          6. Ability to handle an external event-driven batch
          7. Intelligent scheduling — decision-making based on predefined conditions
          8. Security features
          9. Additional reporting, auditing, and history tracking features
      3. Centralized enterprise scheduling
        1. Challenges in today's batch processing
          1. Processing time
          2. Batch window length
          3. Batch monitoring and management
          4. Cross-time zone scheduling
          5. Resource utilization
          6. Maintenance and troubleshooting
          7. Reporting
          8. Reacting to changes
        2. The solution
          1. Processing time and resource utilization
          2. Batch monitoring and management
          3. Cross-time zone scheduling
          4. Maintenance and troubleshooting
          5. Reporting
          6. Reacting to changes
      4. From batch scheduling to workload automation
        1. Batch scheduling: Static scheduling
        2. The Workload Automation concept
        3. Dynamic batch processing with virtualization technology and Cloud computing
        4. Integration with real-time system, workload reusability
      5. Summary
    10. 2. Exploring Control-M
      1. Control-M overview
        1. Control-M road map
        2. Key features
        3. Supported platforms
        4. The Control-M way
          1. Control-M job
            1. Job conditions
            2. Resources
          2. Submitting jobs
          3. Post processing
        5. From the user's perspective - Control-M/Enterprise Manager
          1. Control-M Enterprise Manager GUI Client
          2. Control-M Desktop
          3. Control-M Configuration Manager
          4. Reporting Facility
        6. Control-M's Optional Features
          1. Control-M Control Modules
          2. Control-M/Forecast and BMC Batch Impact Manager
            1. Control-M/Forecast
            2. BMC Batch Impact Manager
          3. BMC Batch Discovery
      2. Control-M Architecture and Components
        1. Control-M/Enterprise Manager
          1. Control-M/Enterprise Manager Server Components
            1. Naming Service
            2. Control-M Configuration Server
            3. Control-M/Enterprise Manager Configuration Agent
            4. GUI Server
            5. Gateway process (GTW)
            6. Global Alert Server (GAS)
            7. Global Condition Server (GCS)
            8. Control-M Web Server
        2. Control-M/Server
          1. Control-M/Server processes
            1. SU: Supervisor
            2. SL: Job Selector
            3. TR: Job Tracker
            4. NS: Agent Communication Process
            5. CE: New Day and EM Communication Process
            6. CS: Server Process
            7. LG: Logger Process
            8. WD: Watchdog Process
            9. RT: Internal Communication Router
            10. CA: Configuration Agent
        3. Control-M/Agent
          1. AG: Agent Listener, Request Handler
          2. AT: Agent Tracker
          3. AR: Agent Router Process
          4. UT: Utility Process
        4. Agentless Technology
        5. Control-M/Control Modules
      3. How do Organizations Work With Control-M?
      4. Where to Start?
        1. General Product information
        2. Official Education and Certification
        3. Getting a Job in Control-M
      5. Summary
    11. 3. Building the Control-M Infrastructure
      1. Three ages to workload automation
        1. Stone age
        2. Iron age
        3. Golden age
      2. Planning the Batch environment
        1. Control-M sizing consideration
          1. Total number of batch jobs run per day
          2. Total number of job execution hosts
          3. Number of datacenters
          4. Amount of concurrent GUI users
        2. Use Control-M/Agent or go Agentless
        3. Production, development, and testing
        4. Control-M high availability requirements
          1. Control-M in a clustered environment
          2. Control-M/Server mirroring and failover
            1. Control-M/Server database mirroring
            2. Control-M/Server failover
          3. Control-M node group
          4. High availability by virtualization technology
      3. Pre-installation technical considerations
        1. Environment compatibility
        2. Choices of database
        3. System configuration requirements
          1. Linux Kernel parameters
            1. Shared memory
            2. Semaphores
          2. User limits
          3. Other requirements
          4. Storage space related considerations for Control-M
        4. Firewall requirements
          1. Between Control-M/Enterprise Manager Clients and Server Components
          2. Between Control-M/Enterprise Manager Server Components and Control-M/Server
          3. Between Control-M/Server and Control-M/Agent
          4. Agentless remote hosts
          5. Database
        5. Last things to make sure of before the installation starts
      4. Installation
        1. Install Control-M/Enterprise manager server components
          1. Download and execute the check_req script
          2. Create a Linux user and allocate space for Control-M/EM
          3. Configuring the system to meet installation requirements
          4. Preparing the installation media
          5. Installation
          6. Post-installation tasks
        2. Install Control-M/Enterprise manager clients
          1. Preparing the installation media
          2. Installation
          3. Post-installation tasks
        3. Installing Control-M/Server
          1. Installation in Linux environment
            1. Pre-installation
            2. Installation
            3. Post-installation tasks
          2. Installation in a Windows environment
            1. Pre-installation tasks
            2. Installation
            3. Post-installation tasks
        4. Installing Control-M/Agent
          1. Installation in Linux environment
            1. Pre-installation tasks
            2. Installation
            3. Post-installation tasks
          2. Installation in a Windows environment
      5. Summary
    12. 4. Creating and Managing Batch Flows with Control-M GUI
      1. The Control-M way — continued
        1. Contents of a job definition
          1. "What" #1: job type
          2. "What" #2: task type
          3. "Who" #1 — owner of the job
          4. "Who" #2 — author of the job
          5. "Where" #1 — job's execution host
          6. "Where" #2 — storing job definitions
            1. Datacenter/Table/Job
            2. Application/Group/Job
          7. "When" #1 — job's scheduling date
            1. Defining a job's scheduling date
            2. Calendars
            3. Rule-Based Calendar (RBC)
            4. Retro job
          8. "When" #2 — time frame for job submission
          9. "When" #3 — cyclic jobs
          10. "When" #4 — manual confirmation jobs
          11. "When" #5 — job condition
          12. "When" #6 — resource and job priority
            1. Quantitative resource
            2. Control resource
          13. "When" #7 — time zone
          14. What happens right after the job's execution is completed?
            1. PostProc
            2. Step
          15. Autoedit facility
            1. Autoedit variables
            2. Autoedit expressions and functions
        2. Lifecycle of a job
          1. Write/Load, Upload/Download, Order/Force, and Hold
          2. State of a job
        3. New Day Procedure (NDP)
          1. Active job ordering
          2. Active job cleaning
          3. Control-M Date and Odate
        4. User Daily
      2. Working with Control-M Desktop and EM GUI Client
        1. Control-M Desktop — the Workspace
        2. Control-M/EM GUI client — Active ViewPoint
        3. Defining and running jobs
          1. Creating the first job — Hello World!
          2. Write, Upload, and Order the job
            1. Write
            2. Upload
            3. Order
          3. Monitor and Control the Job
            1. Job Sysout
            2. Rerun a Job
            3. Job Log
            4. Job Statistics
          4. Modifying and rerunning the job
            1. Modifying the static job definition
            2. Modifying the active job instance
        4. A more complicated job flow
          1. Defining SMART table, application, and group
          2. Building cyclic jobs
          3. Utilizing the Autoedit facility
            1. Job submission variables
            2. User-defined Variables
            3. System Variables
          4. Linking jobs with job conditions
          5. Defining Global Conditions
            1. Deciding the Global Condition pre-fix
            2. Registering the Global Condition pre-fix
          6. Creating calendars
          7. Adding job post-processing and job steps
            1. Post-processing
            2. Job steps
          8. Working with Resources
            1. Quantitative Resource
            2. Control Resources
          9. Having a "Start" job
      3. Summary
    13. 5. Administrating the Control-M Infrastructure
      1. Additional component installations
        1. Installation of BIM and Forecast
          1. Installation
          2. Post-installation tasks
          3. Configuring BIM web interface
        2. Installation of Control Modules
          1. Pre-installation considerations
          2. Installation — Control-M for database
          3. Installation — Control Module for Advanced File Transfer
            1. Interactive installation
            2. Silent installation
          4. Installation — Control-M Business Process Integration Suite
          5. Post-installation tasks
            1. Importing CM-specific job editing forms
            2. Installing CM utility add-ons into the CCM and Control-M/EM server
      2. Expanding and updating the batch environment
        1. Ongoing installation of Control-M/Agents and Control Modules
        2. Installing multiple Control-M/Agents on the same host
        3. Defining Agentless remote hosts
          1. Unix/Linux remote host (using SSH)
          2. Windows remote host (using WMI)
        4. Applying Control-M fix packs and patches
          1. When to apply fix packs and patches
          2. How to apply fix packs and patches
          3. Fix pack and patch installations in our environment
        5. Installing additional Control-M GUI clients
      3. Frequent administration tasks
        1. Stop/start components
          1. Manually stop/start components
            1. Control-M/EM server components
            2. Control-M/Server components
            3. Control-M/Agent components
          2. Configuring automatic startup script
        2. Defining additional GUI users and groups
          1. Authorization of configuration items
            1. Active tab
            2. Tables and calendars
            3. Prerequisite conditions and global conditions
            4. Quantitative and control resources
            5. Owners
            6. Privileges
            7. Member of
        3. Customizing Control-M GUI
          1. Control-M/EM GUI
          2. Control-M Desktop
      4. Summary
    14. 6. Advanced Batch Scheduling and Management
      1. Importing existing batch processing tasks
        1. Importing CRON jobs into our environment
          1. For Host ctm-demo-linux-01
          2. For Host ctm-demo-linux-02
      2. Enhance the file processing batch flow
        1. Control-M filewatch
          1. Technical background
            1. Invoking methods
            2. Filewatch rules
          2. Defining filewatch job
            1. Adding the job
            2. Defining filewatch rules
            3. SMART table level autoedit variable
        2. Advanced file transfer
          1. Technical background
            1. Verifying destination file size after transfer
            2. Verifying checksum
            3. Restarting from the point of failure
            4. Encryption and compression
            5. Pre and post (transfer) actions
            6. Filewatcher
          2. Implementing AFT jobs
            1. Creating an AFT account
            2. Defining the AFT job
        3. Control-M for database
          1. Technical background
          2. Implementing Database CM jobs
            1. Creating a database connection account
            2. Defining Database CM Job
      3. Advanced batch management
        1. ViewPoints
          1. Viewing jobs in Active ViewPoint
            1. Find feature
            2. Dynamic filter
            3. Print
          2. Performing job actions in Active ViewPoint
            1. Delete/Undelete
            2. Kill
            3. Force OK/Force OK with no post processing
            4. Why/Enhanced Why
            5. Bypass
            6. Branch Menus and Neighborhood
            7. Critical Path
          3. The Time Machine — Archived ViewPoint
          4. Creating ViewPoint
            1. Hierarchy
            2. Collection and default filter
        2. Forecasting
          1. Job scheduling plan forecasting
          2. Forecast ViewPoint
        3. Managing batch flows as services
          1. Defining services
          2. Monitoring services
      4. Control-M reporting facility
        1. Type of reports
        2. Creating a report
        3. Automated reporting
          1. The emreportcli utility
          2. Reporting job
      5. Summary
    15. 7. Beyond Everyday Administration
      1. GUI alternative — command-line utilities
        1. Control-M/Server utilities
        2. Control-M/Agent utilities
      2. Securing the environment
        1. User authentication: External authentication
          1. Configuring LDAP parameters
          2. Converting existing GUI users to authenticate with LDAP
          3. Associating EM user groups with LDAP groups
        2. User privilege: Control-M/Server security
          1. Defining group-level security
          2. Defining user-level security
          3. Enabling Control-M/Server security
        3. Job ordering and submission: User exit
        4. Job execution: Control-M/Agent security
          1. Security for Windows Control-M/Agents
          2. Security for Unix/Linux Control-M/Agents
          3. Control-M/Server utility authorizations
        5. Inter-component communication — firewall
          1. Between Control-M/EM server components and GUI clients
          2. Between Control-M/Server and Agents
        6. Inter-component Communication — SSL
          1. Implementing SSL
        7. Auditing
          1. Enabling and configuring auditing
          2. Producing auditing report
      3. Control-M mirroring and failover
        1. Pre-implementation tasks
          1. Installing and configuring the secondary Control-M/Server
          2. Configuring Control-M/Agents
          3. Testing the secondary Control-M/Server
        2. Initializing mirroring and failover
        3. Switching to mirroring and failover
          1. Switching over to mirroring
          2. Switching over to failover
        4. Recovering from mirroring and failover
          1. Recovering from mirroring
          2. Recovering from failover
      4. Perfecting Control-M
        1. Housekeeping
          1. Active environment-related housekeeping
            1. Statistic average calculation
            2. Historical statistic average cleaning
            3. Job condition cleaning
            4. Exporting Control-M/Server Log (IOALOG/ctmlog)
          2. Database-related housekeeping
            1. Control-M/Server database statistics calculation
            2. Control-M/Server health check
            3. Control-M/Server database hot backup
            4. Control-M/EM data backup
          3. Filesystem-related housekeeping
          4. Component status checking
        2. NDP tuning
          1. Things happening during NDP
            1. Removing old ctmlog
            2. Removing old job statistic information
            3. Sending Sysout cleanup trigger
            4. Deleting conditions
            5. After active job cleaning and ordering
          2. Shortening NDP
            1. Removing old job statistic information outside NDP
            2. Ordering jobs outside NDP
        3. Other configurations items
          1. Control-M/EM: MaxOldDay and MaxOldTotal
          2. Control-M/EM: Default AverageTime
          3. Control-M/Server: New Day Time
          4. Control-M/Server: Simple Mail Transfer Protocol parameters
          5. Control-M/Server: shout destination tables
      5. Summary
    16. 8. Road to Workload Automation
      1. Integrating Control-M with business processes
        1. Building the environment
        2. Interacting with BPI interfaces
          1. Technical background
          2. Defining an account
          3. Triggering job ordering
            1. Creating a project in soapUI
            2. Sending SOAP request to the web services
        3. Taking parallel processing to the next level
          1. Merging the two file processing job flows
          2. Enabling parallel processing
            1. Adding table ID into condition names
            2. Modifying the filesystem directory structure and job scripts
            3. Updating the quantitative resource
        4. Implementing control module for BPI jobs
          1. Technical background
          2. Defining accounts
            1. Web service account
            2. Message queue account
          3. Creating jobs
            1. Web service job
            2. Message queue jobs
          4. Updating the quantitative resource
        5. End-to-end testing
      2. Managing batch jobs as workloads
        1. Running jobs in node groups
          1. Technical background
          2. Creating node groups
          3. Making necessary changes to the environment and jobs
            1. The big picture
            2. Making changes to jobs and scripts
            3. Making changes to quantitative resources
          4. Putting into action
        2. Defining and managing workloads
          1. Technical background
          2. Defining workloads
          3. Putting into action
      3. Into the Cloud
        1. Technical background
        2. Defining accounts
        3. Defining jobs
          1. Modifying the file processing job flow
          2. Defining CM for Cloud jobs
        4. End-to-end testing
      4. Summary
    17. Index

Product information

  • Title: BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation
  • Author(s): Qiang Ding
  • Release date: October 2012
  • Publisher(s): Packt Publishing
  • ISBN: 9781849682565