WORKFLOW AUTOMATION MODELS

Sun, 22 Apr 2018 23:59:42 -0659

Workload automation sounds simple at its surface. A business consumer (e.g., a department head or a senior executive) may state requirements in “if this, then that” form:

If a department budget file arrives, validate it and send to the accounting system for processing.
At midnight, backup and index the payroll database.
On demand, generate an email to a prospect list of with an attached new product brochure.

They ask information technology staff “What’s so hard about that?” Business consumers are familiar with mobile apps that help them simply script their daily interactions email, calendars, and spreadsheets, and they expect this simplicity to be present in their enterprise applications. In fact, many IT staffs start down this path of writing scripts using the “if this, then that” form, assigning developers or support staff to create command scripts and Windows Task Scheduler and crontab entries to perform routine tasks. But the simplicity of these scripts erode over time. Corporate policies often force such scripts to address additional demands such as:

Failover and backup of the scripts and the underlying jobs being executed
Security requirements to restrict access to jobs
Audit requirements of job creation, submission, pausing, interrupts, and restarts
Centralized monitoring of hundreds to thousands of these tasks
Error recovery that becomes a mission critical component of the daily operation of the enterprise And as these requirements are addressed the scripts become large, complex, often undocumented, and difficult to maintain. Such scripts are often created by a single person, creating a single point of failure when that individual takes leave or exits the company.

Yet constructing workload automation tasks that are simple enough for the business consumer yet comprehensively support the enterprise’s needs is indeed a reasonable goal. How does one achieve such a goal? Many engineering activities use models to make things simpler to grasp while hiding unnecessary complexities. Models can be graphical, as well as textual and even mathematical.

To better understand how models are useful in workload automation, let’s take the first example requirement above and start to expand what fully supporting it requires:

If a department budget file arrives, validate it and send to the accounting system for processing.
Listen on the SFTP server located in the company’s data center DMZ for the arrival of a file with the extension .BUDGET.
Determine if the file is empty (has zero bytes)
If true, send an email notification to the customer that sent the file.
Locate the customer ID in the file’s name using a regular expression
Looking up email address information in a customer database using the customer ID as a key.
Update the customer database record with a note that an empty file has been sent, and cannot be processed until a corrected file is sent.
Append to the file the extension ‘PENDING CUSTOMER RESPONSE’. …

Continuing to expand the requirement can lead to many pages of details – burying the consumer in the implementation details of – what is perceived by them – a simple and often performed need. Not shown in the above text are all the paths that may be taken in the event of errors or exceptional condition, e.g., the SFTP server is down, or no file is present at the expected time. So what’s needed to manage this complexity back to the supposed simple request of the business consumer? Two powerful strategies for managing complexity are those of abstraction and information hiding. From Techopedia,

Abstraction: “the act of representing essential features without including the background details or explanations.”
Information-Hiding: “a way in which clients could be shielded from internal program workings.”

Abstraction and information-hiding work in concert to provide flexibility and simplicity by focusing attention on the essential details of the work being automated while hiding the lower-level details of how that is accomplished. Powerful yet simple to use workload automation platforms provide modeling features that hide low level details and focus on the key items of interest (the key “abstractions”) in a request. For example, workload automation models should consider the following common workflow concepts:

Triggers – an abstraction that waits or listens for an event to occur, and when that event occurs they “fire” or execute, returning a result appropriate for the trigger. A timer trigger, for example, listens for the arrival of a time and when that happens it fires and flows to the next step. The arrival of a file (which returns information about the file) and the existence of the database in a given state (which returns true or false depending upon if the database condition is met) are other instances of triggers.
Actions – a different abstraction that performs a service, such as running an operating system command or performing a database query. Actions return results too, such as the exit code of running a command, or a list of rows retrieved from a database query.
Flows – flows are the connectors, or “glue”, between actions and triggers. Flows contain the results generated by actions and triggers (and data you may want to add) and routes this information to the next steps.
Workflows – workflows are flowcharts consisting of the above items. In the context of a workflow the above actions, triggers, and flows interact to generate an intended result.

Each of the above abstractions hides, and makes visible when needed, its own implementation details. Triggers may hide/expose the conditions when they fire, or the interval that they wait before firing. Actions may hide/expose the command they are to run, or the maximum time the command should be allowed to run before timing out.

Workflow designers through their models can exploit abstraction and information-hiding to facilitate the creation of understandable and easy to use workflows without overwhelming the business consumer with low level details. For example, the example requirement can be modeled with the following, either graphically or in text:

Define a trigger that checks for a file’s presence. Within that trigger specify the details of file server host names, file transfer protocol, user credentials, and directories and file names to scan for.
Define an action that performs a validation routine against the file. Within that action specify the command or program code to use and additional details as to working directories, timeout conditions, where to route error or output messages, etc.
Define an action that copies the file to a location the accounting system recognizes. Within that action specify server name and directory names to move files to, as well as renaming the file with a suffix or date processed extension.
Define flows between the above triggers and actions. Then define the properties of the flows, such as timeout conditions, which flow to take depending upon the result of the prior action or trigger, and exceptional or error flow conditions.

The above models can be constructed using a number of modeling approaches such as text (e.g., business rules or formal executable specifications) or graphics. For example, a graphical modeling tool that supports creating icons (for the abstractions) and assigning properties to them (for the implementation details) is one approach. Visio and draw.io are two tools one could consider. But workload automation models created using these tools would not actually run workloads. Ideally a workload automation “platform” would provide support for such modeling, as well as provide an environment within which these models would execute. Below is one example of modeling just the abstractions, while hiding the low-level details, in a common workflow.

In Summary

Describing requirements using “if this, then that” is a simple, but not sufficient, means of modeling workload automation. A more powerful and comprehensive approach can be achieved by isolating the fundamental concepts of workloads (abstractions such as triggers, actions and flows), and modeling these into workflows. Ideally a workload automation “platform” supports directly executing these models making the distance from concept to running workflows a mouse click away.

PROVISIONING WORKFLOWS TO SIMPLIFY BATCH SCHEDULE DEPLOYMENTS

Sun, 22 Apr 2018 23:59:42 -0659

Designing and deploying batch job schedules can be complex and intimidating. Coordinating dependencies, balancing CPU and disk resources, and tracking down errors can be time-consuming and error-prone.

But after design, how do you deploy your workflows to maximize their utility for a wide variety of uses?

Workflow provisioning provides a solution. By taking your best practice workflows and embedding variables within them that are resolved at runtime, and adding a provisioning workflow to configure and submit your workflows, you can take your workflows and use them repeatedly.

Such a facility requires that your workflow engine or scheduler allows data entry at the time the workflow is submitted for scheduling. The entered data is used to “provision” the workflow with needed values for the variables defined in the workflow.

Say we want a workflow that we will use repeatedly to start other workflows. We want this initiating workflow to provision other workflows with their name, time to run, command to execute, and email addresses for sending success and failure notifications. We will call this initiating workflow “EasyWorkflow.”

First, we add a provision form to our EasyWorkflow workflow. The form itself is defined using Json and will be rendered when we submit EasyWorkflow for execution. A snippet of the Json is illustrated below.

The complete version of the above form collects five pieces of information with the following variable names:

ENGINENAMESPACE – The namespace of where to run the SimpleCommand workflow
TIMEEXPRESSION – A cron expression or relative time expression of when to run
COMMAND – A batch script, shell script, or command line to execute
FAILEDCONTACT – An email address to notify when the command fails
SUCCESSCONTACT – An email address to notify when the command succeeds

EasyWorkflow contains a flow chart action which is used to initiate the other flow chart. This flow chart action is used to fetch another flow chart and inject the values collected from the above form into the flow chart specified (in this case a workflow named SimpleCommand). SimpleCommand will be initiated separate and apart from EasyWorkflow.

The SimpleCommand workflow is shown below. It simply runs a specified command within a process action based on a time expression specified within a timer trigger.

The definition for how EasyWorkflow will start SimpleCommand is shown below. EasyWorkflow will submit SimpleCommand for execution with the values from its form by turning on “Initialize from Flow Context”. EasyWorkflow will submit SimpleCommand to the scheduler engine with the variable value with the name ENGINENAMESPACE collected from the form above.

When EasyWorkflow is submitted from the repository of workflows to the scheduler for execution, it renders the following form to the submitting user. Default values are provided to be edited for your specific needs. In this case the command is being scheduled to run at 5 AM using the time expression “0 0 0 5” (i.e., the 0th millisecond, 0th seconds, 0th minutes, 5th hour of every day).

I’M LATE, I’M LATE. HANDLING THE LATE SCHEDULING OF JOBS

Sun, 22 Apr 2018 23:59:42 -0659

A scheduled batch job might not run on time (i.e., be “late”) for many reasons - for example:

the scheduler might be down for an extended period of time, during which one or more jobs would normally have run. there is not enough concurrency available to process jobs in a timely manner, so a queued job may not be reached for scheduling until after it’s time to execute has passed. A required resource for your job may not be available so your job must wait. This is common when your job performs file transfers from a remote service and the remote service is down. Such circumstances can often be handled manually by pausing or restarting jobs, but this can quickly become overwhelming, especially when many jobs are involved.

To handle this, jobs need a defined way of telling the scheduler what the absolute latest time the job can run without being skipped. This is referred to as a “late time window.”

For example, a late time window of “+5m” (five minutes) means that the job can fire up to five minutes past its scheduled firing time. A late time window of “+1d” would allow the job to fire up to one day late.

If the job attempts to run but the late time window has elapsed, the scheduler will simply determine the next scheduled date, skip the late job, and continue running as normal (unless makeup firing is enabled – see further down in this post).

Scheduled Date/Time	Late Time Window Date on Clock Now Time Expression for Next Run Result
April 30, 2020 08:00:00	+6H	April 30, 2020 10:30:00	+6H	Actual date is within the late time window, so the job fires. Only one firing was missed so the job continues as normal, scheduling the next firing for 14:00:00.
April 30, 2020 08:00:00	+5m	April 30, 2020 08:06:00	+6H	Actual date is not within the late time window, the job does not fire. The scheduler simply schedules the next execution at 14:00:00 as though no firing was missed.
April 30, 2020 08:00:00	+30m	April 30, 2020 08:25:00	+10m	Actual date is within the window, so the job fires. Two additional firings were missed (08:10:00 and 08:20:00); the job fires again for each of those before setting the next date (April 30, 2020 08:30:00).
April 30, 2020 08:00:00	+1H	April 30, 2020 10:30:00	+1H	Actual date is not within the late time window, the job does not fire (even though one of the missed firings was within the window). The next firing date (April 30, 2020 11:00:00) is scheduled.

MAKEUP FIRING ENABLED This property is only evaluated if the late time window has elapsed but there are one or more missed firings.

If makeup firing is enabled, at least one firing was missed, but that firing did not fall within the late window, then Flux will fire a single makeup firing before advancing to the next scheduled trigger date.

This is useful in cases where several firings were missed and did not fall within the late time window, but you still want to fire once to compensate for the missed firings.

Examples - Assume the scheduler was unavailable from April 30, 2020 07:59:00 to April 30, 2020 15:00:00

Scheduled Date/Time	Late Time Window	Date on Clock Now	Time Expression for Next Run	Enable Makeup Firing	Result
April 30, 2020 08:00:00	Not specified April 30, 2020 15:00:00 +6H Not specified (default is false)	The trigger fires at April 30, 2020 15:00:00. The next date (April 30, 2020 20:00:00) is scheduled.
April 30, 2020 08:00:00	+6H	April 30, 2020 15:00:00	+6H	true	Late time window has elapsed but makeup firing is enabled, so the trigger fires at April 30, 2020 15:00:00. The next date (April 30, 2020 20:00:00) is scheduled.
April 30, 2020 08:00:00	+6H	April 30, 2020 15:00:00	+6H	false	Late time window elapsed and makeup firing is disabled, so the trigger does not fire. The next date (April 30, 2020 20:00:00) is scheduled.
April 30, 2020 08:00:00	+1H	April 30, 2020 10:32:00	+5m	true	Late time window has elapsed but makeup firing is enabled, so the the trigger fires. The next date (April 30, 2020 10:35:00) is scheduled.

For additional details of Flux’s handling of time schedules see FIX-ME-ADD-LINKhttps://doc.flux.ly/display/flux80/Timer+Trigger or contact support@flux.ly.

DATES, TIMES, AND CALENDARS – OH MY!

Sun, 22 Apr 2018 23:59:42 -0659

Workload automation projects generally start simple. The first step is often setting up the schedule of when to run jobs. Schedules can start out straightforward, but over time the kinds and types of schedules needed to be addressed evolve and grow increasingly complex.

A commonly used time expression language is that of Unix Cron. We here at Flux have extended the Cron syntax to support the wide variety of schedules seen by our customers. Flux time expressions have 11 fields. Position (1) ms (2) sec (3) min (4 )hr (5) day_in_month (6) month (7) weekday (8) day_in_year (9) week_in_month (10) week_in_year (11) year as well as support for relative time expressions (e.g., +35s, +1m, +2H, +7d, +2w, +1M, +1y)

Consider the following kinds of schedules and their associated time expressions:

Run a job on a specified date or time, such as midnight on the fifth day of each month, or every 15 minutes between 9 am and 5 pm. [0 0 0 0 5] and [0 0 0,15,30,45 9-17]
Run a job every 15 hours from now for only Monday – Friday [0 0 0 +15 * * MON-FRI]
Run a job at two or more times that are unrelated to each other such as midnight and 4:45 am. [(0 0 0 0|0 0 45 4)]
Run a job on a date or time relative to now (or another date and time), such as 2 months from now or every 10th Wednesday [+2M] and [0 0 0 0 * * WED * +10]
Run a job starting from a start date and time to an end date and time, firing at regular intervals such as January 1-15 firing at 3 PM. [0 0 0 15 * * * 1-15]

Time expressions support a wide range of scheduling needs, but how does one define holidays, vacations, or other intervals when schedules should, or should not run?

BUSINESS INTERVALS AKA BUSINESS CALENDARS

A Business Interval, generally known as a “Business Calendar,” defines periods of time. These intervals are then applied during the scheduling of workflows to determine whether a workflow may (or may not) run. Unlike ordinary calendars, Business Intervals are not required to be oriented to calendar days and may even have resolution to the second. Picture the calendar as a timeline running from left to right – Business Intervals define the ranges of time that are either included or excluded. Included ranges of time define when workflows may run, and excluded ranges of time define the periods that the workflows may not run.

The “timeline” concept of business intervals means that they do not need to be confined to 24-hour periods – even if it crosses from one calendar day to the next.

Business Intervals can be applied to time expressions to help them calculate their future firing times. Intervals can be used in timer and delay triggers, in calculating timeouts for actions or triggers, and to specify an allowable time window for a File Exist Trigger to monitor for files.

DEFINING BUSINESS INTERVALS / BUSINESS CALENDARS Business Intervals are themselves described with time expressions. They are generally a collection of dates and times using the time expressions that define the allowed processing intervals. For example, here are the time expressions for U.S. Federal holidays. How these work is described further below using shifting.

NEW_YEAR = "^y?sat{<fri}{}?sun{>mon}{}";
MLK = "^y@3mon";
PRESIDENTS = "^y>feb@3mon";
MEMORIAL = "^y>may$M<mon^d";
INDEPENDENCE = "^y>jul+3d?sat{<fri}{}?sun{>mon}{}";
LABOR = "^y>sep@mon";
COLUMBUS = "^y>oct@2mon";
VETERANS = "^y>nov+10d?sat{<fri}{}?sun{>mon}{}";
THANKSGIVING = "^y>nov@4thu";
CHRISTMAS = "^y>dec+24d?sat{<fri}{}?sun{>mon}{}";

USING BUSINESS INTERVALS / CALENDARS

Calendars add meaning to dates. A day can now, for example, be a business- or non-business day, a weekday, a weekend, or a holiday. A time expression of “0 0 0 0 5” (the fifth day of the month) can be amended to “0 0 0 0 5b” to now point to the fifth business day of the month. The “b” indicates the value is to be calculated with the associated business interval.

SHIFTING

NEW_YEAR = ^y?sat{<fri}{}?sun{>mon}{}

Take when New Year’s Day is celebrated as a holiday in a business calendar If it occurs on a Saturday, take off the preceding Friday. If it occurs on a Sunday, take off the following Monday. This is referred to as date “shifting.”

For instance, to run a task on the first business day on or after the 10th calendar day of the month, use “10>1b” in the Days-of-month column. If the 10th calendar day of the month falls on a holiday, then the “>1b” portion will “push” the expression ahead to the next business day. On the other hand, if the 10th calendar day of the month is already a business day, then the “>1b” portion does nothing, because the 10th calendar day of the month is already a business day.

These techniques and symbols can be used in any of the Cron-style time expression fields, not just Days-of-month and Day-of-year fields.

ROLLOVER

FIRST_MONDAY_ON_OR_AFTER_THE_28th = 0 0 0 0 28»1MO

The rollover operator (“»”) provides similar functionality to the right shift operator (“>”). The difference between rollover and right shift is that when the rollover operator attempts to satisfy the constraints of the Cron-style time expression, it can “rollover” into the next Cron column on the right in search of a match, unlike the right shift operator.

SUMMARY

The combination of powerful time expressions and business calendars yield a flexible scheduling platform that address complex scheduling needs. Embedded within Flux is a time expression tester that makes defining and testing time expressions easy.