Skip to main content

Overview of Operationalization

Operationalization refers to a general class of platform features that enable repeated application of Designer Cloud Powered by Trifacta Enterprise Edition on production data. Whether deployed in a single flow or across all flows in your environment, operationalization features broaden the scope of wrangled data, simplify job execution, and enable these processes on a repeated or scheduled basis.

In the following sections, you can review short summaries of specific features and explore more detailed information on them.

Single-Flow Operations

These features can be applied to individual flows to simplify job execution.

Parameterization

Parameterization enables you to specify parameters that capture variability in your data source paths or names. For example, you can parameterize the names of folders in your filepaths to capture files within multiple folders. Or, you can parameterize your inputs to capture datasets named within a specific time range. Nested folders of data can be parameterized, too.

Parameter types:

  • dataset parameters: Parameterize the input paths to your data, allowing you to process data in parallel files and tables through the same flow.

  • output parameters: Parameterize the output paths for your results.

  • flow parameters: Define parameters that can be applied in your flows, including recipe steps.

    Tip

    You can apply overrides to any parameter at the flow level. These parameter override values are applied to any parameter that is referenced within the flow for any supported parameter type.

Parameter formats:

Note

Some of the following may not be available in your product edition.

Parameter Type

Description

Pattern

Use regular expressions or Wrangle in your paths or queries to sources to capture a broader set of inputs.

Wildcard

Replace parts of your paths or queries with wildcards.

Datetime

You can specify parameterized Datetime values in one of the supported formats.

Variable

Variable values can be specified as overrides during import, job execution, and output.

Parameterization is available for the following:

File systems

Input

Output

Date/time

Pattern

Variable

Timestamp

Variable

Relational sources

Input

Output

Timestamp

Variable

Timestamp

Variable

Note

For relational data, parameterization is applied to custom SQL queries used to import the data. For more information, see Enable Custom SQL Query.

For more information, see Overview of Parameterization.

Scheduling

The scheduling feature, also known as Automator, enables you to schedule the execution of individual flows on a specified frequency. Frequencies can be specified through the Trifacta Application through a simple interface or, if needed, in a modified form of cron syntax.

Tip

Automator is often used with parameterization to fully automate data preparation processes in Designer Cloud Powered by Trifacta Enterprise Edition.

For more information, see Overview of Automator.

Job monitoring

After a job has been launched, detailed monitoring permits you to track the progress of your job during all phases of execution. Status, job stats, inputs, outputs and a flow snapshot are available through the Trifacta Application. For more information, see Overview of Job Monitoring.

Email notifications

After a job has completed, you can send email notifications to stakeholders based on the success or failure of the job.

Note

This feature must be enabled. See Workspace Settings Page.

These notifications are defined within Flow View. See Email Notifications Page.

Webhooks

Webhook notifications let you define outgoing HTTP messages to any REST API. The message form and body can be customized to include job execution metadata. For more information, see Create Flow Webhook Task.

Deployment Manager

The Deployment Manager is a separate environment that can be enabled for the execution of production flows under limited access. Flows in development are exported from your default (Dev) instance and then imported to the Production instance, the Deployment Manager, where you can configure the periodic execution of the flow. For more information, see Overview of Deployment Manager.

Orchestration

Orchestration is a set of functionality that supports the scheduled execution of jobs across multiple flows. These jobs could be external processes, other flows, or even HTTP requests.

Terms

Term

Description

plan

A plan is a sequence of tasks that are executed on one or more flows to which you have access. To orchestrate tasks, you build a plan. A plan can be scheduled for execution, triggered manually, or invoked via API.

trigger

A task is executed based on a trigger. A trigger is a condition under which a task is executed. In many cases, the trigger for a task is based on the schedule for the plan.

task

A task is a unit of execution in the platform. For example, one type of task is the execution of a flow, which executes all recipes in the flow, as well as the flow's upstream dependencies.

snapshot

A snapshot of the plan is captured, and the plan is executed against this snapshot. For more information on snapshots, see "Plan execution" below.

Task types

The following types of tasks are available.

Type

Description

flow task

An ad-hoc or scheduled execution of the transformations required to produce one or more selected outputs from a flow.

HTTP task

A request submitted to a third-party server as part of a plan run.

Slack task

Send a message with information about the plan run to a specified Slack channel.

Limitations

  • You cannot specify parameter overrides to be applied to plans specifically.

    • Plans inherit parameter values from the objects referenced in the plan's tasks.

    • If overrides are applied to flow parameters, those overrides are passed to the plan at the time of flow execution.

      Tip

      Prior to plan execution, you can specify parameter overrides at the flow level. These values are passed through to the plan for execution. For more information, see Manage Parameters Dialog.

For this release:

  • The only type of plan task that is supported is Run Flow.

  • Tasks are defined in a linear, non-branching sequence based on the trigger.

Basic workflow

You create a plan and schedule it using the following basic workflow.

  1. Create the plan. A plan is the container for definition of the tasks, triggers, and other objects. See Plans Page.

  2. In Plan View, you specify the objects that are part of your plan. See Plan View Page.

    1. Schedule: The schedule defines the set of triggers that queue the plan for execution.

      1. Trigger: A trigger defines the schedule and frequency at which the plan is executed. A plan can have multiple triggers (e.g. monthly versus weekly executions).

    2. Task(s): Next, you specify the tasks that are executed in order.

      1. Flow task: A flow task includes the specification of the flow to run and the outputs from the flow to generate.

        Note

        You can select the outputs from the recipe that you wish to generate. You do not need to generate all outputs.

        Note

        When a flow task is executed, the execution plan works back from the selected outputs to execute all of the recipes required to generate the output, including the upstream dependencies of those recipes.

        See Plan View for Flow Tasks.

      2. HTTP task: An HTTP task is a request issued when it is triggered from the application to a target endpoint. This request supports a variety of API methods. See Plan View for HTTP Tasks.

      3. Slack task: A Slack task is a message between Designer Cloud Powered by Trifacta Enterprise Edition and a specified Slack channel that is triggered within the plan. See Plan View for Slack Tasks.

      4. Continue building tasks in a sequence.

  3. As needed, you can apply override values to any flow parameters that are included in the tasks of your recipe. These overrides are applied during a plan run. For more information, see Manage Parameters Dialog.

  4. To test:

    1. Click Run now.

    2. To track progress, click the Runs link.

    3. In the Run Details page, you can track the progress.

    4. The first task is executed and completes, before the second task is started.

    5. Individual tasks are executed as separate jobs, which you can track through the Jobs page. See Jobs Page.

    6. When the plan has completed, you can verify the results through the Job details page. See Job Details Page.

  5. If you are satisfied with the plan definition and your test run, the plan will execute according to the scheduled trigger.

Plan scheduling

Through the Plan View page, you can configure the scheduled executions of the plan. Plan schedules are defined using triggers.

  • These schedules are independent of schedules for individual flows.

  • You cannot create schedules for individual tasks.

Plan execution

When a plan is triggered for execution, a snapshot of the plan is taken. This snapshot is used to execute the plan. Tasks are executed in the sequence listed in Plan View.

Important notes:

Note

Any subsequent changes to the flows, datasets, recipes, and outputs referenced in the plan's tasks can affect subsequent executions of the plan. For example, subsequent removal of a dataset in a flow referenced in a task can cause the task to fail to execute properly.

At the flow level, you can define webhooks and email notifications that are triggered based on the successful generation of outputs. When you execute a plan containing an output with one of these messages, the message is triggered and delivered to stakeholders.

Note

Webhook messages and email notification cannot be directly triggered based on a plan's execution. However, you can create HTTP-based tasks to send messages based on a plan task's execution.

Tip

When a flow email notification is triggered through a plan, the internal identifier for the plan is included in the email.

See "Webhooks" and "Email notifications" above.

Enable

Enable the following setting:

Plans feature

Plan sharing, import, and export must also be enabled.

For more information, seeWorkspace Settings Page.

The following flags must be enabled for the orchestration service to correctly function.

Steps:

  1. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

  2. Locate the following settings. Verify that they are set to true:

    "webapp.orchestrationWorkers.enabled": true,
"orchestration-service.enabled": true,
"orchestration-service.autoRestart": true,

  3. You can choose to enable the following task types:

    Task type

    Setting

    Description

    HTTP task

    feature.orchestration.httpTasks.enabled

    When true, you can configure plan tasks to deliver a REST request over HTTP or HTTPS to a specified endpoint, including endpoints in the Designer Cloud Powered by Trifacta platform.

    Slack task

    feature.orchestration.slackTask.enabled

    When true, you can configure plan tasks to deliver messages to a specified Slack channel.

    Email task

    feature.orchestration.emailTask.enabled

    This feature is not yet available.

    Flow task

    This feature is automatically enabled when Plans feature is enabled. See above.

    These tasks execute a specific output on a selected flow.

  4. Save your changes and restart the platform.

Logging

Logging information on plan execution is captured in the orchestration-service.log. This log file can be downloaded as part of a Support Bundle. For more information, see Support Bundle Contents.

You can configure aspects of how this log captures service events. For more information, see Configure Logging for Services.

In this section: