PDI to Jupyter Notebook

The Data Validator transform enables you to define validation rules that check input data across different fields in each row. When the validator encounters a row that violates one or more validation rules, it generates an error or exception.

You can capture all validation errors by configuring an error handling hop from this transform, which will provide you with a comprehensive list of any validation failures that occur during processing.

The Replace In String transform performs simple search and replace operations on String fields.

It also supports regular expressions and group references. group references are picked up in the replace by string as $n where n is the number of the group.

You can also use hexadecimal replacements for special characters.

The Calculator transform offers built-in functions that operate on your input field values.

Beyond the function arguments (Field A, Field B, and Field C), you need to define the return type for each function.

After calculations are complete, you can exclude specific fields from the output using the Remove option. This feature helps eliminate temporary values that aren't needed in your final pipeline.

The Calculator delivers significantly faster execution compared to custom JavaScript scripts.

The Group By transform organizes rows from a data source according to one or more specified fields, creating a single row for each distinct group. Additionally, it can compute aggregate values like sums, averages, or counts for each group.

Typical applications include determining average sales figures by product category or tallying inventory quantities for each item type.

This step requires sorted input data to function properly. When working with unsorted data, only identical consecutive rows will be grouped together correctly. Furthermore, if data is sorted externally before entering the Transformation, differences in case sensitivity within the grouping fields may lead to unexpected results.

For scenarios involving unsorted input data, consider using the Memory Group By transform instead, which can handle data regardless of its initial order.

The Text File Output transform exports data to text file formats, most commonly generating CSV files that can be opened in spreadsheet applications like Excel.

This transform also supports creating fixed-width files by specifying field lengths in the fields configuration tab. You have two options for defining the output structure: use an existing Schema Definition or manually configure the field layout. When working with a Schema Definition, pair this transform with the Schema Mapping transform to align your incoming data stream with the chosen schema structure.