Merge Streams

The Transformation underlines the ‘rules’ for manipulating data streams. Each data stream must have the same structure / layout, before they can be merged.

In this guided demonstration, you will merge datastreams based on a set of rules:

• Add constant step

Merge datastreams

1. Text File Input

The Text File Input step is used to read data from a variety of different text-file types. The most commonly used formats include Comma Separated Values (CSV files) generated by spreadsheets and fixed width flat files.

The Text File Input step provides you with the ability to specify a list of files to read, or a list of directories with wild cards in the form of regular expressions. In addition, you can accept filenames from a previous step making filename handling more even more generic.

1. Examine both the orders.txt and description.txt.

2. Configure the Text file input steps to point to, and retrieve the data from each of the files.

2. Add Constant

The Add constant values step is a simple and high performance way to add constant values to the stream.

Why - in order to merge the streams each stream has to have the same layout.

1. The PRODUCTDESCRIPTION field is added to the ‘orders stream’ to ensure the data stream matches the ‘description’ data stream.

3. Select Values

The Select Values step is useful for selecting, removing, renaming, changing data types and configuring the length and precision of the fields on the stream. These operations are organized into different categories:

• Select and Alter - Specify the exact order and name in which the fields could be placed in the output rows

• Remove - Specify the fields that could be removed from the output rows

• Meta-data - Change the name, type, length and precision (the metadata) of one or more fields

Each of the Select values ensures that each data stream is consistent in layout before merging. Each field must be in the correct order within the data stream so that mappings are successful.

Select values

4. RUN

1. Run the Transformation.

2. Click on the Dummy step and ‘Preview’.

Preview data

As you can see we have 2 merged streams ..