Merge Streams

Workshop - Merge data streams

The transformation underlines the ‘rules’ for manipulating data streams. Each data stream must have the same data stream fields / order / data type, before they can be merged.

In this workshop, you will need to add a 'description' to the data stream:

• Add constant step

Merge datastreams

1. Text File Input

Text File input

The Text File Input step is used to read data from a variety of different text-file types. The most commonly used formats include Comma Separated Values (CSV files) generated by spreadsheets and fixed width flat files.

The Text File Input step provides you with the ability to specify a list of files to read, or a list of directories with wild cards in the form of regular expressions. In addition, you can accept filenames from a previous step making filename handling more even more generic.

1. Examine both the orders.txt and description.txt.

2. Configure the Text file input steps to point to, and retrieve the data from each of the files.

2. Add Constant

Add Constant

The Add constant values step is a simple and high performance way to add constant values to the stream.

Why - in order to merge the streams each stream has to have the same layout.

1. The PRODUCTDESCRIPTION field is added to the ‘orders stream’ to ensure the data stream matches the ‘description’ data stream.

3. Select Values

Select Values

The Select Values step is useful for selecting, removing, renaming, changing data types and configuring the length and precision of the fields on the stream. These operations are organized into different categories:

• Select and Alter - Specify the exact order and name in which the fields could be placed in the output rows

• Remove - Specify the fields that could be removed from the output rows

• Meta-data - Change the name, type, length and precision (the metadata) of one or more fields

Each of the Select values ensures that each data stream is consistent in layout before merging. Each field must be in the correct order within the data stream so that mappings are successful.

Select values

4. RUN

RUN

1. Run the Transformation.

2. Click on the Dummy step and ‘Preview’.

Preview data

As you can see we have 2 merged streams ..