MinIO

Access S3 type Object Store -VFS ..

Workshop - MinIO

MinIO is a high-performance, Kubernetes-native object storage system designed for cloud-native applications. Built from the ground up to be compatible with Amazon S3, MinIO offers a lightweight yet powerful alternative for organizations looking to deploy object storage in their own infrastructure.

At its core, MinIO provides distributed object storage with performance characteristics. It's capable of handling millions of operations per second and can store petabytes of data while maintaining sub-millisecond latency. This performance is achieved through a simplified architecture that eliminates complex dependencies and optimizes for modern hardware capabilities.

One of MinIO's key strengths lies in its versatility. It can be deployed virtually anywhere - from bare metal servers to public, private, and edge cloud environments. Organizations particularly value its seamless integration with Kubernetes, making it an ideal choice for containerized environments. MinIO's open-source nature also provides transparency and flexibility that many enterprises require for their data infrastructure needs.

Please ensure you have completed the following setup: MinIO

The following content has been automatically generated by an AI system and should be used for informational purposes only. We cannot guarantee the accuracy, completeness, or timeliness of the information provided.

Any actions taken based on this content are at your own risk. We recommend seeking qualified expertise or conducting further research to validate and supplement the information provided.

To create a new Transformation

Any one of these actions opens a new Transformation tab for you to begin designing your transformation.

By clicking File > New > Transformation
By using the CTRL-N hot key

Add some data

Before we connect to Pentaho Data Integration, lets add some data to a MinIO bucket.

Log into MinIO.

Username: minioadmin

Password: minioadmin

Acknowledge the license agreement.

Click the 'Create Bucket' link.
Enter: sales-data & 'Create Bucket'.

Click on the Upload button.

Upload the sales_data.csv from:

Windows

C:\Pentaho\design-tools\data-integration\samples\transformations\files

Linux

~/Pentaho/design-tools/data-integration/samples/transformations/files

Virtual File Systems

PDI allows you to establish connections to most Virtual File Systems (VFS) through VFS connections. These connections store the necessary properties to access specific file systems, eliminating the need to repeatedly enter configuration details.

Once you've added a VFS connection in PDI, you can reference it whenever you need to work with files or folders on that Virtual File System. This streamlines your workflow by allowing you to reuse connection information across multiple steps.

For instance, if you're working with Hitachi Content Platform (HCP), you can create a single VFS connection and then use it throughout all HCP transformation steps. This approach saves time and ensures consistency by removing the need to re-enter credentials or access information for each data operation.

Start Pentaho Data Integration.

Windows - PowerShell

Set-Location C:\Pentaho\design-tools\data-integration
.\spoon.bat

Linux

cd
cd ~/Pentaho/design-tools/data-integration
./spoon.sh

Create a New Transformation.
Drag and drop a Text File input step onto the canvas.
Click: 'View' Tab.
Right mouse click on VFS Connections > New.
Enter the following details:

Setting

Value

Connection Name

MinIO:sales-data

Connection Type

Amazon S3/Minio/HCP

Description

Connection to sales-data bucket

S3 Connection Type

Minio/HCP

Access Key

minioadmin

Secret Key

minioadmin

Endpoint

http://localhost:9000 [MinIO Container IP]

Signature Version

AWSS3V4SignerType

Root Folder Path

Test the connection.

The CSV file input step doesn't support VFS - it only supports local file paths. This is a design restriction in the CSV file input step for performance optimization.

Double-click on the Text file Input step.

Browse & Add the VFS - sales-data/sales_data.csv

Click on the Content tab and configure with the following settings:

Click on the Fields tab > Get Fields which will parse the file:

From here you can Preview the rows.

PreviousStorage NextSMB

Last updated 2 hours ago