RESTful API

Using weather REST API ..

Use Case - Weather API

Accurate weather information and forecasts are essential elements in critical decision-making scenarios. Both enterprises and individual consumers rely on current and future weather patterns to run their daily operations and manage risks successfully.

As such, the increased demand for accurate weather information has pushed the developer community to invest in building reliable weather apps.

So in this workshop we're going to:

• use a weather API.

• pull the data into PDI.

• display dataset in a CDE dashboard - Pentaho Server service.

CDE weather dashboard

---

Setup

The following section is for Reference only.

The CDE has been activated.

You may require to create an Openweathermap account.

Follow the steps outlined below to activate CDE and test the API call with Postman.

Postman

1. Activate CDE

To activate the Community Dashboard Editor (CDE) plugin, you will need to change the configuration of several .xml files in the Pentaho solutions folder as described below. Verify that you have the appropriate permissions to read, write, and execute commands in the specified directories in the instructions.

1. Ensure the Pentaho Server is stopped.

cd
cd /opt/pentaho/server/pentaho-server
sudo ./stop-pentaho.sh

1. Uncomment the following lines:

/opt/pentaho/server/pentaho-server/pentaho-solutions/system/pentaho-cdf-dd/plugin.xml

cd
cd /opt/pentaho/server/pentaho-server/pentaho-solutions/system/pentaho-cdf-dd
sudo nano plugin.xml

1. Locate the following two commented blocks in this file and remove the comment tags from these blocks.

<operation>
    <id>EDIT</id>
    <perspective>wcdf.edit</perspective>
</operation>

<overlays>
    <overlay id="launch" resourcebundle="content/pentaho-cdf-dd/lang/messages">
        <button id="launch_new_cde" label="${Launcher.CDE}" command="Home.openFile('${Launcher.CDE}', '${Launcher.CDE_TOOLTIP}', 'api/repos/wcdf/new');$('#btnCreateNew').popover('hide');"/>
    </overlay>
    <overlay id="startup.cde_dashboard"  resourcebundle="content/pentaho-cdf-dd/lang/messages" priority="1">
        <menubar id="newmenu">
            <menuitem id="new-cde_dashboard" label="${Launcher.CDE}" command="mantleXulHandler.openUrl('${Launcher.CDE}','${Launcher.CDE_TOOLTIP}','api/repos/wcdf/new')" />
        </menubar>
    </overlay>
</overlays>

1. Save.

CTRL + O
Enter
CTRL + X

1. Uncomment the following lines:

/opt/pentaho/server/pentaho-server/pentaho-solutions/system/pentaho-cdf-dd/settings.xml

cd
cd /opt/pentaho/server/pentaho-server/pentaho-solutions/system/pentaho-cdf-dd
sudo nano settings.xml

1. Locate the block at end of the file, 'Defining the new-toolbar-button' and remove the comments tags from this block

<new-toolbar-button>1,New CDE Dashboard,CDE Dashboard,api/repos/wcdf/new</new-toolbar-button>

1. Locate the block at end of the file, 'Defining the new-toolbar-button' and remove the comments tags from this block.

2. Save.

CTRL + O
Enter
CTRL + X

1. Save your changes.

2. Restart the Pentaho Server.

cd
cd /opt/pentaho/server/pentaho-server
sudo ./start-pentaho.sh

---

Community Dashboard Editor (CDE)

To verify CDE is activated, do the following.

1. Log on to the Pentaho User Console.
2. From the Home page, click the Create New button.

3. From the menu that displays, select the New CDE Dashboard option. You can now begin creating your first CDE dashboard.

CDE Dashboard

2. Weather API

1. Sign up for a free Openweather account.

Сurrent weather and forecast

Link to Openweather

1. Once registered, generate an API key.

Generate a key

1. Read the API documentation for deployment - Geocoding

Geocoding API

Link to Openweathermap API

http://api.openweathermap.org/geo/1.0/direct?q={city name},{state code},{country code}&limit={limit}&appid={API key}

Parameters		Description
q	required	City name, state code (only for the US) and country code divided by comma. Please use ISO 3166 country codes.
appid	required	Your unique API key (you can always find it on your account page under the "API key" tab)
limit	optional	Number of the locations in the API response (up to 5 results can be returned in the API response)

Example

http://api.openweathermap.org/geo/1.0/direct?q=London&limit=5&appid={API key}

3. Postman

Postman is an API platform for building and using APIs.

1. Sign up - for free - & download Postman.

Postman

You can also install Postman as a VS Extension or from Snap.

1. Once logged into your account you can test your API weather call and test the output of various parameters.

Test

The new 'One Call' API requires the location latitude & logitude.

This method usues a deprecated Geocoding API to resolve city, state, country - ISO 3166.

Query Params	Value
https://api.openweathermap.org/data/2.5/weather?
q=
{city name}	london
{state code}	England
{country code}	GB
appid	{API key}

Optional Params

1. Edit the query to set the mode & units.

Params	Description
mode	Possible values are xml and html. If you don't use the mode parameter format is JSON by default.
units	Units of measurement. standard, metric and imperial units are available. If you do not use the units parameter, standard units will be applied by default.
lang	You can use this parameter to get the output in your language.

https://api.openweathermap.org/data/2.5/weather?q={city name},{state code},{country code}&appid={API key}&mode=xml&units=metric

x

Resources

REST API documentation API information for building custom solutions of Pentaho products.

Welcome to the Hitachi Vantara Documentation Portal

Link to Pentaho Pro Suite API

CDF API Pentaho Data Integration and Analytics CDF API documentation.

Welcome to the Hitachi Vantara Documentation Portal

Link to Pentaho CDF API

Pentaho Data Integration and Analytics Platform JavaScript API.

Welcome to the Hitachi Vantara Documentation Portal

Link to Pentaho Javascript API

Pentaho Data Integration

The CDE is a Pentaho service.

x

x

1. Transformation

weather API

Weather API

1. Start Pentaho Pro Suite.

cd
cd /Scripts
./start_pro.sh

1. Open the following transformation in Pentaho Data Integration.

~/Workshop--Data-Integration/Labs/Module 7 - Workflows/API/tr_weather_cda.ktr

1. Browse through the following steps.

Get variables

As you already know, Kettle variables can be used in Jobs and Transformations wherever you see a dollar sign. There is one more way to use variables—getting their values as fields of your stream. For this, you have to use the Get Variables step.

With the Get Variables step, you can get the value for one or more variables. The kind of variable can be any of the Kettle variables types you just learned the variables defined in the kettle.properties file, internal variables, for example, ${user.dir}, named parameters, or other Kettle variables.

1. Double-click on the step to view settings.

Get variables

1. The variables & default values are pulled from 'Named Parameters' set in the Transformation properties.

Default Named parameters

REST Client

The REST Client transformation step enables you to consume RESTfull services.

For a simple http GET request you can also use: http client step.

1. Double-click on the step to view settings.

HTTP Client step

Based on the openweathermap API documentation:

• the URL calls the service to return the weather data based on:

${city}=London

${country}=GB

${units}=metric

${mode}=xml

• 'response' is return to the 'results' data stream field.

• the fields are 'mapped' in the fileds tab.

map parameter fields to URL

Get data from XML

This step provides the ability to read data from any type of XML file using XPath specifications.

For further details: Get data from XML

1. Double-click on the step to view settings.

Get data from XML - defined in a field

The response from the API call - results - is the data source.

1. Click on the 'Content' tab.

Determine XPath

To determine the XPath, you will need to copy and paste a snippet of the API call.

The root is: /current

1. Click on Fields tab.

Set the XPath / data Type to Nodes and attributes to retrieve the field values

Row Normaliser

The Row Normaliser step converts the columns of an input stream into rows. You can use this step to normalize repeating groups of columns.

1. Double-click on the step to view settings.

Row Normaliser

Select values

The Select values step can perform all the following actions on fields in the PDI data stream:

• Select values

• Remove values

• Rename values

• Change data types

• Configure length and precision of values

x

x

x

2. Data Access

Community Data Access (CDA) allows you to gather and combine data from several data sources into a single structure, which you can then use in dashboards. Driven by the need to unify access to the Pentaho data layer, CDA was developed to create an abstraction layer between a CTools dashboard and the physical connections to different databases.

CDA has three main objectives:

• Combine data from several sources

• Ensure security while accessing the data (avoiding, for instance, SQL injection problems)

• Ease data exports

1. View the following CDA file.

~/Workshop--Data-Integration/

<?xml version="1.0" encoding="UTF-8"?>
<CDADescriptor>
   <DataSources>
      <Connection id="weather" type="kettle.TransFromFile">
         <KtrFile>tr_weather_cda.ktr</KtrFile>
         <variables datarow-name="city"/>
         <variables datarow-name="country"/>
       </Connection>
   </DataSources>
   
   <DataAccess access="public" cache="true" cacheDuration="3600" connection="weather"
               id="current"
               type="kettle">
      <Columns/>
      <Parameters>
         <Parameter default="London" name="city" type="String"/>
         <Parameter default="GB" name="country" type="String"/>
      </Parameters>
      <Query>conditions_normalized</Query>
      <Output indexes="0,1"/>
   </DataAccess>
   
 </CDADescriptor>

x

x

3. Repository

1. Define a connection to the 'Pentaho' Repository.

Pentaho Repository

1. Create a folder: Public / weather

x

x

x

x