Example Jupyter notebooks show how to enrich data with Open Datasets

The example Jupyter notebooks for Azure Open Datasets explain how to load open datasets, and use them to enrich demo data. The techniques include use of Apache Spark and Pandas to process data.

Important

When working in a non-Spark environment, Open Datasets allows downloads of only one month of data at a time with certain classes, to avoid MemoryError problems with large datasets.

Load NOAA Integrated Surface Database (ISD) data

Notebook	Description
Load one recent month of weather data into a Pandas dataframe	Learn how to load historical weather data into your favorite Pandas dataframe.
Load one recent month of weather data into a Spark dataframe	Learn how to load historical weather data into your favorite Spark dataframe.

Join demo data with NOAA ISD data

Notebook	Description
Join demo data with weather data - Pandas	Join a one-month demo dataset of sensor locations with weather readings in a Pandas dataframe.
Join demo data with weather data – Spark	Join a demo dataset of sensor locations with weather readings in a Spark dataframe.

Join NYC taxi data with NOAA ISD data

Notebook	Description
Taxi trip data enriched with weather data - Pandas	Load NYC green taxi data (over one month) and enrich it with weather data in a Pandas dataframe. This example overrides the method get_pandas_limit and balances data load performance with the amount of data.
Taxi trip data enriched with weather data – Spark	Load NYC green taxi data, and enrich it with weather data, in Spark dataframe.

Next steps

• Tutorial: Regression modeling with automated machine learning and an open dataset
• Python SDK for Open Datasets
• Azure Open Datasets catalog
• Create Azure Machine Learning dataset from Open Dataset