Share via

Using Custom Python Libraries with U-SQL

The U-SQL/Python extensions for Azure Data Lake Analytics ships with the standard Python libraries and includes pandas and numpy. We've been getting a lot of questions about how to use custom libraries. This is very simple!

Introducing zipimport

PEP 273  (zipimport) gave Python's import statement the ability to import modules from ZIP files. Take a moment to review the zipimport documentation  before we proceed.

Here are the the basics:

  • Ccreate a module (a .py file, etc.)
  • ZIP up the module into a .zip file
  • Add the full path of the zip file to sys.path
  • Import the module

Build and test a simple zipped package

Before you try this with U-SQL, first master the mechanics of zipimport on your own box.

Create a file called with the following contents:

 # demo module
hello_world = "Hello World! This is code from a custom module"

This module defines a single variable called hello_world.

Create a zip file called that contains the at the root .

  • In Windows you can create right-click on and select Send to compressed folder
    • This will create a file called
  • Rename to
    • NOTE: This renaming step isn't strictly mandatory when using zipimport, but will help highlight how the process will work

Create a Python file in the same folder as

 import sys
sys.path.insert(0, '')
import mymodule

Your folder should contain:


Now run the program


The output should look like this:

 Hello World! This is code from a custom module

Deploying Custom Python Modules with U-SQL

First upload the file to your ADLS store - in this case we will upload it to the root of the default ADLS account for the ADLA account we are using - so its path is "\"

Now, run this U-SQL script


// is inside the file

DECLARE @myScript = @"
import sys
sys.path.insert(0, '')
import mymodule

def usqlml_main(df):
 del df['number']
 df['hello_world'] = str(mymodule.hello_world)
 return df

@rows = 
 SELECT * FROM (VALUES (1)) AS D(number);

@rows =
 REDUCE @rows ON number
 PRODUCE hello_world string
 USING new Extension.Python.Reducer(pyScript:@myScript);

OUTPUT @rows
 TO "/demo_python_custom_module.csv"
 USING Outputters.Csv(outputHeader: true);

It will produce a simple CSV file with "Hello World! This is code from a custom module" as a row.