Install Python packages with sqlmlutils
Applies to: SQL Server 2019 (15.x) Azure SQL Managed Instance
This article describes how to use functions in the sqlmlutils package to install new Python packages to an instance of Machine Learning Services on SQL Server and on Big Data Clusters. The packages you install can be used in Python scripts running in-database using the sp_execute_external_script T-SQL statement.
This article describes how to use functions in the sqlmlutils package to install new Python packages to an instance of Azure SQL Managed Instance Machine Learning Services. The packages you install can be used in Python scripts running in-database using the sp_execute_external_script T-SQL statement.
ملاحظة
You cannot update or uninstall packages that have been preinstalled on an instance of SQL Managed Instance Machine Learning Services. To view a list of packages currently installed, see List all installed Python packages.
For more information about package location and installation paths, see Get Python package information.
ملاحظة
The sqlmlutils package described in this article is used for adding Python packages to SQL Server 2019 or later. For SQL Server 2017 and earlier, see Install packages with Python tools.
- You must have SQL Server Machine Learning Services installed with the Python language option.
Install Azure Data Studio on the client computer you use to connect to SQL Server. You can use other database management or query tools, but this article assumes Azure Data Studio.
Install the Python kernel in Azure Data Studio. You can also install and use Python from the command line, and you can use an alternative Python development environment such as Visual Studio Code with the Python Extension.
The version of Python on the client computer must match the version of Python on the server, and packages you install must be compliant with the version of Python you have. For information on which version of Python is included with each SQL Server version, see Python and R versions.
To verify the version of Python on a particular SQL Server instance, use the following T-SQL command.
EXECUTE sp_execute_external_script @language = N'Python', @script = N' import sys print(sys.version) '
The Python package library is located in the Program Files folder of your SQL Server instance and, by default, installing in this folder requires administrator permissions. For more information, see Package library location.
Package installation is specific to the SQL instance, database, and user you specify in the connection information you provide to sqlmlutils. To use the package in multiple SQL instances or databases, or for different users, you'll need to install the package for each one. The exception is that if the package is installed by a member of
dbo
, the package is public and is shared with all users. If a user installs a newer version of a public package, the public package is not affected but that user will have access to the newer version.Before adding a package, consider whether the package is a good fit for the SQL Server environment.
We recommend that you use Python in-database for tasks that benefit from tight integration with the database engine, such as machine learning, rather than tasks that simply query the database.
If you add packages that put too much computational pressure on the server, performance will suffer.
On a hardened SQL Server environment, you might want to avoid the following:
- Packages that require network access
- Packages that require elevated file system access
- Packages used for web development or other tasks that don't benefit by running inside SQL Server
The Python package tensorflow cannot be installed using sqlmlutils. For more information and a workaround, see Known issues in SQL Server Machine Learning Services.
To use sqlmlutils, you first need to install it on the client computer that you use to connect to SQL Server.
If you'll be using sqlmlutils in Azure Data Studio, you can install it using the Manage Packages feature in a Python kernel notebook.
- In a Python kernel notebook in Azure Data Studio, click Manage Packages.
- Click Add new.
- Enter "sqlmlutils" in the Search Pip packages field and click Search.
- Select the Package Version you want to install (the latest version is recommended).
- Click Install and then Close.
If you'll be using sqlmlutils from a Python command prompt or IDE, you can install sqlmlutils with a simple pip command:
pip install sqlmlutils
You can also install sqlmlutils from a zip file:
- Make sure you have pip installed. See pip installation for more information.
- Download the latest sqlmlutils zip file from https://github.com/microsoft/sqlmlutils/tree/master/R/dist to the client computer. Don't unzip the file.
- Open a Command Prompt and run the following commands to install the sqlmlutils package. Substitute the full path to the sqlmlutils zip file you downloaded - this example assumes the downloaded file is
c:\temp\sqlmlutils-1.0.0.zip
.pip install --upgrade --upgrade-strategy only-if-needed c:\temp\sqlmlutils-1.0.0.zip
Using sqlmlutils, you can add Python packages to a SQL instance. You can then use those packages in your Python code running in the SQL instance. sqlmlutils uses CREATE EXTERNAL LIBRARY to install the package and each of its dependencies.
In the following example, you'll add the text-tools package to SQL Server.
If the client computer you use to connect to SQL Server has Internet access, you can use sqlmlutils to find the text-tools package and any dependencies over the Internet, and then install the package to a SQL Server instance remotely.
On the client computer, open Python or a Python environment.
Use the following commands to install the text-tools package. Substitute your own SQL Server database connection information (if you use Windows Authentication, you don't need the
uid
andpwd
parameters).
On the client computer, open Python or a Python environment.
Use the following commands to install the text-tools package. Substitute your own SQL Server database connection information.
import sqlmlutils
connection = sqlmlutils.ConnectionInfo(server="server", database="database", uid="username", pwd="password")
sqlmlutils.SQLPackageManager(connection).install("text-tools")
If the client computer you use to connect to SQL Server doesn't have an Internet connection, you can use pip on a computer with Internet access to download the package and any dependent packages to a local folder. You then copy the folder to the client computer where you can install the package offline.
Open a Command Prompt and run the following command to create a local folder that contains the text-tools package. This example creates the folder
c:\temp\text-tools
.pip download text-tools -d c:\temp\text-tools
Copy the
text-tools
folder to the client computer. The following example assumes you copied it toc:\temp\packages\text-tools
.
Use sqlmlutils to install each package (WHL file) you find in the local folder that pip created. It doesn't matter in what order you install the packages.
In this example, text-tools has no dependencies, so there is only one file from the text-tools
folder for you to install. In contrast, a package such as scikit-plot has 11 dependencies, so you would find 12 files in the folder (the scikit-plot package and the 11 dependent packages), and you would install each of them.
Run the following Python script. Substitute the actual file path and name of the package, and your own SQL Server database connection information (if you use Windows Authentication, you don't need the uid
and pwd
parameters). Repeat the sqlmlutils.SQLPackageManager
statement for each package file in the folder.
Run the following Python script. Substitute the actual file path and name of the package, and your own SQL Server database connection information. Repeat the sqlmlutils.SQLPackageManager
statement for each package file in the folder.
import sqlmlutils
connection = sqlmlutils.ConnectionInfo(server="yourserver", database="yourdatabase", uid="username", pwd="password"))
sqlmlutils.SQLPackageManager(connection).install("text_tools-1.0.0-py3-none-any.whl")
You can now use the package in a Python script in SQL Server. For example:
EXECUTE sp_execute_external_script
@language = N'Python',
@script = N'
from text_tools.finders import find_best_string
corpus = "Lorem Ipsum text"
query = "Ipsum"
first_match = find_best_string(query, corpus)
print(first_match)
'
If you would like to remove the text-tools package, use the following Python command on the client computer, using the same connection variable you defined earlier.
sqlmlutils.SQLPackageManager(connection).uninstall("text-tools")
The sqlmlutils package contains a number of functions for managing Python packages, and for creating, managing, and running stored procedures and queries in a SQL Server. For details, see the sqlmlutils Python README file.
For information about any sqlmlutils function, use the Python help function. For example:
import sqlmlutils
help(SQLPackageManager.install)
For information about Python packages installed in SQL Server Machine Learning Services, see Get Python package information.
For information about installing R packages in SQL Server Machine Learning Services, see Install new R packages on SQL Server.