Key benefits

Simplifies the U-SQL database development, management and deployment experience

Allows management of database environment for local build and run

Try it now

Known limitation for public preview

Extending SSIS with big data capability using ADLA

• Edit the inline U-SQL script in ADLA Task to call table valued functions and stored procedures in your U-SQL databases.
• Use the U-SQL files stored in ADLS or Azure Blob Storage by leveraging Azure Data Lake Store File System Task and Azure Blob Download Task.
• Use the U-SQL files from local file directly using SSIS File Connection Manager.
• Use an SSIS variable that contains the U-SQL statements. You can also use SSIS expression to generate the U-SQL statements dynamically.

Using SSIS to move cloud compute results back to on-premises

Try it yourself

Step 1: Connect Azure Data Lake and Log Analytics

Step 2: Create a query that identifies when a path or file type is used

Query File Paths, Names and Types

AzureDiagnostics
| where ( ResourceProvider == "MICROSOFT.DATALAKESTORE" )
| where ( Path_s contains "/webhdfs/v1/##YOUR PATH##")

• Query for a specific path

  | where ( Path_s contains "/webhdfs/v1/MY/SAMPLE/PATH/")

• Query for a file name and/or type

  | where ( Path_s endswith "MYFILE.EXT")

• Query for both file path and file name/type.

  | where ( Path_s contains "/webhdfs/v1/MY/SAMPLE/PATH/") AND where ( Path_s endswith "MYFILE.EXT")

  or

  | where ( Path_s contains "/webhdfs/v1/MY/SAMPLE/PATH/")
  | where ( Path_s endswith"MYFILE.EXT")

Query Folder/File Creation or Deletion

• A new file gets created in a path:

   AzureDiagnostics
   | where ( ResourceProvider == "MICROSOFT.DATALAKESTORE" )
   | where ( OperationName == "create" )
   | where ( Path_s contains "/webhdfs/v1/##YOUR PATH##")

• A new folder is created in a path

   AzureDiagnostics
   | where ( ResourceProvider == "MICROSOFT.DATALAKESTORE" )
   | where ( OperationName == "mkdirs" )
   | where ( Path_s contains "/webhdfs/v1/##YOUR PATH##")

• A folder/path is deleted in a path

   AzureDiagnostics
   | where ( ResourceProvider == "MICROSOFT.DATALAKESTORE" )
   | where ( OperationName == "delete" )
   | where ( Path_s contains "/webhdfs/v1/##YOUR PATH##")

Query on specific account

| where ( Resource == "myadlsacct" ) 

Using Other Query Operations

• open
• rename
• append
• checkaccess

• setexpiry - when the file expiration is set.
• modifyaclentries - when the ACL for a file/folder is modified.

AzureDiagnostics 
| where ( ResourceProvider == "MICROSOFT.DATALAKESTORE" ) 
| where ( Resource == "myadlsacct" ) 
| where ( OperationName == "create" ) 
| where ( Path_s contains "/webhdfs/v1/##YOUR PATH##")

Step 3: Create an alert when an event is detected or a threshold is reached.

The bottom line for a best practice...

// The following @@FeaturePreviews flags are only needed while the features are in preview
// Special runtimes may be required if the feature is still in private preview
// Contact us at usql at microsoft dot com if you need access to the private preview 

SET @@FeaturePreviews = "EnableParquetUdos:on"; // Public Preview
SET @@FeaturePreviews = "DataPartitionedOutput:on"; // Private Preview (needs special runtime)

@input = 
    EXTRACT order_key long,
            part_key int,
            supp_key int,
            linenumber int,
            quantity int,
            extendedprice double,
            discount double,
            tax double,
            returnflag string,
            linestatus string,.
            ship_date DateTime,
            commit_date DateTime,
            receipt_date DateTime,
            ship_instruct string,
            ship_mode string,
            comment string
    FROM "/input/tpch/biglineitems.tbl"
    USING Extractors.Csv();

OUTPUT @input
TO "/output/parquet/biglineitems/supplier={supp_key}/data.parquet"
USING Outputters.Parquet(
         columnOptions: 
         "1~snappy,2~snappy,3~snappy,4~snappy,5~snappy,6~snappy,7~snappy,8~snappy,9~snappy,10~snappy,11~snappy,
          12~snappy,13~snappy,14~snappy,15~snappy"
      );

// As of time of publishing this blog, this script has to be run with a private runtime to enable 
// fast file sets with Parquet Extractors (otherwise this script will time out during compilation)

SET @@FeaturePreviews = "EnableParquetUdos:on"; 
SET @@FeaturePreview = "InputFileGrouping:on";

// Would be better to use a custom extractor that does not actually parse the file for this use case, 
// since we do not make use of any of the data columns and only need the number of files and their file lengths.

@data = 
    EXTRACT order_key long,
            supp_key int,
            file_size = FILE.LENGTH()
    FROM "/output/parquet/biglineitems/supplier={supp_key}/data.parquet"
    USING Extractors.Parquet();

@result =
    SELECT COUNT(DISTINCT supp_key) AS supp_count,
           MIN(file_size) AS min_file_sz,
           MAX(file_size) AS max_file_sz
    FROM @data;

OUTPUT @result
TO "/output/parquetfiles_stats.csv"
USING Outputters.Csv(outputHeader : true);

SET @@FeaturePreviews = "EnableParquetUdos:on";
SET @@FeaturePreviews = "InputFileGrouping:on";

@data = 
    EXTRACT order_key long,
            supp_key int,
            linenumber int,
            quantity int,
            extendedprice double,
            discount double,
            tax double,
            returnflag string,
            linestatus string,
            ship_date DateTime,
            commit_date DateTime,
            receipt_date DateTime,
            ship_instruct string,
            ship_mode string,
            comment string
    FROM "/output/parquet/biglineitems/supplier={supp_key}/data.parquet"
    USING Extractors.Parquet();

@result =
SELECT * FROM @data WHERE supp_key BETWEEN 200 AND 299 AND ship_mode IN("AIR", "RAIL", "SHIP");

OUTPUT @result
TO "/output/suppliers200to299.csv"
USING Outputters.Csv(outputHeader : true);

CREATE TEMPORARY VIEW LineItemView
USING org.apache.spark.sql.parquet
OPTIONS (
  path "adl://mryscentraladls.azuredatalakestore.net/output/parquet/biglineitems"
)

SELECT * FROM LineItemView WHERE supplier BETWEEN 200 AND 300 AND ship_mode IN('AIR', 'RAIL', 'SHIP');

1. the ability to generate many files from a rowset in a single statement, providing a way to dynamically partition the data for future use with Hadoop or Spark, or to provide individual files for customers. This has been our top customer ask on the ADL Feedback forum --and now it is in private preview!
2. the ability to handle many small files. We recommend that you make your files large enough for the processing to be efficient (300MB to 4GB is a good range), but often, your file formats (e.g., images) or data ingestion pipelines (e.g., EventHub archives) are not able to reach that size. Thus, we are adding the ability to group several files into a vertex to increase efficiency and lower cost of your job (we have seen 10 to 30 times improvement in some customer jobs!).

1. Use the new AU modeler to optimize your jobs cost/performance trade-off!
2. Extract from files that use the Windows code pages!
3. Augment your script with job information through @@JobInfo!
4. Light-weight, self-contained script development with in-script C# named lambdas and script-scoped U-SQL objects!

• Pending and Upcoming Deprecations and Breaking Changes
  • U-SQL jobs will introduce an upper limit for the number of table-backing files being read
  • Built-in extractors will change mapping of empty fields from zero-length string to null with quoting enabled
  • Disallow `DECLARE EXTERNAL` inside packages, procedures and functions
  • Removal of undocumented use of `CLUSTER` | `CLUSTERED BY` in `CREATE INDEX` and `CREATE TABLE`
  • Calling static members of types in a `SELECT` expression that also references a column having the same name as the type will need to fully qualify the type name or rename the column
  • Disallow U-SQL identifiers in C# delegate bodies in scripts
  • Strengthening of Read after DML check
• Breaking Changes
  • The following previously announced deprecation items are now removed
  • API changes for the cognitive extension libraries
• Major U-SQL Bug Fixes, Performance and Scale Improvements
  • Input File Set scales orders of magnitudes better
  • U-SQL Python, R and Cognitive Extensions bug fixes and performance and scale improvements
• U-SQL Preview Features
  • Input File Set uses less resources when operating on many small files (Public Preview)
  • Built-in Parquet Extractor and Outputter (Public Preview)
  • Automatic GZip compression on `OUTPUT` statement (Public Preview)
  • Built-in ORC Extractor and Outputter (Private Preview)
  • Data-driven Output Partitioning with `OUTPUT` fileset (Private Preview)
  • A limited flexible-schema feature for U-SQL table-valued function parameters is now available for private preview
• New U-SQL capabilities
  • U-SQL adds job information system variable `@@JOBINFO`
  • U-SQL adds support for computed file property columns on `EXTRACT`
  • DiagnosticStream support in .Net User-code
  • Built-in Text/Csv/Tsv Extractors and Outputters support ANSI/Windows 8-bit codepage encodings
  • U-SQL supports ANSI SQL `CASE` expression
  • U-SQL adds C# `Func`-typed variables in `DECLARE` statements (named lambdas)
  • U-SQL adds temporary, script-bound meta data objects with `DECLARE` statements
  • The `ORDER BY FETCH` clause can be used with all query expressions
  • The EXTRACT expression's schema can be specified with a table-type
  • The `EXTRACT`, `REDUCE` and `COMBINE` expressions now support a `SORTED BY` assertion
  • The `REQUIRED` clause for UDO invocations now allows `NONE`
  • The `EXTRACT` expressions now support the `REQUIRED` clause to support column pruning in user-defined extractors
  • U-SQL adds compile-time user errors and warnings
  • U-SQL Cognitive Library additions
• Azure Data Lake Tools for Visual Studio New Capabilities
  • ADL Tools for VisualStudio provides an improved Analytics Unit modeler to help improve a job's performance and cost
  • Job Submission's simple interface now makes it easier to change the allocated AUs
  • The stage tool tip is simplified and makes it easier to find the Vertex Operator View
  • Improved visualization of the job execution graph inside a vertex
  • The job stage graph and job execution graph now indicates if the stage contains user-defined operators and what language they have been authored in
  • New "Data" tab for enumerating all input and output data
  • Job View includes a link to the diagnostic folder
  • U-SQL compilation errors are now shown in the "Error List" window
  • U-SQL Project supports MSBuild
• Azure Portal Updates
  • The portal provides an improved Analytics Unit modeler to help improve a job's performance and cost

What is U-SQL?

Step 1: Read the U-SQL Tutorial

Step 2: Submit a U-SQL job through the Azure Portal

Step 3: Run some of the U-SQL samples in the Azure Portal

• Go to your ADLA account
• Click on Sample Scripts (on the left, or on the top)
• If you are prompted to install sample data, do so. It will copy a few MBs of sample data into your ADLS account
• Select any sample – for example Query a TSV file and click Submit

Step 4: Submit a U-SQL Job with Data Lake Tools for Visual Studio

Step 5: Learn how a U-SQL script executes

• JSON Sample
• Python Example
• R Sample
• UDO Sample

•  How to define and use a UDO in U-SQL
• Python in U-SQL
• R in U-SQL
• Submitting and Monitoring U-SQL Jobs in the Azure Portal
• USQL programmability guide
• usql.io gives you tutorials, samples and links to developer tools
• USQL-Reference contains reference documentation

Optimize for cost

Optimize for speed

Customize it!

How does the AU Analyzer work?

How do we calculate the recommendations?

Try this feature today!

Connect your Azure Data Lake Analytics account to Log Analytics

Create the query: Azure Data Lake Analytics AUs assigned

search *
 | where Type == "AzureDiagnostics"
 | where ResourceProvider == "MICROSOFT.DATALAKEANALYTICS"
 | where OperationName == "JobEnded

search *
 | where Type == "AzureDiagnostics"
 | where ResourceProvider == "MICROSOFT.DATALAKEANALYTICS"
 | where OperationName == "JobEnded"
 | summarize AggregatedValue = sum(Parallelism_d)  by bin(TimeGenerated, 1d)

Creating the Log Analytics alert

• Use the query from the previous step
• Set the sum of AUs to 50 as the threshold (you can use any number that reflects your own threshold)
• Set the trigger to 0: whenever the threshold is breached
• Set the period and frequency for 24 hours.

Conclusion

Connect your Azure Data Lake account to Log Analytics

Create the Log Analytics Alert

• In the Search query field,  paste the specific query that will trigger the alert. For this example, we will track when a new folder is created in a Data Lake Store account: [caption id="attachment_9935" align="alignnone" width="414"] Figure 5: Log search query[/caption]

• The alert logic can be based on a number of results such as the total number of events tracked (create a folder in Data Lake Store, submit a job in Data Lake Analytics, etc), or a specific metric value such as a sum of the events or aggregation of values from the query (total data read, total number of AUs assigned, total duration of the jobs ran in a window of time, etc). [caption id="attachment_9925" align="alignnone" width="509"] Figure 6: The two main types of alerts - based on number of results or a single metric value[/caption]

• The period and frequency indicate the rolling window of time that needs to be evaluated, and how frequently to check it, respectively.

Conclusion

Find the best AU allocation for your job

How accurate is the analyzer?

Try this feature today!

Learn more