DEPLOY RESOURCE (U-SQL)

2017-11-02

Summary

The RESOURCE statement identifies the data to be used as a resource.

Syntax

Deploy_Resource_Statement :=                                                                             
      'DEPLOY' 'RESOURCE' file_path_URI { ',' file_path_URI}.

Using a Resource

Files have to be in ADLS or WASB. Files are deployed to vertex and are accessible from any custom code. The file paths of deployed resources have to be unique among all deployed resources in a script and the additional files included when referencing assemblies as they are put into a single directory during compilation and execution. Ensure that all resources have unique file names or in case of identical files use only one copy. Otherwise, you will receive the error message E_CSC_USER_SAMERESOURCEWITHDIFFERENTPATH: Script contains <N> different resources with identical file name.

Size Limits

Single resource file limit is 400MB.
Overall limit for deployes resource files is 3GB.

Examples

The examples can be executed in Visual Studio with the Azure Data Lake Tools plug-in.
The scripts can be executed locally. An Azure subscription and Azure Data Lake Analytics account is not needed when executed locally.
For simplicity, the example(s) with user-defined code make use of Code-Behind for assembly management. The main advantage of Code-Behind is that the tooling will register the assembly file and add the REFERENCE ASSEMBLY statement automatically. To use Assembly registration instead of Code-Behind, see Using Assemblies: Code-Behind vs. Assembly Registration Walkthrough.

Basic Example

// a small file containing text
DEPLOY RESOURCE "/Samples/Data/helloworld.txt";

@departments =
  SELECT * 
  FROM (VALUES
      (31, "Sales"),
      (33, "Engineering"),
      (34, "Clerical"),
      (35, "Marketing")
    ) AS D( DepID, DepName );

@departments =
    SELECT DepID, DepName, System.[IO].File.ReadAllText("helloworld.txt") AS Message
    FROM @departments;

OUTPUT @departments 
TO "/ReferenceGuide/Concepts/Resource/Test.txt"
USING Outputters.Tsv();

RESOURCE with a Processor
HelloWorldProcessor
c# code is placed in the associated Code-Behind .cs file. See usage in next section, below.

using Microsoft.Analytics.Interfaces;

namespace ReferenceGuide
{
    [SqlUserDefinedProcessor]
    public class HelloWorldProcessor : IProcessor
    {
        private string hw;

        public HelloWorldProcessor()
        {
            this.hw = System.IO.File.ReadAllText("helloworld.txt");
        }

        public override IRow Process(IRow input, IUpdatableRow output)
        {
            output.Set<int>("DepID", input.Get<int>("DepID"));
            output.Set<string>("DepName", input.Get<string>("DepName"));
            output.Set<string>("HelloWorld", hw);
            return output.AsReadOnly();
        }
    }
}

Using RESOURCE with a Processor
Using Code-Behind from previous section, above.

// a small file containing text
DEPLOY RESOURCE "/Samples/Data/helloworld.txt";

@departments =
  SELECT * 
  FROM (VALUES
      (31, "Sales"),
      (33, "Engineering"),
      (34, "Clerical"),
      (35, "Marketing")
    ) AS D( DepID, DepName );


@departments =
     PROCESS @departments
     PRODUCE DepID int,
             DepName string,
             HelloWorld string
     USING new ReferenceGuide.HelloWorldProcessor();

OUTPUT @departments 
TO "/ReferenceGuide/Concepts/Resource/Test2.txt"
USING Outputters.Tsv();

Other Examples utilizing DEPLOY RESOURCE:

Share via