For this use case, the SSIS project needed to deployed to a SQL 2012 instance. In SSDT the project was configured for SQL 2012 and all seemed well.

If you viewed the dtsx file in notepad, the PackageFormatVersion reflected the change that was made to the project.

To deploy the project, the ISPAC file to deploy to a SQL 2012 SSIS Catalog. Note that the client machine had SQL 2014 client tools running on it. It will be pretty common for DBAs to have the version of SQL client tools that reflect the highest version in their environment. While we would all love to have all our SQL instances on the latest and greatest version of SQL Server, that is simply not realistic in most companies. Here you can see the package was deployed to a SQL 2012 instance

After the project was deployed, an attempt was made to run the package and it failed. The error message states the version number is not valid and shows the PackageFormatVersion is version 8, even though it was version 6 when the project was built.

You can also see the PackageFormatVersion in the internal.packages table in SSISDB on the SQL 2012 version… it is set to version 8.

Getting around this problem is pretty easy. Copy the ISPAC file to the server hosting SQL2012. You can then run the ISPAC file from an RDP session on the SQL 2012 server in order to use SQL 2012 client tools. You can see that now the package is using the PackageFormatVersion of 6. Note that since SSIS versions packages, that is why you see 2 entries for the package.

The package ran successfully!

I hope if you stumble across this error, you will find this blog post and get an easy resolution to the problem.

Lisa

[sql]
USE [PolyBaseSandbox]
GO
 
--create external data source to the hadoop sandbox
--here I used the internal IP address 
CREATE EXTERNAL DATA SOURCE [HDP24] WITH (TYPE = Hadoop, LOCATION = N'hdfs://10.0.0.4:8020')
GO
 
 
-- create file format for the sample within the hadoop sandbox
CREATE EXTERNAL FILE FORMAT [TSV] WITH (FORMAT_TYPE = DelimitedText, FORMAT_OPTIONS (FIELD_TERMINATOR = N'\t', DATE_FORMAT = N'MM/dd/yyyy', USE_TYPE_DEFAULT = False))
GO
 
 
 
--create the external table to the sample data in the sandbox
CREATE EXTERNAL TABLE [dbo].[sample_07]
(
[code] [nvarchar](255) NULL,
[description] [nvarchar](255) NULL,
[total_emp] [int] NULL,
[salary] [nvarchar](255) NULL
)
WITH (LOCATION = N'/apps/hive/warehouse/sample_07', DATA_SOURCE = HDP24, FILE_FORMAT = TSV)
 
GO
 [/sql]

[sql]
USE [PolyBaseSandbox]
GO

--create a new file format since this table is in ORC
CREATE EXTERNAL FILE FORMAT [ORC] WITH
(FORMAT_TYPE = ORC)

GO


--create table using the new file format but existing data source from previous script
CREATE EXTERNAL TABLE[dbo].[truck_mileage]
(
[truckid] nvarchar(255) NULL,
[driverid] nvarchar(255) NULL,
[rdate] nvarchar(255) NULL,
[miles] bigint  NULL,
[gas] bigint  NULL,
[mpg] float NULL
)
WITH (LOCATION = N'/apps/hive/warehouse/truck_mileage', DATA_SOURCE = HDP24, FILE_FORMAT = ORC)


select top 100 * from truck_mileage order by trucked;
[/sql]

[sql]
--Similar to lab 2.6.3 - select the data just to see it
SELECT truckid, avg(mpg) avgmpg FROM truck_mileage GROUP BY truckid;


--Create a table to persist the data
CREATE TABLE [dbo].[truck_mpg](
[truckid] [nvarchar](50) NULL,
[avgmpg] [float] NULL
) ON [PRIMARY]

GO

--Insert data into the table by querying the external table
--Now you have the aggregate data that you need persisted locally in SQL Server
--There is no need to store all the sensor data if you don't need all the details
INSERT INTO truck_mpg
SELECT truckid, avg(mpg) avgmpg FROM truck_mileage GROUP BY trucked;
[/sql]

1. Create a Resource Group: I find that it is best to create everything I need for a particular lab setup within 1 resource group in Azure. That way when I am done with it, I can just delete the entire resource group and everything in it. I named mine RG_Polybase
2. Create Windows Server 2012 R2 VM with SQL 2016 within RG_Polybase.
3. Install the Polybase feature on the SQL VM. Note the SQL install media will be in a folder on the C drive. For this version it is c:\SQLServer_13.0_Full\ . Since this is a test environment, I just used the install GUI and default options for Polybase. There is a prerequisite of installing Java Runtime . You will need to make sure and install the 64bit Java runtime. The default download on the Java download site will be a 32 bit, so you will need to make sure you select the correct one.

   

4. Configure Polybase for Hortonworks Hadoop: (hadoop connectivity reference)
   
5. Restart SQL Server (and Polybase services) for the configuration change to take effect.
6. Create HW Sandbox VM within RG_Polybase resource group. This will ensure that the 2 VMs will easily be able to connect to one another.

   This link from Hortonworks walks you right through the process: https://hortonworks.com/hadoop-tutorial/deploying-hortonworks-sandbox-on-microsoft-azure/

• ssh into the VM and run the following command as the root user (sudo): ambari-admin-password-reset

• You will be prompted to enter the password. Ambari server will restart.

Java and Polybase Services:

Files:

System Databases:

Database objects:

I know this is a super short post, but I wanted to get my slides from my 2 sessions up here pronto. There is even bonus material from my "Why is My Query Slow?" session! (by "bonus material" I mean stuff I didn't get to because I talk to much)

I packaged both PPTs and some scripts used in demos from "Why is My Query Slow" and "SQL 2014 SSIS Catalog"  presentations into 1 zip file.  Have at it! 

LisaGDogfoodSessionSlides.zip

Note: I am going to assume that you readers are at least slightly aware of project and package parameters and how they work. If not, that's ok, you will still get something out of this. You may just need to go do a little more research to better understand it.

Anyway, I have a package where I have 8 files I plan to load into 8 tables. This means that I have 8 flat file connection managers in my SSIS package. This is a quick and dirty package I am making for my internal team to use which means we will be taking these 8 files that we have generated and load them into a db on our local SQL instances to do some brief analysis. I want to parameterize the package so I can just hand over the dtsx file to my colleagues to reuse for themselves. This is not a traditional SSIS use case, but the problem is common….

I right click on the connection manager for one of the flat files and select "parameterize". I create a new parameter with the filepath and filename

This is great! Just a few clicks and I created a parameter. Now here is the problem…. I create 8 of these for all the flat file connection managers.

Problem: Now when I hand the package off to a team member, he/she will need to modify 8 parameters for their environment. Since I know that all 8 files will be in the same directory, that just seems silly.

Solution: First of all, do not right click on the connection manager and select "parameterize". Sounds silly but it will eventually be parameterized.

Create a parameter at the package or project level. For my use case, I am using package level for portability, but I highly recommend using the project deployment model and project level parameters. You can see my example here:

Now you go within each connection manager in the properties window and click on the ellipse button next to Expressions (highlighted).

You will then select "Connection String" as the property and click on the ellipse again to get into the expression builder.

For the expression, you drag and drop the InputFilePath parameter, then add the plus sign and the filename in quotes.

Then click on OK and notice you will now see the F(x) now with the connection manager icon. You see this same icon after you parameterize the connection manager via the GUI with right click "parameterize". This different method simply allows you to apply a parameter and an expression.

Now simply repeat this process for all other connection managers. The only difference in the expression will be the file name.

Whether you run the package in SSDT or configure the package in the SSIS Catalog, you just need to modify the file path in 1 place to alter all 8 of my connection managers. You can see this in SSMS:

So when you are creating SSIS packages, think about ways to minimize the number of configurations needed. This will help minimize configurations needed during deployment.

The drawback to this approach is that you are now assuming that all those files will always be in the same directory. Make sure you consider whether or not that is what you want to require within your process. Rarely is there a solution that is ideal for every use case, but depending on your needs, this can be another solution to help simplify your package parameters.

Wow! It has been a long time since I have blogged. Microsoft is firing on all cylinders lately and keeping us Premier Field Engineers very busy. I am going to try very hard to get back to some regular blogging about topics that I feel are good points for customers. The other day I had to go back and remind myself about index rebuild option changes. I have customers that run varying versions of SQL as I am sure many of you are as well. They are also upgrading to newer versions. With version changes, sometimes the general database maintenance does not always get updated. This is not always required, but it can really help improve uptime and impact of index maintenance when you look at new enhancements. When working with multiple versions we can all get confused on which version includes which feature. I came up with a little chart to help show index rebuild enhancements as it pertains to online/offline, partition level, and LOB data types.

Here is a short explanation on what you can/cannot do with the different versions:

SQL 2008R2: You cannot do online index rebuilds at the partition level and you cannot do online index rebuilds of a clustered index with LOB data types (varchar(max), text, image, etc)

SQL 2012: You cannot do online index rebuilds at the partition level.  You CAN, however, do online index rebuilds of the whole index of all data types except text, ntext, image.  This means that tables with varchar(max), nvarchar(max), and varbinary(max) columns can be rebuilt online in SQL 2012

SQL 2014: You CAN do online index rebuilds at the partition level.  You CAN do online index rebuilds of varchar(max), nvarchar(max), and varbinary(max) data types.  Just like with SQL 2012, text, ntext, image are not included.  These are legacy data types which really should not be phased out over time.

Key difference between online and offline index rebuilds

Offline: A Sch-M lock is placed at the table level throughout the rebuild.  This is a Schema Modification lock as if the schema is getting modified.  The same type of lock when you add a column, etc. This blocks any DML or DDL operation.

Online: A Sch-S lock is placed at the table level throughout the rebuild. This is a Schema Stability lock.  Any select also gets a Sch-S lock. This just makes sure that the table is not getting changed while it is doing its work.  At the end of the online index rebuild there is a quick Sch-M lock in order to publish the rebuild changes. This typically takes less than a second, but this request can get blocked as well as block others (while it is waiting).  This is where Managed Lock Priority comes into play to help tweak what you want to happen in those cases.  It can allow other processes to jump in front of it in line and not get blocked while the index rebuild is waiting for a lock.

TIP: Don't let partition level rebuild confuse you when it comes to locking:

In SQL 2012 and below, even though you may be only rebuilding 1 partition, there is a Schema Modification lock (sch-m) at the TABLE level.  This blocks other DML operations for the entire table.  The benefit of the partition level index rebuild here is that it takes less time to do only 1 partition, but the entire table is still inaccessible during the rebuild.  You may think that you are rebuilding the index of an old partition that none of your queries will be using, therefore there should be no blocking, right? This is not the case

Also keep in mind that online index rebuild options are only available in Enterprise Edition of SQL Server!

Here is a link with some good guidelines on online index rebuilds. Note that with any BOL article, you can click on the "other versions" link. That is a great way to compare and see how a particular feature differs between SQL Server versions.

https://msdn.microsoft.com/en-us/library/ms190981(v=sql.120).aspx

I know I touched upon Managed Lock Priority, but I did not explain it. That will be a follow up blog post.

Lisa Gardner

One of my fellow PFEs, Joe McTaggart  gave an awesome presentation on how to use XPerf to troubleshoot IO issues!  My presentation was showing some free performance analysis and troubleshooting tools available from Codeplex.  I have to thank another one of my fellow PFEs Tim Chapman for working on it with me! I am attaching the slides and scripts that a few folks requested.

Lisa

SQLDiagAndSQLNexus.zip