Siddharth Tandon's Blog

Workaround for Identity column in APS

SiddharthTandon1990 — Thu, 24 Sep 2015 23:58:00 GMT

While working on a problem which involves migrating SSIS package from SQL Server to APS, there was extensive use of Identity column in data warehouse, and as there is no support for Identity column in APS till now, I have to come up with some workaround.

Adding to this problem there is one more limitation, that while dumping data to APS using SQL Server PDW Destination Adapter you have to provide a mapping for destination column (which in normal scenario one will left it out for Identity columns), which means that for identity column you have to provide a dummy column with a dummy value.

So, I have written a stored procedure, which will update the Identity column value from dummy value to the next biggest value already present in the table + 1 (and if table does not have any entry, then it will take 1 as the starting point)

This SP expects 4 parameters

TableName: Table Name
Shema: Schema of the table
IdentityColName: Identity column name
UpdatableValue: Dummy value which needs to be updated

CREATE PROC [dbo].[uspUpdateIdentityColumnValue] @TableName [varchar](100),@Schema [varchar](100),@IdentityColName [varchar](100),@UpdatableValue [int] AS

BEGIN

/*****************************************************************************

PROCEDURE NAME: [uspUpdateIdentityColumnValue]

AUTHOR: Siddharth

CREATED: 09/22/2015

Example:

DECLARE

@IdentityColName varchar(100) = 'TableID'

,@TableName varchar(100) = ‘TableName'

,@Schema varchar(100) = 'dbo'

,@UpdatableValue int = -1

EXEC [dbo].[uspUpdateIdentityColumnValue] @TableName,@Schema,@IdentityColName,@UpdatableValue

VERSION HISTORY:

Date Change Description

Sep 22, 2015 Created

****************************************************************************/

BEGIN TRY

DECLARE @ErrorMessage NVARCHAR(4000);

DECLARE @ErrorSeverity INT;

DECLARE @ErrorState INT;

DECLARE @TempTableName varchar(100)

,@TableFullName varchar(100)

,@TempTableFullName varchar(100)

,@Sql nvarchar(4000)

,@IdentityColDataType varchar(50)

,@ColNames varchar(1000) = ''

,@Index int = 1

,@Count int

---------Create table names

SET @TempTableName = @TableName + '_Tmp_001'

SET @TempTableFullName = @Schema + '.'+@TempTableName

SET @TableFullName = @Schema + '.'+@TableName

IF object_id('tempdb.dbo.#ColNames') IS NOT NULL

DROP TABLE dbo.#ColNames

----------------Get All column names

CREATE TABLE dbo.#ColNames WITH(LOCATION = USER_DB, CLUSTERED COLUMNSTORE INDEX,DISTRIBUTION = REPLICATE) AS

SELECT ROW_NUMBER() OVER(ORDER BY COLUMN_NAME) ID, COLUMN_NAME

FROM INFORMATION_SCHEMA.COLUMNS

WHERE Table_NAME = @TableName

AND TABLE_SCHEMA = @Schema

AND COLUMN_NAME != @IdentityColName

SET @IdentityColDataType = (SELECT DATA_Type

FROM INFORMATION_SCHEMA.COLUMNS

WHERE Table_NAME = @TableName

AND TABLE_SCHEMA = @Schema

AND COLUMN_NAME = @IdentityColName)

-----------------concatenate the list of columns

SET @Count = (SELECT COUNT(*) FROM #ColNames)

WHILE(@Index<=@Count)

BEGIN

SET @ColNames = @ColNames + (SELECT COLUMN_NAME FROM #ColNames WHERE Id = @Index) + ','

SET @Index+=1

END

SET @ColNames = (SELECT SUBSTRING(@ColNames,1,LEN(@ColNames)-1))

IF OBJECT_ID(@TempTableFullName) IS NOT NULL

BEGIN

SET @Sql = N'DROP TABLE ' + @TempTableFullName

EXEC sp_executesql @Sql

END

SET @Sql = '

DECLARE @MaxID int

SET @MaxID = (SELECT ISNULL(MAX(' + @IdentityColName + '),0) FROM ' + @TableFullName +' WHERE ' + @IdentityColName + '! = ' + CAST(@UpdatableValue AS varchar(5))+ ')

CREATE TABLE ' + @TempTableFullName + '

WITH (CLUSTERED COLUMNSTORE INDEX, DISTRIBUTION = REPLICATE)

SELECT ' + @IdentityColName + ',' + @ColNames+'

FROM ' + @TableFullName + '

WHERE ' + @IdentityColName + '! = ' + CAST(@UpdatableValue AS varchar(5)) + '

UNION

SELECT ISNULL(CAST(ROW_NUMBER() OVER(ORDER BY ' + @IdentityColName + ') + @MaxID AS ' + @IdentityColDataType + '),-1) AS ''' +@IdentityColName + ''',' + @ColNames+'

FROM ' + @TableFullName + '

WHERE ' + @IdentityColName + ' = ' + CAST(@UpdatableValue AS varchar(5)) + '

--SELECT @SQl

IF LEN(@Sql) > 3999

BEGIN

SET @ErrorMessage = 'String exceeds limit'

SET @ErrorSeverity = 16

SET @ErrorState = -1

RAISERROR (@ErrorMessage,

@ErrorSeverity,

@ErrorState

);

END

ELSE

BEGIN

EXEC sp_executeSQL @Sql

SET @Sql = N'

IF OBJECT_ID(''' + @TableFullName + ''') IS NOT NULL

DROP TABLE ' + @TableFullName + '

IF OBJECT_ID(''' + @TempTableFullName + ''') IS NOT NULL

RENAME OBJECT ' + @TempTableFullName + ' TO ' + @TableName + '

EXEC sp_executesql @Sql

END

IF object_id('tempdb.dbo.#ColNames') IS NOT NULL

DROP TABLE dbo.#ColNames

END TRY

BEGIN CATCH

SET @ErrorMessage = ERROR_MESSAGE()

SET @ErrorSeverity = ERROR_SEVERITY()

SET @ErrorState = ERROR_STATE()

RAISERROR (@ErrorMessage,

@ErrorSeverity,

@ErrorState

);

IF(@@TRANCOUNT > 0)

ROLLBACK;

END CATCH

END

Limitations

In the script, I am using dynamic SQL, and there is a limitation on the maximum size of nvarchar i.e. 4000. So the script might break in case the script exceeds the limit

Truncate Table referenced by a Foreign key

SiddharthTandon1990 — Wed, 20 May 2015 10:02:00 GMT

Last night, I was truncating some tables from an alien database, and while truncating one table I got below error, which is logical because tables were related by foreign key constraint

Msg 4712, Level 16, State 1, Line 1

Cannot truncate table '' because it is being referenced by a FOREIGN KEY constraint.

That particular database contains 100 tables and on creating the Database Diagram, I found that there are more than 15 tables related to each other.

Now to truncate all the tables, I have written a recursive procedure, which will perform following operation, starting from the leaf table (table which is not referenced by any table)

1. Drop the constraint

2. Truncate the table

3. Create the constraints back

/****** Object: StoredProcedure [dbo].[uspTruncateTableWithForeignKey] Script Date: 7/17/2015 1:45:02 PM ******/

SET ANSI_NULLS ON

SET QUOTED_IDENTIFIER ON

/*****************************************************************************

PROCEDURE NAME: [uspTruncateTableWithForeignKey]

AUTHOR: Siddharth Tandon

CREATED: 04/10/2015

DESCRIPTION: This will truncate the table (sent as parameter) and all the tables related to it

PARAMETERS

@TableName: Name of the tbale to truncate

@CosntraintName: Pass NULL

@SchemaName: Schema of the table

EXEC [dbo].[uspTruncateTableWithForeignKey] 'TableName',NULL,'dbo'

****************************************************************************/

CREATE PROC [dbo].[uspTruncateTableWithForeignKey]

@TableName varchar(100)

,@ConstraintName varchaR(1000) = NULL

,@SchemaName varchar(100)

BEGIN

DECLARE

@Index int = 1

,@Count int = 0

,@TmpTableName varchar(100)

,@Sql nvarchar(1000)

,@TmpConstraintName varchar(1000)

,@TmpSchemaName varchar(100)

,@TmpColumnName varchar(100)

,@TmpPKTableName varchaR(100)

,@TmpPKSchemaName varchar(100)

,@TmpPKColumnName varchar(100)

,@BaseTableName varchar(100)

,@TmpDeleteAction varchaR(100)

,@TmpUpdateAction varchar(100)

DECLARE @Tables TABLE

(

ID int IDENTITY(1,1),

TableName varchar(100),

ConstraintName varchaR(1000),

SchemaName varchar(1000)

)

INSERT INTO @Tables (TableName,ConstraintName,SchemaName)

SELECT OBJECT_NAME(parent_object_id),name,SCHEMA_NAME(schema_id)

FROM sys.foreign_keys

WHERE OBJECT_NAME(referenced_object_id) = @TableName

AND OBJECT_SCHEMA_NAME(referenced_object_id) = @SchemaName

IF object_id('tempdb..#tmpRelationship') IS NULL

BEGIN

SET @BaseTableName = @TableName

SET @ConstraintName = NULL

CREATE TABLE #tmpRelationship

(

ID int IDENTITY(1,1)

,ConstraintName nvarchar(1000)

,FKTableName varchar(100)

,FKColumnName varchar(100)

,FKTableSchemaName varchar(100)

,PKTableNAme varchar(100)

,PKColumnName varchar(100)

,PKTableSchemaName varchaR(100)

,DeleteAction varchar(100)

,UpdateAction varchar(100)

)

END

SELECT @COUNT = COUNT(*)

FROM @Tables

IF(@Count > 0)

BEGIN

WHILE(@Index<=@Count)

BEGIN

SELECT

@TmpTableName = TableName

,@TmpConstraintName = ConstraintName

,@TmpSchemaName = SchemaName

FROM @Tables

WHERE ID = @Index

EXEC [dbo].[uspTruncateTableWithForeignKey] @TmpTableName,@TmpConstraintName,@TmpSchemaName

SET @Index += 1

END

IF(@ConstraintName IS NOT NULL)

BEGIN

INSERT INTO #tmpRelationship

(

ConstraintName

,FKTableName

,FKColumnName

,FKTableSchemaName

,PKTableNAme

,PKColumnName

,PKTableSchemaName

,DeleteAction

,UpdateAction

)

SELECT

KCU1.CONSTRAINT_NAME AS FKConstraint

,KCU1.TABLE_NAME AS FKTable

,KCU1.COLUMN_NAME AS FKColumn

,KCU1.TABLE_SCHEMA AS FKSchema

,KCU2.TABLE_NAME AS ReferencedTable

,KCU2.COLUMN_NAME AS ReferencedColumn

,KCU2.TABLE_SCHEMA AS ReferencedSchema

,RC.DELETE_RULE

,RC.UPDATE_RULE

FROM INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS AS RC

INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS KCU1

ON KCU1.CONSTRAINT_CATALOG = RC.CONSTRAINT_CATALOG

AND KCU1.CONSTRAINT_SCHEMA = RC.CONSTRAINT_SCHEMA

AND KCU1.CONSTRAINT_NAME = RC.CONSTRAINT_NAME

INNER JOIN INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS KCU2

ON KCU2.CONSTRAINT_CATALOG = RC.UNIQUE_CONSTRAINT_CATALOG

AND KCU2.CONSTRAINT_SCHEMA = RC.UNIQUE_CONSTRAINT_SCHEMA

AND KCU2.CONSTRAINT_NAME = RC.UNIQUE_CONSTRAINT_NAME

AND KCU2.ORDINAL_POSITION = KCU1.ORDINAL_POSITION

WHERE KCU1.TABLE_NAME = @TableName

------------Drop all constraints related to it

SET @Sql = N'ALTER TABLE ['+@SchemaName+'].['+@TableName+'] DROP CONSTRAINT ['+@ConstraintName+']'

EXEC sp_Executesql @Sql

END

SET @Sql = 'TRUNCATE TABLE ' + ISNULL(@SchemaName,'dbo') + '.[' + @TableName + ']'

EXEC sp_executesql @Sql

IF(@BaseTableName = @TableName)

BEGIN

SELECT @COUNT = COUNT(*)

FROM #tmpRelationship

SET @Index = 1

IF(@Count > 0)

BEGIN

WHILE(@Index<=@Count)

BEGIN

SELECT

@TmpTableName = FKTableName

,@TmpConstraintName = ConstraintName

,@TmpSchemaName = FKTableSchemaName

,@TmpColumnName = FKColumnName

,@TmpPKTableName = PKTableNAme

,@TmpPKColumnName = PKColumnName

,@TmpPKSchemaName = PKTableSchemaName

,@TmpDeleteAction = DeleteAction

,@TmpUpdateAction = UpdateAction

FROM #tmpRelationship

WHERE ID = @Index

SET @Sql = N'ALTER TABLE ['+@TmpSchemaName+'].['+@TmpTableName+']

WITH NOCHECK ADD FOREIGN KEY([' + @TmpColumnName + '])

REFERENCES ['+@TmpPKSchemaName+'].['+@TmpPKTableName+'] (['+@TmpPKColumnName+'])

ON DELETE '+ @TmpDeleteAction + '

ON UPDATE ' + @TmpUpdateAction

EXEC sp_Executesql @Sql

SET @Index += 1

END

Use Microsoft Translator APIs & SQL CLR for language translation in SQL Server

SiddharthTandon1990 — Mon, 13 Apr 2015 09:29:00 GMT

Recently I got the chance to explore the Microsoft Translator API.

As per technet gallery, the API is written in C# using the public Microsoft Translator APIs and extended within SQL Server using a CLR stored procedure and a CLR user-defined function. It can be used to translate text between any of the currently 50 supported languages.

Read here to configure and use the API

Compare Look up transformation over Truncate load in case of Incremental load

SiddharthTandon1990 — Wed, 08 Apr 2015 07:07:12 GMT

While populating data warehouse, we have to do incremental load, and to do so, there are multiple ways to do that.

One is old traditional way of truncating the data in destination and loading it with the fresh data from source, and then move it to data warehouse. But this has many consequences. But our discussion will move around the comparisons between the two ways

Second way is to use lookup and check which all rows are present in destination, and perform necessary insert and updates. This will utilize the memory comparisons and minimum amount of writes will be performed

In traditional method a lot of disk writes are preformed, whereas in case of look up very few writes will happen. This improves the performance multifold

I will create two packages one utilizing the first method, and the other utilizing the power of second method, and then we will compare the performance