SQL Server: Service Broker Team Blog

External Activator Security

luis vargas [MSFT] — Mon, 19 Jul 2010 00:54:53 GMT

This short post deals with security and permission-related aspects of External Activator.

Selecting External Activator service account

When you install External Activator, you are asked to choose the service account (the Windows account that External Activator service will run as). The choice is from well-known Windows service accounts and a custom user with a password. For more information on the well-known accounts, please refer Sql Server Books Online or MSDN pages. The recommended service account for External Activator is a local or domain user.

Local security groups created by External Activator

The External Activator setup application will also create one or two (depending on the Operating System) local security groups for administrative purposes.

The first group, SSB EA Admin, is created regardless of the OS used. Members of this group have permission to start/stop the service, view the trace (log) file and modify the configuration file. The purpose of this group is to provide sufficient privilege separation so that the user(s) configuring and running External Activator doesn’t have to be a box admin.

The second group, SSB EA Service, is only created on down-level OSes and its purpose is to make changing External Activator’s service account easier (more on that below).

Changing External Activator’s service account

On newer OSes (starting from Vista/Server 2008) changing External Activator’s service account is as simple as setting the new account in Services MMC snap-in (Win-R –> services.msc). The reason it works is that ACLs on External Activator’s files are created based on Service SID, which is independent of service’s “real” service account (more about Per Service SID can be found here). On down-level OSes using the Services MMC snap-in is not enough, because changing the External Activator’s service account there won’t automatically change the file ACLs to the new account. This is something you need to do yourself, but in order to make it easier, the SSB EA Service local group has been introduced. The files are ACLed to be accessible by members of SSB EA Service group rather than by the External Activator’s service account directly. Therefore, when changing service account, it’s enough to add the new account to that group (and possibly remove the old one), without the need to actually touch any file ACLs. You can do that from a command line window:

net localgroup “SSB EA Service” /add “”
net localgroup “SSB EA Service” /delete “”

Local security groups created by External Activator

One of the previous posts provided a detailed description of all the Sql Server permissions that External Activator needs in order to work properly. It all starts with creating a Sql Server login for External Activator to use. There are several options how this login may be created, depending on the topology of the services:

External Activator runs on a machine in different domain from the machine hosting Sql Server. Even though External Activator doesn’t support Sql authentication when connecting to the notification service, you can still make connections across domain boundaries. The trick is to use “mirrored accounts”, which boils down to creating a local/domain windows user on the database server with the same username and password as the External Activator’s service account. The service account must be a local or domain user (you can’t use e.g. NT AUTHORITY\NETWORK SERVICE). Once you create a Sql Server login from that user and grant necessary Sql Server permissions to that login, External Activator will be able to connect, even from outside of the database server’s domain.
External Activator runs on different machine from Sql Server, but both machines are in the same domain or in trusted domains.
1. 1. If External Activator runs as NT AUTHORITY\NETWORK SERVICE or NT AUTHORITY\SYSTEM, the login may be created from machine account (e.g. MyDomain\EaMachine$ - note the dollar sign necessary for machine accounts).
  2. If External Activator runs as domain user, the login may be created from that user directly (e.g. MyDomain\EaUser).
  3. If External Activator runs as local user, you have to resort to using mirrored accounts as described above.
  4. NT AUTHORITY\LOCAL SERVICE is not supported as External Activator service account in a multi-machine deployment.
External Activator runs on the same machine as the Sql Server it’s connecting to. In this case, in addition to the above possibilities, you can also use the following:
1. 1. If External Activator runs as local user, the login may be created from the local user (e.g. MachineName\EaUser).
  2. If the OS is Vista or higher, the login may be created using the service SID (i.e. NT SERVICE\SSBExternalActivator).
  3. If the OS is older than Vista, the login may be created using the SSB EA Service local group (i.e. MyMachine\SSB EA Service). Note however that it won’t work if External Activator runs as NT AUTHORITY\SYSTEM. Local system is a special account and even if it belongs to SSB EA Service security group, that group’s token won’t be passed to Sql Server when External Activator tries to log in, hence the login will fail.
    
    The advantage of using Service SID/SSB EA Service group as a base for Sql login is that nothing needs to be done on Sql Server side when External Activator’s service account is changed.

Sample activated application

pmarciniak — Wed, 10 Mar 2010 11:50:00 GMT

In today’s post we're providing a sample application skeleton that may be used to play with External Activator, as well as serve as a base for writing custom activated applications.

A file with source code of the application is attached to this post and you can get it by clicking here.

The sample assumes a usage scenario where the activated application is used to offload some CPU-intensive computation from Sql Server. Whenever there’s some work to be done, Sql Server creates a conversation and sends a RequestMessage on it. The message contains some application-specific binary payload. The activated application receives such message, performs the computation and sends back a ResponseMessage containing the results of the computation. At this point Sql Server may decide to end the conversation or keep sending more RequestMessages on the same conversation. Upon receiving error or end conversation messages, the activated application will end its end of the conversation (and write an entry to Windows Event Log in case of error message). For the purpose of this sample, the “CPU-intensive computation” is just calculating an MD5 hash of the incoming payload.

Note that some users may prefer to insert the result directly into database table rather than sending a response message. However, this sample chooses a response message approach, since it provides better decoupling between the activated application and database schema.

The application is somewhat generic in the sense that it is passed the location of the queue it should read messages from as command line arguments. That way, it may be compiled once and used for servicing more than just single application queue. In order to make it work, you need to make sure the first 4 command line arguments External Activator passes to the app are:

The Sql Server instance to connect to.
The name of the database where the application queue is located.
The name of the schema where the application queue is located.
The name of the application queue.

Luckily, External Activator is able to generate these arguments based on the contents of the event notification message received (the Sql-generated message that triggers launching the user app). All you need to do is to configure the LaunchInfo section of the EAService.config file like this:

    c:\test\MessageProcessingApplication.exe
    %sqlserver% %database% %schema% %queue%
    c:\test

One more thing: The example app requires a specific Windows Event Log source to exist and we didn’t want to force you to play with the code using admin privileges. Therefore, prior to running the main app, the event source needs to be explicitly created. For your convenience, the code includes helpers to create and destroy the event log source. In order to create it, simply execute the app with CreateLogSource command line argument (for deleting the source, use DeleteLogSource). You need to do that as administrator (elevated).

That's it for now; next time we’ll provide a step-by-step guide how to configure External Activator so that this simple app may be tested, as well as how to troubleshoot any issues.

Program.cs

Get Started With Using External Activator

junan_msft — Mon, 18 May 2009 14:47:00 GMT

In the blog post Announcing Service Broker External Activator, we introduced Service Broker External Activator and showed what benefits a broker user can get from using it. In this article, we'll get you started with using external activator in four steps:

· How to create a notification service

· How to create an event notification to associate your user queue with the notification service

· How to modify external activator configuration file to connect to the notification service you just defined and to launch applications when messages are arriving at your user queues that are being monitored

· A few things External Activator expect you to do

To begin with, external activator must connect to a notification service before it can do anything useful. If you don't have a notification service yet, here is the script you can use to create one:

-- switch to the database where you want to define the notification service

USE my_db

-- create a queue to host the notification service

CREATE QUEUE my_notif_queue

-- create event notification service

CREATE SERVICE my_notif_svc

ON QUEUE my_notif_queue

(

[https://schemas.microsoft.com/SQL/Notifications/PostEventNotification]

)

Next, let's create an event notification object so whenever messages have arrived at the user queue you are interested in (my_user_queue), notifications will be posted to the notification service we just created above:

CREATE EVENT NOTIFICATION my_evt_notif

ON QUEUE my_user_queue

FOR QUEUE_ACTIVATION

TO SERVICE 'my_notif_svc' , 'current database'

The above example assumes my_user_queue and my_notif_svc reside in the same database. In the case of my_notif_svc is in another database, 'current database' should be replaced with the broker instance GUID where my_notif_svc is defined in.

Assume you have already downloaded the Service Broker External Activator MSI package and installed external activator to C:\Program Files\Service Broker\External Activator\. Suppose the message processing application that you want external activator to invoke is in c:\test\myMessageReceiver.exe, and your notification database server is running on my_pc01. Here is what your configuration file will look like (C:\Program Files\Service Broker\External Activator\config\EAService.config):

...

<Description>my notification serviceDescription>

<Unencrypted>server=my_pc01;database=my_db;Application Name=External Activator;Integrated Security=true;Unencrypted>

ConnectionString>

NotificationService>

NotificationServiceList>

<ServerName>my_pc01ServerName>

<DatabaseName>my_dbDatabaseName>

<SchemaName>dboSchemaName>

<QueueName>my_user_queueQueueName>

OnNotification>

<ImagePath>c:\test\myMessageReceiver.exeImagePath>

<CmdLineArgs>whatever cmd-line arguments you need to pass to your receiver applicationCmdLineArgs>

<WorkDir>c:\testWorkDir>

LaunchInfo>

ApplicationService>

ApplicationServiceList>

...

We now have specified notification service name, notification database connection string, the four-part user queue name whose activities we like to watch, and the message-receiving application we like External Activator to invoke when messages are coming in. We have also configured the min attribute of the <Concurrency/> element to 1, which means External Activator will launch a single instance of c:\test\myMessageReceiver.exe upon the first QUEUE_ACTIVATION notification message received for my_user_queue. The max attribute is set to 4, meaning as many as four instances of the same application can be launched if service broker sees my_user_queue are not being drained fast enough (e.g., messages keep accumulating). We recommend max to be set to the number of CPU cores of the machine where External Activator is deployed (my_pc01) to take full advantage of the machine power.

A couple of things External Activator expects you to do it right include:

· The windows login-account external activator service is running under needs to have the set of permissions that are listed in Security Implications section of C:\Program Files\Service Broker\External Activator\bin\\ssbea.doc in order to connect to the notification service and database to read notification messages from the notification service queue (my_notif_queue). Assuming the service account is my_domain\my_username, we have provided the SQL scripts below that can be used to set up the minimum set of permissions required by external activator. Please refer to Service Broker Identity and Access Control page for more information about what permissions are expected by service broker applications, and Service Broker Tutorials for more information about broker programming in general.

-- switch to master database

USE master

-- create a sql-login for the same named service account from windows

CREATE LOGIN [my_domain\my_username] FROM WINDOWS

-- switch to the notification database

USE my_db

-- allow CONNECT to the notification database

GRANT CONNECT TO [my_domain\my_username]

-- allow RECEIVE from the notification service queue

GRANT RECEIVE ON my_notif_queue TO [my_domain\my_username]

-- allow VIEW DEFINITION right on the notification service

GRANT VIEW DEFINITION ON SERVICE::my_notif_svc TO [my_domain\my_username]

-- allow REFRENCES right on the notification queue schema

GRANT REFERENCES ON SCHEMA::dbo TO [my_domain\my_username]

· Your application (c:\test\myMessageReceiver.exe), when launched, must issue RECEIVEs and consume messages from your user queue (my_user_queue) before it exits for service broker to post more event notifications when either your user queue is not completely drained at the time your application finishes or there are new messages coming in later after your application quits. For more details about how broker activations work, check out Understanding When Activation Occurs of SQL Server Books Online.

If you have done all the above necessities, now try to start your external activator service, and send a few messages to your user queue, you should probably be able to see the messages are read and processed. But if not, in our next EA blog article, and we'll show how you can trouble-shoot external activator to find out what have gone wrong. Stay tuned! :)

Service Broker Wait Types

priyankporwal — Mon, 01 Dec 2008 15:08:00 GMT

SQL server engine keeps track of wait operations (aka wait types) performed by all its executing threads, either to serialize access to protected structures or to wait for asynchronous events/notifications. Sys.dm_os_wait_stats DMV can be used to get the statistics for all wait types and can potentially point to performance issues and code paths with high degrees of contention.

Service Broker threads use 12 different wait types. The sections below describe these wait types in detail and their expected values depending on usage of specific Service Broker features.

[Note: Let avg_wait_time_ms = wait_time_ms/waiting_tasks_count for each wait type from the DMV]

1. BROKER_CONNECTION_RECEIVE_TASK

Each Service Broker (and Database Mirroring) connection endpoint has a list of buffers posted to receive data from the network. There are two threads working on this list, one that posts buffers for receive and one that processes them after receiving the data.

This wait type is charged whenever these threads attempt to access this list to add or remove buffers.

Waiting_tasks_count and wait_time_ms for this wait type should both be proportional to the amount of network data received by all Service Broker (and Database Mirroring) connection endpoints and avg_wait_time_ms should be a really small value.

2. BROKER_ENDPOINT_STATE_MUTEX

This wait type is charged each time there is some state change for a Service Broker (or Database Mirroring) connection endpoint during the connection establishment (i.e. handshake) phase - e.g. initialization before connect or after accept, login negotiation (authentication, encryption), validation, error, arbitration and error. This wait type is also charged per connection endpoint every time sys.dm_broker_connections (or sys.dm_db_mirroring_connections) DMV is queried to serialize access to each connections handshake state.

Avg_wait_time_ms for this wait type should be very small and the wait_time_ms and waiting_tasks_count should both be proportional to the number of times Service Broker (or Database Mirroring) establishes connection with some other SQL server instance and the number of times sys.dm_broker_connections (or sys.dm_db_mirroring_connections) DMV is queried.

Rapidly increasing values of wait_time_ms and waiting_tasks_count for this DMV could indicate very frequent connection establishment (and teardown). Since service broker transport tears down connections after ~90 seconds of inactivity, these values can increase if applications use service broker once every ~90 seconds.

3. BROKER_EVENTHANDLER

Each SQL server instance has a primary event handler thread for processing Service Broker startup/shutdown and timer events. This thread never goes away and is always either waiting for such events or processing them.

This wait type is charged each time Service Broker's primary event handler waits for instance startup/shutdown or any dialog timer events (dialog timeouts) and mirrored routes timeouts.

Wait_time_ms for this wait type should approximately be equal to the interval since instance startup. Waiting_tasks_count merely indicates the number of times the primary event handler had to wait due to absence of any events.

Neither of these two fields in the DMV indicates any performance issue in the engine. If Service Broker is not being used at all (either directly or through DBMail, Event Notification), then max_wait_time_ms and wait_time_ms would approximately be the same and waiting_tasks_count would be really small value.

4. BROKER_INIT

This wait type is charged each time Service Broker fails to initialize internal broker managers for any database. Service Broker waits for about 1 second before re-attempting to initialize broker for same database. These events should be rare.

Waiting_tasks_count for this wait type is the number of times Service Broker failed to initialize broker on any database. Wait_time_ms will be proportional to waiting_tasks_count with avg_wait_time_ms being close to 1 second.

High or increasing values of waiting_tasks_count for this wait type indicate some problem in the SQL instance.

5. BROKER_MASTERSTART

This wait type is charged only during instance startup, when Service Broker is waiting for master database to startup.

Waiting_tasks_count should be just 1 and wait_time_ms should be really small for this wait type.

6. BROKER_RECEIVE_WAITFOR

This wait type is charged once per WAITFOR RECEIVE SQL statement, where the statement execution waits for messages to arrive in the user queue.

Waiting_tasks_count must be same as the number of times such statements have been executed and wait_time_ms should be the total time their execution had to wait before messages arrived or WAITFOR timeout for each.

If avg_wait_time_ms is much higher than expected, errorlog and profiler events should be checked on both initiator and target server instances for potential problems.

7. BROKER_REGISTERALLENDPOINTS

This wait type is charged only during instance startup, when Service Broker is waiting for all endpoint types to be registered, so that it can start Broker and/or Database Mirroring endpoints.

Waiting_tasks_count should be just 1 and wait_time_ms should be really small for this wait type.

8. BROKER_SERVICE

This wait type is charged when next hop destination list associated with a target service/broker instance pair gets updated or re-prioritized due to addition or removal of a dialog to the target service/broker instance pair. Service Broker sends messages to these next hop destinations in the order of their priority and hence it needs to serialize access to destination-list and their effective priority changes.

Waiting_tasks_count and wait_time_ms for this wait type merely indicate the number of times Service Broker had to serialize access to these internal structures with avg_wait_time_ms being really small.

9. BROKER_SHUTDOWN

This wait type is charged only during instance shutdown, when Service Broker waits a few seconds for its primary event handler and all connection endpoints to shutdown.

Waiting_tasks_count and wait_time_ms for this wait type should be both 0 unless instance shutdown has already started.

10. BROKER_TASK_STOP

Service Broker has several task handlers to execute broker internal tasks related to transmission of messages, asynchronous network operations and processing of received messages.

This wait type is charged only when one of these task handlers is stopping due to absence of broker internal tasks. The task handler waits for maximum 10 seconds before getting destroyed in case it needs to be restarted to execute some task.

Waiting_tasks_count and wait_time_ms should both be small values for heavy Service Broker usage scenarios. In addition, every 5 seconds, Service Broker schedules an internal cleanup task that does not do much work when broker is not being used. But, it causes one of the task handlers to wake-up, restart, execute the task and then start waiting again. As a result, even though Service Broker is not used at all, waiting_tasks_count and wait_time_ms for this wait type keep increasing, proportional to the interval since instance startup with avg_wait_time_ms being close to 5 seconds.

11. BROKER_TO_FLUSH

For performance reasons Service Broker maintains all dialog state (TO - transmission object) in memory as well as in temporary tables on disk. Every time a TO is updated, it is scheduled to be flushed lazily to the temporary table on disk. Service Broker employs an always alive lazy flusher task to do this job.

This wait type is charged when the TO lazy flusher task is waiting for some TOs to be saved to the temporary tables. The lazy flusher sleeps for 1 second before waiting again for ~1 second for TOs to be saved.

If Service Broker is not used at all, wait_time_ms and waiting_tasks_count for this wait type should be proportional to the duration since instance startup, with avg_wait_time_ms being close to ~1 second. When Service Broker is used heavily these columns should have lowe values since the lazy flusher will be busy as well.

12. BROKER_TRANSMITTER

Service Broker has a component known as the Transmitter which schedules messages from multiple dialogs to be sent across the wire over one or more connection endpoints. The transmitter has 2 dedicated threads for this purpose.

This wait type is charged when these transmitter threads are waiting for dialog messages to be sent using the transport connections.

High values of waiting_tasks_count for this wait type point to intermittent work for these transmitter threads and are not indications of any performance problem. If service broker is not used at all, waiting_tasks_count should be 2 (for the 2 transmitter threads) and wait_time_ms should be twice the duration since instance startup.

Example: Broker wait types statistics after 1 hour (3,600,000 ms) of idle system

Service Broker Wait Type	waiting_tasks_count	wait_time_ms
BROKER_CONNECTION_RECEIVE_TASK	0	0
BROKER_ENDPOINT_STATE_MUTEX	0	0
BROKER_EVENTHANDLER	3	81*
BROKER_INIT	0	0
BROKER_MASTERSTART	0	0
BROKER_RECEIVE_WAITFOR	0	0
BROKER_REGISTERALLENDPOINTS	0	0
BROKER_SERVICE	0	0
BROKER_SHUTDOWN	0	0
BROKER_TASK_STOP	724	3634180
BROKER_TO_FLUSH	1762	1804005
BROKER_TRANSMITTER	2	0*

* Service Broker's primary event handler and the transmitter threads are still waiting for some dialog activity to wake them up. Since wait_time_ms gets updated only after the wait is over, we see 0/low values for these wait types.

Announcing Service Broker External Activator

junan_msft — Fri, 21 Nov 2008 13:07:00 GMT

The Microsoft SQL Server Service Broker External Activator (EA) is distributed in the Microsoft SQL Server 2008 Feature Pack. It is an extension of the internal activation feature and lets you move the logic for receiving and processing service broker messages from the database engine service to an application executable that runs outside the database engine service. This can provide a higher level of scale-out performance by moving processing loads from the database server to another computer. The activation application process can also run under a different windows account from the database engine process. This gives administrators additional control over the resources that the activation application can access. Installation packages for x86, x64, and ia64 architectures can be downloaded from here.

Specifically, here are a list of benefits a user can get from using EA:

· Access non-database resources such as files, network connections in their message processing applications

· Offload computation-intensive work out of SQL box and deploy it to a different machine

· Leverage legacy code libraries that are not accessible within T-SQL stored procedures

· Message processing solution that adapts and scales well by specifying the MIN and MAX number of instances of application processes to run

After installed, EA runs as a NT service. It watches the user configured notification queue for QUEUE_ACTIVATION events. As soon as a notification message is seen, it responds by launching an activation application specified in the configuration file to process messages received in the notification-related user queue. The notification queue, the user queue and the event notification associating them must be predefined before EA can run. A configuration file will tell EA which notification queue to watch on and what application to launch when a QUEUE_ACTIVATION event is received. In addition, a user can configure EA in a way that EA can launch enough number of application instances to consume messages from the user queue in a timely manner without a wasteful use of system resources.

In v1 release, EA only supports:

· Listening on a single notification service for queue activation events

· Only integrated security is allowed in notification database connection string. SQL login and password are not supported just yet.

· All user applications are launched under the same user credential as what EA is running under

Please check out more details on how to setup and start using EA by reading the installed user doc bin\\SSBEA.doc in your install folder (for example, the English culture code is EN).

We'll blog more about EA on topics such as configuration and deployment, tracing and trouble-shooting, some corner cases in our next several posts. Stay tuned! :-)

Securing a dialog with certificates

portegys — Tue, 26 Aug 2008 15:37:00 GMT

This sample shows how to set up a secure dialog using certificates. Service broker will always have a level of security at the transport level, which may include encryption, but this is at a server level of granularity. It does not secure conversations on a database-to-database basis. If this is required, then dialog security can be used. Dialog security is also end-to-end as opposed to the point-to-point connection-based security provided by transport security. Since conversations may entail multiple hops through the use of forwarding, dialog security can provide authentication and one-time encryption at the terminating services. Certificate-based authentication also allows users to specify a window of time in which authentication will be honored.

The initiator and target certificates must be exchanged in order for them to authenticate each other. This "out of band" exchange should be done with a high level of trust, since a certificate bearer will be able to begin dialogs and send messages to service broker services in the authenticating server.

Running the sample

This sample requires two server instances on different machines to avoid a port collision. It is essential that the servers are configured to enable communication protocols. In this example, we will be using TCP, so use the SQL Server Configuration Manager to make sure TCP is enabled on both servers. To keep things simple, Windows authentication is used for transport security. The transport security sample shows how to use certificates for this if needed.
Run the scripts, in order:

Initiator endpoint setup. Target endpoint setup. Initiator service setup. Target service setup. Initiator certification of target. Target certification of initiator. Initiator message send. Target message receive. Initiator cleanup. Target cleanup.

Scripts

--------------------------------------------------------------------

-- Script for dialog security sample.

-- This file is part of the Microsoft SQL Server Code Samples.

-- This source code is intended only as a supplement to Microsoft

-- Development Tools and/or on-line documentation. See these other

-- materials for detailed information regarding Microsoft code samples.

-- THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF

-- ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

-- THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

-- PARTICULAR PURPOSE.

--------------------------------------------------------------------

-- Set up an initiator service broker endpoint for dialog

-- certificate-based security.

-- Modify domain_name and target_host in script to suit configuration.

USE master;

-- Create the broker endpoint using Windows authentication.

IF EXISTS (SELECT * FROM sys.endpoints WHERE name = 'service_broker_endpoint')

DROP ENDPOINT service_broker_endpoint;

CREATE ENDPOINT service_broker_endpoint

STATE = STARTED

AS TCP (LISTENER_PORT = 4022)

FOR SERVICE_BROKER (AUTHENTICATION = Windows);

-- Create a login for the target machine (target_host) in the shared domain

-- (domain_name). This assumes the availability of Kerberos authentication.

-- Note: the '$' is significant.

CREATE LOGIN [domain_name\target_host$] FROM Windows;

-- Grant the target connection access to the endpoint.

GRANT CONNECT ON ENDPOINT::service_broker_endpoint TO [domain_name\target_host$];

--------------------------------------------------------------------

-- Script for dialog security sample.

-- This file is part of the Microsoft SQL Server Code Samples.

-- This source code is intended only as a supplement to Microsoft

-- Development Tools and/or on-line documentation. See these other

-- materials for detailed information regarding Microsoft code samples.

-- THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF

-- ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

-- THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

-- PARTICULAR PURPOSE.

--------------------------------------------------------------------

-- Set up a target service broker endpoint for dialog

-- certificate-based security.

-- Modify domain_name and initiator_host in script to suit configuration.

USE master;

-- Create the broker endpoint using Windows authentication.

IF EXISTS (SELECT * FROM sys.endpoints WHERE name = 'service_broker_endpoint')

DROP ENDPOINT service_broker_endpoint;

-- Use Windows for authentication.

CREATE ENDPOINT service_broker_endpoint

STATE = STARTED

AS TCP (LISTENER_PORT = 4022)

FOR SERVICE_BROKER (AUTHENTICATION = Windows);

-- Create a login for the initiator machine (initiator_host) in the shared domain

-- (domain_name). This assumes the availability of Kerberos authentication.

-- Note: the '$' is significant.

CREATE LOGIN [domain_name\initiator_host$] FROM Windows;

-- Grant the initiator connection access to the endpoint.

GRANT CONNECT ON ENDPOINT::service_broker_endpoint TO [domain_name\initiator_host$];

--------------------------------------------------------------------

-- Script for dialog security sample.

-- This file is part of the Microsoft SQL Server Code Samples.

-- This source code is intended only as a supplement to Microsoft

-- Development Tools and/or on-line documentation. See these other

-- materials for detailed information regarding Microsoft code samples.

-- THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF

-- ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

-- THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

-- PARTICULAR PURPOSE.

--------------------------------------------------------------------

-- The initiator creates a database, queue, service, target route,

-- and certificate for the dialog to the target service.

-- Modify target_host and location of stored certificate in script

-- to suit configuration.

USE master;

-- Create initiator database.

IF EXISTS (SELECT * FROM sys.databases WHERE name = 'initiator_database')

DROP DATABASE initiator_database;

CREATE DATABASE initiator_database;

USE initiator_database;

-- Create a message queue.

CREATE QUEUE initiator_queue;

-- Create a service with a default contract.

CREATE SERVICE initiator_service ON QUEUE initiator_queue ([DEFAULT]);

-- Create a route to the target service.

CREATE ROUTE target_route

WITH SERVICE_NAME = 'target_service',

ADDRESS = 'tcp://target_host:4022';

-- Create a user who is authorized for the initiator service.

IF NOT EXISTS (SELECT * FROM sys.sysusers WHERE name = 'initiator_user')

CREATE USER initiator_user WITHOUT LOGIN;

ALTER AUTHORIZATION ON SERVICE::initiator_service TO initiator_user;

-- A master key is required to use certificates.

BEGIN TRANSACTION;

IF NOT EXISTS (SELECT * FROM sys.symmetric_keys WHERE name = '##MS_DatabaseMasterKey##')

CREATE MASTER KEY ENCRYPTION BY PASSWORD='Password#123'

COMMIT;

-- Create a certificate and associate it with the user.

IF EXISTS (SELECT * FROM sys.certificates WHERE name = 'initiator_dialog_cert')

DROP CERTIFICATE initiator_dialog_cert;

CREATE CERTIFICATE initiator_dialog_cert

AUTHORIZATION initiator_user

WITH SUBJECT = 'Dialog certificate for initiator';

-- Backup to a file to allow the certificate to be given to the target.

BACKUP CERTIFICATE initiator_dialog_cert

TO FILE = 'c:\initiator_dialog.cert';

--------------------------------------------------------------------

-- Script for dialog security sample.

-- This file is part of the Microsoft SQL Server Code Samples.

-- This source code is intended only as a supplement to Microsoft

-- Development Tools and/or on-line documentation. See these other

-- materials for detailed information regarding Microsoft code samples.

-- THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF

-- ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

-- THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

-- PARTICULAR PURPOSE.

--------------------------------------------------------------------

-- The target creates a database, queue, service, initiator route,

-- and certificate for the dialog to the initiator service.

-- Modify initiator_host and location of stored certificate in script

-- to suit configuration.

USE master;

-- Create target database.

IF EXISTS (SELECT * FROM sys.databases WHERE name = 'target_database')

DROP DATABASE target_database;

CREATE DATABASE target_database;

USE target_database;

-- Create a message queue.

CREATE QUEUE target_queue;

-- Create a service with a default contract.

CREATE SERVICE target_service ON QUEUE target_queue ([DEFAULT]);

-- Create a route to the initiator service.

CREATE ROUTE initiator_route

WITH SERVICE_NAME = 'initiator_service',

ADDRESS = 'tcp://initiator_host:4022';

-- Create a user who is authorized for the target service.

IF NOT EXISTS (SELECT * FROM sys.sysusers WHERE name = 'target_user')

CREATE USER target_user WITHOUT LOGIN;

ALTER AUTHORIZATION ON SERVICE::target_service TO target_user;

-- A master key is required to use certificates.

BEGIN TRANSACTION;

IF NOT EXISTS (SELECT * FROM sys.symmetric_keys WHERE name = '##MS_DatabaseMasterKey##')

CREATE MASTER KEY ENCRYPTION BY PASSWORD='Password#123'

COMMIT;

-- Create a certificate and associate it with the user.

IF EXISTS (SELECT * FROM sys.certificates WHERE name = 'target_dialog_cert')

DROP CERTIFICATE target_dialog_cert;

CREATE CERTIFICATE target_dialog_cert

AUTHORIZATION target_user

WITH SUBJECT = 'Dialog certificate for target';

-- Backup to a file to allow the certificate to be given to the initiator.

BACKUP CERTIFICATE target_dialog_cert

TO FILE = 'c:\target_dialog.cert';

----------EXCHANGE CERTIFICATES BEFORE PROCEEDING---------------

-- The initiator and target certificates must be exchanged in order for them to

-- authenticate each other. In a production system, this "out of band" exchange

-- should be done with a high level of trust, since a certificate bearer will be

-- able to begin dialogs and send messages to the secured service.However, assuming

-- the sample is being used on a development system, the exchange may be simple

-- remote copies.

--------------------------------------------------------------------

-- Script for dialog security sample.

-- This file is part of the Microsoft SQL Server Code Samples.

-- This source code is intended only as a supplement to Microsoft

-- Development Tools and/or on-line documentation. See these other

-- materials for detailed information regarding Microsoft code samples.

-- THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF

-- ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

-- THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

-- PARTICULAR PURPOSE.

--------------------------------------------------------------------

-- The initiator creates a target user certified by the target certificate,

-- binds it to the target service, and grants it send access to the initiator

-- service.

-- Modify location of stored certificate in script to suit configuration.

USE initiator_database;

-- Create a user for the target.

IF NOT EXISTS (SELECT * FROM sys.sysusers WHERE name = 'target_user')

CREATE USER target_user WITHOUT LOGIN;

IF EXISTS (SELECT * FROM sys.certificates WHERE name = 'target_dialog_cert')

DROP CERTIFICATE target_dialog_cert;

-- Associate the target user with the target certificate.

CREATE CERTIFICATE target_dialog_cert

AUTHORIZATION target_user

FROM FILE = 'c:\target_dialog.cert';

-- Bind the target service to the target user.

IF EXISTS (SELECT * FROM sys.remote_service_bindings WHERE name = 'remote_target_service_binding')

DROP REMOTE SERVICE BINDING remote_target_service_binding;

CREATE REMOTE SERVICE BINDING remote_target_service_binding

TO SERVICE 'target_service'

WITH USER = target_user;

-- Allow the target to send to the initiator service.

GRANT SEND ON SERVICE::initiator_service TO target_user;

--------------------------------------------------------------------

-- Script for dialog security sample.

-- This file is part of the Microsoft SQL Server Code Samples.

-- This source code is intended only as a supplement to Microsoft

-- Development Tools and/or on-line documentation. See these other

-- materials for detailed information regarding Microsoft code samples.

-- THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF

-- ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

-- THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

-- PARTICULAR PURPOSE.

--------------------------------------------------------------------

-- The target creates an initiator user certified by the initiator certificate,

-- binds it to the initiator service, and grants it send access to the target

-- service.

-- Modify location of stored certificate in script to suit configuration.

USE target_database;

-- Create a user for the initiator.

IF NOT EXISTS (SELECT * FROM sys.sysusers WHERE name = 'initiator_user')

CREATE USER initiator_user WITHOUT LOGIN;

-- Associate the initiator user with the initiator certificate.

IF EXISTS (SELECT * FROM sys.certificates WHERE name = 'initiator_dialog_cert')

DROP CERTIFICATE initiator_dialog_cert;

CREATE CERTIFICATE initiator_dialog_cert

AUTHORIZATION initiator_user

FROM FILE = 'c:\initiator_dialog.cert';

-- Bind the initiator service to the initiator user.

IF EXISTS (SELECT * FROM sys.remote_service_bindings WHERE name = 'remote_initiator_service_binding')

DROP REMOTE SERVICE BINDING remote_initiator_service_binding;

CREATE REMOTE SERVICE BINDING remote_initiator_service_binding

TO SERVICE 'initiator_service'

WITH USER = initiator_user;

-- Allow the initiator to send to the target service.

GRANT SEND ON SERVICE::target_service TO initiator_user;

--------------------------------------------------------------------

-- Script for dialog security sample.

-- This file is part of the Microsoft SQL Server Code Samples.

-- This source code is intended only as a supplement to Microsoft

-- Development Tools and/or on-line documentation. See these other

-- materials for detailed information regarding Microsoft code samples.

-- THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF

-- ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

-- THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

-- PARTICULAR PURPOSE.

--------------------------------------------------------------------

-- The initiator creates a dialog and sends a message to the target service.

USE initiator_database;

-- Create a message.

DECLARE @message varchar(max);

SELECT @message = 'Hello from initiator';

-- Create an encrypted dialog to the target service.

DECLARE @handle uniqueidentifier;

BEGIN DIALOG CONVERSATION @handle

FROM SERVICE initiator_service

TO SERVICE 'target_service'

WITH ENCRYPTION = ON;

-- Send the message.

SEND ON CONVERSATION @handle (@message);

PRINT 'Message sent to target_service on conversation:';

PRINT @handle;

--------------------------------------------------------------------

-- Script for dialog security sample.

-- This file is part of the Microsoft SQL Server Code Samples.

-- This source code is intended only as a supplement to Microsoft

-- Development Tools and/or on-line documentation. See these other

-- materials for detailed information regarding Microsoft code samples.

-- THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF

-- ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

-- THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

-- PARTICULAR PURPOSE.

--------------------------------------------------------------------

-- The target waits for and receives a message from the initiator.

USE target_database;

DECLARE @message_body varchar(max);

-- Wait for the message.

WAITFOR(

RECEIVE TOP(1)

@message_body=message_body

FROM target_queue

);

PRINT 'Target received message: ' + @message_body;

--------------------------------------------------------------------

-- Script for dialog security sample.

-- This file is part of the Microsoft SQL Server Code Samples.

-- This source code is intended only as a supplement to Microsoft

-- Development Tools and/or on-line documentation. See these other

-- materials for detailed information regarding Microsoft code samples.

-- THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF

-- ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

-- THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

-- PARTICULAR PURPOSE.

--------------------------------------------------------------------

-- Clean up the initiator.

-- Modify domain_name and target_host in script to suit configuration.

-- The recommended method is for the initiator to inform the target
-- with an "end of stream" message which causes the target to end the
-- conversation.

USE master;

IF EXISTS (SELECT * FROM sys.databases WHERE name = 'initiator_database')

DROP DATABASE initiator_database;

IF EXISTS (SELECT * FROM sys.endpoints WHERE name = 'service_broker_endpoint')

DROP ENDPOINT service_broker_endpoint;

IF EXISTS (SELECT * FROM sys.syslogins WHERE name = 'domain_name\target_host$')

DROP LOGIN [domain_name\target_host$];

--------------------------------------------------------------------

-- Script for dialog security sample.

-- This file is part of the Microsoft SQL Server Code Samples.

-- This source code is intended only as a supplement to Microsoft

-- Development Tools and/or on-line documentation. See these other

-- materials for detailed information regarding Microsoft code samples.

-- THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF

-- ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

-- THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

-- PARTICULAR PURPOSE.

--------------------------------------------------------------------

-- Clean up the target.

-- Modify domain_name and initiator_host in script to suit configuration.

-- For simplicity, the dialog is terminated by dropping the database.
-- The recommended method is for the initiator to inform the target
-- with an "end of stream" message which causes the target to end the
-- conversation.

USE master;

IF EXISTS (SELECT * FROM sys.databases WHERE name = 'target_database')

DROP DATABASE target_database;

IF EXISTS (SELECT * FROM sys.endpoints WHERE name = 'service_broker_endpoint')

DROP ENDPOINT service_broker_endpoint;

IF EXISTS (SELECT * FROM sys.syslogins WHERE name = 'domain_name\initiator_host$')

DROP LOGIN [domain_name\initiator_host$];

Fast data push tuning

portegys — Tue, 12 Aug 2008 15:30:00 GMT

A common use of service broker is the "data push" scenario in which messages are asynchronously sent to a destination such as a data warehouse for storage and processing with minimal impact on the source application. Two frequent concerns are whether service broker can handle a proposed work load, and how to "tune" a broker application so that it can achieve the required performance. Since various applications impose different constraints and are hosted within differing computing and networking configurations, there is no "one size fits all" answer.

This sample offers a means of estimating the performance of a broker application and tuning it to suit a given load and configuration. The user does this by setting several application parameters, such as message volume, message size, and processing time, as well as several internal parameters, such as number of initiator transactions, number of dialogs, etc. On the initiator, the output is the time to send the specified number of messages and the time for the messages to be transmitted to the target. On the target, the output is the time to receive and process the messages, which allows the user to obtain an estimate of whether the initiator is overrunning the target, something to avoid for a sustained high volume message load.

The sample also implements a number of recommended practices for using service broker, and can serve as an example of how to build the service broker part of a data push application. Of particular significance, it is recommended that batch messaging be done where possible. On the initiator side, this refers to sending messages on a set of "reusable" dialogs to avoid the overhead of creating a dialog per message. The dialog pool sample shows how to do this. On the target, batching refers to receiving a set of messages at a time, which can significantly improve performance.

As an illustration of the effect of reusing dialogs, data pushes of 10000 messages were performed from an initiator instance to a target. Each message was 1000 bytes. Two extreme cases were evaluated and compared. In the first case, a dialog was created for each message. In the second, all messages were sent under the same dialog. Performance was measured in terms of the time for the application to send all messages, the time for the messages to be transmitted to the target, and the time for the target to process the messages. The results show that the single dialog case is approximately ten times faster for all of these metrics.

As mentioned for the above, these are "end of the continuum" cases. There are some trade-off considerations for using more or fewer dialogs. More dialogs can ease errror handling in the case of a dialog failure, for example, since fewer messages need recovery action. "Recycling" dialogs by periodically replacing dialogs in a shared pool is another way of minimizing the impact of dialog failures. This sample and the dialog pool sample show how recycling may be implemented. One of the major reasons to use more dialogs, however, it to achieve concurrent processing on the target. This is due to the fact that a dialog is associated with a conversation group lock which allows only a single receiving procedure on the target. Thus if all messages are in the same dialog, it will serialize message reception on the target. As an illustration of how more dialogs allow more target concurrency, again using the 10000 message data push, a processing time of 10ms per message is also imposed on the target. Using ten dialogs instead of one results in halving the total processing time on the target, with no significant impact on the sending times.

The previous illustrations underscore the purpose of this sample: to allow a user to tune an application to achieve performance goals given a particular system and networking configuration.

Parameters

----------------------------------------------------
-- The data push parameters.
--
-- Application parameters:
-- message_quantity: number of messages sent.
-- message_size: size of message in bytes.
-- message_processing_time: time for target to process a message.
--    Format: 'hh:mm:ss:xxx' hh=hours, mm=minutes, ss=seconds, xxx=milliseconds
--
-- Internal parameters:
-- number_initiator_transactions: number of initiator transactions used.
--    Notes: 1. Fewer is more efficient since each transaction entails an overhead.
--           2. Messages are actually sent when transaction commits, so sending a large
--              number of messages in a transaction can result in increased latency.
-- initiator_transaction_delay: delay time between initiator transactions.
--    Format: 'hh:mm:ss:xxx' hh=hours, mm=minutes, ss=seconds, xxx=milliseconds
--    Notes: 1. A transaction can be thought of as a burst of message_quantity /
--              number_initiator_transactions messages. This delay specifies a time
--              to wait before the next transaction is run.
--           2. This parameter can be used to simulate message traffic distributed
--              over time.
-- number_dialogs: number of dialogs used to send messages.
--    Notes: 1. Message ordering only guaranteed with a dialog.
--           2. Multiple dialogs allows concurrent processing on target.
--           3. Dialog creation is expensive; dialog reuse is employed here.
-- dialog_recycle_max_messages: maximum number messages sent on a dialog before
--    recycling the dialog. Recycling is defined as ending the old dialog and
--    beginning a new one. A value of -1 indicates no recycling.
--    Notes: 1. Larger is more efficient since is minimizes the overhead of
--              creating dialogs.
--           2. Larger can complicate dialog error processing.
-- number_target_procedures: number of activated target procedures to receive messages.
--    Notes: 1. A target proc locks all messages in a dialog when it receives first message
--              for a dialog, blocking other procs from processing these messages.
--           2. Thus more dialogs yields increased concurrent processing. However, unless
--              dialog recycling is used, this should be set to number_dialogs, which
--              can utilize a target proc for each dialog.
-- max_messages_per_receive: maximum number of messages per target receive call.
--    Notes: 1. Larger is more efficient, but can complicate transaction error processing.
--           2. The maximum value can be set to message_quantity / number_dialogs.
--
-- Example:
--
-- I want to send 100000 messages in sets of 10000 with a delay of 10 seconds between
-- each set. This calls for 10 transactions. Each message is 100 bytes and the target
-- message processing time is 10 ms. The messages are independent of each other, so use
-- 5 dialogs and target procedures to get some concurrent processing on the target. Allow
-- each target proc to receive 2000 messages at a time. Do not recycle dialogs.
--
-- INSERT INTO data_push_parameters
--       VALUES
--       (
--       100000,
--       10000,
--       '00:00:00:010',
--       10,
--       '00:00:10:000',
--       5,
--       -1,
--       5,
--       2000
--       );
--
--

Running the Sample

This sample is normally run between two server instances on different machines using Windows transport security. However, it can easily be configured to perform a "loop around" data push in the same database by skipping the indicated sections of the initiator and target setup scripts. For the two server case, it is essential that the servers are configured to enable communication protocols. In this example, we will be using TCP, so use the SQL Server Configuration Manager to make sure TCP is enabled on both servers.
Edit the Common setup script and set the desired parameters. Make sure the edits are performed identically on both servers.
Run the scripts, in order:

Common setup (both servers). Initiator setup. Target setup. Initiator send. The message sending start and end times are printed. Target monitor. The message processing time is printed. Cleanup. (both servers).

Scripts

--------------------------------------------------------------------

-- Script for fast data push sample.

-- This file is part of the Microsoft SQL Server Code Samples.

-- This source code is intended only as a supplement to Microsoft

-- Development Tools and/or on-line documentation. See these other

-- materials for detailed information regarding Microsoft code samples.

-- THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF

-- ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

-- THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

-- PARTICULAR PURPOSE.

--------------------------------------------------------------------

----------------------------------------------------

-- Common setup for fast data push.

-- Before running, replace the configuration-dependent

-- domain_name and partner_host names.

----------------------------------------------------

USE master;

---------------------------------------------------------------------

-- Create the broker endpoint using Windows authentication.

-- On a secure network, encryption may be disabled to improve speed:

-- (AUTHENTICATION = Windows, ENCRYPTION = DISABLED)

-- This step can be skipped if services are in the same database instance.

---------------------------------------------------------------------

IF EXISTS (SELECT * FROM sys.endpoints WHERE name = 'service_broker_endpoint')

DROP ENDPOINT service_broker_endpoint;

CREATE ENDPOINT service_broker_endpoint

STATE = STARTED

AS TCP (LISTENER_PORT = 4022)

FOR SERVICE_BROKER (AUTHENTICATION = Windows);

-- A procedure to create a Windows login and grant it endpoint connection permission.

IF EXISTS (SELECT name FROM tempdb.sys.procedures WHERE name LIKE '#usp_windows_login_for_broker_endpoint%')

DROP PROCEDURE #usp_windows_login_for_broker_endpoint;

CREATE PROCEDURE #usp_windows_login_for_broker_endpoint (

@domain_name VARCHAR(100),

@login_name VARCHAR(50),

@endpoint_name VARCHAR(50))

BEGIN

SET NOCOUNT ON;

DECLARE @query VARCHAR(1000);

-- Create the login.

SET @query =

'IF EXISTS (SELECT * FROM sys.syslogins WHERE name = ''' + @domain_name + '\' + @login_name + ''')

DROP LOGIN [' + @domain_name + '\' + @login_name + ']';

EXEC (@query);

SET @query = 'CREATE LOGIN [' + @domain_name + '\' + @login_name + '] FROM Windows';

EXEC (@query);

-- Grant the login connection access to the endpoint.

SET @query = 'GRANT CONNECT ON ENDPOINT::' + @endpoint_name + ' TO [' + @domain_name + '\' + @login_name + ']';

EXEC (@query);

END;

-- Create a login for the partner machine (partner_host) in the

-- shared domain (domain_name) and grant it endpoint connection permission.

-- This assumes the availability of Kerberos authentication.

-- Note: the '$' is significant.

EXEC #usp_windows_login_for_broker_endpoint 'domain_name', 'partner_host$', 'service_broker_endpoint';

---------------------------------------------------------------------

-- Create the data push database.

---------------------------------------------------------------------

IF EXISTS (SELECT * FROM sys.databases WHERE name = 'data_push_database')

DROP DATABASE data_push_database;

CREATE DATABASE data_push_database;

USE data_push_database;

-- Create messages and contract.

CREATE MESSAGE TYPE data_push_message VALIDATION = NONE;

CREATE MESSAGE TYPE end_of_stream;

CREATE CONTRACT data_push_contract

(

data_push_message SENT BY INITIATOR,

end_of_stream SENT BY INITIATOR

);

----------------------------------------------------

-- The data push parameters.

-- Application parameters:

-- message_quantity: number of messages sent.

-- message_size: size of message in bytes.

-- message_processing_time: time for target to process a message.

-- Format: 'hh:mm:ss:xxx' hh=hours, mm=minutes, ss=seconds, xxx=milliseconds

-- Internal parameters:

-- number_initiator_transactions: number of initiator transactions used.

-- Notes: 1. Fewer is more efficient since each transaction entails an overhead.

-- 2. Messages are actually sent when transaction commits, so sending a large

-- number of messages in a transaction can result in increased latency.

-- initiator_transaction_delay: delay time between initiator transactions.

-- Format: 'hh:mm:ss:xxx' hh=hours, mm=minutes, ss=seconds, xxx=milliseconds

-- Notes: 1. A transaction can be thought of as a burst of message_quantity /

-- number_initiator_transactions messages. This delay specifies a time

-- to wait before the next transaction is run.

-- 2. This parameter can be used to simulate message traffic distributed

-- over time.

-- number_dialogs: number of dialogs used to send messages.

-- Notes: 1. Message ordering only guaranteed with a dialog.

-- 2. Multiple dialogs allows concurrent processing on target.

-- 3. Dialog creation is expensive; dialog reuse is employed here.

-- dialog_recycle_max_messages: maximum number messages sent on a dialog before

-- recycling the dialog. Recycling is defined as ending the old dialog and

-- beginning a new one. A value of -1 indicates no recycling.

-- Notes: 1. Larger is more efficient since is minimizes the overhead of

-- creating dialogs.

-- 2. Larger can complicate dialog error processing.

-- number_target_procedures: number of activated target procedures to receive messages.

-- Notes: 1. A target proc locks all messages in a dialog when it receives first message

-- for a dialog, blocking other procs from processing these messages.

-- 2. Thus more dialogs yields increased concurrent processing. However, unless

-- dialog recycling is used, this should be set to number_dialogs, which

-- can utilize a target proc for each dialog.

-- max_messages_per_receive: maximum number of messages per target receive call.

-- Notes: 1. Larger is more efficient, but can complicate transaction error processing.

-- 2. The maximum value can be set to message_quantity / number_dialogs.

-- General note: for simplicity, @message_quantity should be evenly divisible

-- by @number_initiator_transactions x @number_dialogs, since this allows a

-- constant number of messages to be sent per dialog per transaction. "Remainder"

-- messages will not be sent to the target.

-- Example:

-- I want to send 100000 messages in sets of 10000 with a delay of 10 seconds between

-- each set. This calls for 10 transactions. Each message is 100 bytes and the target

-- message processing time is 10 ms. The messages are independent of each other, so use

-- 5 dialogs and target procedures to get some concurrent processing on the target. Allow

-- each target proc to receive 2000 messages at a time. Do not recycle dialogs.

-- INSERT INTO data_push_parameters

-- VALUES

-- (

-- 100000,

-- 10000,

-- '00:00:00:010',

-- 10,

-- '00:00:10:000',

-- 5,

-- -1,

-- 5,

-- 2000

-- );

CREATE TABLE data_push_parameters (

message_quantity BIGINT NOT NULL,

message_size INT NOT NULL,

message_processing_time CHAR(12) NOT NULL,

number_initiator_transactions INT NOT NULL,

initiator_transaction_delay CHAR(12) NOT NULL,

number_dialogs INT NOT NULL,

dialog_recycle_max_messages BIGINT NOT NULL,

number_target_procedures INT NOT NULL,

max_messages_per_receive BIGINT NOT NULL);

-- Insert parameter values.

TRUNCATE TABLE data_push_parameters;

INSERT INTO data_push_parameters

(

message_quantity,

message_size,

message_processing_time,

number_initiator_transactions,

initiator_transaction_delay,

number_dialogs,

dialog_recycle_max_messages,

number_target_procedures,

max_messages_per_receive

)

VALUES

(

10000,

1000,

'00:00:00:000',

-1,

1000

);

-- Check parameters.

DECLARE @message_quantity BIGINT;

DECLARE @number_initiator_transactions INT;

DECLARE @number_dialogs INT;

DECLARE @i BIGINT;

DECLARE @string VARCHAR(50);

SET @message_quantity = (SELECT message_quantity FROM data_push_parameters);

SET @number_initiator_transactions = (SELECT number_initiator_transactions FROM data_push_parameters);

SET @number_dialogs = (SELECT number_dialogs FROM data_push_parameters);

SET @i = @message_quantity / (@number_dialogs * @number_initiator_transactions);

SET @i = @i * @number_dialogs * @number_initiator_transactions;

IF @message_quantity > @i

BEGIN

SET @i = @message_quantity - @i;

SET @string = (SELECT CAST( @i AS VARCHAR(50)));

PRINT 'Warning: @message_quantity is not evenly divisible by @number_dialogs * @number_initiator_transactions';

PRINT @string + ' messages will not be sent to the target';

END;

--------------------------------------------------------------------

-- Script for fast data push sample.

-- This file is part of the Microsoft SQL Server Code Samples.

-- This source code is intended only as a supplement to Microsoft

-- Development Tools and/or on-line documentation. See these other

-- materials for detailed information regarding Microsoft code samples.

-- THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF

-- ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

-- THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

-- PARTICULAR PURPOSE.

--------------------------------------------------------------------

----------------------------------------------------

-- Initiator setup for fast data push.

-- Before running, customize the configuration-dependent

-- routing to the target service.

----------------------------------------------------

USE data_push_database;

-- The data push procedure: send messages to target.

CREATE PROCEDURE usp_data_push

BEGIN

SET NOCOUNT ON;

-- Get initiator parameters.

DECLARE @message_quantity BIGINT;

DECLARE @message_size INT;

DECLARE @number_initiator_transactions INT;

DECLARE @initiator_transaction_delay CHAR(12);

DECLARE @number_dialogs INT;

DECLARE @dialog_recycle_max_messages BIGINT;

SET @message_quantity = (SELECT message_quantity FROM data_push_parameters);

SET @message_size = (SELECT message_size FROM data_push_parameters);

SET @number_initiator_transactions = (SELECT number_initiator_transactions FROM data_push_parameters);

SET @initiator_transaction_delay = (SELECT initiator_transaction_delay FROM data_push_parameters);

SET @number_dialogs = (SELECT number_dialogs FROM data_push_parameters);

SET @dialog_recycle_max_messages = (SELECT dialog_recycle_max_messages FROM data_push_parameters);

-- Create a message payload of the requested size.

DECLARE @payload VARCHAR(MAX);

DECLARE @char VARCHAR(MAX);

SET @char = '0';

SELECT @payload = REPLICATE(@char, @message_size);

-- Loop controls.

DECLARE @messages_per_transaction BIGINT;

DECLARE @messages_per_dialog_transaction BIGINT;

DECLARE @transaction_counter INT;

DECLARE @message_counter BIGINT;

-- Compute messages to send per dialog per transaction:

-- @message_quantity / (@number_initiator_transactions x @number_dialogs)

-- Note that integer arithmetic may result in "remainder" messages that will not

-- be sent.

SET @messages_per_transaction = @message_quantity / @number_initiator_transactions;

SET @messages_per_dialog_transaction = @messages_per_transaction / @number_dialogs;

-- Error variables.

DECLARE @error_number INT;

DECLARE @error_message VARCHAR(4000);

-- Show start time.

SELECT GETDATE() AS 'Start sending';

-- Create a table containing requested number of dialogs.

DECLARE @dialogs TABLE (idx INT, handle UNIQUEIDENTIFIER, recycle_counter BIGINT);

DECLARE @idx INT;

DECLARE @handle UNIQUEIDENTIFIER;

DECLARE @recycle_counter BIGINT;

SET @idx = 0;

WHILE @idx < @number_dialogs

BEGIN

BEGIN DIALOG CONVERSATION @handle

FROM SERVICE initiator_service

TO SERVICE 'target_service'

ON CONTRACT data_push_contract

WITH ENCRYPTION = OFF;

INSERT INTO @dialogs (idx, handle, recycle_counter) VALUES (@idx, @handle, 0);

SET @idx = @idx + 1;

END

-- Loop through transactions.

SET @transaction_counter = 0;

WHILE @transaction_counter < @number_initiator_transactions

BEGIN

BEGIN TRANSACTION;

-- Loop through dialogs.

SET @idx = 0;

WHILE @idx < @number_dialogs

BEGIN

-- Send a batch of messages for dialog.

SET @handle = (SELECT handle FROM @dialogs WHERE idx = @idx);

SET @recycle_counter = (SELECT recycle_counter FROM @dialogs WHERE idx = @idx);

SET @message_counter = 0;

WHILE @message_counter < @messages_per_dialog_transaction

BEGIN

-- Time to recycle dialog?

IF @dialog_recycle_max_messages <> -1 AND

@recycle_counter = @dialog_recycle_max_messages

BEGIN

-- Inform target to end dialog.

SEND ON CONVERSATION @handle MESSAGE TYPE end_of_stream;

-- Replace the current dialog.

BEGIN DIALOG CONVERSATION @handle

FROM SERVICE initiator_service

TO SERVICE 'target_service'

ON CONTRACT data_push_contract

WITH ENCRYPTION = OFF;

UPDATE @dialogs SET handle = @handle WHERE idx = @idx;

SET @recycle_counter = 0;

END

-- Send a message.

BEGIN TRY

BEGIN

SEND ON CONVERSATION @handle MESSAGE TYPE data_push_message (@payload);

END

IF @dialog_recycle_max_messages <> -1

BEGIN

SET @recycle_counter = @recycle_counter + 1;

END

SET @message_counter = @message_counter + 1;

END TRY

BEGIN CATCH

SET @error_number = ERROR_NUMBER();

SET @error_message = ERROR_MESSAGE();

-- Dialog is faulty?

DECLARE @dialog_error INT;

SET @dialog_error = 1;

DECLARE @dialog_state VARCHAR(2);

SET @dialog_state = (SELECT state FROM sys.conversation_endpoints

WHERE conversation_handle = @handle);

IF @@ROWCOUNT = 1

BEGIN

-- Good dialog is starting or conversing.

IF @dialog_state = 'SO' OR @dialog_state = 'CO'

BEGIN

SET @dialog_error = 0;

END

IF @dialog_error = 1

BEGIN

-- Record the error.

INSERT INTO initiator_processing_errors VALUES(@handle, @error_number,

@error_message, NULL, NULL, NULL, NULL, 0);

-- Replace dialog and continue sending.

BEGIN DIALOG CONVERSATION @handle

FROM SERVICE initiator_service

TO SERVICE 'target_service'

ON CONTRACT data_push_contract

WITH ENCRYPTION = OFF;

UPDATE @dialogs SET handle = @handle WHERE idx = @idx;

SET @recycle_counter = 0;

END

ELSE

BEGIN

-- Record the error and return error.

INSERT INTO initiator_processing_errors VALUES(@handle, @error_number,

@error_message, NULL, NULL, NULL, NULL, 0);

RETURN 1;

END

END CATCH

END

UPDATE @dialogs SET recycle_counter = @recycle_counter WHERE idx = @idx;

SET @idx = @idx + 1;

END

COMMIT;

SET @transaction_counter = @transaction_counter + 1;

-- Wait for next transaction.

IF @transaction_counter < @number_initiator_transactions

BEGIN

WAITFOR DELAY @initiator_transaction_delay;

END

-- Gracefully end dialogs by informing target.

BEGIN TRANSACTION;

SET @idx = 0;

WHILE @idx < @number_dialogs

BEGIN

SET @handle = (SELECT handle FROM @dialogs WHERE idx = @idx);

BEGIN

SEND ON CONVERSATION @handle MESSAGE TYPE end_of_stream;

END

SET @idx = @idx + 1;

END

COMMIT;

-- Show end time.

SELECT GETDATE() AS 'End sending';

RETURN 0;

END;

-- Resends all pending messages in sys.transmission_queue

-- belonging to an old colversation on a new conversation.

CREATE PROCEDURE usp_resend_pending (@old_handle UNIQUEIDENTIFIER)

BEGIN

SET NOCOUNT ON;

DECLARE @message_type_name SYSNAME;

DECLARE @message_body VARCHAR(MAX);

-- Get a new dialog.

DECLARE @handle UNIQUEIDENTIFIER;

BEGIN DIALOG CONVERSATION @handle

FROM SERVICE initiator_service

TO SERVICE 'target_service'

ON CONTRACT data_push_contract

WITH ENCRYPTION = OFF;

-- Declare a cursor to iterate over all the pending messages.

-- It is important to keep the message order and to keep the original message type.

DECLARE cursor_pending CURSOR LOCAL FORWARD_ONLY READ_ONLY

FOR SELECT message_type_name, message_body

FROM sys.transmission_queue

WHERE conversation_handle = @old_handle

ORDER BY message_sequence_number;

OPEN cursorPending;

FETCH NEXT FROM cursor_pending INTO @message_type_name, @message_body;

WHILE (@@FETCH_STATUS = 0)

BEGIN

-- Resend the message on the new conversation

SEND ON CONVERSATION @handle MESSAGE TYPE @message_type_name (@message_body);

FETCH NEXT FROM cursor_pending INTO @message_type_name, @message_body;

END

CLOSE cursor_pending;

DEALLOCATE cursor_pending;

-- Signal end of stream to target.

SEND ON CONVERSATION @handle MESSAGE TYPE end_of_stream;

END;

-- Activated store proc for the initiator to receive messages.

-- Dialogs are gracefully ended by the target after receiving

-- an end_of_stream message from the initiator; the end dialog

-- message is then processed here. This method is recommended

-- to avoid "fire and forget" message loss. One message per

-- invocation is OK here for expected low-volume load.

CREATE PROCEDURE initiator_queue_activated_procedure

BEGIN

SET NOCOUNT ON;

DECLARE @conversation_handle UNIQUEIDENTIFIER,

@message_type_name SYSNAME,

@message_body VARCHAR(MAX);

-- Error variables.

DECLARE @error_number INT;

DECLARE @error_message VARCHAR(4000);

DECLARE @error_severity INT;

DECLARE @error_state INT;

DECLARE @error_procedure SYSNAME;

DECLARE @error_line INT;

BEGIN TRY

BEGIN TRANSACTION;

-- Wait 5 seconds for a message.

WAITFOR (

RECEIVE TOP(1)

@conversation_handle = conversation_handle,

@message_type_name = message_type_name,

@message_body = message_body

FROM initiator_queue), TIMEOUT 5000;

IF @@ROWCOUNT = 1

BEGIN

IF @message_type_name = 'https://schemas.microsoft.com/SQL/ServiceBroker/EndDialog'

BEGIN

-- Target is ending dialog normally.

END CONVERSATION @conversation_handle;

END

ELSE IF @message_type_name = 'https://schemas.microsoft.com/SQL/ServiceBroker/Error'

BEGIN

-- Record the error.

WITH XMLNAMESPACES ('https://schemas.microsoft.com/SQL/ServiceBroker/Error' AS ssb)

SELECT

@error_number = CAST(@message_body AS XML).value('(//ssb:Error/ssb:Code)[1]', 'INT'),

@error_message = CAST(@message_body AS XML).value('(//ssb:Error/ssb:Description)[1]', 'VARCHAR(4000)');

INSERT INTO initiator_processing_errors VALUES(@conversation_handle, @error_number,

@error_message, NULL, NULL, NULL, NULL, 0);

-- Can messages be resent?

IF (@error_number IN (-8489, -8462, -9719, -28052))

BEGIN

-- Resend the messages on a new dialog.

EXEC usp_resend_pending @conversation_handle;

END

ELSE

BEGIN

-- Save the messages in a side table to be processed later.

INSERT INTO unsent_messages

SELECT message_type_name, message_body FROM sys.transmission_queue

WHERE conversation_handle = @conversation_handle;

END

-- End the conversation.

END CONVERSATION @conversation_handle;

END

COMMIT;

END TRY

BEGIN CATCH

SET @error_number = ERROR_NUMBER();

SET @error_message = ERROR_MESSAGE();

SET @error_severity = ERROR_SEVERITY();

SET @error_state = ERROR_STATE();

SET @error_procedure = ERROR_PROCEDURE();

SET @error_line = ERROR_LINE();

IF XACT_STATE() = -1

BEGIN

-- The transaction is doomed. Only rollback possible.

-- Note: 5 consecutive rollbacks will disable the queue!

ROLLBACK TRANSACTION;

-- Record the error.

BEGIN TRANSACTION;

INSERT INTO initiator_processing_errors VALUES(NULL, @error_number, @error_message,

@error_severity, @error_state, @error_procedure, @error_line, 1);

COMMIT;

END

ELSE IF XACT_STATE() = 1

BEGIN

-- Record error and commit transaction.

INSERT INTO initiator_processing_errors VALUES(NULL, @error_number, @error_message,

@error_severity, @error_state, @error_procedure, @error_line, 0);

COMMIT;

END

END CATCH

END;

-- Create the initiator queue with activated procedure.

CREATE QUEUE initiator_queue

WITH ACTIVATION (

STATUS = ON,

MAX_QUEUE_READERS = 1,

PROCEDURE_NAME = initiator_queue_activated_procedure,

EXECUTE AS OWNER);

-- Create initiator service.

CREATE SERVICE initiator_service ON QUEUE initiator_queue (data_push_contract);

-- Any user can send on the service.

GRANT SEND ON SERVICE::initiator_service TO PUBLIC;

-- This table stores unsent messages.

IF EXISTS (SELECT name FROM sys.tables WHERE name = 'unsent_messages')

DROP TABLE unsent_messages;

CREATE TABLE unsent_messages ( message_type_name SYSNAME, message_body VARCHAR(MAX) );

-- Table to store processing errors.

IF EXISTS (SELECT name FROM sys.tables WHERE name = 'initiator_processing_errors')

DROP TABLE initiator_processing_errors;

CREATE TABLE initiator_processing_errors (error_conversation UNIQUEIDENTIFIER, error_number INT,

error_message VARCHAR(4000), error_severity INT, error_state INT, error_procedure SYSNAME NULL,

error_line INT, doomed_transaction TINYINT)

---------------------------------------------------------------------

-- Routing.

-- Skip the following if services are in the same database instance.

---------------------------------------------------------------------

-- Create a route to the target service.

CREATE ROUTE target_route

WITH SERVICE_NAME = 'target_service',

ADDRESS = 'tcp://target_host:4022';

-- In msdb, create an incoming route to the initiator service.

USE msdb;

CREATE ROUTE initiator_route

WITH SERVICE_NAME = 'initiator_service',

ADDRESS = 'local';

USE data_push_database;

--------------------------------------------------------------------

-- Script for fast data push sample.

-- This file is part of the Microsoft SQL Server Code Samples.

-- This source code is intended only as a supplement to Microsoft

-- Development Tools and/or on-line documentation. See these other

-- materials for detailed information regarding Microsoft code samples.

-- THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF

-- ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

-- THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

-- PARTICULAR PURPOSE.

--------------------------------------------------------------------

----------------------------------------------------

-- Target setup for fast data push.

-- Before running, customize the configuration-dependent

-- routing to the initiator service.

----------------------------------------------------

--------------------------------------------------------------------

-- Poison message processing:

-- Although this should not occur here because messages are not

-- processed in any significant way, there are times that

-- an application might find itself temporarily unable to process

-- a message. The temptation is then to roll back the receive

-- transaction and try again. The danger of doing this is that

-- 5 consecutive roll backs on a queue will disable it.

-- If a queue does become disabled, possibly due to a doomed

-- transaction, a BROKER_QUEUE_DISABLED event notification can

-- be used as a recovery mechanism. You can also use the TRY-CATCH

-- construct to process transaction errors as shown below.

--------------------------------------------------------------------

USE data_push_database;

-- Activated store proc for the target to receive messages.

CREATE PROCEDURE target_queue_activated_procedure

BEGIN

SET NOCOUNT ON;

-- Variable table for received messages.

DECLARE @receive_table TABLE(

queuing_order BIGINT,

conversation_handle UNIQUEIDENTIFIER,

message_type_name SYSNAME,

message_body VARCHAR(MAX));

-- Cursor for received message table.

DECLARE message_cursor CURSOR LOCAL FORWARD_ONLY READ_ONLY

FOR SELECT

conversation_handle,

message_type_name,

message_body

FROM @receive_table ORDER BY queuing_order;

DECLARE @conversation_handle UNIQUEIDENTIFIER,

@message_type_name SYSNAME,

@message_body VARCHAR(MAX);

-- Count processed messages.

DECLARE @message_counter BIGINT;

SET @message_counter = 0;

-- Error variables.

DECLARE @error_number INT;

DECLARE @error_message VARCHAR(4000);

DECLARE @error_severity INT;

DECLARE @error_state INT;

DECLARE @error_procedure SYSNAME;

DECLARE @error_line INT;

-- Get target parameters.

DECLARE @message_processing_time CHAR(12);

SET @message_processing_time = (SELECT message_processing_time FROM data_push_parameters);

DECLARE @max_messages_per_receive BIGINT;

SET @max_messages_per_receive = (SELECT max_messages_per_receive FROM data_push_parameters);

-- Receive messages for available conversation groups.

BEGIN TRY

WHILE (1=1)

BEGIN

BEGIN TRANSACTION;

-- Receive max available messages into the table.

-- Wait 5 seconds for messages.

WAITFOR (

RECEIVE TOP(@max_messages_per_receive)

queuing_order,

conversation_handle,

message_type_name,

message_body

FROM target_queue

INTO @receive_table

), TIMEOUT 5000;

IF @@ROWCOUNT = 0

BEGIN

COMMIT;

BREAK;

END

-- Process the messages.

OPEN message_cursor;

WHILE (1=1)

BEGIN

FETCH NEXT FROM message_cursor

INTO @conversation_handle,

@message_type_name,

@message_body;

IF (@@FETCH_STATUS != 0) BREAK;

-- Process a message.

-- If an exception occurs, catch and attempt to recover.

BEGIN TRY

IF @message_type_name = 'data_push_message'

BEGIN

-- Process the message for the specified amount of time.

WAITFOR DELAY @message_processing_time;

SET @message_counter = @message_counter + 1;

END

ELSE IF @message_type_name = 'end_of_stream'

BEGIN

-- Initiator is signaling end of message stream: end the dialog.

END CONVERSATION @conversation_handle;

END

ELSE IF @message_type_name = 'https://schemas.microsoft.com/SQL/ServiceBroker/Error'

BEGIN

-- If the message_type_name indicates that the message is an error,

-- record the error and end the conversation.

WITH XMLNAMESPACES ('https://schemas.microsoft.com/SQL/ServiceBroker/Error' AS ssb)

SELECT

@error_number = CAST(@message_body AS XML).value('(//ssb:Error/ssb:Code)[1]', 'INT'),

@error_message = CAST(@message_body AS XML).value('(//ssb:Error/ssb:Description)[1]', 'VARCHAR(4000)');

INSERT INTO target_processing_errors VALUES(@conversation_handle, @error_number,

@error_message, NULL, NULL, NULL, NULL, 0);

END CONVERSATION @conversation_handle;

END

END TRY

BEGIN CATCH

SET @error_number = ERROR_NUMBER();

SET @error_message = ERROR_MESSAGE();

SET @error_severity = ERROR_SEVERITY();

SET @error_state = ERROR_STATE();

SET @error_procedure = ERROR_PROCEDURE();

SET @error_line = ERROR_LINE();

IF XACT_STATE() = -1

BEGIN

-- The transaction is doomed. Only rollback possible.

-- This could disable the queue if done 5 times consecutively!

ROLLBACK TRANSACTION;

-- Record the error.

BEGIN TRANSACTION;

INSERT INTO target_processing_errors VALUES(NULL, @error_number, @error_message,

@error_severity, @error_state, @error_procedure, @error_line, 1);

COMMIT;

-- For this level of error, it is best to exit the proc

-- and give the queue monitor control.

-- Breaking to the outer catch will accomplish this.

RAISERROR ('Message processing error', 16, 1);

END

ELSE IF XACT_STATE() = 1

BEGIN

-- Record error and continue processing messages.

-- Failing message could also be put aside for later processing here.

-- Otherwise it will be discarded.

INSERT INTO target_processing_errors VALUES(NULL, @error_number, @error_message,

@error_severity, @error_state, @error_procedure, @error_line, 0);

END

END CATCH

END

CLOSE message_cursor;

DELETE @receive_table;

COMMIT;

END

END TRY

BEGIN CATCH

-- Process the error and exit the proc to give the queue monitor control

SET @error_number = ERROR_NUMBER();

SET @error_message = ERROR_MESSAGE();

SET @error_severity = ERROR_SEVERITY();

SET @error_state = ERROR_STATE();

SET @error_procedure = ERROR_PROCEDURE();

SET @error_line = ERROR_LINE();

IF XACT_STATE() = -1

BEGIN

-- The transaction is doomed. Only rollback possible.

-- This could disable the queue if done 5 times consecutively!

ROLLBACK TRANSACTION;

-- Record the error.

BEGIN TRANSACTION;

INSERT INTO target_processing_errors VALUES(NULL, @error_number, @error_message,

@error_severity, @error_state, @error_procedure, @error_line, 1);

COMMIT;

END

ELSE IF XACT_STATE() = 1

BEGIN

-- Record error and commit transaction.

-- Here you could also save anything else you want before exiting.

INSERT INTO target_processing_errors VALUES(NULL, @error_number, @error_message,

@error_severity, @error_state, @error_procedure, @error_line, 0);

COMMIT;

END

END CATCH

-- Increment processed message counter.

BEGIN TRANSACTION;

DECLARE @counter BIGINT;

SET @counter = (SELECT TOP(1) counter FROM target_message_counter);

SET @counter = @counter + @message_counter;

UPDATE target_message_counter SET counter = @counter;

COMMIT;

END;

-- Get number of activated target procedures parameter.

DECLARE @number_target_procedures INT;

SET @number_target_procedures = (SELECT number_target_procedures FROM data_push_parameters);

-- Create the target queue with specified number of activated procedures.

DECLARE @query VARCHAR(500);

DECLARE @string VARCHAR(50);

SET @string = (SELECT CAST( @number_target_procedures AS VARCHAR(50)));

SET @query = 'CREATE QUEUE target_queue

WITH ACTIVATION (

STATUS = ON,

MAX_QUEUE_READERS = ' + @string + ',

PROCEDURE_NAME = target_queue_activated_procedure,

EXECUTE AS OWNER)';

EXEC (@query);

-- Create target service.

CREATE SERVICE target_service ON QUEUE target_queue (data_push_contract);

-- Any user can send on the service.

GRANT SEND ON SERVICE::target_service TO PUBLIC;

-- Table to count processed messages.

IF EXISTS (SELECT name FROM sys.tables WHERE name = 'target_message_counter')

DROP TABLE message_counter;

CREATE TABLE target_message_counter (counter BIGINT NOT NULL);

INSERT INTO target_message_counter VALUES (0);

-- Table to store processing errors.

IF EXISTS (SELECT name FROM sys.tables WHERE name = 'target_processing_errors')

DROP TABLE target_processing_errors;

CREATE TABLE target_processing_errors (error_conversation UNIQUEIDENTIFIER, error_number INT,

error_message VARCHAR(4000), error_severity INT, error_state INT, error_procedure SYSNAME NULL,

error_line INT, doomed_transaction TINYINT)

---------------------------------------------------------------------

-- Get size of a message queue.

-- Method used is faster than SQL count operator.

---------------------------------------------------------------------

CREATE PROCEDURE usp_get_queue_size ( @queue_name VARCHAR(50) )

BEGIN

SELECT p.rows

FROM sys.objects AS o

JOIN sys.partitions AS p ON p.object_id = o.object_id

JOIN sys.objects AS q ON o.parent_object_id = q.object_id

WHERE q.name = @queue_name

AND p.index_id = 1;

END;

---------------------------------------------------------------------

-- Routing.

-- Skip the following if services are in the same database instance.

---------------------------------------------------------------------

-- Create a route to the initiator service.

CREATE ROUTE initiator_route

WITH SERVICE_NAME = 'initiator_service',

ADDRESS = 'tcp://initiator_host:4022';

-- In msdb, create an incoming route to the target service.

USE msdb;

CREATE ROUTE target_route

WITH SERVICE_NAME = 'target_service',

ADDRESS = 'local';

USE data_push_database;

--------------------------------------------------------------------

-- Script for fast data push sample.

-- This file is part of the Microsoft SQL Server Code Samples.

-- This source code is intended only as a supplement to Microsoft

-- Development Tools and/or on-line documentation. See these other

-- materials for detailed information regarding Microsoft code samples.

-- THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF

-- ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

-- THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

-- PARTICULAR PURPOSE.

--------------------------------------------------------------------

---------------------------------------------------------------------

-- Initiator data push.

-- This script can also be run on multiple connections to achieve transactional

-- concurrency. Since each connection will send message_quantity messages

-- with number_initiator_transactions transactions and number_dialogs

-- dialogs, these parameter quantities should be divided by the number

-- of connections to get the same total quantities.

---------------------------------------------------------------------

USE data_push_database;

-- Send the messages. Output sending time.

EXEC usp_data_push;

---------------------------------------------------------------------

-- Wait for transmission queue to be empty, signifying the

-- reception and acknowledgement of all messages by the target.

-- Use this efficient checking method every 5 seconds.

---------------------------------------------------------------------

DECLARE @count BIGINT;

WHILE (1=1)

BEGIN

SET @count =

(SELECT p.rows

FROM sys.objects AS o

JOIN sys.partitions AS p ON p.object_id = o.object_id

WHERE o.name = 'sysxmitqueue');

IF (@count = 0) BREAK;

WAITFOR DELAY '00:00:05:000';

END;

SELECT GETDATE() AS 'End transmission';

-- View processing errors.

SELECT * FROM initiator_processing_errors;

-- View unsent messages.

SELECT * FROM unsent_messages;

--------------------------------------------------------------------

-- Script for fast data push sample.

-- This file is part of the Microsoft SQL Server Code Samples.

-- This source code is intended only as a supplement to Microsoft

-- Development Tools and/or on-line documentation. See these other

-- materials for detailed information regarding Microsoft code samples.

-- THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF

-- ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

-- THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

-- PARTICULAR PURPOSE.

--------------------------------------------------------------------

----------------------------------------------------

-- Monitor target.

-- A count of processed messages is kept in the

-- target_message_counter table.

-- The usp_get_queue_size procedure can also be used to

-- monitor the size of the target queue.

----------------------------------------------------

USE data_push_database;

-- Wait for message_quantity messages to be processed.

DECLARE @message_quantity BIGINT;

SET @message_quantity = (SELECT message_quantity FROM data_push_parameters);

DECLARE @count BIGINT;

WHILE (1=1)

BEGIN

SET @count = (SELECT TOP(1) counter FROM target_message_counter);

IF (@count >= @message_quantity) BREAK;

WAITFOR DELAY '00:00:05:000';

END;

SELECT GETDATE() AS 'Messages processed';

-- View message processing errors.

SELECT * FROM target_processing_errors;

-- Clear processed message counter for next run.

UPDATE target_message_counter SET counter = 0;

--------------------------------------------------------------------

-- Script for data push sample.

-- This file is part of the Microsoft SQL Server Code Samples.

-- This source code is intended only as a supplement to Microsoft

-- Development Tools and/or on-line documentation. See these other

-- materials for detailed information regarding Microsoft code samples.

-- THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF

-- ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

-- THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

-- PARTICULAR PURPOSE.

--------------------------------------------------------------------

----------------------------------------------------

-- Cleanup for fast data push.

-- Before running, replace the configuration-dependent

-- domain_name and partner_host names.

----------------------------------------------------

USE master;

IF EXISTS (SELECT * FROM sys.databases WHERE name = 'data_push_database')

DROP DATABASE data_push_database;

USE msdb;

IF EXISTS (SELECT * FROM sys.routes WHERE name = 'initiator_route')

DROP ROUTE initiator_route;

IF EXISTS (SELECT * FROM sys.routes WHERE name = 'target_route')

DROP ROUTE target_route;

USE master;

IF EXISTS (SELECT * FROM sys.endpoints WHERE name = 'service_broker_endpoint')

DROP ENDPOINT service_broker_endpoint;

IF EXISTS (SELECT * FROM sys.syslogins WHERE name = 'domain_name\partner_host$')

DROP LOGIN [domain_name\partner_host$];

Reusing dialogs with a dialog pool

portegys — Thu, 24 Jul 2008 19:55:00 GMT

As noted in various Service Broker sources, it is often advantageous to minimize the overhead of creating dialogs to send messages on. This blog shows how to create a shared pool of dialogs to be able to reuse dialogs instead of creating new ones. The dialog pool is a variation of Remus Rusanu's reusing and recycling conversations as shown in his blog. One of the main differences is that the dialog pool is keyed only on services and contract, not SPID. This allows the same SPID to obtain multiple dialogs from the pool should the need arise. As importantly, different SPIDs can reuse the same dialog sequentially instead of creating two of them. Measurements show equivalent performance using the dialog pool compared to the SPID-based reuse scheme.

The following code shows how to get, free and delete dialogs from a dialog pool table. Initially empty, a new dialog is created in the pool when a request for an existing free dialog cannot be met. Thus the pool will grow during bursts of high demand.

The dialog pool entries also contain creation time and send count fields that ease the auditing and "recycling" of dialogs in the pool based on application requirements. Recycling consists of gracefully ending an existing dialog between services and beginning a new one. If done prudently, this technique can ease the handling of dialog errors by limiting the number of messages affected. For example, the application may choose to contrain a dialog to a certain number of messages before it is recycled. This might also be done according to the age of a dialog. See the end of the usp_send procedure for an example of recycling.

An example application that exercises the dialog pool is also included.

--------------------------------------------------------------------------

-- Dialog Pool Sample.

-- This sample shows how to create and use a shared pool of reuseable dialogs.

-- The purpose of reusing dialogs is to reduce the overhead of creating them.

-- The sample also shows how dialogs in the pool can be "recycled" by deleting

-- dialogs based on application criteria, such as number of messages sent.

-- This sample is largely based on Remus Rusanu's tutorials on reusing and

-- recycling conversations (rusanu.com/blog).

-- Contents: dialog pool and application using the pool.

----------------------------------------------------

USE master

--------------------------------------------------------------------------

-- Create demo database section

--------------------------------------------------------------------------

IF EXISTS (SELECT name FROM sys.databases WHERE name = 'SsbDemoDb')

DROP DATABASE [SsbDemoDb];

CREATE DATABASE [SsbDemoDb]

USE [SsbDemoDb];

-- Create master key

IF NOT EXISTS(SELECT name FROM sys.symmetric_keys WHERE name = '##MS_DatabaseMasterKey##')

CREATE MASTER KEY ENCRYPTION BY PASSWORD='Password#123'

--------------------------------------------------------------------------

-- Dialog pool section

--------------------------------------------------------------------------

-- The dialog pool table.

-- Obtain a conversation handle using from service, to service, and contract.

-- Also indicates age and usage of dialog for auditing purposes.

--------------------------------------------------------------------------

IF EXISTS (SELECT name FROM sys.tables WHERE name = 'DialogPool')

DROP TABLE [DialogPool]

CREATE TABLE [DialogPool] (

FromService SYSNAME NOT NULL,

ToService SYSNAME NOT NULL,

OnContract SYSNAME NOT NULL,

Handle UNIQUEIDENTIFIER NOT NULL,

OwnerSPID INT NOT NULL,

CreationTime DATETIME NOT NULL,

SendCount BIGINT NOT NULL,

UNIQUE (Handle));

--------------------------------------------------------------------------

-- Get dialog procedure.

-- Reuse a free dialog in the pool or create a new one in case

-- no free dialogs exist.

-- Input is from service, to service, and contract.

-- Output is dialog handle and count of message previously sent on dialog.

--------------------------------------------------------------------------

IF EXISTS (SELECT name FROM sys.procedures WHERE name = 'usp_get_dialog')

DROP PROC usp_get_dialog

CREATE PROCEDURE [usp_get_dialog] (

@fromService SYSNAME,

@toService SYSNAME,

@onContract SYSNAME,

@dialogHandle UNIQUEIDENTIFIER OUTPUT,

@sendCount BIGINT OUTPUT)

BEGIN

SET NOCOUNT ON;

DECLARE @dialog TABLE

(

FromService SYSNAME NOT NULL,

ToService SYSNAME NOT NULL,

OnContract SYSNAME NOT NULL,

Handle UNIQUEIDENTIFIER NOT NULL,

OwnerSPID INT NOT NULL,

CreationTime DATETIME NOT NULL,

SendCount BIGINT NOT NULL

);

-- Try to claim an unused dialog in [DialogPool]

-- READPAST option avoids blocking on locked dialogs.

BEGIN TRANSACTION;

DELETE @dialog;

UPDATE TOP(1) [DialogPool] WITH(READPAST)

SET OwnerSPID = @@SPID

OUTPUT INSERTED.* INTO @dialog

WHERE FromService = @fromService

AND ToService = @toService

AND OnContract = @OnContract

AND OwnerSPID = -1;

IF @@ROWCOUNT > 0

BEGIN

SET @dialogHandle = (SELECT Handle FROM @dialog);

SET @sendCount = (SELECT SendCount FROM @dialog);

END

ELSE

BEGIN

-- No free dialogs: need to create a new one

BEGIN DIALOG CONVERSATION @dialogHandle

FROM SERVICE @fromService

TO SERVICE @toService

ON CONTRACT @onContract

WITH ENCRYPTION = OFF;

INSERT INTO [DialogPool]

(FromService, ToService, OnContract, Handle, OwnerSPID,

CreationTime, SendCount)

VALUES

(@fromService, @toService, @onContract, @dialogHandle, @@SPID,

GETDATE(), 0);

SET @sendCount = 0;

END

COMMIT

END;

--------------------------------------------------------------------------

-- Free dialog procedure.

-- Return the dialog to the pool.

-- Inputs are dialog handle and updated send count.

--------------------------------------------------------------------------

IF EXISTS (SELECT name FROM sys.procedures WHERE name = 'usp_free_dialog')

DROP PROC usp_free_dialog

CREATE PROCEDURE [usp_free_dialog] (

@dialogHandle UNIQUEIDENTIFIER,

@sendCount BIGINT)

BEGIN

SET NOCOUNT ON;

DECLARE @rowcount INT;

DECLARE @string VARCHAR(50);

BEGIN TRANSACTION;

-- Release dialog by setting OwnerSPID to -1.

UPDATE [DialogPool] SET OwnerSPID = -1, SendCount = @sendCount WHERE Handle = @dialogHandle;

SELECT @rowcount = @@ROWCOUNT;

IF @rowcount = 0

BEGIN

SET @string = (SELECT CAST( @dialogHandle AS VARCHAR(50)));

RAISERROR('usp_free_dialog: dialog %s not found in dialog pool', 16, 1, @string) WITH LOG;

END

ELSE IF @rowcount > 1

BEGIN

SET @string = (SELECT CAST( @dialogHandle AS VARCHAR(50)));

RAISERROR('usp_free_dialog: duplicate dialog %s found in dialog pool', 16, 1, @string) WITH LOG;

END

COMMIT

END;

--------------------------------------------------------------------------

-- Delete dialog procedure.

-- Delete the dialog from the pool. This does not end the dialog.

-- Input is dialog handle.

--------------------------------------------------------------------------

IF EXISTS (SELECT name FROM sys.procedures WHERE name = 'usp_delete_dialog')

DROP PROC usp_delete_dialog

CREATE PROCEDURE [usp_delete_dialog] (

@dialogHandle UNIQUEIDENTIFIER)

BEGIN

SET NOCOUNT ON;

BEGIN TRANSACTION;

DELETE [DialogPool] WHERE Handle = @dialogHandle;

COMMIT

END;

--------------------------------------------------------------------------

-- Application setup section.

--------------------------------------------------------------------------

-- Send messages from initiator to target.

-- Initiator uses dialogs from the dialog pool.

-- Initiator also retires dialogs based on application criteria,

-- which results in recycling dialogs in the pool.

--------------------------------------------------------------------------

-- This table stores the messages on the target side

IF EXISTS (SELECT name FROM sys.tables WHERE name = 'MsgTable')

DROP TABLE MsgTable

CREATE TABLE MsgTable ( message_type SYSNAME, message_body NVARCHAR(4000))

-- Activated store proc for the initiator to receive messages.

CREATE PROCEDURE initiator_queue_activated_procedure

BEGIN

DECLARE @handle UNIQUEIDENTIFIER;

DECLARE @message_type SYSNAME;

BEGIN TRANSACTION;

WAITFOR (

RECEIVE TOP(1) @handle = [conversation_handle],

@message_type = [message_type_name]

FROM [SsbInitiatorQueue]), TIMEOUT 5000;

IF @@ROWCOUNT = 1

BEGIN

-- Expect target response to EndOfStream message.

IF @message_type = 'https://schemas.microsoft.com/SQL/ServiceBroker/EndDialog'

BEGIN

END CONVERSATION @handle;

END

COMMIT

END;

-- Activated store proc for the target to receive messages.

CREATE PROCEDURE target_queue_activated_procedure

BEGIN

-- Variable table for received messages.

DECLARE @receive_table TABLE(

queuing_order BIGINT,

conversation_handle UNIQUEIDENTIFIER,

message_type_name SYSNAME,

message_body VARCHAR(MAX));

-- Cursor for received message table.

DECLARE message_cursor CURSOR LOCAL FORWARD_ONLY READ_ONLY

FOR SELECT

conversation_handle,

message_type_name,

message_body

FROM @receive_table ORDER BY queuing_order;

DECLARE @conversation_handle UNIQUEIDENTIFIER;

DECLARE @message_type SYSNAME;

DECLARE @message_body VARCHAR(4000);

-- Error variables.

DECLARE @error_number INT;

DECLARE @error_message VARCHAR(4000);

DECLARE @error_severity INT;

DECLARE @error_state INT;

DECLARE @error_procedure SYSNAME;

DECLARE @error_line INT;

DECLARE @error_dialog VARCHAR(50);

BEGIN TRY

WHILE (1 = 1)

BEGIN

BEGIN TRANSACTION;

-- Receive all available messages into the table.

-- Wait 5 seconds for messages.

WAITFOR (

RECEIVE

[queuing_order],

[conversation_handle],

[message_type_name],

CAST([message_body] AS VARCHAR(4000))

FROM [SsbTargetQueue]

INTO @receive_table

), TIMEOUT 5000;

IF @@ROWCOUNT = 0

BEGIN

COMMIT;

BREAK;

END

ELSE

BEGIN

OPEN message_cursor;

WHILE (1=1)

BEGIN

FETCH NEXT FROM message_cursor

INTO @conversation_handle,

@message_type,

@message_body;

IF (@@FETCH_STATUS != 0) BREAK;

-- Process a message.

-- If an exception occurs, catch and attempt to recover.

BEGIN TRY

IF @message_type = 'SsbMsgType'

BEGIN

-- process the msg. Here we will just insert it into a table

INSERT INTO MsgTable values(@message_type, @message_body);

END

ELSE IF @message_type = 'EndOfStream'

BEGIN

-- initiator is signaling end of message stream: end the dialog

END CONVERSATION @conversation_handle;

END

ELSE IF @message_type = 'https://schemas.microsoft.com/SQL/ServiceBroker/Error'

BEGIN

-- If the message_type indicates that the message is an error,

-- raise the error and end the conversation.

WITH XMLNAMESPACES ('https://schemas.microsoft.com/SQL/ServiceBroker/Error' AS ssb)

SELECT

@error_number = CAST(@message_body AS XML).value('(//ssb:Error/ssb:Code)[1]', 'INT'),

@error_message = CAST(@message_body AS XML).value('(//ssb:Error/ssb:Description)[1]', 'VARCHAR(4000)');

SET @error_dialog = CAST(@conversation_handle AS VARCHAR(50));

RAISERROR('Error in dialog %s: %s (%i)', 16, 1, @error_dialog, @error_message, @error_number);

END CONVERSATION @conversation_handle;

END

END TRY

BEGIN CATCH

SET @error_number = ERROR_NUMBER();

SET @error_message = ERROR_MESSAGE();

SET @error_severity = ERROR_SEVERITY();

SET @error_state = ERROR_STATE();

SET @error_procedure = ERROR_PROCEDURE();

SET @error_line = ERROR_LINE();

IF XACT_STATE() = -1

BEGIN

-- The transaction is doomed. Only rollback possible.

-- This could disable the queue if done 5 times consecutively!

ROLLBACK TRANSACTION;

-- Record the error.

BEGIN TRANSACTION;

INSERT INTO target_processing_errors VALUES(NULL, @error_number, @error_message,

@error_severity, @error_state, @error_procedure, @error_line, 1);

COMMIT;

-- For this level of error, it is best to exit the proc

-- and give the queue monitor control.

-- Breaking to the outer catch will accomplish this.

RAISERROR ('Message processing error', 16, 1);

END

ELSE IF XACT_STATE() = 1

BEGIN

-- Record error and continue processing messages.

-- Failing message could also be put aside for later processing here.

-- Otherwise it will be discarded.

INSERT INTO target_processing_errors VALUES(NULL, @error_number, @error_message,

@error_severity, @error_state, @error_procedure, @error_line, 0);

END

END CATCH

END

CLOSE message_cursor;

DELETE @receive_table;

END

COMMIT;

END

END TRY

BEGIN CATCH

-- Process the error and exit the proc to give the queue monitor control

SET @error_number = ERROR_NUMBER();

SET @error_message = ERROR_MESSAGE();

SET @error_severity = ERROR_SEVERITY();

SET @error_state = ERROR_STATE();

SET @error_procedure = ERROR_PROCEDURE();

SET @error_line = ERROR_LINE();

IF XACT_STATE() = -1

BEGIN

-- The transaction is doomed. Only rollback possible.

-- This could disable the queue if done 5 times consecutively!

ROLLBACK TRANSACTION;

-- Record the error.

BEGIN TRANSACTION;

INSERT INTO target_processing_errors VALUES(NULL, @error_number, @error_message,

@error_severity, @error_state, @error_procedure, @error_line, 1);

COMMIT;

END

ELSE IF XACT_STATE() = 1

BEGIN

-- Record error and commit transaction.

-- Here you could also save anything else you want before exiting.

INSERT INTO target_processing_errors VALUES(NULL, @error_number, @error_message,

@error_severity, @error_state, @error_procedure, @error_line, 0);

COMMIT;

END

END CATCH

END;

-- Table to store processing errors.

IF EXISTS (SELECT name FROM sys.tables WHERE name = 'target_processing_errors')

DROP TABLE target_processing_errors;

CREATE TABLE target_processing_errors (error_conversation UNIQUEIDENTIFIER, error_number INT,

error_message VARCHAR(4000), error_severity INT, error_state INT, error_procedure SYSNAME NULL,

error_line INT, doomed_transaction TINYINT)

-- Create Initiator and Target side SSB entities

CREATE MESSAGE TYPE SsbMsgType VALIDATION = WELL_FORMED_XML;

CREATE MESSAGE TYPE EndOfStream;

CREATE CONTRACT SsbContract

(

SsbMsgType SENT BY INITIATOR,

EndOfStream SENT BY INITIATOR

);

CREATE QUEUE SsbInitiatorQueue

WITH ACTIVATION (

STATUS = ON,

MAX_QUEUE_READERS = 1,

PROCEDURE_NAME = [initiator_queue_activated_procedure],

EXECUTE AS OWNER);

CREATE QUEUE SsbTargetQueue

WITH ACTIVATION (

STATUS = ON,

MAX_QUEUE_READERS = 1,

PROCEDURE_NAME = [target_queue_activated_procedure],

EXECUTE AS OWNER);

CREATE SERVICE SsbInitiatorService ON QUEUE SsbInitiatorQueue;

CREATE SERVICE SsbTargetService ON QUEUE SsbTargetQueue (SsbContract);

-- SEND procedure. Uses a dialog from the dialog pool.

IF EXISTS (SELECT name FROM sys.procedures WHERE name = 'usp_send')

DROP PROC usp_send

CREATE PROCEDURE [usp_send] (

@fromService SYSNAME,

@toService SYSNAME,

@onContract SYSNAME,

@messageType SYSNAME,

@messageBody NVARCHAR(MAX))

BEGIN

SET NOCOUNT ON;

DECLARE @dialogHandle UNIQUEIDENTIFIER;

DECLARE @sendCount BIGINT;

DECLARE @counter INT;

DECLARE @error INT;

SELECT @counter = 1;

BEGIN TRANSACTION;

-- Will need a loop to retry in case the dialog is

-- in a state that does not allow transmission

WHILE (1=1)

BEGIN

-- Claim a dialog from the dialog pool.

-- A new one will be created if none are available.

EXEC usp_get_dialog @fromService, @toService, @onContract, @dialogHandle OUTPUT, @sendCount OUTPUT;

-- Attempt to SEND on the dialog

IF (@messageBody IS NOT NULL)

BEGIN

-- If the @messageBody is not null it must be sent explicitly

SEND ON CONVERSATION @dialogHandle MESSAGE TYPE @messageType (@messageBody);

END

ELSE

BEGIN

-- Messages with no body must *not* specify the body,

-- cannot send a NULL value argument

SEND ON CONVERSATION @dialogHandle MESSAGE TYPE @messageType;

END

SELECT @error = @@ERROR;

IF @error = 0

BEGIN

-- Successful send, increment count and exit the loop

SET @sendCount = @sendCount + 1;

BREAK;

END

SELECT @counter = @counter+1;

IF @counter > 10

BEGIN

-- We failed 10 times in a row, something must be broken

RAISERROR('Failed to SEND on a conversation for more than 10 times. Error %i.', 16, 1, @error) WITH LOG;

BREAK;

END

-- Delete the associated dialog from the table and try again

EXEC usp_delete_dialog @dialogHandle;

SELECT @dialogHandle = NULL;

END

-- "Criterion" for dialog pool removal is send count > 1000.

-- Modify to suit application.

-- When deleting also inform the target to end the dialog.

IF @sendCount > 1000

BEGIN

EXEC usp_delete_dialog @dialogHandle ;

SEND ON CONVERSATION @dialogHandle MESSAGE TYPE [EndOfStream];

END

ELSE

BEGIN

-- Free the dialog.

EXEC usp_free_dialog @dialogHandle, @sendCount;

END

COMMIT

END;

------------------------------------------------------------------------------------

-- Run application section

------------------------------------------------------------------------------------

-- Send some messages

exec usp_send N'SsbInitiatorService', N'SsbTargetService', N'SsbContract', N'SsbMsgType', N'This is a well formed XML Message1.'

exec usp_send N'SsbInitiatorService', N'SsbTargetService', N'SsbContract', N'SsbMsgType', N'This is a well formed XML Message2.'

exec usp_send N'SsbInitiatorService', N'SsbTargetService', N'SsbContract', N'SsbMsgType', N'This is a well formed XML Message3.'

exec usp_send N'SsbInitiatorService', N'SsbTargetService', N'SsbContract', N'SsbMsgType', N'This is a well formed XML Message4.'

exec usp_send N'SsbInitiatorService', N'SsbTargetService', N'SsbContract', N'SsbMsgType', N'This is a well formed XML Message5.'

-- Show the dialog pool

SELECT * FROM [DialogPool]

-- Show the dialogs used.

SELECT * FROM sys.conversation_endpoints;

-- Check whether the TARGET side has processed the messages

SELECT * FROM MsgTable

TRUNCATE TABLE MsgTable

Using multiple routes in Service Broker

pmarciniak — Mon, 14 Jul 2008 16:31:00 GMT

One of main Service Broker components is routing. Whenever you want your messages to leave the database they originate in, you need to provide routes. Setting up routes may become complicated, so if you're making your first steps in Service Broker area, I suggest staying within single database. Once you have an idea of how Service Broker conversations work, it's time to move one of the communicating services to a different database, or even different server. For that you'll need routes. For a syntax of route creation, see T-SQL reference at https://msdn.microsoft.com/en-us/library/ms186742.aspx. A route is basically a matching between logical conversation endpoint and an address of the machine that hosts the service. The logical endpoint may be specified in two ways:

By giving just the service name
By giving the service name and additionally a broker instance identifier, which ties the route to a specific instance of the service (broker instance ID is just a database identifier in broker terms, so in other words it specifies a database the service is deployed in).

When such mappings (in the form of routes) are defined, each time a message needs to be sent to the specified service (and optionally in the specified database), it will use the address provided.

In this post I would like to cover somewhat more advanced topic of using multiple routes for the same service name. There are two main reasons you would like to do so, namely:

Load balancing
High availability

I'll go over these two scenarios, describing what happens in each case and providing examples.

To understand the rest of this post, you need to be aware that a service of given name may exist in multiple databases. A pair is required to uniquely identify service deployment in a database with this broker instance ID (broker instance ID is simply a guid; I'll just call it "broker ID" from now on). You can specify which instance of target service you want to talk to by providing target broker ID in the BEGIN DIALOG statement. But you may specify just the target service name and in such case the broker ID remains "open", i.e. messages are sent to whichever instance (with whatever broker ID) of the target service is known. Once an acknowledgement comes back from the target service, the target broker ID it carries is set in sys.conversation_endpoints table and all further messages sent from initiator are directed specifically to that broker ID. If a broker receives a message carrying broker ID other than its own, it simply drops it, concluding that it was probably meant for other instance of a service of the same name.

Load balancing

If no broker ID is specified in the BEGIN DIALOG statement and all matching routes specify broker ID, Service Broker will pick one of the available broker IDs from the routing table and direct messages to the chosen broker ID. To avoid distributing messages from a single dialog among different service instances, load balancing doesn't simply pick a route randomly every time it is needed. It employs a mechanism called deterministic routing: each time a message needs to be sent on a dialog started without specifying broker ID and all your routes to the target service are load-balancing routes (i.e. they contain broker IDs), Service Broker performs a hash of dialog ID and, based on that hash, picks one from the set of possible broker IDs. Dialog ID is a parameter that is available from the moment of dialog creation and stays the same during the whole lifetime of a dialog, so as long as the routing table doesn't change while conversations are active, every message of a given dialog will be sent to the same target service instance, because the same broker ID will be always picked up based on the dialog ID hash. Note that all this refers only to messages that are being sent before the first ack comes back from the target. When it does, the target's broker ID is locked at the initiator side and load balancing mechanism is no longer used in the sending process.

Load balancing example

It's now time for an example. Let's assume that we start our dialogs from InitiatorService on a machine named ServerA. The service is deployed in DatabaseA. For the sake of simplicity, let broker IDs in the example be equal to database names (in reality they would be GUIDs that have nothing to do with database names). The target of our dialogs is TargetService, which is deployed in two databases: DatabaseB located on ServerB, and DatabaseC located on ServerC.

For the load balancing mechanism to start working, you need to set up the following routes:

DatabaseA on ServerA:
CREATE ROUTE [LoadBalancingRoute1] WITH SERVICE_NAME = 'TargetService', BROKER_INSTANCE = 'DatabaseB', ADDRESS = 'tcp://ServerB:4022';
CREATE ROUTE [LoadBalancingRoute2] WITH SERVICE_NAME = 'TargetService', BROKER_INSTANCE = 'DatabaseC', ADDRESS = 'tcp://ServerC:4022';
DatabaseB on ServerB:
CREATE ROUTE [ReturnRoute] WITH SERVICE_NAME = 'InitiatorService', BROKER_INSTANCE = 'DatabaseA', ADDRESS = 'tcp://ServerA:4022';
DatabaseC on ServerC:
CREATE ROUTE [ReturnRoute] WITH SERVICE_NAME = 'InitiatorService', BROKER_INSTANCE = 'DatabaseA', ADDRESS = 'tcp://ServerA:4022';

Unless you have dropped the default ‘AutoCreatedLocal' routes in msdb's of all servers, this will suffice. If you followed the recommended practices and dropped the default routes, you'll need the following routes as well to close the routing loop:

msdb of ServerA:
CREATE ROUTE [LocalRoute] WITH SERVICE_NAME = 'InitiatorService', BROKER_INSTANCE = 'DatabaseA', ADDRESS = 'LOCAL';
msdb of both ServerB and ServerC:
CREATE ROUTE [LocalRoute] WITH SERVICE_NAME = 'TargetService', BROKER_INSTANCE = 'DatabaseB', ADDRESS = 'LOCAL';

The BROKER_INSTANCE parts of LoadBalancing1 and LoadBalancing2 routes are REQUIRED. Without them the load balancing mechanism won't work (I'll explain later what happens in such case). Note that I specified broker ID for return routes as well, even though there is only one instance of InitiatorService. It may seem redundant, but it is always a good thing to specify broker ID in a route if it is known at the time of route creation. This may save you from headaches in the future, when something changes in the way services are deployed.

Now, having these routes in place, each time you start a dialog in DatabaseA, one instance of the target service will be chosen randomly with even distribution among the provided TargetService instances and the dialog will be bound to that instance (to be specific, it's not the choice being made that is random - the randomness comes from random values of dialog IDs).

You've got your load balancing, but there are two gotchas you need to remember:

Of course you need to start your dialogs without specifying broker ID of the target service in the BEGIN DIALOG statement. If you do specify it, you're preventing the load balancing mechanism from doing its work of choosing the instance for you.
You cannot have any routes in DatabaseA that point to the target service and do not specify broker ID. Such routes have higher priority for dialogs started without specifying broker ID, so your load balancing routes won't be considered at all. For more information on route matching priority, take a look at https://msdn.microsoft.com/en-us/library/ms166052.aspx.

Unfair load balancing

Deterministic routing, explained before, is a reason why it is impossible to provide "uneven" load balancing, based on machines' processing power. One might think that if ServerC is two times more powerful than ServerB, it would be a nice idea to create routes as follows, doubling the chances of the fast machine for being picked up, and effectively making 2/3 of the traffic hit the more powerful server:

CREATE ROUTE [LoadBalancingRoute1] WITH SERVICE_NAME = 'TargetService', BROKER_INSTANCE = 'DatabaseB', ADDRESS = 'tcp://ServerB:4022';
CREATE ROUTE [LoadBalancingRoute2] WITH SERVICE_NAME = 'TargetService', BROKER_INSTANCE = 'DatabaseC', ADDRESS = 'tcp://ServerC:4022';
CREATE ROUTE [LoadBalancingRoute3] WITH SERVICE_NAME = 'TargetService', BROKER_INSTANCE = 'DatabaseC', ADDRESS = 'tcp://ServerC:4022';

Unfortunately, this won't work, as deterministic routing uses dialog ID hash to choose particular broker ID and not particular route, so the number of routes with the same broker ID doesn't change the probability of that ID being picked up.

High availability

As explained above, providing multiple routes for the same service instance doesn't influence the load balancing behavior in any way. But it is still an important scenario. Why would anyone want to create multiple routes to the same service instance? The answer is availability.

Imagine that you only have single instance of the target service (in DatabaseB on ServerB), but the traffic between ServerA and ServerB needs to go through one of two available forwarders (e.g. network boundary nodes). You don't care which forwarder your traffic goes through, but in the event one of them goes down, you would like the traffic to start flowing through the other one. Here's how Service Broker helps you achieving this functionality. When multiple routes match the target of a dialog, (and it is not a load balancing scenario described above), broker doesn't arbitrarily choose to use one of the routes. Instead it passes all matching routes to the underlying transport layer, which tries to deliver the message utilizing all the routing information it got. This may just mean sending the message on all the routes simultaneously, but may also be based on previous attempts to connect to given target, connection latency etc. You shouldn't assume anything regarding this behavior. It is unspecified and may change without any notice in the documentation. Treat it as a black box that knows what it is doing.

Wait a minute! So shouldn't all "matching routes" be passed to the transport also in load balancing scenario described before? Actually, that's something else. The catch is that load balancing chooses the target broker ID for a dialog first, so only the route with the chosen broker ID is considered a "matching route". If there are two or more routes with the chosen broker ID, they will indeed be all passed to the transport, even in a load balancing scenario.

High availability example

Let's go through an example of how to set up a high availability scenario. As mentioned before, now there is only one instance of TargetService, deployed in DatabaseB on ServerB. Let the two mentioned forwarder nodes be named GatewayA and GatewayB. ServerA and ServerB cannot directly communicate with each other (e.g. due to cross-domain trust relationship issues). The routes that need to be in place are as follows:

DatabaseA on ServerA:
CREATE ROUTE [HighAvailabilityRoute1] WITH SERVICE_NAME = 'TargetService', BROKER_INSTANCE = 'DatabaseB', ADDRESS = 'tcp://GatewayA:4022';
CREATE ROUTE [HighAvailabilityRoute2] WITH SERVICE_NAME = 'TargetService', BROKER_INSTANCE = 'DatabaseB', ADDRESS = 'tcp://GatewayB:4022';
DatabaseB of ServerB:
CREATE ROUTE [ReturnRoute1] WITH SERVICE_NAME = 'InitiatorService', BROKER_INSTANCE = 'DatabaseA', ADDRESS = 'tcp://GatewayA:4022';
CREATE ROUTE [ReturnRoute2] WITH SERVICE_NAME = 'InitiatorService', BROKER_INSTANCE = 'DatabaseA', ADDRESS = 'tcp://GatewayB:4022';
msdb of both GatewayA and GatewayB:
CREATE ROUTE [ForwardingRoute] WITH SERVICE_NAME = 'TargetService', BROKER_INSTANCE = 'DatabaseB', ADDRESS = 'tcp://ServerB:4022';
CREATE ROUTE [ForwardingReturnRoute] WITH SERVICE_NAME = 'InitiatorService', BROKER_INSTANCE = 'DatabaseA', ADDRESS = 'tcp://ServerA:4022';

If your gateway nodes serve as forwarders for multiple services, you may opt for defining more generic TRANSPORT routes on them and naming your services accordingly, so that you don't have to worry about providing connectivity for each specific service pair, thus decreasing the administrative burden.

For the example to work, you will also need the following msdb routes (that you may have already):

msdb of ServerA:
CREATE ROUTE [LocalRoute] WITH SERVICE_NAME = 'InitiatorService', BROKER_INSTANCE = 'DatabaseA', ADDRESS = 'LOCAL';
msdb of ServerB:
CREATE ROUTE [LocalRoute] WITH SERVICE_NAME = 'TargetService', BROKER_INSTANCE = 'DatabaseB', ADDRESS = 'LOCAL';

If, for some reason, you don't know target broker ID at the time of setting the routes at ServerA, it's OK, you may omit the BROKER_INSTANCE = ‘DatabaseB' part and both routes will still be passed to the transport. However, when creating multiple routes for the same service without specifying broker IDs in them, it is very important to make sure they all point to the same instance of the service, merely providing alternative ways of reaching it. Using multiple routes to different service instances without specifying broker ID for each of them may easily lead to problems such as multiple target endpoint creation (described in details below), so you should never do it. I really can't think of a scenario where it would be desired.

Multiple target endpoint creation

Let's see how such multiple target endpoint situation could happen. Imagine that in the initiator database you have routes to two different instances of TargetService (on ServerB and ServerC), but you don't specify broker ID for them, just the service name and address.

Once you begin your dialog and send the first message, both these routes are passed to the transport layer and let's say it sends the message on both of them.
The message hits ServerB first and a dialog endpoint is created there (an entry in DatabaseB.sys.conversation_endpoints shows up).
After some time, the broker in DatabaseB sends an acknowledgement back. The acknowledgement contains the broker ID of the database hosting TargetService on ServerB (i.e. DatabaseB).
After a little while (connection to ServerC might have taken longer to be established) the message transmitted in step 1 hits ServerC and a dialog endpoint is created there as well. ServerB and ServerC don't know anything about each other.
ServerC sends an acknowledgement cointaining its own broker ID.

If the business logic that processes first message of a dialog triggers some external action that should be carried out only once for each dialog, you've already run into problems, because it has been executed twice. But let's see what may happen next.

The ack from ServerB arrives at ServerA and locks the target broker ID for the conversation to DatabaseB.
The ack from ServerC arrives as well, but its broker ID doesn't match the one already fixed for the dialog, so an error is sent back to ServerC.
Now, InitiatorService tries to send second message on the dialog, again both routes are passed to the transport layer, but the transport layer might keep choosing the one to ServerC (perhaps it thinks that ServerB is inaccessible or slow). The message now carries the broker ID of DatabaseB (since the first ack locked it) so ServerC keeps dropping it and the broker conversation cannot continue.

Well, the transport logic will probably try ServerB eventually, but anyway that's certainly not a behavior one's looking for. Note that we didn't provide any matching in the routes between target server addresses and broker IDs, so there is no way for Service Broker to act smart in this case.

So how come this problem doesn't occur in a load balancing scenario? Because of how deterministic routing works. As long as the routing table doesn't change while conversations are active, each message of a given dialog will be sent to the same TargetService instance, so the risk of creating multiple target endpoints is avoided. What if you cannot avoid changing routes when new conversations are being started and you really care not to fall into the multiple target endpoints scenario? Well, you'll have to implement some kind of a three-way handshake and defer executing any business logic at the target side until you get second message from the initiator, because receiving it means that it's your broker ID that has been saved in initiator's sys.conversation_endpoints table.

As you can see, it's always a good idea to provide broker IDs in created routes. The only exception is a situation when the initiator server doesn't know about target server location, number of instances and broker IDs of the target service. In such case there is usually a dedicated node in the topology that takes care of routing, load balancing etc. It is justified to have a route without broker ID in the initiating database, which would delegate all the messages to the intermediate node for processing. Setting the broker ID by the initiator might for example prevent that node from doing load balancing on its own or choosing an appropriate target service instance based on some business logic.

Putting it all together

Finally, let me just quickly mention that it is also possible to combine the two multiple route features: load balancing and high availability. Hopefully that's obvious at this point, but let me just provide a short example of how the routes in the initiator database would need to be created. In this example we'll have two TargetService instances, just as in the load balancing example, but access to each one will be available via two dedicated forwarders: GatewayA, GatewayB for accessing ServerB, and GatewayC, GatewayD for accessing ServerC. The routes in DatabaseA will have to be created as follows:

CREATE ROUTE [LoadBal1Fwd1] WITH SERVICE_NAME = 'TargetService', BROKER_INSTANCE = 'DatabaseB', ADDRESS = 'tcp://GatewayA:4022';
CREATE ROUTE [LoadBal1Fwd2] WITH SERVICE_NAME = 'TargetService', BROKER_INSTANCE = 'DatabaseB', ADDRESS = 'tcp://GatewayB:4022';
CREATE ROUTE [LoadBal2Fwd1] WITH SERVICE_NAME = 'TargetService', BROKER_INSTANCE = 'DatabaseC', ADDRESS = 'tcp://GatewayC:4022';
CREATE ROUTE [LoadBal2Fwd2] WITH SERVICE_NAME = 'TargetService', BROKER_INSTANCE = 'DatabaseC', ADDRESS = 'tcp://GatewayD:4022';

Now, when we begin a dialog and send a message to TargetService, one of the two instances of the service will be chosen in the load balancing process, based on dialog ID hash, as described before. But since two routes match the chosen broker ID, they will both be passed to the transport layer and it will do its heuristic to determine which forwarder to use for sending the message. The multiple target endpoint creation problem doesn't exist in this case. Dialog ID is established between initiator and target and not changed by any forwarding machine, so whichever gateway a message will flow through, it will carry the same dialog ID. Therefore even if the same message is sent to the target server from both gateways, it will be able to recognize that it is the same dialog and receive the message that arrives first, treating the other one as a duplicate, thus dropping it.

Real Time Data Integration with Service Broker and Other SQL Techniques

Jang — Wed, 09 Jul 2008 15:40:00 GMT

This article discusses how to use various SQL technologies to accomplish real time data integration between SQL Server instances. It provides a set of sample code to help users with their development. The document focuses on the usage of each technology which is incorporated into the data integration service. Please refer to provided links for detail information about the technologies.

Real time data integration definition

Real time data integration supports event-driven data movement and transformation between SQL Server instances which host databases with different schemas. The data integration should be transparent to source systems without significantly impacting the systems when events are captured and delivered. The technique also supports an intermediate format which allows decoupling of schemas between source and destination systems. It allows either system to change schemas without breaking the application in the other system. The data integration provides fast and efficient data delivery to a destination in an event-driven model, without polling the source system for new data.

Sales data integration

The real time data integration demo shows the sales data integration between the databases, AdventureWorks (AW) and AdventureWorksDW (AWDW). The data integration service catches the sales data change on AW, and transforms the data in the schema supported in AW onto a general XML format. The service sends the data in XML onto AWDW, and transforms it to correspond to the AWDW schema.

The demo uses the sample databases on SQL Server. Please refer to the link for the detail information about the databases [https://msdn.microsoft.com/en-us/library/ms124659.aspx]. Users can download and install the databases for SQL Server 2008 from the following link [https://technet.microsoft.com/en-us/library/ms124501(SQL.100).aspx].

Techniques

Change tracking

Change tracking provides a mechanism to query for changes to data and to access information related to the changes. This solution provides answer to the following questions. What rows have changed for a user table? What are the latest data in the rows? Change Tracking requires small amount of storage for each changed row, while it only works for getting the latest data. Please refer to the following link for detail information about Change Tracking [https://msdn.microsoft.com/en-us/library/bb933874(SQL.100).aspx]. If an application requires information about all the changes and the intermediate values of the changed data then it should use Change Data Capture (CDC). Please refer to the following document for the comparison of two techniques [https://msdn.microsoft.com/en-us/library/cc280519(SQL.100).aspx]. We plan to write another document which shows how to use CDC as a change tracking option. The following code block shows how to enable Change Tracking on the database and table levels.

ALTER DATABASE AdventureWorks

SET CHANGE_TRACKING = ON

(CHANGE_RETENTION = 2 DAYS, AUTO_CLEANUP = ON)

ALTER TABLE [AdventureWorks].[Sales].[SalesOrderHeader]

ENABLE CHANGE_TRACKING

WITH (TRACK_COLUMNS_UPDATED = ON)

ALTER TABLE [AdventureWorks].[Sales].[SalesOrderDetail]

ENABLE CHANGE_TRACKING

WITH (TRACK_COLUMNS_UPDATED = ON)

Changed data in XML

After setting change tracking on a database and tables the change tables are populated with the data change information when data is inserted, deleted or updated on the tables. The data integration service uses the following code block to fetch the change information and create an XML file with the data change. Using CHANGETABLE function it creates change tracking information for the tables, ‘SalesOrderHeader’ and ‘SalesOrderDetail’. The code generates an XML document containing the information using the FOR XML mode. In the XML file the root, top-level element is named with ‘Sales’, and each sales order header corresponds to an element named with ‘SalesOrderHeader’. A ‘SalesOrderHeader’ element contains one or more ‘SalesOrderDetail’ elements which describe the data change information on the table, ‘SalesOrderDetail’. INNER JOIN clauses make sure that all the change data information is retrieved from the tables.

SET @changeReportXML =

(

SELECT SYS_CHANGE_OPERATION, c_soh.SalesOrderID,

(

SELECT SYS_CHANGE_OPERATION, c_sod.SalesOrderID,

c_sod.SalesOrderDetailID

FROM CHANGETABLE

(

CHANGES [AdventureWorks].[Sales].[SalesOrderDetail],

@last_sync_version

) AS c_sod

INNER JOIN [AdventureWorks].[Sales].[SalesOrderDetail] sod

ON sod.SalesOrderDetailID = c_sod.SalesOrderDetailID

WHERE c_soh.SalesOrderID = c_sod.SalesOrderID

FOR XML PATH ('SalesOrderDetail'),

type, ELEMENTS XSINIL

)

FROM CHANGETABLE (

CHANGES [AdventureWorks].[Sales].[SalesOrderHeader],

@last_sync_version

) AS c_soh

INNER JOIN [AdventureWorks].[Sales].[SalesOrderHeader] soh

ON soh.SalesOrderID = c_soh.SalesOrderID

WHERE @salesOrderID = c_soh.SalesOrderID

FOR XML PATH ('SalesOrderHeader'),

root('Sales'),ELEMENTS XSINIL

);

Change notification

SQL Server provides several mechanisms for notifying data change to an application. For example, Trigger [https://msdn.microsoft.com/en-us/library/ms189599.aspx] and Query Notification (QN) [https://msdn.microsoft.com/en-us/library/ms130764.aspx]. Trigger provides a simple way for the notification, while only supporting synchronous mechanism. QN supports asynchronous notification and rich filtering semantics. However QN cannot be configured in TSQL within SQL Server. In the Real Time Data integration demo we use a technique integrating Service Broker and Trigger. It provides a simple way to support event notification implementing asynchronous semantic in TSQL within SQL Server. The following code block shows the event notification on the demo.

CREATE TABLE ConversationHandle

(conversationHandle uniqueidentifier);

--Create a dialog to send all the transactions on

BEGIN TRANSACTION

DECLARE @conversationHandle uniqueidentifier

--Create a new conversation on the table

BEGIN DIALOG @conversationHandle

FROM SERVICE AsynchTriggerInitiatorService

TO SERVICE N'AsynchTriggerTargetService'

ON CONTRACT [AsynchTriggerContract]

WITH ENCRYPTION = OFF;

INSERT ConversationHandle (conversationHandle)

VALUES (@conversationHandle)

COMMIT;

-- TRIGGER for initiating the change tracking demo

CREATE TRIGGER ChangeTrackingTrigger

ON [AdventureWorks].[Sales].[SalesOrderHeader]

AFTER INSERT, DELETE, UPDATE

BEGIN TRANSACTION;

DECLARE @conversationHandle uniqueidentifier;

SELECT TOP (1) @conversationHandle = conversationHandle

FROM ConversationHandle;

SEND ON CONVERSATION @conversationHandle

MESSAGE TYPE [AsynchTriggerMessageType]

COMMIT;

Reliable data movement

Service Broker provides asynchronous and reliable data movement. It supports TSQL programming model built on SQL Server database engine. Please refer to the following link for the detail information about Service Broker [https://technet.microsoft.com/en-us/sqlserver/bb671396.aspx].

The data integration service uses multiple conversations for message delivery to increase throughput. Using multiple dialogs brings the data parallelism on the receiving side. Multiple threads can receive and process the messages in the dialogs independently. However, initiating the conversations brings load to a system. Therefore right amounts of conversations should be chosen smartly. In the real time data integration demo we choose four conversations for processing the messages with high throughput. In the real time data integration service we initiate four dialogs, and store them onto a table. The demo uses the dialogs for sending messages about changed data information. Please refer to the following code block for the dialog creation. The following code blocks present the procedure for sending an XML message using Service Broker. The procedure uses the four conversations evenly distributed messages based on the sales order ID ( SET @dialogHandleID = @salesOrderID % 4) . Because messages for the same sales order ID are delivered in a single conversation it is guaranteed that the messages are delivered exactly once in order manner.

CREATE PROCEDURE SendChanges

BEGIN

DECLARE @last_sync_version bigint;

DECLARE @salesOrderID bigint;

DECLARE @dialogHandleID INT;

DECLARE @dialogHandle uniqueidentifier;

DECLARE @changeReportXML XML;

DECLARE @next_baseline bigint;

DECLARE @TotalDialogs INT;

DECLARE @logMsg VARCHAR(MAX);

BEGIN TRANSACTION;

SELECT TOP (1) @last_sync_version = lastVersion

FROM LastVersion;

SET @TotalDialogs = 4;

--Create a cursor on the change table for [SalesOrderHeader]

DECLARE cursorChangeOrderHeader

CURSOR FORWARD_only READ_ONLY

FOR SELECT SalesOrderID FROM

CHANGETABLE (

CHANGES [SalesOrderHeader], @last_sync_version) AS Cursor_CH

ORDER BY SYS_CHANGE_VERSION;

--Open the cursor on the change table for [SalesOrderHeader]

--Loop for each changed sales order id

OPEN cursorChangeOrderHeader

WHILE(1=1)

BEGIN

FETCH NEXT FROM cursorChangeOrderHeader INTO @salesOrderID;

--If there is no more changed sales order then exit

IF (@@FETCH_STATUS != 0) BREAK;

--<Fetching changed data and creating XML file.>

--Find the conversation handle for the sales order

--from the dialog handle table

SET @dialogHandleID = @salesOrderID % @TotalDialogs;

SELECT @dialogHandle = dialogHandle

FROM DialogHandles

WHERE ID = @dialogHandleID;

--Capture last version info

SELECT @next_baseline = SYS_CHANGE_VERSION

FROM CHANGETABLE (

CHANGES[AdventureWorks].[Sales].[SalesOrderHeader],

@last_sync_version) as c_soh

WHERE @salesOrderID = SalesOrderID;

--Send the message using Broker

SEND ON CONVERSATION @dialogHandle

MESSAGE TYPE [RealTimeDImessagetype](@changeReportXML);

END

CLOSE cursorChangeOrderHeader;

DEALLOCATE cursorChangeOrderHeader;

UPDATE LastVersion SET lastVersion = @next_baseline;

COMMIT;

END;

Activation

Activation allows message processing logic to be launched when a message arrives on a Service Broker queue. When an internal activation is used for processing messages a stored procedure is declared on a Service Broker queue, and invoked on a background thread when a message arrives. A user can also specify an executable for processing the messages as an external activator. For example, SQL Server Integration Services (SSIS) can be used as an external activation procedure to process messages. Please refer to the following link for the code sample and document of External Activator [https://www.codeplex.com/SQLSrvSrvcBrkr/Release/ProjectReleases.aspx?ReleaseId=3853].

In the real time data integration services demo we use internal activators to process event notification messages on the initiator service as well as changed data information messages on the target service. We briefly mention how the services process the messages in the activation procedures.

Message processing in the initiator

The real time data integration initiator handles messages from two different services. One of the services is an asynchronous event notification service, and the other is a real time data integration target service. A single service in the initiator handles the message from the two different sources based on message types and service names. The following pseudo-code block describes the message processing logic in the initiator.

WHILE there is any message on ‘RealTime_DI_Initiator_queue’

RECEIVE a message FROM the queue

IF message type is ‘EndDialog’ THEN

END CONVERSATION

ELSE IF message type is ‘ERROR’ THEN

IF service name is ‘RealTime_DI_Initiator_Service’ THEN

Raise error;

Create a new dialog;

Resend pending messages using the dialog;

Replace old dialog with the new one;

END CONVERSATION (old dialog);

IF service name is ‘Asynchronous_Trigger_Target_Service’ THEN

Raise error;

END CONVERSATION;

ELSE IF message type is ‘Asynchronous triggering’ THEN

RECEIVE

WHERE conversation_handle is identical with this message’s handle

EXEC SendChanges PROCEDURE

Message processing in the target

The message processing procedure in the real time data integration target receives messages from a target queue, and transforms the messages from the XML format into a supported schema. A simple and straightforward way to process messages is to receive a message from the queue and to transform it one by one until all the messages are processed on the queue. However, the mechanism may hurt the performance of the data integration target. Instead of receiving a single message and transform it the data integration target service uses a cursor-based processing mechanism. It receives all the messages from the target queue, and stores in a temporary table. A cursor iterates the table to fetch a message and process it to covert from an XML format to a desired schema. The following code block shows the activation procedure on the target.

CREATE PROCEDURE ProcessMessagesDW

BEGIN

DECLARE @handle uniqueidentifier;

DECLARE @messageBody XML;

DECLARE @tableMessages TABLE(

queuing_order BIGINT,

conversation_handle UNIQUEIDENTIFIER,

message_body VARBINARY(MAX));

DECLARE cursorMessages CURSOR

FORWARD_ONLY READ_ONLY

FOR SELECT conversation_handle,

message_body

FROM @tableMessages

ORDER BY queuing_order;

WHILE(1=1)

BEGIN

BEGIN TRANSACTION;

WAITFOR(RECEIVE

queuing_order,

conversation_handle,

message_body

FROM [RealTimeDItargetqueue]

INTO @tableMessages), TIMEOUT 1000;

IF(@@ROWCOUNT = 0)

BEGIN

COMMIT;

BREAK;

END

OPEN cursorMessages;

WHILE(1=1)

BEGIN

FETCH NEXT FROM cursorMessages

INTO @handle, @messageBody;

IF(@@FETCH_STATUS != 0)

BREAK;

END

CLOSE cursorMessages;

DELETE FROM @tableMessages;

COMMIT;

END

DEALLOCATE cursorMessages;

END

Data transformation

After receiving messages the target service transforms the received messages, and populates tables with the changed data information from the messages. The received messages are in XML format. The service processes each of the messages to obtain required information from the message using TSQL language coupled with integrated XML support. The following code block shows a sample of the transformation using TSQL. On this example, the transformation is occurred only for the data insert event.

INSERT INTO [AdventureWorksDW].[dbo].[FactInternetSales]

SELECT

N1.SOH.value('CustomerID[1]', 'int')

AS [CustomerKey]

,N2.SOD.value('SpecialOfferID[1]', 'int')

AS [PromotionKey]

,N2.SOD.value('CarrierTrackingNumber[1]','NCHAR(9)')

AS CarrierTrackingNumber

,N1.SOH.value('PurchaseOrderNumber[1]','NVCHAR(25)')

AS [CustomerPONumber]

FROM

@messageBody.nodes('/Sales/SalesOrderHeader') N1(SOH)

CROSS APPLY soh.nodes('SalesOrderDetail') N2(SOD)

WHERE

N1.SOH.value('CustomerType[1]', 'CHAR') = 'I'

AND N2.SOD.value('SYS_CHANGE_OPERATION[1]','CHAR')='I'

SQL Server Integration Services (SSIS) also provides the data transformations. SSIS supports various forms of data transformation between heterogeneous sources. Please refer to the following link for more detail information about SSIS [https://technet.microsoft.com/en-us/sqlserver/bb671392.aspx].

This document discusses about real-time data integration technologies with the coordination of a set of powerful SQL Server technologies. The service provides reliable and transparent data integration between instances. The service is composed with the following technologies.

Data tracking: Change tracking

Change notification: Triggers and Service Broker

Reliable data movement: Service Broker

Activation: Internal, Blocking with WAITFOR RECEIVE

Transformation: TSQL with XML support

The complete code list for the demo can be found on the following link [https://www.codeplex.com/SQLSrvSrvcBrkr/Release/ProjectReleases.aspx?ReleaseId=15139].