On the other hand, if you want to develop a custom application to load data to Azure Data Lake Store, you could leverage the Data Lake Store .Net SDK to load data. Refer to this article for more details: https://azure.microsoft.com/en-us/documentation/articles/data-lake-store-get-started-net-sdk/. 

First is that it needs additional precision/scale information when passing decimal type data, otherwise the decimal data would be rounded like long type value. It is very hidden as there is no exception thrown, and we can only find it by manually checking the data. In the meantime, as ADO.Net DataTable class doesn't contain precision/scale information, it means you cannot directly pass the DataTable as the parameter. You may either use IEnumerator<SqlDataRecord> or use DbDataReader created from DataTable and reset the schema table. Refer to this article which also discusses such issue.

Second is that the data is populated by position instead of column names, although you have column names defined in DataTable/DbDataReader and column names defined in table type. As there is no exception thrown, it is hard to realize if the data is corrupted for incorrect orders. Refer to this article which also discusses this issue.

When meet such warning, the first thing is to identify which assembly has conflicts. To do that, you just need to add a property in msbuild, e.g. "msbuild /fileLoggerParameters:LogFile=MyLog.log;Verbosity=detailed". The detailed build logs would be written to the file, which contains a log like "There was a conflict between "<assembly>, Version=<version1>" and "<assembly>, Version=<version2>"." to tell you the conflict assembly.

For msb3276, I can repro the build warning in Visual Studio 2013. You could add an assembly binding "<bindingRedirect oldVersion="0.0.0.0 - 2.0.0.0" newVersion="2.0.0.0"/>" in app.config/web.config to resolve the warning. Refer to https://msdn.microsoft.com/en-us/library/eftw1fys(v=vs.110).aspx. Or set the "AutoGenerateBindingRedirects" property to true in the project file. For more information, see https://go.microsoft.com/fwlink/?LinkId=294190. This warning can be reproed with the following dependencies trees:

           Assembly C V1.0.0.0

                      |

                 Project B                  Assembly C V2.0.0.0

                      |                               |

                      Executable Project A

For msb3277, I can repro the build warning in Visual Studio 2013. I found two kinds of situations could cause this warning. One case can be resolved by assembly binding, but the second one cannot.

(1)  The first case can be resolved by adding an assembly binding "<bindingRedirect oldVersion="0.0.0.0 - 2.0.0.0" newVersion="1.0.0.0" />", but cannot be resolved by setting the "AutoGenerateBindingRedirects" property. This case can be reproed with the following dependencies trees:

           Assembly C V2.0.0.0

                      |

                 Project B                  Assembly C V1.0.0.0

                      |                               |

                     Executable Project A

 (2) For the second case, it seems no easy way to resolve the warning but to use the same version of assembly instead. This case can be reproed with the following dependencies trees:

Assembly C V2.0.0.0 

              |

Assembly D           Assembly C V1.0.0.0

                |               |

                 Project B                  Assembly C V1.0.0.0

                      |                               |

                      Executable Project A

For msb3247, I can repro the build warning in Visual Studio 2012 <lower version may also repro>. The above three kinds of dependencies trees would cause the same warning msb3247 in VS2012, so such warning code cannot tell you whether you can resolve it with assembly binding. However, there is a difference that if the warning messages contain a hint like "Consider app.config remapping of assembly... ", it is like the msb3276 in VS2013 and you could resolve it by assembly binding. Otherwise, it is like the msb3277 in VS2013 and there are two situations/solutions.

1. Download PsTools from https://technet.microsoft.com/en-us/sysinternals/bb897553.aspx.
2. Copy the bits to the VM.
3. Run command "psexec -i -s mmc" to view the wanted certificates.

PsTools is also a convenient way to let you run your program as System account. Just run "psexec -i -s cmd.exe", and do anything as System account in the popup command line.

This article aims to introduce the third way, which takes advantage of Data-Driven Test for such requirement.

The following Method 3 is a sample to use Data-Driven Test. It would logically split a test case into several sub test cases. To some extend, such method doesn't have the short comings in the former two methods.

But I think there still exists some open issues for Method 3. Would further investigate whether there is a better solution.

• The first issue is the test case display name. With method 3, the sub test cases' names would be like "TestTransfer (Data Row 0)" and "TestTransfer (Data Row 1)", which are not informative.
• The second issue is that the xml file may be also duplicate, if I have several kinds of parameters and want to test all combinations of them.
• E.g. if there is another parameter "DataType", the xml file would be like "1024, txt", "1048576, txt", "1024, jpg", "1048576, jpg".

Method 1: Create new test methods for each variation. E.g.

        [TestMethod()]
        public void TestTransfer_1KB()
        {
            TestTransfer(1 * 1024);
        }

        [TestMethod()]
        public void TestTransfer_1MB()
        {
            TestTransfer(1 * 1024 * 1024);
        }

        private void TestTransfer(long dataSize)
        {
            // ...
        }

Method 2: Put all variations into a single method. E.g.

        [TestMethod()]
        public void TestTransfer()
        {
            var allDataSize = new long[] { 1 * 1024, 1 * 1024 * 1024 };
            foreach (var dataSize in allDataSize)
            {
                TestTransfer(dataSize);
            }
        }

Method 3: Data-Driven Test. E.g.

        public TestContext TestContext { get; set; }

        [TestMethod()]
        [DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML", "TransferTest.xml", "Transfer", DataAccessMethod.Sequential)]
        public void TestTransfer()
        {
            TestTransfer(long.Parse(TestContext.DataRow["DataSize"].ToString()));
        }

TransferTest.xml:

 <Settings>
  <Transfer>
    <DataSize>1024</DataSize>
  </Transfer>
  <Transfer>
    <DataSize>1048576</DataSize>
  </Transfer>
</Settings>

I have done a performance comparison between the Blosc and the .Net built in Deflate compression algorithm.

Here is my PC environment:

• CPU: Intel Xeon CPU E5-1620 3.6 GHz
• RAM: 16.0 GB

For Blosc, I use the following settings:

• Number of threads: 1
• Set as single thread for fair comparison. Using multi-threads will improve the throughput a lot.

• Compression level: 9

• It is the desired compression level and must be a number between 0 (no
  compression) and 9 (maximum compression).

• Whether do shuffle: No

• Type size: 8

Test BaseLine:

• Implement a mock data reader. The pure read throughput is several GB/S, which won’t be the bottleneck for DB insertion.
• The schema is 4 columns of string, and each field is a Guid.
• The data size is set as 1.6 GB with around 10 million rows.
• Use SqlBulkCopy.WriteToServerAsync(reader, token) to insert data, and hard code BatchSize as 1000.
• The throughput is 14 MB/S. 

Improvement 1:

• Use the sync API in SqlBulkCopy instead of the async API.
• The improved throughput is 30 MB/S.
• The performance of sync API is better than the async one, which is same as my previous finding in SqlDataReader.

Improvement 2:

• Use an adaptive BatchSize logic based on row size, instead of a hard coded value.
• My logic is to set the BatchSize as (8 MB)/(row size), where the row size can be estimated in our application.
• The further improved throughput is 42 MB/S.

Improvement 3:

• Set the target database as bulk-logged recovery mode, and use table lock for SqlBulkCopy. Refer to this article: https://msdn.microsoft.com/en-us/library/ms190422%28v=sql.105%29.aspx
• The further improved throughput is 60 MB/S.

Row size and Performance:

• Did some tests based on different schema. The performance can reach to 60 MB/S if the row size is larger than 100 bytes.
• For those very small rows (e.g. only contain 4 int value), the performance will be downgraded to 35 MB/S.

1. Six Do's
1. Do the right thing  
1. Process/PMs
2. Do it right
1. Understanding
3. Do it efficiently
4. Do it safely
5. Do it predictably (in estimated time)
1. The above three items is about design, which is developer's responsibility.
6. Do it sustainably
1. Manager
2. Discussion: possible issues in a project
1. External dependencies
1. The dependencies change.
2. Tight schedule
1. Scope change
3. Requirement change/add
4. Quality problems
5. Overly complex design
6. Conclusion: most are caused by design change
3. Discussion: the difference between procedural design and object oriented design
1. Procedure consider two concepts: behavior (procedure) and data
2. OO consider two concepts: entities and relationship, where each entity contains behavior and status (data).
4. Three magic questions to know whether a design is good
1. How can I test it? 
1. The easier the better
2. What do I need to know (if change it)?
1. The less the better
3. What do I need to do?
1. The less the better
5. Two questions to know whether an interface is good.
1. Assertive or Inquisitive?
2. Stable or Unstable?
6. Most comments are not needed.  Here is the six kinds of comments and only the sixth is valuable.
1. Not a comment.  
1. e.g. author, time (version controller need to cover such information instead of comments)
2. Useless  
1. e.g. repeat the method name
3. Fear  
1. Comment the legacy code instead of deleting it.
4. What
1. Explain what the code do. The good method name should explain.
5. How
1. Explain how the code do. The clear code implementations should explain.
6. Why
1. Only this is valuable as the code cannot show it.
7. Three steps to look into a design diagram
1. Look for relationship
1. Static: inheritance
2. Dynamic: interface
2. Abstract class: variation
3. Interface
1. Stable or unstable
8. Three essential skills of a  developer
1. Programming   =  Design    -> Implementation
2. Design             = Problem -> Design
3. Refactor          = Design    -> Another Design
9. The "Constants.cs"  file is a bad thing
1. It has poor cohesion.
1. Break the second magic question: need to know the whole design of the system if need to change the file.
10. Criterions to measure code quality
1. Clarity  
1. Good names. No need to explain code with comments.
2. No Redundancy
1. Redundancy: A change to one thing would necessitate a similar change to all the redundant ones.
3. Strong Cohesion
1. "One Responsibility in One Place"
4. Intentional Coupling
1. The technical nature of the relationship between two entities.
2. Good coupling: intentional. Vs. Bad coupling: accidental.
3. Encapsulation aims to improve coupling and cohesion   
1. Hidden things cannot be coupled to. 
2. Cohesive issues are easier to hide.
3. Need a balance between encapsulation and redundancy.
5. Focus
1. The locality of responsibility.
2. Need a balance between focus and cohesion.
11. Context of a design:

12. There are two separated aspects to design a system. Never mixing the two aspects when doing design.
1. How the system runs?
1. Design the interface/relationship between objects.
2. How to construct the system?
1. Factory is responsible for construction.
13. Design is about the problem, not the implementation.
14. Advice from the Gang of Four
1. Design to interfaces
2. Favor object delegation over class inheritance
3. Consider what should be variable in your design … and "encapsulate the concept that varies"
15. The Strategy Pattern
1. We need a different behavior (variation) at different times (dynamic) for different clients (reuse) requesting the service.
2. GoF Intent: Define a family of algorithms, encapsulate each one, and make them interchangeable. Strategy lets the algorithm vary independently from clients that use it.
16. Commonality-Variability Analysis
1. Commonality analysis is the search for common elements that helps us understand how family members are the same.
2. Those things that are found by commonality analysis can often become the abstract types.
17. The Template Method Pattern
1. GoF Intent: Define the skeleton of an algorithm in an operation, deferring some steps to subclasses. Template Method lets subclasses redefine certain steps of an algorithm without changing the algorithm's structure.
2. E.g. Commonality: the steps of several algorithms. Variation: each specific algorithm
18. The Bridge Pattern
1. GoF Intent: De-couple an abstraction from its implementation so that the two can vary independently.
19. The Adapter Pattern
1. We need to change the interface of a class that has the right stuff but has the wrong interface.
2. GoF Intent: Convert the interface of a class into another interface clients expect. Adapter lets classes work together that couldn't otherwise because of incompatible interfaces.
20. The Façade
1. We want to use a subset of the functionality of a complex system, or to simplify its interface, or to improve how it is used, or to provide OO access to a non-OO system.
2. GoF Intent: Provide a unified interface to a set of interfaces in a subsystem. Façade defines a higher-level interface that makes the subsystem easier to use.
21. The Analysis Matrix
1. Motivation: Information overload:
1. More information than you need.
2. No good way to organize it.
3. No way to tell what is important.
4. No way to see what the cases are.
2. Each column represents a given case, Each row is used to represent the concept that case is an example of. Build the matrix as we go.
22. The Abstract Factory
1. We need to create families of objects for particular situations. That is, particular clients need particular sets of instantiations.
2. GoF Intent: Provide an interface for creating families of related or dependent objects without specifying their concrete classes.
23. The Singleton Pattern
1. GoF Intent: Ensure a class only has one instance, and provide a global point of access to it.
24. The Family of Proxy Patterns
1. We need to add functionality before or after accessing an object in such a way that the client object doesn't know it is talking to a different object.
2. GoF Intent: Provide a surrogate or placeholder for another object to control access to it.
3. Different kinds of proxy patterns
1. The Protection  Proxy Pattern
2. The Cache Proxy Pattern
3. The Virtual Proxy Pattern
4. The Remote Proxy Pattern
25. The Decorator Pattern
1. GoF Intent: Attach additional responsibilities to an object dynamically. Decorators provide a flexible alternative to sub-classing for extending functionality.
2. A common example of decorator pattern is the Streaming IO: FileStream, MemoryStream, CompressionStream…
26. Chain of Responsibility
1. Have a set of objects that can perform a task but the requester of the task does not want to know which one actually does it.

Here is the different codes for the two APIs:

• DownloadToStream: blob.DownloadToStream(stream);
• OpenRead:                var blobStream = blob.OpenRead();      blobStream.CopyTo(stream);

The following table gives the test result of different APIs. The duration is the average of 10 times' running.