1. A) Weekly Data processed: To show scale of the platform
2. B) Weekly cost per TB of Data Processed: To show efficiency of the platform.
3. C) Availability: Platform is available X % in a week.

• Monitoring (is my system up & running, understand peak load patterns)
• Understanding the usage pattern of customers
• Derive insights which helps them identify areas to grow business
• A/B test etc.

• Instrumentation framework team : Write framework which can be used by product or feature teams to log things consistently
• Logs Collection team : Write framework which can go to client, servers, api's and collect logs into Big Data environment.
• Big Data environment : Ecosystem of Hadoop (Map Reduce) and Hive etc.
• Data Processing team (typically called as DataWarehouse team) :
  • Once data is made available in Big Data environment, write code to shape data from unstructured to structured , apply schema adjustors etc.
  • Once data is available in structured shape, apply business logics, process it to different aggregates ( and/or traditionally facts or dimensions)
  • Build DataWarehouse in either big data environment (Hive) or Push data to Massively Parallel Processing (MPP) systems. Typically MPP systems have concurrency limits so you will need to scale out them with multiple clusters but they give you good query experience. Some companies keeps their DataWarehouse in Hive & do not invest in MPP systems layers. I think both have advantages and disadvantages.
• OLAP layer : Build OLAP layer for self service BI

EXEC('INSERT INTO DatabaseName.SchemaName.Tablename values (1)') AT [PDWLinkedServer]

This blog is focused on testing aspects of ETL(Extract, Transform, Load) part of Business Intelligence (BI) Applications. Lets talk a bit about BI Application first to bring everyone on same page and then we will go into testing strategy. Typically BI Application means building a DataWarehouse, then some DataMarts and finally a Reporting layer which could be Self Service BI (Eg. Microsoft Power BI) or Canned reports (Eg SSRS) or Power Reports. There are several others less common ways like building a UI layer to present data or futuristic ways of building a Windows 8 App as reporting layer. Other important thing to understand about BI systems is they are 'after the fact' reporting systems. 

Testing ETLs have always been challenge, complicated and always keeping test managers on toes.

Below diagram depicts sequence of steps to be followed to test ETLs. This is purely around test execution part. Refer to this blog post to learn more about MiniDB concept.

Hope this helps you, enjoy testing ETLs.

-Naren

Views or Opinions expressed in this blog represent only author and not any company or organization.

Percentage of Revenue by Region

Distribution of Sales by Region

No. of defects per release

Ever wondered what leadership or business teams would do by seeing these metrics. What actions are taken, if  a metric is up or down in an organization.

Some time, we can make a good guess if we know the system or business process well but then its not documented well that, that's exactly a CEO or CFO would do, when he scans and finds in a BI report that revenue is down by few percentages.

I feel, Metrics need to be followed up with at least 3 action items based on that number otherwise its just loses the very essence & context of initial vision and business value.

As a engineering team member which built BI applications, I feel there is a section or rule missing in the reports. 

"There is no section which talks about next steps if a metric is up or down"

Its always left to interpretation of the viewer to come up with his own action or set of changes in the business processes. I feel, these metrics can be more effective, if during requirements stage or in the course of development, business users can come up with at least 3 ACTION ITEMS for each metrics performance. These would provide the perspective a metric needs to be looked from, and more importantly, what's the next steps :-) , these next steps would be changing over course of change in business processes.

 Imaginary example:-

A CFO views BI report metric 'Revenue by Geography' and notices that revenue is down by 5%. Now example next steps could be:-

-If this metric is down by 5% or more, then immediate meeting is needed and business owner of BI application needs to schedule it. Outcome would be steps to increase revenue in region.

-If this metric is down by  less than 5%, then business owner would need to come up with a plan to increase revenue in region within next 3 weeks.

-If this metric is up by 5% or more, business owner should do drill down into more areas and share best practices with other folks in next 3 weeks.

-If this metric is up by less than 5%, than this is good news and normal business expectations are met.

Hope you enjoyed reading the post and walk away with some thoughts for the day!

 Let’s get started:-

Connect to SQL server instance and create a simple table in any of your test or playground database as.

CREATE TABLE UserDetails
 
 (
 
 Id INT IDENTITY(1,1),
 
 UserName VARCHAR(6),
 
 Pwd VARCHAR(32),
 
 SSNNumber SMALLINT,
 
 Addres NVARCHAR(100),
 
 ZipCode smallint
 
 )

Let’s insert some data into this table.

INSERT INTO UserDetails
 
 VALUES
 ('Shubh'
 ,'5f4dcc3b5aa765d61d8327deb882cf98'
 ,123
 ,'House Number1, Street Number1, Apt Number1, ZipCode1, State1'
 ,12345),
 ('Khushi'
 ,'6f4dcc3b5aa765d61d8327deb882cf97'
 ,456
 ,'House Number2, Street Number2, Apt Number2, ZipCode2, State2'
 ,17891),
 ('Ashi'
 ,'7f4dcc3b5aa765d61d8327deb882cf96'
 ,789
 ,'House Number3, Street Number3, Apt Number3, ZipCode3, State3'
 ,23456)

 Now, assume this table is used to store information about registered users of a website.

Next a business users comes in to website and searches for customer name using username as Shubh and in the background below SQL query is run to get the results for that specific username. (Assume that business user is authorized from website security perspective to search a username). 

SELECT * FROM UserDetails WHERE UserName = 'Shubh';

Output displays below data, which is expected, all is well:

Now, if a user enter input as 'OR 1=1--

This makes the query look like this, which means SQL injected by manipulating way you enter value in search box:

SELECT * FROM UserDetails WHERE UserName = ''OR 1=1--';

And the result is a disaster (S1/P1 bug), as it shows all users and their information.

Try out the above example and hope this helps! Important thing is to always test for SQL injection from a tester perspective, and push for input validations in code.

Is your team building Windows 8 Modern App for customers? As part of QA team let me assure you it’s a good  thing to be in that project! But we ready to get exposed to new things to test, its lot of interesting stuff for QA community to learn and quickly come to terms of changing software trends.

Testing a Windows 8 Modern Applications is interesting and happening space. Having tested couple of these applications made me write this post to share some of the things which are different and need more attention than normal testing approach.

As much as tester is required to understand the intended functionality of the Win8 apps, they need to understand Win 8 Modern App principles themselves thoroughly to really test a true Win 8 app than just any HTML5/XAML application.

Watch here about Windows 8 Modern Application guidelines, you will find many bugs by just validating against these guidelines during tests.

Testing data in your application would be specific to requirements, however some of the important things to consider during different phases of development are listed below in tabular format. This list focuses on QA tasks which are important or different than conventional application testing.

Hope this would help QA community to test Windows 8 Modern Apps better and delight customers!

Happy New Year and Wish you happy reading.

-Naren

Views or Opinions expressed in this blog represent only author and not any company or organization or person.

So i started to think back and re-looked at defination of Regression Testing, top of world's top search engines bring up this page as first result (Bing and Google).

https://en.wikipedia.org/wiki/Regression_testing

I looked at history of page and it was last edited on 29-March-2012.

Lets look at defination...

Regression testing is any type of software testing that seeks to uncover new software bugs, or regressions, in existing functional and non-functional areas of a system after changes, such as enhancements, patches or configuration changes, have been made to them.

The intent of regression testing is to ensure that a change, such as a bugfix, did not introduce new faults.^[1] One of the main reasons for regression testing is to determine whether a change in one part of the software affects other parts of the software.^[2]

Common methods of regression testing include rerunning previously run tests and checking whether program behavior has changed and whether previously fixed faults have re-emerged. Regression testing can be used to test a system efficiently by systematically selecting the appropriate minimum set of tests needed to adequately cover a particular change.

I for some reason dont agree to this defination for reasons explained further, this sounds like assuming many things which have evolved in test space.

- Testing technique is mostly Blackbox

-Tester does not knowledge of internal system in terms of they dont look into code to know dependency of objects or dont do any impact analysis while writting Test Plan

- They wait till Regression test cycle at end of Test Phase to identify if system have any new bugs or regression bugs

I feel these assumptions are not so true nowourdays, with Test organizations moving towards more of SDET profile and more focus on White Box techniques like code reviews by Test team, Code Coverage techniques, Peer reviews, Inspections at all stages; testers are better equipped to analyze and make decision which areas to regress in the application rather than regressing the entire application. Let me try to explain what i mean with visual diagram.

Lets say your application consists of below modules from A to F, you had regression suite with say 300 test cases which have coverage for all areas, and it costs 10 days to run that regression suite.

Now in next release C has been modified to C++, which impacts A and F, due to indirect impact or data flow lets say it also impacts B not less than A and F. So below diagram shows areas which are impact due to this change in yellow or light yellow.

Now if we go by definations which had assumptions i listed earlier, i would still need same effort and resource for regression testing.

Here comes problem, i dont agree to that concept. I feel now with tools, techniques, more SDET skills, i know that only A, B and F are affected due to change in this release and it has no what so ever impact on E and D. So I should be testing only A, B and F in regression testing, which is going to save time for me and make my project manager more happy :-)

Ofcourse my approach assumes that you have the skills and knowledge to identify A, B, F as impacted areas with confidence in first place but thats where we are heading isnt it.. Testers are getting more skilled day by day with lots of focus on technical skills and domain knowledge.

We should get better at doing regression testing with evolution of testing space, get smarter and bring in more productivity every time we do something, do your regression cycles faster and target right areas which would yield potential bugs rather than executing same regression suite which was never re-visited and keep on taking standard days forever!!!

-Naren

Views or Opinions expressed in this blog represnt only author and not any company or organization or person.

I am sharing top 3 reasons for automation failure in this post which is kind of little modified from my post on blogspot  @ https://infomine-software.blogspot.com/#!/2010/10/why-does-test-automation-fail-every-now.html:-

1-  How many tester's are really tester, may be 50%, rest 50% were circumferenced into testing world. (Dont beat me for that:-))

A good tester generally does not like coding, he/she likes to be always in test execution phase, hitting severity one bugs all the time..... bottom line, A good tester generally does not like creating scripts. So how do we motivate such folks to start writting automation scripts/code, well show them value of automation to find bugs, that may just fly with them.

 2-  Well done, we have got the automation done - date X.................................................................Scripts are not working, due to environment or other issues on date X + 2 months..........Most of us will be able to relate to this scenario, why do scripts fail, because they were probably poor in the first place. Person maintaining scripts are not deep into ocean of automation. Its hard but thats how it goes down!!!

You need deep knowledge of any tools or technology you use for automation, folks floating on surface can give few demo's or write good paper's but for a long term automation ROI, you need someone to do automation design for you whom you can call as SME in say Coded UI or Middle tier web services. Having someone in the automation team who can lead that R&D effort will save from lot of re-work in future.

3- Biggest tempation for a tester working on test automation is technology and coding. With little code understanding automation tester's make mistake of targeting low hanging fruits. So it works for a release and then die's or involves so much manual effort before running it, that there is absolutely no ROI.

Automate for ROI, without productivity gains automation is waste. Future of Automation is in ROI to start with, next level is to find bugs with automation. Keep that in mind from very begining, low hanging fruits are not always a good choice to run after especially when it comes to automation.

My vision of automation goes something like below:-

Good Tester's write testcases in a predefined manner (or using some test case writting tool) and say Run, he/she never has to worry about code. So who handles code, a dedicate person from test team who works with Dev team to understand their code structure, suggests them to insert some hooks, which will be used by automation framework. This automation framework just calls the hooks and results scripts from test cases.

From a management perspective if i have to implement above vision, i would have folks who have good knowledge of automation as directors and good testers as actors to build a successful automation story!!!

-Naren

Views or Opinions expressed in this blog represnt only author and not any company or organization or person.

Test teams are often asked, "How do you plan to improve test productivity?" AND there is an opportunity in this question for every test organization!!!

Well, there is no one answer for it!!!

On an average overall test activities effort is around 50% of development effort, which can definitely be improved. Test teams needs to start moving towards more tailored test processes which suit their business model.

A key challenge for any organization is always how to reduce COST for customer. Today the customers can be made happier

using cost reduction rather than giving ornamental features.

COST, Timelines and Quality are three most important priorities for business in today’s highly competitive market. If we don’t reduce cost for customers, they will do it in their own way!!! As the IT industry faces challenges due to economical slowdown, we need to find different ways to reduce cost for our customer so that they still see smooth ROI, and remain our customer’s.

This paper talks about causes which refrain test teams from being more productive and suggests measures to overtake them. Use of Automation right from test case preparation till test reporting along with examples/tools available at MS is explained in later sections.

Testers does spend most of their time in test case preparation, execution, and bug reporting and finally test status reporting. This paper will talk about where we lose opportunity to improve productivity at every step. Also, suggesting some productivity tools to do our job better and in more efficient manner. A suggested test lifecycle

Complete paper is attached for your further read. Please note that this paper was written with respect to my then team at Microsoft, so any mentions had to be read with this note in mind.

-Naren

Views or Opinions expressed in this blog represnt only author and not any company or organization or person.

Improve productivity in Testing.pdf