Automation Testing versus Manual Testing Guidelines

Artículo
02/07/2005

I met with my team’s automation experts a few weeks back to get their input on when to automate and when to manually test. The general rule of thumb has always been to use common sense. If you’re only going to run the test one or two times or the test is really expensive to automation, it is most likely a manual test. But then again, what good is saying “use common sense” when you need to come up with deterministic set of guidelines on how and when to automate?

Pros of Automation

If you have to run a set of tests repeatedly, automation is a huge win for you
It gives you the ability to run automation against code that frequently changes to catch regressions in a timely manner
It gives you the ability to run automation in mainstream scenarios to catch regressions in a timely manner (see What is a Nightly)
Aids in testing a large test matrix (different languages on different OS platforms). Automated tests can be run at the same time on different machines, whereas the manual tests would have to be run sequentially.

Cons of Automation

It costs more to automate. Writing the test cases and writing or configuring the automate framework you’re using costs more initially than running the test manually.
Can’t automate visual references, for example, if you can’t tell the font color via code or the automation tool, it is a manual test.

Pros of Manual

If the test case only runs twice a coding milestone, it most likely should be a manual test. Less cost than automating it.
It allows the tester to perform more ad-hoc (random testing). In my experiences, more bugs are found via ad-hoc than via automation. And, the more time a tester spends playing with the feature, the greater the odds of finding real user bugs.

Cons of Manual

Running tests manually can be very time consuming
Each time there is a new build, the tester must rerun all required tests – which after a while would become very mundane and tiresome.

Other deciding factors

What you automate depends on the tools you use. If the tools have any limitations, those tests are manual.
Is the return on investment worth automating? Is what you get out of automation worth the cost of setting up and supporting the test cases, the automation framework, and the system that runs the test cases?

Criteria for automating

There are two sets of questions to determine whether automation is right for your test case:

Is this test scenario automatable?

Yes, and it will cost a little
Yes, but it will cost a lot
No, it is no possible to automate

How important is this test scenario?

I must absolutely test this scenario whenever possible
I need to test this scenario regularly
I only need to test this scenario once in a while

If you answered #1 to both questions – definitely automate that test

If you answered #1 or #2 to both questions – you should automate that test

If you answered #2 to both questions – you need to consider if it is really worth the investment to automate

What happens if you can’t automate?

Let’s say that you have a test that you absolutely need to run whenever possible, but it isn’t possible to automate. Your options are

Reevaluate – do I really need to run this test this often?
What’s the cost of doing this test manually?
Look for new testing tools
Consider test hooks

Comments

Anonymous
February 07, 2005
Hi Sarah,

Interesting post. I have recently blogged about my opinion on the same subject here: http://www.teknologika.com/blog/General/Testing/TenetsOfTestingPartTwo.html.

Regards,

Bruce McLeod
bruce at teknologika dot com
Anonymous
February 08, 2005
Thanks. In my position I have been mulling this over for a while. I think i will have use for your framework. I was just assigned another product of ours and I need to decide what I can and should automate.
Anonymous
February 08, 2005

Automation makes you capitalize on invariants, hence test families. But what about adapting to the slightest changes in UIs? Do you simply throw a broken test case to the trashcan, or do you fix it?

A big con of automation is that it doesn't find you any new bugs. If you want to find those, you need to either find those yourself by hand, or rely on somebody else's work.

Are you still using VisualTest for runs? How will that evolve will UIs going more towards IDless controls in UIs?
Anonymous
February 09, 2005
Stephane,

"Do you simply throw a broken test case to the trashcan, or do you fix it?"
That's a good point about Automation that i missed in the above guidelines. There is a cost with Automation to maintain the test case. Almost always, you fix the test case, unless the app has changed in such a way that the test is no longer valid. For example, say you test for a feature, but the feature is modified or cut, then you would most likely destroy the test case. One good rule of thumb is don't automate if you know the UI will be in flux. The slightest changes in UI can cause automation to fail. When you know the UI has been finalized, excluding any fixes to bugs, that's the time to automate.

"A big con of automation is that it doesn't find you any new bugs."
I disagree with this statement. The point of automation is to find regressions. Regressions are bugs that appear because of some change in the code. For example, say a developer fixes a bug, but because of the bug fix, another bug appears, the second bug is called a regression.

"Are you still using VisualTest for runs?"
I've never used VisualTest. We have our own automation framework.

Thanks,
-sara
Anonymous
February 09, 2005
Great post. I would like to add yet another dimension to the decision - the human factor. Having worked with testing teams in three continents I believe that building a good automated test suite, one that truly delivers the anticipated ROI, cannot be achieved by your average Joe. Programming great test cases that really work is much like, well, programming... Therefore, when you consider whether to automate a test suite, I recommend you seriously consider the level of expertise of your test team. Leaving automation to a non-expert would probably lead to a lot of frustrations and no ROI. This all applies to functional or regression test. When it comes to load test, obviously automation is absolutely mandatory. Gone are the days when you summoned 40 employees into a big lab and asked them to hit the Enter key at the same time...
Anonymous
February 27, 2005
The comment has been removed
Anonymous
March 01, 2005
Nilesh, I think the approach of manual testing all test cases for multiple passes and then running those same test cases with automation for the final "phase" is not a real world scenario.

Ideally most projects would have manual testing and automation going on at the same time, or even have automation being coded before there is anything to test. In an ideal and agile world, your automation for a feature under test should be completed before the feature is code complete. This is an aspect of test driven development. Also keep in mind that automation is not limited to GUI testing or driving the application through GUI manipulation. That is, most people think of GUI and they say UI. UIs encompass more than only GUIs. All testing requires UI in that calling a function/method of a DLL or webservice is interacting with a UI (a non-graphical UI).

So, if you have a detailed design spec or you practice Scrum or other agile/xp methodologies, the QA people will know what is being devloped before it is done, and they can code some automation prior to the functionality even being "code-complete". It is up to the team/group to decide if the automation is run prior to code complete and thus should pass once the feature goes in or if the automation is run only after you know the feature goes in.

All the while, as Sara has already detailed and as is the crux of her blog entry here, there are classifications of testing that make no sense to be automated. Thus, each project has some balance to find between what is to be automated, when to automated it, how it will be automated, who will automate it, and how is manual testing complementing what is not being automated.

For too long many teams/groups have pictured QA in terms of how can automated testing compliment manual testing and improve quality. In reality, automation needs to have a larger and earlier focus and manual testing is the compliment to fill the holes where it makes no sense time-wise, feature-wise, or skill-wise to automate a test. ADditionally, manual testing may certainly be needed for tests to be run that are planned to be implemented via automation at a later time, but there will alsways be testing that won't be and shouldn't be automated.
Anonymous
March 01, 2005
Ken,

I got the gist of ur reply...But isn't manual testing necessary atleast for one pass? Or should we automate all the passes??

Thanks
Nilesh
Anonymous
March 01, 2005
Ken,

I got the gist of ur reply...But isn't manual testing necessary atleast for one pass? Or should we automate all the passes??

Thanks
Nilesh
Anonymous
March 02, 2005
The comment has been removed
Anonymous
March 02, 2005
The comment has been removed
Anonymous
March 07, 2005
Nilesh,
Anonymous
March 08, 2005
What are test hooks ?

Amol.
Anonymous
April 11, 2005
Hi my name is sakthi and my e-mail address will be sakthisankar@msn.com I need some notes regarding the testing can any on send me regarding the automated tools.
Anonymous
April 18, 2005
Hi Saraford,
You have really mastered the art of explainingTesting .I would be happy if I could get some tips like this at my mail.
Anonymous
April 21, 2005
The comment has been removed
Anonymous
May 12, 2005
Hi...
Thanks for great Post..Some are asking when we have to automate?

Is it in the Initial Testing Phase or in subsequent Testing..

Let me tell you my experience..

Many times we go to automation for regression testing and performance testing.Based on type of testing we'll opt type of tool.
For example we can opt WinRunner for Regression and LoadRunner for Performance.

As Saraford said, one should be cautious how automation is going to help our testing in the next versions.
Anonymous
May 31, 2005
I came across this great article by Sara Ford about when to automate. I've read lots of articles concering this topic and they all seem to boil down to the few questions that sara outlined. I believe that one criteria that's over looked sometimes is if th
Anonymous
May 31, 2005
Cons of manual testing:

The tester gets more and more bored by his work. Before soon, he either 1) gets depressed, 2) starts looking for another team/company. Both ways, productivity dives and soon enough you loose working force in which you invested education/training money.

:)
Anonymous
June 18, 2005
Hai sara,
Its clear and brief. well its quite interesting. I am new to this field. Gave me a brief description of the manual and automated testing.
Anonymous
July 11, 2005
really good stuff
Anonymous
September 12, 2005
hi
very good posting.
i need a clear classification of practicle approach for TESTING. what happens in Real time senario after geathering requirements and till delivary stage? i need how to proceed with SRS, Test plans Test cases, Which Test Cases To Be automated?
can any one help me ramineni999@gmail.com
Anonymous
September 29, 2005
Nice article , but the author should be more clear in explaining and if this one of more depth , it would have been more useful for us.

If it is possible please give when should we plan automating the test cases ( during SRS or while creating Test Plans )
Anonymous
October 19, 2005
The comment has been removed
Anonymous
October 19, 2005
The comment has been removed
Anonymous
October 24, 2005
Hi,
There seems to be good knowledge sharing posts in your blogs. If you take my opinon about Automation vs Manual then i would say that its not only monetary factors or size of application that makes us to decide what to go for. What i think that apart from these factors a Project Manager need to consider;
1. How much quality the development team delivers to the testers (Less bugs or more bugs)?
2. What is the project deadline ?
3. How critical is the project ? Say if you are making an application for NASA than you need to do manual and automation also and i would suggest that the application should also be tested even if it goes live.
4. How familiar is the user with the technology ? (Novice / Beginner / Advanced)
5. What will be the users reaction if he finds a bug ? Say if the application goes live and if the user discovers a bug then how will he handle it. Will he protect himself from the bug and pray not to see that bug again (Eg. Win XP OS) or will he stop the application and pick up the phone and will yell to the support team (Like we do today with call center guys) which will definately make an impact on users impression for vendors quality

And this list can go on . I think based on these factors and cost of resources (money , labour , time)involved you can decide whether to go either for manual or automation or both .
Anonymous
October 27, 2005
Very good one for a fresher to understand.
Can I have detail notes regarding Manual Vs Automation testing as i have to take a session in my company for freshers. Pls help me out?
Anonymous
November 26, 2005
Sara -- good post, and a good set of rules of thumb to work with. But I also agree with your early comment on using common sense. It is a very contextual consideration. For my two cents worth, look at my editorial at http://www.sirius-sqa.com/newsletter1.html.

Sincerely,
David Gilbert
Anonymous
December 10, 2005
The comment has been removed
Anonymous
February 21, 2006
hi

iam very impressed about manual and automation .Can you explain detailed explanation regarding the live environment in companies.
Anonymous
February 26, 2006
more notes about manual versus automated testing
Anonymous
March 20, 2006
The comment has been removed
Anonymous
April 05, 2006
I am new to testing field.Currently working as a developer from last 2 years and want to shift to testing soon can any one provide me notes regrading testing from basics.
my mail id id abhips686266@rediffmail.com
Anonymous
April 23, 2006
I am new to testing field.Currently working in mech field and want to shift to testing soon can any one provide me notes regrading testing from basics.
my mail id is sjdhote@gmail.com
Anonymous
April 30, 2006
Hi Sara,
I am very much impressed with this topic.Its really a time to appreciate u r topic.According to my point of view i think tools cannot be intellegent than human brains.Rather manual testing takes time to complete the testing but its reliable,whereas automation tools leaves behind some important bugs which are difficult to find by tools.Tools are needed during the time of testing the load and performance.Is this section gives a point of view.If so pls feel free to send any comments.
My mail id is:srikant_2977@yahoo.co.in
srikant_ctc@rediffmail.com
Anonymous
May 05, 2006
Sir/Madam,
This time i am workin in softare company they put me on Manual Testing.Here is not Automated Testing.Are there scope of Manual testing.Please Suggest me.My Email ID is harish_usa@rediff.com
Anonymous
May 10, 2006
Hi Sara,
This is a good artical and I am impresed. Sara, currently I am doing Manual testing on Web Applications. Please guide me should I shift to Automation testing and which is the right tool for that. And will you please tell me should I go for the CSTE exam though I am doing manual testing.

my mail Id is : - sam_sps@yahoo.co.in or
sameer.bhate@rediffmail.com

Thank you

With Warm Regards

Sameer
Anonymous
June 01, 2006
Respected mam or sir,

Quest:-How Can we apply manual testing on web based parojat or like java based project.

Thanks
mam
Anonymous
June 04, 2006

hi sara,

i am very new to the testing enviornment specially manual , what are the intiatiives need to be taken to make myself perfect on manual testing, and can you please list out the websites which quote examples of manual testing.

thank u
Anonymous
June 12, 2006
The comment has been removed
Anonymous
June 12, 2006
Hi, I am venky. I would like to know some sites regarding Manual testing tat will guide me. I wish to know sites that allowsme to practice Manual testing with samples and other stuff.
Anonymous
June 13, 2006
The comment has been removed
Anonymous
July 05, 2006
Came through the google search, Found this article, and it's nice. Probably a better insight for people who are still confused about whether to go for automation or not for particular testing tasks or modules.
Anonymous
July 17, 2006
I do manual testing on Web based applicatons and I want to change the format of the test scripts that the company has used in the past. TDo you know of a resource I could use to see different types of formats? We also just started Agile developing.

thanks much
Anonymous
July 17, 2006
can u tellme which is going to rule in future? Manual testing r Automation Testing
Anonymous
July 18, 2006
I have two question to ask and as always your help will be greatly appreciated.

How do you answer this question if you were me as I am totally new to testing feild

Q1- Can you explain your experience with writing test scripts from scratch using automation tools such as Silktest, Winrunner, Test Director, SilkPerformer, homegrown, etc).

Q2- Can you explain your experience with, and philosophy about, creating an architecture/harness for automated tests. What was the hardest thing about implementing it?
Anonymous
July 24, 2006
Excellent job, very useful information. Thank you.
Anonymous
July 24, 2006
Dear Sir/madam,

Sub: Application for suitable post (Master of computer applications)

Having come to understand that there is an opportunity for a Post-graduation in MCA, in your esteemed organization, I offer myself as a candidate for the same. I request you to consider my application favorably.

I have enclosed my Resume that will be helpful to you to know my educational details. I shall produce the certificates of all qualifications mentioned therein when I am called for an interview.

I promise to discharge my duties, when appointed, to the best of my abilities.

Thanking you.

Yours faithfully,

Shivappa.Seemikeri

Cell No:9880058464

Enclr:Resume
Anonymous
August 02, 2006
Hi! Madam
I’m sateesh new to testing field. I want more information about manual testing plz send manual testing notes thanks madam
Anonymous
August 07, 2006
Hi,

please let me some manual testing tips.
if anybody has Manual testing notes or questioning answering in details plz send me on my mail id tahir@sebiz.net or send me URL

Thanx 'n' regards,
Tahir AHmad
Anonymous
August 15, 2006
Hi Sara,
your information is very clear and intresting. I am new to this field and I would like to know more details about Manual and Automated testing. If you or any one have notes about both Manual and Automated testing, please provide me. That should be very helpful to me and would be great.
My emai id : srireddys@gmail.com.

Thanks & regards,
Sree
Anonymous
August 15, 2006
No Problem
Anonymous
August 15, 2006
Hi, Ms. Sara

Thanks for this guidance, i am new to this feild,i really appreciate your notes.Please provide some more information if possible on Software Testing.

Regards

Shivani
Anonymous
August 19, 2006
Hi Sara & All,

Its really great to know how u all are asking & sharing ur questions on the Board. From now onward I also want to become the part of the family.

Regards,
Sheetal
Anonymous
August 20, 2006
Hi, Ms. Sara ,

I’m raj kumar have a 1 years in testing field. I want more information about manual as well as Automation testing plz send manual and Automation testing notes. and also sand me latetest Question for QTP and Load Runner and Guide me . My Mail ID-

rajkumar.saw@gmail.com

Thanks and regards,

Raj kumar
Anonymous
August 22, 2006
Hi,

Need notes on software testing. Further if i cud get some scripts that automate testing. I am new to this so if u cud help me.
Anonymous
August 22, 2006
Hi, Ms. Sara,

A great article. Well incase if u have notes on Software Testing and Samples of Automation Script. I would like to receive one. They will really help me to further understand and enhance my knowledge. Also using them i can enhance my skills. Please guide and help me...

Thanking You,
Manav
emailid: ankurjain_ojsb@rediffmail.com
Anonymous
August 24, 2006
Dear sarah Madam,
I am Sundar. I would like to know some sites regarding to practice Manual testing with samples and other stuff.

If you or any one have notes about both Manual and Automated testing, please provide me. That should be very helpful to me and would be great.

Thank you very much.

Kind regards,
Sundar.
Anonymous
August 28, 2006
hi sara

its clear now of automated and manual testing,

can u tell what role Unix shell,perl scripts in testing

and what are the prospects of the above

regards
srinivasan
Anonymous
September 16, 2006
The comment has been removed
Anonymous
September 18, 2006
Hi! Madam
I’m Praveen Kumar.N new to testing field. I want more information about manual testing plz send manual testing notes thanks madam .This is my mail id:

navathi_0522@yahoo.com
Anonymous
September 18, 2006
The comment has been removed
Anonymous
September 18, 2006
The comment has been removed
Anonymous
September 30, 2006
Respected Madem, I am iqbal i have visited this site it is great, it really helps people specially the freshears. I am really thankfull to you for providing this material. I have some questions of which i am not getting proper answers, can i get it form you. I yes please send me a mail i will let you know the questions. Thank You.
Anonymous
May 19, 2007
PingBack from http://www.bugtest.info/2005/automation-testing-versus-manual-testing.html
Anonymous
May 29, 2009
PingBack from http://paidsurveyshub.info/story.php?title=sara-ford-s-weblog-automation-testing-versus-manual-testing-guidelines
Anonymous
June 01, 2009
PingBack from http://woodtvstand.info/story.php?id=6521

Compartir a través de

Automation Testing versus Manual Testing Guidelines

Comments

Recursos adicionales