There are many types and approaches to testing. In this article, we discuss some of these different techniques by comparing them to testing pens on sale in a stationery shop. The shop represents the software application. Features within the application are represented by pots containing different pens. We need to test each pen. What testing approach could be applied to ensure that each pen meets the criteria specified?
Manual Testing
There will be a specified criteria that each pen must meet. For example, you expect the pen to be the exact color, width and type advertised on the pot. There will also be an expectation that the pen will actually work. This information will be included in a test case.
Each test case runs manually. Any pen that fails to meet the specified criteria is rejected. A bug report is filed for each rejected pen.
A test script might include the following steps:
| Step | Action | Expected Result |
| 1 | Remove lid from the pen | The pen has a lid which can be easily removed from the main part of the pen. |
| 2 | Check the pen type | The pen type matches the type specified on the pot. |
| 3 | Measure the nib | The width should match the width specified on the pot. |
| 4 | Write something onto a piece of paper | The pen should contain ink which flows onto the paper as the user is writing.
The color of the ink should match the color specified on the pot. |
Automated Testing
Checking each pen individually is likely to be very time consuming, especially if there is a large quantity of pens. Initial testing could be completed a lot sooner by introducing some test automation.
Test automation would not only find ‘buggy’ pens quicker, but automation could also potentially improve the precision and accuracy of the tests. This is especially the case for the width of the nib.
A typical pen will have a nib that is about 0.8mm wide, although sizes can vary. While it is possible to measure such small widths using a ruler, an automated measurement using a microscope would provide a more accurate value. The nib could be measured to the nearest micrometer instead of a millimeter (0.8mm is 800 microns). A microscope connected to a camera could be used to take a photograph of the pen. This would allow the process to be automated. The automated test can then assess the shape and size of the nib. Checking the size of the nib determines if it meets the defined criteria, assessing the shape can also inform the tester if there are any deformities that could cause issues when writing with the pen.
Of course, automated testing also takes a lot of time to develop. The cost of automating a test may not be justified. In the event that a test cannot be automated or is unsuitable for automation, the test can still be run manually.
The automated tests would also need updating if the requirements change. Pens could be incorrectly rejected if the tests are not updated. It would also be recommended to manually test a selection of pens to ensure that the automated tests are working as designed.
Data-driven Testing
Data-driven tests are an excellent way to utilize automated testing. There will be a data table containing all the conditions including test inputs and verifiable outputs. Each row in the table will be a new iteration.
In this example, the test input is the pen and the verifiable outputs are the type of pen, width of the nib and ink color. The iteration passes if all outputs are met. There is a choice of ending the entire test or continuing with the remaining iterations should an iteration fail. In this example, it would be more beneficial to run all iterations until completion as each iteration is independent of the others.
| Input | Type | Width of nib | Ink color |
| Pen 1 | Ball point | 0.8mm | Blue |
| Pen 2 | Ball point | 1.1mm | Black |
| Pen 3 | Fountain | 1.5mm | Red |
| Pen 4 | Fountain | 1.9mm | Blue |
When the specification changes, only the data table needs to be updated, not the automated test script. This makes updating the tests with changing requirements a lot quicker.
Exploratory Testing
Scripted testing (automated or manual) should not be the only method for testing these pens. We know what the acceptance criteria is for each pen, and we also know which pens meet this acceptance criteria. However, there are some other checks that need to be made. These are ones that are based on instinct or gut feeling. For these checks, the tester needs to have a large amount of skill, experience and understanding of the application. This allows recognizing potential issues with only a brief summary of explaining the expected behavior.
Some things to look out for will include potential usability issues – How easy is it to write with the pen? Does the ink transfer to the paper smoothly? How does the ink look on paper?
There may also be issues with the acceptance criteria that needs to be highlighted. The width of the nib may be as specified but if the user can’t write with a pen because the nib is too wide then it will be useless.
Ad Hoc Testing
There are some who think that Ad Hoc testing is the same as exploratory testing. They are not. With exploratory testing, there is some knowledge of the expected requirements. The user knows what the expected behavior of each pen should be. The tester is able to take an unscripted but logical approach to testing based on this information.
With ad hoc testing there is no information about the pens. All the pens are dumped into a single pot. The tester randomly selects a pen and guesses what the pen should do and how to write with it. There is no information about the expected behavior of the pen. There are no details about what the end-user needs.
Other Testing Levels
So far we have only discussed how testing would be applied at the system level. In total there are 4 test levels:
Unit and Integration Testing
For unit testing, we have each individual component of a pen – lid of the pen, the tube that makes of the middle section of the pen, ink cartridge, nib of the pen.
After unit testing, we need to ensure that each of these smaller components work well together. How does the lid fit onto the top of the mid section tube? Can the ink cartridge fit into the mid section tube? Does ink from the cartridge flow to the nib well? Dummy modules (test drivers and stubs) may also be used for integration testing if some components are not yet complete.
New defects introduced to the main product are expensive to fix. The cost of fixing defects increases when found later in the development life cycle. Unit and integration testing reduce the number of new defects introduced to the product.
Acceptance Testing
Of course, the person testing the pens will not be the end-user. This will be the person buying the pen. The pen will have to meet the end-user’s needs, not the tester’s needs. Acceptance testing by the end-user may or may not be as rigorous as the previous testing. A lot of the edge cases are rarely covered and they will not be scrutinizing some of the features as closely (for example they may not care if the nib is 0.7mm instead of 0.8mm).
In most cases, the user will walk into a shop and find a pen that they think will be suitable. They will do some writing with the pen, decide if it is suitable for their needs. If it is not, then the user will find a different pen. They may speak to the shop manager and request a different design. The user may decide to approach a different shop instead. This will happen if they are dissatisfied with the quality and price of the product.
Summary
Testing can be a lengthy process, combining multiple testing methods. The time required to run tests manually can be improved by including some test automation, but there are some things that can’t be defined using acceptance criteria which is why we need some exploratory testing as well.
Unit testing focuses on smaller components within the application. Integration testing looks at how the individual units interact with each other. These tests reduce the risk of introducing defects to the final product.
The product must not only meet the acceptance criteria, but it must also be of high quality and fit for the purpose. It is the client who is paying for the product that has the final say in whether the product meets their needs and can be released.
Acknowledgment
I would like to thank Steve Watson for giving me the idea for this blog. If you want to see the conversation that generated this idea, then check out the comments in my other blog post.
Please share your thoughts in the comments section below 😉
Hi Louise.
Great article – and thanks for the acknowledgement at the end.
I’m reading through and adding comments here as I go, as I find it easier, so hope thats ok:
Manual Testing.
These are good examples, and you could also add others where you dont have a lid, but press a button or twist the pen as well. The one-handed test is a good one here – with one free hand, could I pick up and use a pen easily. If my thumb could pop a lid off, then yes, if my thumb cam press the button at the end, then yes, but twisting a pen one-handed is not something I can do easily, so would fail that scenario. It adds a ‘twist’ (no pun intended) to the tests.
Automated tests.
I struggled with this one – but then I’m thinking as an individual, and not a large company. I would assume a manufacturer of pens has a way of testing x number of pens all at once, which would be part of the manufacturing process. As a consumer purchaser, it would be more difficult to do.
Exploratory & Ad-Hoc testing – I like the way you describe the differences. Its something I struggled with a bit, but I think it makes sense. Both are similar types of approaches, but one is where you know more about what you expect to happen, and one is where you really have no pre-conceived ideas. Exploratory is also looking for the unkowns, but the difference is in picking up a pen which says the nib is 0.8mm, versus one you pick at random, You’ll do various tests with that pen – e.g. can it write on the types of surfaces that I need it to, which may not have been specified properly or at all, but with Exploratory testing you will check that the ink is consistently 0.8mm, and ad-hoc you are looking for it to be readable.
Just out of interest, here’s a question for you – I sent some post the other day, and used a ‘Broad’ Berol pen to address it. The nib was quite fat, so the address wasnt written as neatly as I would have wished. Would you say that I discovered this through an Exploratory or Ad-hoc approach?
Since you were using the pen as an end-user not a tester hired by the company, I would say this was discovered as part of acceptance testing. However, if you were testing this as an employee and you knew it was a ‘broad’ beral pen then I would say exploratory testing, otherwise then it would probably be ad hoc testing.