Tools for session-based testing

Session-based testing is the structured way of exploratory testing developed by James and Jon Bach. In the article Session-Based Test Management we can read :

"What we call a session is an uninterrupted block of reviewable, chartered test effort. By "chartered," we mean that each session is associated with a mission—what we are testing or what problems we are looking for. By "uninterrupted," we mean no significant interruptions, no email, meetings, chatting or telephone calls. By "reviewable," we mean a report, called a session
sheet, is produced that can be examined by a third-party, such as the test manager, that provides information about what happened"

As mentioned above after each session tester hands a report with important information and results. There is an example of Sample Session Report on James Bach page. The session by design should be uninterrupted which means that things like note taking should be done on fly. It can be easy achieved with applications which support session-based testing.

1. Rapid Reporter, exploratory notetaking

This is tool created by Shmuel Gershon. Rapid Reporter is really small, stays always on top of the screen and doesn't interrupt the testing process. You can type into the app all information worth noted. The types of information like : "Installation", "Bug", "Issue" etc. can be tailored to the individual needs. In addition you can easily assign time range for the session.


What is really cool is the possibility of doing screenshot with just one click, moreover this screenshot will be printed next to the inserted comment in a session report.


After the session the app generates session report with all entered information along with screenshot taken during the session. The report can be presented in e.g. HTML format

2. Session Tester

This is tool created by Jonathan Kohl. Before the session you need specify: Tester Name, Mission, Session Length.


Once you start your own session Session Tester window is opened. You can type there all information learned during the session. This app stays also in tray and thanks to this you can get information in form of the balloons about time left. You can also find in this app some nice little feature called "Prime Me". When you click the button you get some random inspiring notes which may help you during the session. e.g.:

- Consider self-reference.
- What are the boundaries?
- Try something negative.
- Testability first.
- Who's your client?
- What's the language?
- Consider the opposite.
- Unless...
etc.

The results of the sessions are initially saved to the xml files but there is also an option to produce session report in formatted HTML file.



3. Atlassian Bonfire
I haven't had chance to work with Bonfire yet but if a picture is worth a thousand words, a video must be worth a million :)




Alek

Numberz Challenge - my approach

Recently Alan Page has posted testing challenge on his private blog - Numberz Challenge. He has attached little application and described it in this way:

"When you press the "Roll!" button, the app generates 5 random numbers between 0 & 9 (inclusive), and sums the numbers in the "Total" above. The stakeholder’s primary objectives are that the numbers are random, and that the summing function is correct."

And here was the challange :

"If you want to play a little game for me and do some testing, test this app and tell me if it’s ready to ship (according to the stakeholder expectations)."

My approach to this challenge:

Learn as much as possible about the stakeholders and application context.

- Are there any other requirements beside those written down?
- What are the important information we need to know prior to testing ?
- Who is the customer ?
- Where this application is going to be used ?
- What are the consequences of shipping defective product ?
- What time is given to test this product ?
- Is this standalone product, or maybe just an part of something bigger ?
- On what OS and hardware this app is going to be used ?
- How long this app is intended to run without restart ?

Test against written requirements.

I clicked the "Roll!" button couple times and I could verify that application generates 5 numbers (0-9 inclusive), that numbers seemed to be randomly generated, that total were calculated correctly. This simple sampling however was not enough. To verify randomness you need way more tests. I was also curious if "Total" is always calculated correctly. To have more samples I hired automation tool. I have started with AutoIt and it worked out perfectly. I could process 10k rolls in about 60 seconds. Because I had pure blackbox approach I didn't know if restarting application might impact the results. So I decided to rolls numbers in 2 way :
- 20.000 times in a row without restarting the application
- 20.000 times but after each 1000 rolls I restart the application.

Based on the new results I've created normal distribution chart for the total.



Here we can spot some deviations from normal distribution e.g. the '20' shouldn't appear more often then '21'. Then I tried to focus on generated numbers to see if all numbers appear equally.



Number 3 has about 13% chance for appearing whereas it should has around 10%. It was first possible bug. I also noticed that for 20k rolls the Total has been 400 times calculated incorrectly which could be another possible bug.

Test against non-written requirements

The application had some problems with closing and I've encountered this on Windows 7 and XP OS. I was trying to observe the memory usage after 50k of rolls and without restarting the application for 24 hours but there wasn't any significant change in memory usage.

Ship / No Ship

Based on the information I learned from Alan about the app and it's context I come up with following conclusion.

Hard to say really when keeping in mind that shipping decision is always business decision. All in all we know that this app doesn’t meet all stakeholder expectation because numbers are not random and total doesn’t always work correctly. On the other hand there will be around 50 rolls per year per car dealer where randomness deviation may be not easily spotted. In addition statistically there may be one bad sum calculation a year (2% * 50). Is this acceptable ? I don’t know but for sure doesn’t meet given requirements.

Here you can read the summary post from Alan : Numberz Winnerz

Alek

Questioning Simple Test

This is the 2nd post about the testing challenge that was given to my testing team during one of the internal testing workshop. The presenter was trying show us how often requirements are misunderstood even by technical people which often leads to making false assumptions.

Major of attendees suggested to turn 1) card #1 (card with 'A' vowel letter on one side) expecting even number on the other side and 2) card #3 (card with '4' even number on one side) expecting vowel on the other side. That was a trap. We can't say based on given requirements that if there is even number on one side there must vowel on the other side. It works only other way around: if there is vowel then there must be even number. It was pure false assumption.
According to presenter we should turn 1) card #1 (with 'A' vowel letter on one side) to verify if there is an even number on the other side as a positive test and 2) card#4 (with '7' odd number on one side) expecting consonant letter only on other side as negative test. This challenge however was not as easy at seemed because you can spot more false assumption in it.

First of all I think there is problem with statement: "Can you optimize testing and have only two cards that you might turn to test and verify that your system works correctly."
We can never ever be 100% sure that something works correctly when it comes to testing software application. And in this exercise according to presenter 2 simple tests were enough to prove that this system works! It's like with testing calculator summing function, when you put '2+2' and you get '4' which is correct but you can never be sure if this '4' was the result of summing function or maybe hardcoded result value for each operation.
Secondly let's focus on this requirement : "system returns playing cards with numbers or characters on both sides". English is not my mother language and maybe that is the problem but based on above statement I can't assume that for each card I will have letter on side and number on other side. I think that "letter or numbers on both sites" suggest that we can also have cards with only letters and only numbers on both sides.

Alek

Simple Test

Imagine a system which returns playing cards with numbers or characters on both sides.
There is one requirement:

If there is a vowel on on side, there must be an even number on the other side

System returned four cards like:


Can you optimize testing and have only two cards that you might turn to test and verify that your system works correctly.

Which cards would you turn ?

Alek

Matura exam and subject for bias

In Poland every year in May 350.000 students take Matura exam (High School Diploma examinations). Below is a chart with points distribution from Matura exam in Polish on foundation level in 2010


Do you see anything interesting in the picture above?

Almost all values fit into normal distribution, however for some values results seem not correct - values between 17-22 points. Actually results doesn't seem strange when you know that to succeed in this exam student needed to have 30% of maximal number of points which in this case equaled to 21 points. According to this chart it looks that people evaluating this exams had tendency to omit giving exactly 20 points. There may be many reasons for such tendency. We don't know if this was done consciously and on purpose. What we do know is that people who are expected to make any evaluation can be easily directed toward some direction because decisions are subject to bias. I've learned during AST Bug Advocacy course that in decision making we can analyze several biasing variables. For example common bias variables are : "Motivation", "Expected Consequences" , "Perceived probability" ,"Perceived importance of the task". We don't know if there were any consequences for examiners for failing student when he got exactly 20 points, but for sure this number had some impact on making pass / fail decision.

The Matura exam in Polish consist of foundation part and second not mandatory part on advance level. This part has no impact on pass / fail decision. Let's see normal distribution of points on advanced level.

There is no strange values around specific numbers. So it looks that with pre-defined boundary point for pass / fail decision, examiners are less likely to fail students which have one or two points below this boundary value.

Same applies to people who evaluate software on daily basis - testers. Our decision regarding Bug / Feature is also subject to bias. Glen Myers in his book "The Art of software Testing" writes that "testers who want to find bugs are more likely to notice program misbehavior then people who want to verify that the program works correctly". What is most interesting testers do this unconsciously.

You can read more about above phenomenon by searching phrase Experimenter Effect and for more about making decision under uncertainty search Signal Detection Theory

Alek

I hate manual testing!

Every now and then when you sit among testers you can hear:

"I hate manual testing !"

I tried once to follow this sentence and asked additional questions:
"What do you specifically hate in manual testing ?"
I received following answers:

"I hate doing repetitive tasks!"
"I hate regression tests !"
"I hate writing test scripts for someone else!"
"I hate following someone else’s test script!"
"I don't want to just click around this like a monkey!”

I replied: "What you would like to have instead?"

"I would like to automate something, I would like to do automation testing!"

It's obvious that regression part of testing is best candidate for automation because people are often inefficient with doing repetitive tasks.
What about preparing test scripts in advance or following test script created by someone else? I think that with: creating test scripts from requirements, writing test scripts without seeing product to test, executing test scripts without possibility to do own research, the testing is simply disappearing. The problem is not in manual testing itself but in treating testing like linear office activity whereas in my opinion testing should go beyond verifying what is to what is expected. Secondly why not take advantage from people having "testing DNA" instead of treating them like clicking monkeys ?

Below is the definition of exploratory testing from Cem Kaner’s blog

"Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project."

I think that with above approach to testing we wouldn't hear often "I hate manual testing!” Do you agree?

Alek

Tester and challanges

One of the common question you can hear during job interview for tester role is : why do you want to work as a Tester ? I can see in this question great opportunity to say about passion for testing, to mention about value tester can create and of course it's chance to mention something about solving problems and challenges.

Since I remember I always liked puzzles. It was sometime after I had started my first job as tester when I realized that this kind work doesn’t need to be necessarily boring and I can have a lot fun from it. When you work as tester you often encounter situation where you touch something new that you have never seen before: new system developed in new technology which works in new business environment and you have to collaborate with different stakeholders.
To create value with your testing you need learn quick and this is often great challenge. Of course tester work doesn't always look like this, depending on context you are sometimes obligated to work for same project for long period, to do repetitive task, to create tones of documents and to create evidence for every single piece of test activity. In such situation there is great opportunity to stay in shape as tester and have some fun from testing - it's Weekend Testing movement. These challenges occur usually during weekends. For those who hasn’t had chance to participate there is a place where some of the weekend’s challenges are archived

You can also create challenges by yourself, simply by challenging your thoughts with colleagues, by working on imaginary problem and ways to tackle it, you can try reverse engineering on some web application. There is a lot of ways to keep your Testing DNA in good shape. Good Luck!

Alek