Numberz Challenge - my approach

Recently Alan Page has posted testing challenge on his private blog - Numberz Challenge. He has attached little application and described it in this way:

"When you press the "Roll!" button, the app generates 5 random numbers between 0 & 9 (inclusive), and sums the numbers in the "Total" above. The stakeholder’s primary objectives are that the numbers are random, and that the summing function is correct."

And here was the challange :

"If you want to play a little game for me and do some testing, test this app and tell me if it’s ready to ship (according to the stakeholder expectations)."

My approach to this challenge:

Learn as much as possible about the stakeholders and application context.

- Are there any other requirements beside those written down?
- What are the important information we need to know prior to testing ?
- Who is the customer ?
- Where this application is going to be used ?
- What are the consequences of shipping defective product ?
- What time is given to test this product ?
- Is this standalone product, or maybe just an part of something bigger ?
- On what OS and hardware this app is going to be used ?
- How long this app is intended to run without restart ?

Test against written requirements.

I clicked the "Roll!" button couple times and I could verify that application generates 5 numbers (0-9 inclusive), that numbers seemed to be randomly generated, that total were calculated correctly. This simple sampling however was not enough. To verify randomness you need way more tests. I was also curious if "Total" is always calculated correctly. To have more samples I hired automation tool. I have started with AutoIt and it worked out perfectly. I could process 10k rolls in about 60 seconds. Because I had pure blackbox approach I didn't know if restarting application might impact the results. So I decided to rolls numbers in 2 way :
- 20.000 times in a row without restarting the application
- 20.000 times but after each 1000 rolls I restart the application.

Based on the new results I've created normal distribution chart for the total.



Here we can spot some deviations from normal distribution e.g. the '20' shouldn't appear more often then '21'. Then I tried to focus on generated numbers to see if all numbers appear equally.



Number 3 has about 13% chance for appearing whereas it should has around 10%. It was first possible bug. I also noticed that for 20k rolls the Total has been 400 times calculated incorrectly which could be another possible bug.

Test against non-written requirements

The application had some problems with closing and I've encountered this on Windows 7 and XP OS. I was trying to observe the memory usage after 50k of rolls and without restarting the application for 24 hours but there wasn't any significant change in memory usage.

Ship / No Ship

Based on the information I learned from Alan about the app and it's context I come up with following conclusion.

Hard to say really when keeping in mind that shipping decision is always business decision. All in all we know that this app doesn’t meet all stakeholder expectation because numbers are not random and total doesn’t always work correctly. On the other hand there will be around 50 rolls per year per car dealer where randomness deviation may be not easily spotted. In addition statistically there may be one bad sum calculation a year (2% * 50). Is this acceptable ? I don’t know but for sure doesn’t meet given requirements.

Here you can read the summary post from Alan : Numberz Winnerz

Alek