WTA07 - when do we really start testing ?

Last Saturday I was participating in Weekend Testing Americas #7. This session realized me once again how little I know :).

The Challenge: Cross the River
Press the round blue button to begin. The goal is to get all eight people across the river. You must respect the following rules:

Only two people can be on the raft at one time.
Only the mother, father & officer can operate the raft.
The mother cannot be left with the sons without the father (or she’ll beat them).
The father cannot be left with the daughters without the mother (or he’ll beat them).
The thief cannot be left with anyone without the officer (or there will be even more beat-downs).

The Mission:
"I’m hiring staff for my IT department. I was told that this simple program will help me in finding the smartest candidates. Your mission: test the program and report how it suits my needs!"

Session from my perspective
I read the mission and downloaded the application. Then I started testing - I was trying to find any problems with this application as fast as I could. Meanwhile I was reading main chat. It was strange that no one was writing about the "findings" or "issues" related to the program. The discussion was mainly focused on the mission. Instead of pure clicking on the program, people were trying to:
- get as much details from the customer as possible
- challenge the mission
- understand what "smartness" we are looking for
- to understand how a piece of software can evaluate the smartness
- to verify if we are smart enough to evaluate others smartness
- and so on

Above questions reminded me, how many times I start testing something without asking for more details and without raising questions. I realized that testing hardly ever start form executing program.

The debate was very inspiring. One of the brillinat thought was from Adam Yuret:

Bugs are meaningless in this context. Like finding that the logo on one side of the sinking titanic is the wrong color.

If I could remember only one thing from the Weekend Testing session it would be from Justin Byers:

Avoid jumping into the software looking for bugs until you know what you’re really testing.

For whole chat session see chat transcript.

Participants: Michael Larsen, Justin Byers, Albert Gareev, Shmuel Gershon, Markus Deibel, Mohinder Khosla, Adam Yuret, Perze Ababa, Phil Kirkham, Aleksander Lipski

Thanks guys for another testing lesson.
Alek