Important skill for an exploratory tester

One of the ways to learn something new or make improvement being a tester is to be challenged by someone more experienced. I've first time experienced such a "way of learning" during coaching session with James Bach. This time the challenge came from Michael Bolton.

"Micheal: One KEY skill for an exploratory tester is to find out what the requirements REALLY are. What ARE the ways of finding out? In general, what are the sources of information on a project?"

I started with the question. What do you mean by requirement?

"Micheal: A requirement is /what someone important wants/ as a part of the completed product.
Notice that /what someone wants/ is not necessarily written down. A requirement is not a piece of paper, or something written on one. The paper and the writing are /representations/, literally re-presentations, of /ideas/."

The above idea has really opened my eyes. How often we rely on written documents only treating them as the one and only one oracle. It's hard even for technical people to articulate what they really want. Now imagine that you need to describe a thing that you haven't seen before. It's hard to put everything on paper and easy to miss something that may become important on the end. It's good to keep in my mind that what we read in formal document is the re-presentation of client idea only. And now question, how often have you aimed with your tests to cover the specification only (assuming that your test mission is for something more than specification coverage) ?

I started answering to challenge with following ideas: people, stakeholders, previous bugs, written specification, developers know-how, service desk and their database of incidents, regulations (eg. FDA), previous versions/products, meetings with clients/stakeholders

Michael cataloged my ideas into:

/References/-> things that we can point to. The reference allows you to say, "It says here..." or "It looks like this..." (previous bug reports, specifications, regulations, previous version of the products)

/Conference/ -> Information obtained by /conference/ - interaction with other people. (people, stakeholders, programmers' know-how, meetings with clients)

/Experience/ -> The program provides with information via /experience/, the actual empirical interaction with the product.

/Inference/->. When you reason about information that you have, extend ideas, weigh competing ideas and figure out what they mean or what they might suggest, that's information from /inference/.

Micheal summed up the challenge with following:

"Michael: When we think about exploring, we think about identifying and developing the information we have. That's a key part of the *learning* mission of exploratory testing."

The coaching session was great. This session was exacting and bring a fun at the same time. Thanks to this session I realized that I need to improve identifying and developing information that I already have. Going beyond what is written in formal document should give me broader perspective and help in creating better tests in the future.


Alek

FDA, Explorotary Testing and Michael Bolton

One of the abbreviations I've learned during a project for one of my clients was "FDA".
FDA ( Food and Drug Administration) is an agency within the Department of Health and Human Services which is responsible for protecting the public health in USA. The project I was involved was strongly "validated" and had to be compliant with FDA regulations (21 CFR Part 11 - Electronic Records and Electronic Signatures).

How these affect testing?
Every single piece of test (usually test script) had to be formally reviewed and signed before and after execution by 3 persons. In total 6 signatures per each test script. My first feeling was that this project is really good tested. But soon I have realized several problems. This huge bureaucracy introduced many consequences for example very long loop between test idea (the moment of creating the script) and test execution (the moment when formally written test step were used for "testing").
Furthermore testers responsible for test scripts creation where seldom executing own test scripts. This approach was compliant with General Principles of Software Validation and guidance for Electronic Signatures Validation provided by FDA. But from my point with such enormous obligatory bureaucracy something which supposed to be "testing" becomes "checking" and regular office work. This 'heavy scripted testing' highlighted the problem of separation between the test creation and execution. Even though we’ve covered all written requirements with test scripts we didn’t go beyond this, we didn’t find out new informations about the product, we didn’t present more insights to stakeholders, we didn't use diversity of tester experiences and skills - simply we were not serving the project as we could do. In addition with huge amount of resources we were still struggling to meet deadlines, leaving people exhausted and unhappy.

Below simple example:
Imagine that there are two testers. Tester A covers requirement with 2 test scripts. One test script per positive and negative test case. Tester B executes those two test scripts. Execution occurs usually month from the moment of test script creation. What are the results of such approach? For example you will never hear statements like:

"Wait, let’s do this test first instead of that test" or
"Hey, I wonder what would happen if…", or
"Bag it, let’s just play around for a while"*

There was simply lack of exploratory testing. For every suggestion of change I heard we have to follow FDA rules and keep "scripting" requirements. This made me wonder, how on earth some government agency can know better than our company what is best for testing ? I went with this problem to Michael Bolton. I was lucky and he found a minute to discuss this with me:
What I learned from this coaching session:

1. There is no procedure and script for producing test scripts.
It means that to some extent, we all apply exploratory approach. When we create any test script we often combine our subjective knowledge about the program, our previous testing experience, and formal requirements. Rarely same approach works for each test script and we need to make decision what is best along the way which can be the argument the every scripted processes "come from" exploratory processes, and not the other way around. Michael also pointed out that "whatever benefit the script provides after you've done your exploration the real benefit came when you found problems "during" the exploration" and I agree with this. From my experience I can say that often more problem are found during test script creation ("exploration") than while actual test script execution

2. There is no single statement in FDA guidance that we have to apply scripted testing to achieve validation and verification.
I have reviewed FDA guidance and regulations and indeed I couldn’t find a word that test scripts are the must. They mention about the evidences, which can be gathered with any approach to testing.
He also suggested to read the principle of the least burdensome approach. This document is not directly related to testing but we can read there few interesting statements for example :

"(...) FDA is an agency committed to fostering innovation and ensuring timely public access to beneficial new products. A least burdensome approach should be used in almost all regulatory activities. Application of the least burdensome principles to premarket requirements will help to reduce regulatory burden and save Agency and industry resources (...)"

I see above as suggestion for well-balanced approach in every aspect of project validated under FDA regulations. For example: why can't we be more flexible, let’s have enough evidences of executed scripted tests while on the other hand give testers freedom to explore the product so they can use their experience and unique testing mind in best possible way.

3. What can we change ?

- Think in terms of reference, inference, conference, and experience. Think in terms of keeping the design activities closely linked to execution. Instead of telling people what steps to follow, provide them with a motivation or a risk or an information mission.

- For the less experienced or less skillful testers, supervise them and coach them. Train them.

- Now, when there are specific things that need to be checked, specific procedures that MUST be followed - specify those.

- Provide them with guidewords for the sorts of problems that you want them to focus on.

- Another thing to get them to do: keep notes of your observations, questions, concerns, confusions. Talk about them after the testing session, or get help during it.

- If someone wants to know /exactly/ what you did, consider this: screen recorder (eg. http://www.bbsoftware.co.uk/BBTestAssistant.aspx )

Alek

*(I borrowed above quotes from Micahel Bolton)

Thinking like a tester and lateral thinking

There is a common belief that testers are "negative" thinkers, that testers complain, that testers like to break stuff, that testers take a special thrill in delivering bad news. While reading "Thinking Like a Tester" chapter from Lesson Learned in Software Testing book we can realize that there is an alternative view. For example: Tester don’t like to break things -> they like to dispel the illusion that things work.
With every lesson form this chapter we learn how tester can develop their mind and how different thinking can help to become better testers. One of the lesson is:

Lesson 21. Good testers thinks technically, creatively, critically and practically

According to authors all kinds of thinking fit into testing. They consider however 4 major categories of thinking as worth to highlight:

Technical thinking - the ability to model technology and understand cases and effects
Creative thinking - the ability to generate ideas and possibilities
Critical thinking - the ability to evaluate ideas and make inferences
Practical thinking- the ability to put ideas into practice

This lesson reminds me that there is another way of thinking which is not a new category but for sure worth to mention in this context - Lateral Thinking. I've read about this term for the first time in one of the book by Edward de Bono : "The Use of Lateral Thinking".
According to Edward de Bono":

Lateral thinking is solving problems through an indirect and creative approach, using reasoning that is not immediately obvious and involving ideas that may not be obtainable by using only traditional step-by-step logic.

We can read in this book that lateral thinking is not always related to solving problems but also helps to find new way of perception of old things and to generate new ideas. De Bono also illustrate lateral thinking in following way :

"You cannot dig a hole in a different place by digging the same hole deeper"

I think it relates very much to testing. Sometimes doing more tests in same way or in the same place is not giving us any new information. To learn something new we have to change direction.

In this book we can also read that people get used to one way of thinking. It reminds me that very often when I see something new (eg. software) my first way of using or running it becomes immediately a new habit - one and the only way of using it. I am limiting myself and forget that this way of using that thing (eg. program) is just one from many. Using lateral thinking may help us in such situations. Applying lateral thinking remind us that we should not be accustomed to the only one way of doing something only because it worked out for the first time.

Alek

Book Review: Lesson Learned in Software Testing

I heard about this book so many times, but thanks to company library it has finally slipped to my hands :)

Lesson Learned in software Testing
A Context-Drive approach
by
Cem Kaner
James Bach
Brat Pettichord

Inspired by Michael Larsen and his book reviews, I decided to write my own book review. I think it will help me with better understanding of this book and also push me to think about it’s content for longer than couple of seconds.

This book consists of advices not only for people in Testing Field but for all who are working around Software Testing and Software Quality.

Chapter 1. The Role of the Tester

Do you agree with below statements ?

- Testers assure the quality
- Testers should be able to block the release
- We have bug-free product
- The product can be tested completely
- Tester should be focused on requirements only
- Programmers are on the opposite site for testers

If you agree with any of above I strongly recommend reading this chapter to see different point of view. Authors present their opinion about many misconceptions regarding role of tester and testing. It's worth to read their arguments and attached examples.
Below couple lessons from this chapter with my own explenation:

"You are headlight of the project"
We don’t drive the project, we don’t make decisions, we are in project to find information about the product and present these information to every stakeholder who is interested in it so they can make better decisions.


"You will not find all bugs"
Authors present their opinion that it is impossible to find all bugs in a product unless your product is very simple or you have limited imagination. During my career I heard sometimes "bug-free software" which simply presents same misconception. We would have "bug-free software" if all the existing bugs were found but we don't have because it's impossible to check every possible place in the product with every possible combination of different circumstances. Even if we have enough resources to achieve this we should still remember the what is feature for someone can be bug for someone else. I like the example which comes from AST BA exam question:

"Suppose we revised a program so that it looked up all display text (e.g. menu names) from a text file on disk, so the user could change the language of the program at any time (e.g. French menu text to Chinese). Suppose too that, because of this change, the new version is much slower than the last, making some people unhappy with it. How we would evaluate this. Is it bug or feature ?"

"Mission drives everything we do"
Having in mind that completely testing is impossible, we should have clear mission on what we are going to achieve with testing. Some example test missions from book :

"Find important bugs fast"
"Prove a general assessment of the quality of the product"
"Do whatever is necessary to satisfy particulate client"

The mission may change from place to place, from context to context and from project to project but still we should have clear guidance for our testing activity.

Alek

P.S.
The initial plan was to create review for each chapter, however I think it's not good idea. The book is packed with so many tips, suggestions and insights that I think that each lesson is worth to review separately. Instead of reviewing all chapters I will try to focus on some of the lessons in later posts

What I learned from Skype coaching with James Bach

I was lucky and had another opportunity for Skype coaching with James Bach. I was interested in learning something more regarding Exploratory Testing (ET).

Basic definitions

James started the session from asking questions, to understand what ET means for me comparing to non-ET. I knew the ET definition, that learning, test design, and test execution are happening in parallel and are mutually supportive, but I had a problem trying to explain it with my own words which simply revealed that I really didn't understand it well.

So we started from beginning, what is testing for me, what is the difference between exploratory and non-exploratory testing ?. James used examples :

If I stand behind you and tell you what to type and what to look at - is it ET ?

If I tell you that I'm doing ET, and you see me type on the keyboard and move the mouse, and I appear to be testing, and you see no script, and I insist that I'm doing ET-- is that ET or not?

So what is exploratory testing ?

All testing that seems free is actually guided by unconscious impulses and we cannot be fully aware of where our ideas come from. ET is also self-managing process and acts upon itself. To sum up all good testing is to some degree exploratory.

How to be better at ET ?

I asked James how to prove thart what we do is somehow exploratory. It appeared to be a silly question. James replied that there is no need for proving that we should rather focus on:

* developing our skills
* learning how to spot biases and ruts
* using variety of methods
* using random testing
* learning from experiences

"The reason we talk about ET is because we want to learn how to manage ourselves well"

Which I see as key concept in improving our testing skill.

How to spot exploratory testing ?

We should start from asking questions:
1. Where do the test procedures come from?
2. Who controls the testing?
3. Is there a feedback loop that modifies testing from within the testing process?

The 2nd part of the session was a challenge. Key question was how many tests you can spot on presented image. It was a trap, no image can be called "test". Of course test is human activity and we can not consciously say that any image consist of "tests". Image can facilitate "tests" only.

It was tough session but I think I did better on it comparing to the previous one. I still see problem however in applying my knowledge. Even though I didn't fall into any obvious traps I couldn’t explain my reasoning well.

Alek

PS. If you want learn more about testing in exploratory way, check Micheal Bolton's resource page

BBST Bug Advocacy from my perspective

BBST Bug Advocacy is the 2nd course served by AST.
After succescfully passing BBST Foundation course, I wanted to participate in the Bug Advocacy. I was quite excited and scared. According to description this course supposed to be harder and more extensive as the 1st one. Indeed these 4 weeks were extremely busy.

Background to the course
During my 5 year of testing, I have never been taught regarding bugs reporting. In each company I was informed only what are the obligatory fields to fulfill in the bug tracking tool, I learned something on ISTQB course but it was still the basics only. I have seen many testers who treat bug reporting as the necessary evil. I have also seen simple bug reports which turned immediately to hard dispute between programmers and testers. That's why I was looking forward for this course because I wanted to improve area where I had definitely lack of skills. This course was designed to improve clearly, unemotional writing and persuasive skills in scope of bug reporting.

Since my first job as a tester I always thought that one and only evidence of my work is the product itself or number of bugs reported (I know the problems involved in using metrics like this :) ) but I never realized that We as Testers are still evaluated by "quality" of our bug reports. How strong our arguments are, how much we help programmers to find and fix problem, how much value we provide with the report or maybe our reports bring to table more vagueness and disrespectful tone only ?


What I liked in Bug Advocacy

- First I had to work on something before I started learning it. In this course I have worked on definitions, I was reporting real bugs for OpenSource project and I was evaluated by others. By Cem Kaner ”You can't learn driving only by reading about it” You work on something, evaluate other’s work and then comeback to your initial thoughts and then try to evaluate yourself to see I you would change anything. Thanks to this approach there is no need to learn something by heart because you can deeply understand everything.

- There is no ready answer, first you have to work on something to understand it. Even knowing one of the definition, you have to prove that you understand it

- This course helped me to understand the diversity of the tester role.

- I realized how often our thinking is biased toward something. Sometimes one little opinion or statement from manger change our behavior and way how we see. See more Signal Detection Theory (I hope I will not be spoiling the course too much).

- I had chance for collaboration and peer review with other students.

What I would’ve changed

- This course was quite intensive. According to the course description it supposed to take 8-10 hours a week - for me it was sometimes 16 hours a week. Even though I would like to be more involved, ask more questions and pay more attention for things I didn’t have time.

- I wish I could translate my thoughts into English text more easily. I think I can communicate well but when it comes to explaining complicated idea with many details I feel I need much more time comparing to native language.

I really enjoyed every lesson from this course. I like to spend time with people who have same passion as me, people who are willing to struggle to gain more experience. I like to be challenged and professionally criticized. In my opinion this is best way to grow in any field - especially in testing. This course helped me also to stay inspired and hungry as a tester. And finally one more time I was reminded that being tester is the ongoing process of learning new things and unlearning some of the obsolete one.

Alek

Software Tester Job Offers, Part2

This post is the follw-up to previous

Hiring good people in IT market is one of the biggest challenge for many mangers but finding "good" testers can be even harder. I've been in few recruitment processes and very often I was only asked for dictionary knowledge. I don't remember if I was ever given a piece of software to test. There are some companies though which don't rely upon candidate's knowledge of dictionary definitions but are looking for their skills.
Very good example is Moolya - company based in India.

Below is the piece of description from their careers page

You could have lots of years of experience and we'd be delighted to see how good it has helped you. Please test a project (...) and send us your test report along with your name(...) If your test report interests us (...) we will provide you with a testing challenge.

There is another example from Atomic Object. To join this company, first you need to prepare answer for one of the question regarding software or your attitude toward work :

What is the most difficult aspect of writing software? What is the best way to address this reality of software development?

Tell us about a situation where you tried to change an organization or introduce an innovation. This doesn’t have to be a situation where you necessarily succeeded.

I really like the idea that as the entry point for recruitment process in these companies you first need to prove your testing skills (by simple testing something), you need to prove your communication skills, you need to respond for challenges. This is great way to look for passionate testers. And on the other hand it is great way to learn something about future employer. This approach also saves time for both part.

Alek