Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

The four-hour tester

There are many ways to learn testing. The 4-hour tester is a way that focuses on simple exercises that take at most one hour to complete. The format is quite simple: briefing, exercise, evaluation.

Why don’t you try one right now?

The idea for this site came to us while we were working on our talk for TestBash Manchester 2016: The 4-hour Tester Experiment. (Which in turn was inspired by Timothy Ferriss’s “The 4-Hour Chef”.)

- Helena Jeret-Mäe and Joep Schuurkes

If you want to contribute a solution to any of the exercises, please submit a pull request to the repository of this site.

Exercises

  1. Interpretation
  2. Modeling
  3. Test design
  4. Note taking
  5. Bug reporting

Interpretation

Words mean different things to different people. And when we string words together into sentences, it gets only worse. So it’s important to be aware of the many different interpretations people might give to one and the same sentence.

A good way to explore these different meanings is play of words. Of the many many options we selected the “Mary had a little lamb.”-heuristic. By stressing different words in a sentence, different interpretations arise:

StatementIn contrast to
Mary had a little lamb… it was hers, not someone else’s
Mary had a little lamb… but she doesn’t have it anymore
Mary had a little lamb… just one, not several
Mary had a little lamb… it was very, very small
Mary had a little lamb… not a goat, a chicken, etc
Mary had a little lamb… but John still has his

And note that we didn’t even yet go into the different possible meanings of “to have”:

  • Mary owned a lamb.
  • Mary gave birth to a small sheep.
  • Mary ate some mutton.
  • Mary conned a mild-mannered person.

As you can see, a lot of different meanings can be found in this simple five-word sentence - if you look for them.

Exercise

Apply “Mary had a little lamb” to the second sentence of this quote: “You can add reminders in Google Calendar. Reminders carry over to the next day until you mark them as done. For example, if you create a reminder to make a restaurant reservation, you’ll see the reminder each day until you mark it as done.” - https://support.google.com/calendar/answer/6285327?hl=en&ref_topic=6076998

Stress different words or groups of words and come up with at least 10 different interpretations.

Evaluation

  1. Has your interpretation of the sentence become richer through this exercise?
  2. How different could two implementations be, if their developers did not share all of these interpretations?
  3. Based on your different interpretations, where do you think are good places to look for bugs?

Sources

Modeling

Briefing

Modeling software under test helps to analyze it, learn about it from different angles to be able to find useful test ideas. Depending on the kind of model you’re building and on your personal preferences you can pick an appropriate format, e.g. a list, a mindmap or a diagram.

Touring explained

One way to explore different perspectives on an application is touring. Like you can tour a city looking for different things (museums, night clubs or theaters), you can also tour an application looking for different kinds of things.

Michael Kelly came up with the following list of possible tours: FCC CUTS VIDS. Take a look at the list. It’s ok if you don’t fully understand all of them. For now we want to focus on the Configuration tour, User tour and the Data tour.

  • Feature tour: Move through the application and get familiar with all the controls and features you come across.
  • Complexity tour: Find the five most complex things about the application.
  • Claims tour: Find all the information in the product that tells you what the product does.
  • Configuration tour: Attempt to find all the ways you can change settings in the product in a way that the application retains those settings.
  • User tour: Imagine five users for the product and the information they would want from the product or the major features they would be interested in.
  • Testability tour: Find all the features you can use as testability features and/or identify tools you have available that you can use to help in your testing.
  • Scenario tour: Imagine five realistic scenarios for how the users identified in the user tour would use this product.
  • Variability tour: Look for things you can change in the application - and then you try to change them.
  • Interoperability tour: What does this application interact with?
  • Data tour: Identify the major data elements of the application.
  • Structure tour: Find everything you can about what comprises the physical product code, interfaces, hardware, files, etc…).

Exercise

Apply the following tours to Google Calendar. Take 5 minutes for each tour.

  1. User tour (instead of five users, do this for one: yourself)
  2. Data tour
  3. Configuration tour

Evaluation

  1. For each tour write down 3 things you would want to test, based on what you learned during each tour.
  2. How did these tours help you to come up with different test ideas?

Sources

Test design

Briefing

When you have a high-level idea about what to test - like for instance from the tours in the Modeling exercise - it’s time to think about what and how exactly you are going to test. That thinking is called test design. Put differently, you have an idea about something you want to find out. Test design is coming up with and designing experiments that will help you to find that something out.

There are many approaches and techniques to test design. For this exercise we will use a part of the testObsessed’s Test Heuristics Cheat Sheet, more specifically the Data Type Attacks section.

Exercise

  1. Review the results of your Data Tour (see the Modeling exercise) and identify 2 or 3 areas you would like to focus your testing on.
  2. Use the Data Type Attacks section in the Test Heuristics Cheat Sheet to test the selected areas in Google Calendar. Time limit 20 minutes.

Evaluation

  1. How did the cheat sheet help you in elaborating your test idea(s) in more detail?
  2. How did the cheat sheet help you create different tests? Which tests seemed to yield more interesting results than others? Why do you think that is the case?
  3. Considering the modeling and the test design exercise, in what ways were the two exercises the same? In what way were they different?

Sources

Note taking

Briefing

Note taking is a skill that supports collecting and later analyzing information gathered during testing. This exercise helps to learn a structure for note taking. Here are some possible uses of note taking:

  • during
    • for yourself: structure thoughts, free up working memory, keep focus
    • for others: pair/mob-testing for shared understanding
  • afterwards (shortly)
    • for yourself: review what you did, decide what you’ll do next, base bug report on, base test report on
    • for others: debrief to devs, PO, other testers, or any other stakeholder
  • afterwards (later)
    • for yourself: did I test that back then? how?
    • for others: audits

Exercise

Pick one feature in Google Calendar to test for 20 minutes. Look for interesting behaviors in this feature. Take notes about experiments and scenarios you try, results you observe, and questions that you come up with. Focus on taking more notes rather than less. Below are suggestions for structuring your notes and using helpful markup to use at the beginning of a bullet point/line.

  • Label the page with a title and a date
  • ? - question
  • ! - suspicious, bring to attention
  • B - bug
  • light bulb - idea for further testing
  • OK - looks good to you

Evaluation

  1. Right after finishing the testing and note taking, start the timer for 3 minutes and write a summary of your testing based on your notes.
  2. Return to your notes the next day. How easy is it get an overview of what you did? How easy is it to understand? Can you reproduce/retrace your steps?
  3. How can you improve your note taking next time?

Bug reporting

Briefing

After finding a bug, you will most likely need to report it. Perhaps this reporting is no more than talking to a developer, but it’s also possible you’ll need to add it to a bug tracker. Whatever form the reporting takes, the basics of a good report remain the same.

One way to remember these basics is Michael Bolton’s PEW heuristic:

  • Problem - describe the problem. The problem is the summary or title of the bug, so be concise, but specific.
  • Example - provide an example that shows the problem: what did you do and observe? The example connects your problem description to observable behavior of the application. Without it, reproduction and/or debugging will become extremely difficult.
  • Why - why is this a problem? Based on what do you consider this problem a bug? The why is the evaluation of the problem and example description. Why is what you described a bug or problem? Based on what are you saying that the behaviour you described is not as it should be?

Exercise

  1. Read the following blog post about an old bug in Google Calendar: http://shkspr.mobi/blog/2014/01/another-google-privacy-flaw/
  2. Using the heuristic write down the Problem, Example and Why of this bug.

Evaluation

Immediately after doing exercise:

  1. How does separating Problem, Example and Why help in writing a clear and easily understandable bug report?
  2. Compare the blog post with your own bug report. Which one does better on which of the three aspects?

After a few hours or days:

  1. Read the bug report you wrote. Focusing only on the bug report evaluate it on
  • understandability
  • clarity of the example
  • communicating why this bug is a bug

Sources

Solutions

Interpretation - Sandeep Garg

Briefing

The exercise asked to apply “Mary had a little lamb” to the second sentence of the quote, I applied it to the first sentence as well. Honestly, The reason was, I actually overlooked that. But, still, that turned out to be helpful.

Exercise

  1. You can add reminders in Google Calendar…. You can add reminders in Google Calendar but you shouldn’t do that.

  2. You can add reminders in Google Calendar…. Only You can add reminders in Google Calendar, nobody else.

  3. You can add reminders in Google Calendar…. Earlier you couldn’t add Reminders. Either only Goals or Events, but now you can add Reminders as well.

  4. You can add reminders in Google Calendar…. You can add only Reminders only in Google Calendar. Not Events or Goals

  5. You can add reminders in Google Calendar…. You can add reminders in Google Calendar only, no other application has this feature.

  6. You can add reminders in Google Calendar…. You can just add Reminders , may not be able to delete or update those reminders.

  7. You can add reminders in Google Calendar…. You Cannot Import from other calendars ( different platform, iPhone Calendar ), You have to Add

  8. You can add reminders in Google Calendar…. You Cannot Import from other calendars ( From a different Google account ), You have to Add

  9. Reminders carry over to the next day… By Default? Even if you don’t opt for ‘Repeat Everyday’ option while setting up a Reminder, Reminder will be carried over to the next day

  10. Reminders carry over to the next day… Only Next Day , may not be at the same time

  11. Reminders carry over to the next day… It doesn’t necessarily mean that original reminder will go away, that will show up on the Calendar for previous days as well

  12. Reminders carry over to the next day… It may result in conflicting** with other Reminder already setup for Next Day at the Same Time OR Overriding an existing Reminder for the Next Day at the Same time OR Corrupting (data loss) both Reminders for the next day**

  13. Reminders carry over to the next day… What if I am in a different Time Zone (time differences) when Reminder carried over?

  14. …until you mark them as done…. “ Done“ can be achieved by Deleting / Updating that reminder?

  15. …until you mark them as done…. “Done” can also be achieved by tapping “ MARK AS DONE“ option in the Calendar App

  16. … until you mark them as done… Using the same medium from where you created it (yours google account on Laptop, Your Mobile’s Calendar App)

  17. … until you mark them as done… From at least one platform from where you created it (yours google account on Laptop, Your Mobile’s Calendar App)

Evaluation

  1. Has your interpretation of the sentence become richer through this exercise?

Yes, Though I already studied “Mary had…” heuristic in past. It always helped in exploring Context, Constraints, Learning about how to read Requirements and Testing Opportunities

  1. How different could two implementations be, if their developers did not share all of these interpretations?

Considering the following Interpretations

  1. You can add reminders in Google Calendar….You cannot Import from other calendars; you have to Add
  2. Reminders carry over to the next day…What if I am in a different Time Zone (time differences) when Reminder carried over?
  3. Reminders carry over to the next day…. It may result in conflicting with other Reminder already setup for Next Day at the Same Time OR Overriding an existing Reminder for the Next Day at the Same time OR Corrupting (data loss) both Reminders for the next day

While developing (Thinking, Designing Solution and Programming, BEFORE start coding), If developers (more precisely in Agile Teams, if team doesn’t work on different interpretations) the end solution may not serve the purpose for which it is intended. Also, it may not have features, fault tolerance, charisma as compared to other comparable products in the market

  1. Based on your different interpretations, where do you think are good places to look for bugs?
  • Reminder CRUD Operations
  • Setting Reminder Frequencies
  • Conflicting Reminders - Application Behaviour
  • Data Corruption
  • Sync Feature
  • Time Zones
  • Import Reminders from other account and vice versa Export

Sources

  • Bit explored the Calendar application on my anDroid phone
  • https://www.youtube.com/watch?v=HVsySz-h9r4
  • https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup
  • https://github.com/Kunena/Kunena-Forum/wiki/Create-a-new-branch-with-git-and-manage-branches

Interpretation - Joep Schuurkes

“You can add reminders in Google Calendar. Reminders carry over to the next day until you mark them as done. For example, if you create a reminder to make a restaurant reservation, you’ll see the reminder each day until you mark it as done.”

Exercise

  1. Reminders carry over to the next day until you mark them as done.
    Reminders, so not events? Is there something else besides reminders and events?

  2. Reminders carry over to the next day until you mark them as done.
    Not hour, not week, not whatever, but day.

  3. Reminders carry over to the next day until you mark them as done.
    What if I cross the international date line?

  4. Reminders carry over to the next day until you mark them as done.
    What else can I do with a reminder? Edit, delete, …?

  5. Reminders carry over to the next day until you mark them as done.
    What other states can a reminder be in?

  6. Reminders carry over to the next day until you mark them as done.
    Can anyone/anything else than me change the status of a reminder?

  7. Reminders carry over to the next day until you mark them as done.
    So when will I be reminded again? At midnight? At some configured time?

  8. Reminders carry over to the next day until you mark them as done.
    Can I do anything else to have them not carry over?

  9. Reminders carry over to the next day until you mark them as done.
    When is the carrying over scheduled by the app?

  10. Reminders carry over to the next day until you mark them as done.
    So multi-select is available?

Evaluation

  1. Has your interpretation of the sentence become richer through this exercise?
    Yes, there is a lot more meaning to distill from the sentence than I initially thought.

  2. How different could two implementations be, if their developers did not share all of these interpretations?
    There definitely could be differences because of the different states a reminder can be in and because of the distinction between the app doing the carrying over and the app re-reminding the user.

  3. Based on your different interpretations, where do you think are good places to look for bugs?
    (1) crossing the international date line or changing the date of the device clock
    (2) state changes of a reminder: different changes by different actors (if possible) in combination with the carrying over

Modeling - Sandeep Garg

Briefing

When I first pulled up Google Calendar (not exactly now, it was a long time back), I was wondering on Why to use this and then How to use this app. Fortunately, this exercise required me to explore the same app.

So, coming to Why, I have reasons… the reasons as a user.

A. I use and admire such applications with a different ‘taste’. Here, ‘taste’ is a subjective thing but being a user, this contributes to biases. The way I look at this application, I want to use it, I want it to fulfill my desires.

B. I usually don’t use Google Calendar and most of the time its Outlook, my official calendar. Also, even in the digital age, I use to circle dates printed on the paper calendar placed on my office desk.

C. Also, I am a bit bad at remembering dates, and I need a calendar, with a useful Reminder facility. Something that uses Data Analytics, so for example, If I set a reminder for doing a practice session for today night 19:00 PM IST and I don’t mark it Done, then Google will set it for next day (as it claims and it does certainly). Fine, but what If I don’t do next day? My desire is, that, it should have the capability to analyze my “Not Done” behavior and probably next day, remind me with the help of some Notification “You are not marking your reminders done, what’s going on” :).

After all, I am just a unique user of this app, and I am ‘probably’ exploring (testing) for my best fit use. I am a user with different “wants” and my perspective of ‘Quality.’

These all characteristics put my perspective as a user into a testing phenomenon called ‘Persona’. I want features, that fulfill my desires, from this product. It may have those, yet It may still be of excellent quality if it doesn’t have those features. But not for me. I would keep searching and if I have to pay some amount, I will.

Exercise

User Tour

So here are few Features I would love to have in this Calendar app

Setting Reminders through Voice - Birthdays (most forgotten), Anniversary, Bill Payment, My Training Days. So basically setting reminders. Through Voice, though (I would Love it). I don’t like typing in apps, supporting voice is needed.

Notification diversity Email, WhatsApp, SMS, Voice assistance

Create my Calendar - I am curious to use this feature, and I need to know what value it adds; how different it is from my standard calendar (Curious to Explore)

Importing and Sharing of Other Calendars - I want to set my dates in sync with at least my Wife and some close friends, so that we can plan something together as we are all working - yes, I can do through some Taps, no issues (It would be great to see this)

Holidays - My work locations and from where people can often contact me - Voice would be excellent, but taps would work (A should have for such a brand)

Privacy of my data

Note: I didn’t explore the application yet except for setting a reminder by typing text and then using Google Now to set a reminder. Google Now is awesome

Data Elements: From Data elements, I interpret the “data” (input, output, stored) that this Calendar Application Takes in, process, store, transport, purge usually during CRUD operations, Migrations, Import / Export, and Reporting to name a few to start with

Data Tour

  1. Voice- When I want to add a reminder, Voice will be a Data which will be processed by Application

  2. Text Data - Strings that I will enter to give a meaningful name to a Reminder

  3. Dates, Calendar Objects - Google Calendar-centric in native language / foreign language

  4. Dates, Calendar Objects - Other platforms

  5. Colours - Colouring different calendars, dates, etc. will require application to process color codes and display accordingly

  6. Data items which we can set in Calendar Settings - Usually they will be fixed and sometimes will be touched, but it would be interesting to see how those settings interact with user data (Date Format, Weather Information, Video Calls, etc.)

Configuration tour

Probably one of the easy one to look at, because here I can concentrate on Settings of this app. What ‘calendar’ app is offering right now from “setting preferences” perspective. However, it would be challenging to prepare a good strategy for testing this important feature

Display Density

Settings - Language, Country, Time Zones, Weather, etc.

Unsubscribe Feature

Labs - Latest ideas to customize the calendar settings

Evaluation

– User Tour - What I would want to Explore, Understand, Check and then Test –

  1. Voice Processing Test - Setting reminder via Voice. It would be my preferred test, as I consider my test mission to ‘evaluate application’s voice processing capabilities, efficiency, and language diversity.’ Maybe, I will use my Indian accent (bit neutral), Maybe I will ask my American friend in Texas to explore, then my Belgian counterpart and my Dutch friend (who sometimes don’t understand her audio recording :)). I would love to use open source tools to add value to these tests.

  2. Google App’s Integration Capabilities - Explore the Options, channels this app gives me to add Birthdays to this calendar. On an Android phone, e.g. I have my Contacts, some have Birthdays added, Anniversaries. Can I add a reminder for someone’s Birthday by saying “Go to a bakery to book a chocolate cake for Charu.” The moment I stop, app start searching for a contact “Charu” (let’s assume by providing me a list of all “Charu” and sorting in order such that selecting the one which is most frequent called?). Then, it adds a reminder for her Birthday today night 12:00 with an B ’day icon“.

  3. Sync Feature Test - Checking out how it syncs different calendar If I go to create my calendar let’s say by choosing some dates from US Holidays, India Holidays, My Vacations and my Wife vacation dates

  4. Exploring APIs used by this application - App making API calls while CRUD operations, Importing/Exporting over the web, Sharing calendars, etc.

– Data Tour - What I would want to Explore, Understand, Check and then Test –

Voice data processing test - It slightly overlaps with Voice Processing Test, but intention is to understand the following

  1. How app weighs different voices while converting into text? How it impacts the application performance, due to different frequency of Voice data

Text Data

  1. Entering reminders in the different languages (maybe via other interfaces) and how are they stored?
  2. What happens when these reminders are updated, deleted or overridden?

Data items which we can set in Calendar Settings

  1. Playing with Data items in Calendar Settings would be more interesting. Video Calls, Weather Information, Date Formats, Timezone specifically
  2. Adding a reminder in IST calendar, importing a reminder from EST one? Updating an event from EST timezone to IST timezone?
  3. Internationalization and Localization Testing

– Configuration Tour_ - What I would want to Explore, Understand, Check and then Test –

  1. Labs ideas about

Automatically Declining Events, Background images for different calendars, Free or Busy

  1. Calendars

Calendars I can only view, Import, Export calendar

  1. General

Speedy Meetings, Event Dimming, Weather, Adding Video Calls, Keyboard Shortcuts

Above tests are interesting from exploring application perspective with intent “how does it behave” and “what is the learning.” Understanding how application is acting with different types of data, combination of data, sequence of data may help in identifying a pattern which will guide the testing

Sources

  • As referenced FCC CUTS VIDS

Blog posts about the four-hour tester

By Ash Winter

By Marvin Bodziuk