I’ve worked with my teammates on VS Core to come up with this list of tips on writing good bug reports.  Take it from me as a tester, the better the bug report, the easier it is to get the issue resolved quickly and adequately. 

Tip #1 – Clear, but minimum Steps to Repro

Let us know what are the minimum steps needed to reproduce your issue, but be clear as possible what the steps are.  When in doubt, include whatever information you think will be helpful.  Better too much info than not enough.  And if we can’t repro, we’ll be in touch =)

The fewer steps required to reproduce an issue, the clearer the root cause becomes for us.  For example, make sure a project is required to reproduce the issue.  If you are running into an issue docking a tool window to a certain location, an open project may not be needed to reproduce the issue.

Also, be as clear as possible when providing repro steps.  Instead of saying, “now reset VS settings”, let us know exactly how you reset the settings, for example, did you use Tools – Import Export Settings, delete registry keys, run devenv /resetusersettings? 

More information on providing solid bug report can be found at http://blogs.msdn.com/productfeedback/archive/2004/07/01/171229.aspx

Examples

Note: I’m just making up these examples.  To the best of my knowledge, these do not exist on any blog, forum, or Product Feedback Center bug.

Not-so-clear bug report

Title:  Tool Window Dragging Issue
Body:  Whenever I try to dock my Tool Window with another tool window, the docking targets never appear.

Clearer bug report

Title:  Tool Window fails to dock with other Tool Windows
Body: 
<version type, build #, general default settings, etc.>
Whenever I try to dock any Tool Window with the Find and Replace tool window, the docking targets never appear.  I can dock tool windows with other tool windows, but just not with this File and Replace tool window.  I verified that I don’t need to have a project opened for this to occur.

Resolution

In the case of the clearer bug report, I could quickly reset to the general default settings and open the Find and Replace tool window.  Since the issue is specific to just the Find and Replace tool window, I can limit my investigation to just it.  In the Default Settings, the Find and Replace tool window is set to “Floating” by default.  In order to dock another tool window to it in this state, you need to first change the Find and Replace tool window to “Dockable” by either Right-Clicking on the title bar or going to the Window menu.

Tip #2 – Version Info

Let us know which version of Visual Studio you are using.  For example, Visual Studio Team Suite, Professional Edition, Standard Edition, or an Express Edition?

Tip #3 – Default Settings Selection

Which default settings did you select on first launch or last reset to?  Don’t remember?  Go to HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\8.0\Profile and look at the LastResetSettingsFile value. 

For example, if you selected the General Development Settings, you’ll see “General.vssettings”.

Tip #4 – Build #

Are you using the official released version of Visual Studio 2005 (build 50727.42)?  If you’re not sure, go to Help – About to check.  You can also use the “Copy Info” button to paste in this info.

Also, let us know if you had a previous version of Visual Studio 2005 installed on your machine.

Tip #5 – Solution Configuration

The more details we know about your solution, the closer we can setup our machines to reproduce the issue.  This information is imperative to reproducing your issue on our machines.

What sort of projects do you have in your solution?

What language is your project(s)? 

What project template(s) are you using? 

Are you using any Web projects? 

For example, here’s what a solution could look like:

MySolution
– WindowsApplication (C#)
– ClassLibrary (C#)
– <Solution Folder>
— ClassLibrary (C#)

Tip #6 – Reporting Crashes

If you happen to encounter a crash, please send the report to Microsoft to help us investigate.  Then let us know the crash ID number you sent in the report.  To find this information, please do the following:

1.  From Start – Run dialog, run "eventvwr.msc"
2.  Open the "Application Log" node.
3.  Look for the newest event with Source="Microsoft Development Environment" and Event="1–1"
4.  Open that event and you’ll see "Bucket" followed by the crash ID number we’re looking for

Tip #7 – Consistent Repro

Let us know if you are consistently running into the issue or if it just happens once in a while.

Also, let us know if the issue reproduces using different projects, if you shut down and restart Visual Studio, etc.

Thanks again for your help making Visual Studio better than ever!

Happy Visual Studio’ing!

These tips apply for all testers, not just Microsofties.  What other tips would you include in this list?

Bend over backwards to help your dev
You’re going to be breaking his/her code for the rest of your careers together, so you want to have a great working relationship between each other.  First off, establish trust with your dev.  If you say you’re going to buddy test something by such-and-such time, get it done.  Actively seek feedback from your dev on what features are lacking testing, how you can do more testing, etc.  And whatever your dev needs from you, like ad-hoc machines in the lab, status of nightly runs, get that info to him/her asap.  It won’t take long for you to see the benefits of going out of your way to help your dev.  And it’s not just about using common sense, it’s about making sure you’re doing your part in helping the team succeed and how you would want your team to help you out when you need it.

Leave appropriate comments in every bug you verify and close
1 year, 6 months, or even just 3 months down the road you won’t remember how you verified that bug fix.  If you can no longer follow the original bug repro instructions (sometimes bugs morph, other times the fix doesn’t allow you to get to the end of the repro), leave comment explaining exactly what you did to verify the fix.  And once again as common sense would dictate, make sure the build number that you verified the fix on is written some where in your bug tracking database.

Never Assume Anything
Never assume that

• Someone else is covering that scenario – Test everything you think of.  A little overlap never hurts.
• Triage (or whoever is reading your bug) will “just get” the bug.  Always be explicit with repros, even when it’s completely obvious (to you) in the attached repro picture.
• A simple scenario could never be broken.  Always, always, always test – even if it is the simplest feature or part of the feature in the world.

Don’t just get it in writing
Email just isn’t enough.  If you’re not going to cover an explicit scenario, get it in writing in your test plan.  If you’ve discovered an issue and the issue isn’t going to be fixed as stated in an email thread, make sure it is in your bug database.  Email is like leaves on a tree.  You get a lot of new email in, you save a lot of email off to the side (like leaves in a large trash bag which makes for excellent tree house furniture), but after a while, saved old email just gets older and unneeded, just like your furniture that is now collecting spiders, so you throw it out.  Make sure any major decisions involving your test strategy or test bed or written in the appropriate document / database / forum, etc.  And yes, that was my worst analogy ever.

Learn from the bugs you missed
Ask yourself why someone filed a bug report in your feature area that was later fixed?  What could you learn from this bug?  Are you missing other similar tests?  This process is actually called “Root Case Analysis”, but you don’t have to do the formal RCA work to learn from bugs that got away from you.  So many times, I find myself staring at my monitor just wondering what I’m missing.  The answer to that question is just doing a query of fixed bugs not opened by me.  Surprise yourself by seeing just how much you can learn by doing this.

Got a tip?  Add it to the list!

View original comments

Finally, the promised post.

Types of Controls

There are three types of controls you’ll discover when automating UI

1. Standard Windows controls
2. Owner-drawn controls – controls that have the functionality of Standard Windows controls, but have custom drawing. These controls will have to implement IAccessible to describe the custom paint job
3. Painted controls – I’ve asked around what the term was, and everyone I asked responded with “they are just painted”. This is where there’s only one window, but within the window, numerous controls are painted.

How automation typically works

Automating UI requires three critical aspects:

1. Capturing controls
2. Manipulating controls
3. Acquiring data from controls

These tasks can usually be achieved via Win32 API calls, MSAA, or in some cases, from the application’s object model when dealing with Standard Windows controls and Owner-drawn controls.

For example, consider capturing a regular button to verify its caption and what it does when clicked. You would first use a tool like Spy to find the control and to get information on the control, like its classname. From the automation framework, you would search for the window with this particular classname among all available windows to get its handle. Using the handle, you’ll be able to use Win32 API calls and MSAA calls to either manipulate the control, ie pressing the button using IAccessible::DoDefaultAction() or acquiring data from it, ie getting the button’s caption using GetWindowText().

But what happens if the control doesn’t have its own window?

Consider File Tabs on the File Tab Channel. See my post on the File Tab Channel for a picture of what this looks like. If you were to use Spy on the File Tab Channel, the first thing you’ll notice is that the individual file tabs do not have their own windows. There’s only one window (the File Tab Channel itself) that hosts the File Tabs. How does one capture the File Tabs if all of them are contained within the same window?

Use Test Hooks

Test hooks are “hidden” messages the developer will write into the application. I call these hidden because you have to ask the developer what messages to send to which window (or look at the code). The tester supplies the developer with a list of requirements, like being able to press the button or get the caption. And the developer in returns gives you back a set of messages to send to a given window that will produce the desired effect, like click the button or return the button’s caption.

Here’s how an automation framework might use the test hook to get the caption of a particular File Tab on the File Tab Channel.

Several things to note:

1. SendMessage() is used to send messages to a control
2. The WM_USER range is typically used to define custom messages to a particular custom control.
3. ATOMs are useful when a string is needed by an external process, which is usually the case in an automation framework. ATOMs are global to Windows.

Const FileTabMessage As Integer = &H8000 + &H1001 ‘ WM_USER + test message app is expecting – note: this is just a number I made up

Const Text As Integer = 1

Const Click As Integer = 2

Private Function GetFileTabText(ByVal fileTabIndex As Integer) As String

‘ send message to test hook requesting the ATOM for the requested file tab’s caption

      Dim atom As IntPtr = SendMessage(MainWindowHwnd, FileTabMessage, fileTabIndex, Text)

      ‘ convert ATOM to String

      Dim buffer As New String(" "c, 255)

      GlobalGetAtomName(atom.ToInt32, buffer, buffer.Length)

      Return buffer.Substring(0, buffer.IndexOf(Nothing))

End Function

View original comments

I met with my team’s automation experts a few weeks back to get their input on when to automate and when to manually test. The general rule of thumb has always been to use common sense. If you’re only going to run the test one or two times or the test is really expensive to automation, it is most likely a manual test. But then again, what good is saying “use common sense” when you need to come up with deterministic set of guidelines on how and when to automate?

Pros of Automation

• If you have to run a set of tests repeatedly, automation is a huge win for you
• It gives you the ability to run automation against code that frequently changes to catch regressions in a timely manner
• It gives you the ability to run automation in mainstream scenarios to catch regressions in a timely manner (see What is a Nightly)
• Aids in testing a large test matrix (different languages on different OS platforms). Automated tests can be run at the same time on different machines, whereas the manual tests would have to be run sequentially.

Cons of Automation

• It costs more to automate. Writing the test cases and writing or configuring the automate framework you’re using costs more initially than running the test manually.
• Can’t automate visual references, for example, if you can’t tell the font color via code or the automation tool, it is a manual test.

Pros of Manual

• If the test case only runs twice a coding milestone, it most likely should be a manual test. Less cost than automating it.
• It allows the tester to perform more ad-hoc (random testing). In my experiences, more bugs are found via ad-hoc than via automation. And, the more time a tester spends playing with the feature, the greater the odds of finding real user bugs.

Cons of Manual

• Running tests manually can be very time consuming
• Each time there is a new build, the tester must rerun all required tests – which after a while would become very mundane and tiresome.

Other deciding factors

• What you automate depends on the tools you use. If the tools have any limitations, those tests are manual.
• Is the return on investment worth automating? Is what you get out of automation worth the cost of setting up and supporting the test cases, the automation framework, and the system that runs the test cases?

Criteria for automating

There are two sets of questions to determine whether automation is right for your test case:

Is this test scenario automatable?

1. Yes, and it will cost a little
2. Yes, but it will cost a lot
3. No, it is no possible to automate

How important is this test scenario?

1. I must absolutely test this scenario whenever possible
2. I need to test this scenario regularly
3. I only need to test this scenario once in a while

If you answered #1 to both questions – definitely automate that test

If you answered #1 or #2 to both questions – you should automate that test

If you answered #2 to both questions – you need to consider if it is really worth the investment to automate

What happens if you can’t automate?

Let’s say that you have a test that you absolutely need to run whenever possible, but it isn’t possible to automate. Your options are

• Reevaluate – do I really need to run this test this often?
• What’s the cost of doing this test manually?
• Look for new testing tools
• Consider test hooks

View original comments

Do you have any software testing topics you want to hear more about? Here are a few suggestions that I’ve already received from readers:

• How do you go about automating?
• How much is automated and how much is manual?
• How does QA fit into the product lifecycle at MS?

Let me know what other topics you’re interested in and I’ll see what I can do…

View original comments