Welcome to my new Testing category. In this category, I plan to discuss what it means to test software, automation versus manual testing, different testing methodologies, black box testing, white box testing, code coverage, and all that jazz.
With four people in my office chanting “Do it! Do it!” I pressed the send button. Last Friday afternoon, I forwarded the Business Week article to Bill Gates. I highlighted the first sentence of the article that mentioned my name and my blog, and, most importantly, I highlighted the last sentence of the first paragraph that read, “And Bill Gates is happy.” I jokingly wrote, “I hope this isn’t a misquote =)” in the body of the email.
A couple of days later, I get a reply back, saying,
I’m happy. Nice article!
It was awesome to grab my coworkers on Monday morning saying, “come check out this mail” just to see the look on their faces (and to here their expletives of course). Before I knew it, I had a small crowd in my office applauding and cheering. Saurabh forwarded the mail around to our product unit. Lots of people have been telling me congratulations. The email hangs outside my door with “Real!” handwritten above the header. The whole experience was pretty funny.
The Business Week article Blogging with the Boss’s Blessing now appears to be available publicly (no subscription required). It should comes out in print really soon.
Fiona, one of the devs on my team VSCore, has put together a really nice sample on how to implement IAccessible for a Custom Push Button. If you’re using standard Windows controls, you get MSAA implemented for free. However, if you’re using custom drawn controls, it is imperative that you implement MSAA (aka IAccessible); otherwise, the control will not be accessible to Assistive Technologies.
There’s a lot of documentation on MSDN how to support MSAA, and now you have an actual sample of how to implement IAccessible! Just point one of the MSAA SDK Testing Tools (Inspect or AccExplorer) at the custom button for a proof of concept.
This sample follows the MSAA SDK guidelines on how to implement IAccessible for a push button.
The sample is located on GotDotNet at
http://www.gotdotnet.com/Community/Workspaces/Workspace.aspx?id=4b5530a0-c900-421b-8ed6-7407997fa979
Enjoy!
The June 28th edition of BusinessWeek has an article called Blogging with the Boss’s Blessing which features me and my blog. BusinessWeek interviewed me last week to discuss why I mix personal information (how I want an autographed photo of Richard Dean Anderson) with technical information (Accessibility, Visual Studio, etc.)
The article is available online right now. The June 28th edition of BusinessWeek should come out in stores next week.
I’m trying to hold back from telling my family until the magazine is in print, so they can run out and buy a copy, but I’m failing miserably. At least now I know my mom has discovered my blog. =) <grins>
While I was giving my CSUN presentation, one of my fellow Microsoft employees attended the Blogging and Accessibility session. He told me that the Blogging and Accessibility session visited my blog and said Hello. For the next couple of days, I was proud to tell people how I participated in two sessions at the same time at CSUN. <grins>
The Accessibility of the Blogging Revolution session proceedings succinctly discuss the following topics:
So, for all you fellow bloggers out there, if there’s only one thing you remember about Accessibility, remember this:
Provide alternative text on images!
I heard this one the other day and wanted to share….
Two strings walk into a bar. The first one says, “Bartender! Bartender! I want a drink!”
The second one says, “Bartender! Bartender! I want a drink too! blaaaaaaaaah Eeeeeeeek yaaaaaaak oooooooh.”
The first one says, “Please excuse my friend. He isn’t null terminated.”
My favorite computer joke of all time….
Once upon a time, a computer programmer drowned at sea. Many <insert profession here> were on the beach and heard him cry out, “F1! F1!”, but no one understood.
Got a favorite computer joke to share?
I can’t wait to go to the local bookstore after work to get a copy of the Song of Susannah. Sunday night I was driving through the Redmond Town Center when I saw a poster in Borders announcing the 6th book of this 20+ year old series. I had to stop the car, so I could find out when it was coming out.
Since Sunday night I’ve been rereading The Dark Tower V: The Wolves of the Calla as fast as I can, including drinking caffeine at night to continue reading. Yeah, flex hours at work are a wonderful thing.
My theory is that Roland is going to find Stephen King sitting on some throne at the top of his Dark Tower announcing he’s retiring. It’s exciting that the last book is coming out this September! (whereas we’ve had to wait years for another book to come along)
Thankee-sai!
My dojo hosted an annual youth karate tournament at a local high school gym a couple of weeks ago. I volunteered to work the concession stand. I figured my superior Walt Disney World Popcorn Vendor cash handling skills weren’t all that rusted. Anyways, I like watching karate competitions.
There were several things about this tournament that I really enjoyed:
It was really cool to see all the kids walk out of the gym as winners. Lord knows parents can get way too competitive sometimes. Just ask me about my 14 years playing soccer, 5 years refereeing, and 2 years as a soccer coach. The best thing I ever saw out on a soccer field was this u-10 girl ask her coach, “did we win?” immediately after I blew the whistle, ending the game. The coach asked her, “did you have fun?” the little girl said, “yes.” The coach replied, “Then you won!” The little girl ran off with the biggest grin on her face.
Zoe’s post Are you a good enough developer to be a Microsoft SDET got me thinking about what I do during a typical day. However, a 3 day period gives a much better impression of an average day in a SDET’s life than just some random day…
Wednesday, May 26th
9am – Arrive at work. I love my Gortex jacket on these miserable, raining Seattle mornings. Today would be a good day to use one of those Expresso Coupons*. I used to get to work at 8am, but because of the lack of sunlight during these bleak mornings, I’m lucky if I’m awake by 8am.
9:15am – Grab a bowl from the kitchen* and some milk. Since milk is provided for free, many of us bring our cereal to work for breakfast. Do mail* while eating breakfast. Setup an OOF mail* that I’m heads-down* on my Focus Day* today.
9:45am – Time to start the UI Consistency Pilot Focus Day. I call this a pilot focus day because it’s just me and another tester running through the test cases before we sign-off on them to make sure the test cases are crisp, deterministic, easy to follow, and have lots of examples. Begin by bashing* on with the Help About dialog as a good warm up. Start raiding bugs*, recording questions and issues in mail, taking snapshots of the UI so I can attach inside the bug reports. A picture speaks a thousand words that I don’t have to type in a bug report =)
I love this type of work that allows me to make an impact on a significant number of people. Since a lot of other people will be using these test cases, I have a lot of motivation today to find good quality bugs*.
11:45am – Reminder* goes off. I’m eating lunch with a PM* I used to work for on VsSettings. I head off to another building, securely putting on my jacket.
1:15pm – Get back into office from lunch. There’s another meeting at 2pm, so I use this time to triage email*.
2pm – My manager has called a team meeting to discuss ideas for Power Toys related to the Help System and other IDE features.
3pm – Back to bashing on some UI and raiding some more bugs. This is one of those times where I really love my job. I have the rest of the afternoon to do nothing but just find and raid bugs. It allows me to get creative (looking for bugs) and really enjoy knowing that I’m finding a lot of good quality bugs for this effort to be used as examples for other testers when they start using this test specification*.
5:30pm – Raid my 19th bug of the day. Although a tester might find three or four bugs within a minute, on average it takes about 5-10 minutes to raid a bug. There’s the looking up whether the bug has already been raided in the database, the report description, minimizing the repro steps*, generating and attaching pictures, and so forth.
5:45pm – Head home, putting on my jacket. Need to eat a light dinner and spend some quality time with the cockatiels before going to karate practice at 7:30.
Thursday, May 27th
9am – Arrive at work on another rainy morning. What did I do before Gortex?? Grab some cereal and some hot tea. Back in Mississippi, we always had sweet tea (iced tea with sugar added while the tea is steeping). When I started working here, I really didn’t want to start drinking coffee in the cold mornings (the mornings here are always cold to me), so I started drinking hot tea. 3 years later, I can drink hot tea without any sugar, but I can’t wait to get my sweet tea fix the next time I go back home.
9:30am – A run* completed Tuesday night and I had 3 failures* I need to have passing as soon as possible. I didn’t have an opportunity yesterday with the UI Consistency effort to investigate these failures, so today I’m considering myself heads-down on automation. I head down to the lab to investigate more closely.
9:45am – discover that one of my test cases is failing due to a slight change in the product. Not sure whether the change is by-design or not. I contact the debugger QA automation contacts asking whether this is a by-design or a known issue. Start investigating the other failures.
10:15am – receive an email from my college’s head of CS department informing all alumni that the newsletter has been updated for this past semester and requesting alumni to send any updates. I reply something along the lines of, “Please see my blog for the latest and greatest information.” I think to myself, “wow, isn’t blogging great.”
10:21am – get a reply from the debugger folks that the issue is by-design. Continue working on other failures in the lab.
11am –Check the calendar and see that I have my VS Accessibility Leadership Team meeting this afternoon. Send out bang mail* requesting everyone make it on time, since there is a lot on today’s agenda.
11:45am – Head back to my office, since I’m finished analyzing* all the failures and have fixed all but that one I needed to talk with the debugger folks about. People on my team stop by asking if I want to go to lunch. Everyone knows that I need my quality human-interaction time.
Noon – head down to the cafeteria.
12:45pm – as I heard this lady tell her dog at the Marymoor park, “Com’on, it’s time to leave. I know, I know, it’s that time again, so sad.” That’s how I feel about having to leave lunch. It’s always that time again and it’s always so sad. I can’t get enough human interaction.
12:50pm – return to the office to check mail. Decide since I have a few minutes, I should try to quickly fix that last remaining failing test case.
1:07pm – start screaming obscenities (and apologize to my dev that shares a wall with me). Outlook reminder didn’t go off, so I’m late to my meeting that I sent bang mail telling people to be on time.
1:10 – 1:30 – meet with my VS Accessibility Leadership team. I love taking meeting notes on my Tablet PC.
1:30 – return to the office. Start fixing that last failing test case.
2:30 – hit a snag while trying to fix that last failing test case. Need to chat with my automation rep* about it since it requires a major rewrite of 2 of my nightly test cases*.
2:35 – Dan is busy. I start generating a report from my blog entry Should VS allow tool windows to maximize and minimize on secondary monitor?
4:30 – Dan walks into his office. I run across the hallway before anyone else can start asking him questions.
5pm – Dan comes up with a great idea that requires very little rewriting. I start working on the test cases.
5:30pm – argh, the workaround is still not working. Dan must have an older build* in his office. Send mail to the debugger team asking for a workaround*. Can’t worry about this anymore today, since I need to run to Karate practice. Tonight’s practice is with the head teacher, a 7th degree black belt. I’ve only met 3rd and 4th degree black belts in my life, so I’m always elated to have any personal time training with someone whose been teaching for 35+ years.
Friday, May 28th
8:30am – Arrive at work. Still need to wear a Gortex jacket. Silently scream as I look out of the windows at all of the clouds and the rain on the way to my office.
8:35am – with my cereal, hot tea, and a bottle of water, I’m ready to finish the UI Consistency testing. I got randomized for a couple of hours on Wednesday when I needed the entire day to dedicate to UI Consistency. (I need a true 1cDay costing) I shut the door with a sticky note “email only”, blast “System of a Down” from my primary computer, and start bashing on the toolbox.
10:55am – Stand up and stretch. I’ve raided 9 bugs this morning, which is pretty good for 2 hours (depending on who you talk to, but in my book, it is pretty good).
11am – I head to the UI Consistency Triage*. Since this is a pilot focus day, we need to triage the 28 bugs I’ve raided in the 1 cDay*. All the bugs look good. One sign that you know a bug was written well is how long does it take people to triage it. The fact that we triaged 28 bugs in an hour is a very, very good sign. Usually, we average 15 bugs an hour for this particular area. Some features are cut and dry whereas others take a little more thought (figuring out the bug and then figuring out what the right fix is).
Noon – Rick, a usability engineer leading the UI Consistency effort, asks me if I want to go to lunch. I tell him sure, since my team hasn’t been going to lunch very much recently.
1pm – time to start rewriting those test cases as Dan suggested. The debugger team sent me a workaround this morning, so it’s now time to see how it does. I must have these test cases passing before I go home, or I’m going to have lots of failures to investigate on Tuesday.
3pm – Yes, the workaround works! All of my test cases are now passing! Sweet. Now, if any of these test cases fails, it is most likely due to a product bug, which i can jump on quickly.
3:30pm – I check on the other person from my QA team* working on the UI Consistency push, making sure the test cases are clear to him and next week’s ETA upon completing the work.
3:45pm – I start writing my status mail, since I’m not going to remember anything I did next week after the 3 day weekend.
4:00pm – start working on a blog entry called “3 days in the life of an SDET”
4:20pm – a developer stops by asking for my opinion on fixing an MSAA issue. We discuss the fix and check with the tester for the impact on automation. Never ever decide to not fix a product bug because it will break the automation. This is a fundamental law of software testing, and being a SDET helps me make sure we’re obeying this law.
4:30pm – Viva la 3 day weekend! Unfortunately, I still have to put on my jacket as I walk outside.
Glossary
Expresso Coupons – awards given to people for morale reasons. Usually received for finding the most bugs in a given hour during a bug bash.
Kitchen – location in the hallways that contains a refrig, drinks, microwave…
Do mail – read, respond, and delete email
OOF mail – a vacation mail notification “I’m Out of Office right now”
heads-down – to be so focused on something, you close your door with a “email-only” note and turn on your vacation mail
Focus Day – a testing term to mean that you’re going to test a particular area of the feature or test against a certain test specification. Difference between a focus day and a bug bash is that there are no Expresso Coupons given out for a focus day
Bashing – testing the hell (forgive me those in the south who are offended by this term) out of a feature, UI, scenario, etc.
raiding bugs – an older term to describe entering a bug report in our bug tracking database. The term comes from Raid, the name of an older application to write bug reports. Raid is also a commercial product to kill bugs, like roaches, spiders, etc.
good quality bugs – bugs that are written so that it is clear what the issue is, it has the minimum number of steps to reproduce the easy, and has a picture attached when needed.
Reminder – an outlook notification to remind users about upcoming meetings during the day.
PM – program manager
triage email – to quickly go through your mail, deciding which mail to answer first and which mail to simply delete
test specification – a subset of a test plan that consists of test cases
minimizing the repro steps – the fewest number of steps required to reproduce an issue. Extremely valuable to developers to know where to start debugging.
bug bash – a day (read: starts at 12:01am and ends 11:59pm) where the testing team does nothing but raid bugs against the product. Prizes in the form of espresso coupons are given out to the person who finds the most bugs per hour. Other prizes include “best bug” (coolest and most interesting bug), “most bugs found” throughout the day, “first bug found”, “last bug found” – yes I have waited for 11:59pm to raid a bug just to win an Espresso coupon. You need lots of Starbucks when you’re in Seattle, so close to summer.
Run – executing a series of automated tests
Failure – after a run is completed, if any of the tests did not work as expected, the test cases are said to have failed
Bang mail – High priority mail. Used conservatively to draw people’s attention quickly to your mail.
Analyzing – to investigate a test case failure from a run, marking the test case appropriate (if it failed due to a product issue, an automation framework issue, etc…)
Automation Representative – the automation expert on the team who does code reviews when checking in new test cases and new automation framework code.
An older build – usually referred to any build older than 2 weeks. A recent build is considered within a week of today’s date. An older build is either good or bad, depending on context. If the tester is looking for another recent build to repro a bug, this is a bad thing. If the tester is trying to determine how long a bug has been repro’ing, this is a good thing.
Workaround – when the shortest path between two points does not connect, an arc is needed.
Triage – a medical term meaning to quickly diagnose wounded personal into categories of “will most likely die”, “50/50”, “will most likely survive.” The “50/50” category gets the most attention. This is the way military triaging was explained to me a long time ago, and MASH episodes seem to confirm this. In the software world, it is slight different. To triage a bug means to quickly diagnose the bug, making sure it has the correct severity and priority set, and either assigning the bug to a developer or resolving the bug as won’t fix, by design, or postponed. A triage consists of three people: dev, test, and PM. Usually, the members of the triage are the leads or managers, so it is essential that bugs contain high-quality well-written bug reports.
cDay – coding Day. A lot of randomness happens around here, from email, to meetings, to people stopping by, automation failures, P0 bugs, etc. One cDay is the same as one day of work without any distractions (aka, no emails, no failures to investigate, no drop everything you’re working on a fix this bug, no meetings). The ratio is 3 cDays / 5 days.
My QA Team – there’s my team (aka everyone on my team that directly reports to the same manager as I do). There’s my QA team (all of the people who report to my manager’s manager). There’s my feature team (all the devs, pms, and testers who work on the same feature). There’s my PU team (or product unit team) – all the people who report to my manager’s manager’s manager.