Input for December 18 Meeting.

2. Overview of testing plan, schedule, budget.

Items #1 and # 2 are online.

3. Assigning preliminary tasks.

Item: D. Formulate test goals (still need some info from CWC)

I'll help with this. The items we need from the CWC are any concerns that they have about:

• Critical tasks.
• Frequently done tasks.
• Tasks done under pressure.
• Known problems.
• Previously dislikes expressed about the old site.
• Previously likes expressed about the old site.
• Specific testing areas.
• Specific target groups (see # 4 below).

These concerns will feed into test goals. A goal is a declarative statement resulting from a question of concern. It establishes an objective to measure. For more information see measures.

I be glad to help write the followup to the CWC and the test goals/measures.

4. Ideas about subject recruitment.

Monday the CWC indicated that the current target audience was the four categories on the home page:

• Prospective students.
• Current students.
• Faculty & Staff.
• Alumni.

But we will need to confirm that in our follow-up with the CWC.

I checked with Judy about obtaining a random sample of potential participants that have representation for all target audiences of the UMD web site. Current students, faculty & staff are all available through the data warehouse. Prospective students might also be accessed that way. We may need to check with the Alumni office for Alumni info. But the good news is that Judy and Deb can provide a random sample of whatever data we need by running the file through SPSS.

After they provide the sample, we could contact potential participants via email. We could easily to convert the example recruiting script and example participant recruiting questionnaire into an email. See:

• Example participant recruiting script.
• Example participant recruiting questionnaire.

Randomization is important to cancel out personal traits and idiosycracies of individuals. (Subjects that volunteer for tests tend have a particular personality type - that becomes a subject variable that you cannot control for. Some may even have ulterior motives.)

5. Student participation in testing team.

After we slot team members to roles, whatever is left over might be good to fill with students. Kathy's idea of using a grad student to fill the role of test facilitator could work out if the student has the right skills, the time to devote to the project and would do it for credit. AND if no other team member is interested in filling that role. The test facilitator is only person who *should* interact with the participant during the session. See:

• Test facilitator checklist.

It would be great if we can get one person to fill this role for the fifteen tests.

6. Other.

As for focus groups some of you know my thoughts. The CWC has conducted focus groups in the past. Now they are letting us do REAL usability testing. A focus group is an inquiry method not a testing method. Focus groups are used to inquire what users want from the web site and their subjective preferences BUT NOT a good method to determine actual usability. A couple of references:

"...Focus groups are very poor vehicles for testing the usability of a site. A public demonstration does not come close to replicating the actual environment of a user navigating a web site. Consequently the suggestions of people in focus groups do not necessarily carry much weight. Sadly focus groups are often used to prove that a particular approach does or doesn't work...a much more appropriate way to study the usability of a prototype or post-launch web site is to conduct individual user testing." - Rosenfeld and Morville, "Information Architecture" p. 171-172.

"Unfortunately, focus groups are a rather poor method for evaluating interface usability...Software products, websites, and other interactive systems also need to be liked by customers, but no amount of subjective preference will make a product viable if users can't use it. To assess whether users can operate an interactive system, the only proper methodology is to sit users down, one at a time, and have them use the system. Because focus groups are groups, individuals rarely get the chance to explore the system on their own; instead, the moderator usually provides a product demo as the basis for discussion. Watching a demo is fundamentally different from actually using the product: There is never a question as to what to do next and you don't have to ponder the meaning of numerous screen options...focus groups can produce inaccurate data because users may think they want one thing when they need another...Another disadvantage to this approach is bias." The Use and Misuse of Focus Groups by Jakob Nielsen.

Both the pre-test and post-test questionnaire during the test session can be used as an inquiry method to give us any subjective information that we need. Ref:

• Example pre-test questionnaire.
• Example post-test questionnaire (part 1).
• Example post-test questionnaire (part 2).

By using the questionnaire in the test session we can save the CWC the added expense of a focus group.

Laura

© 1998-2024 Laura L. Carlson
Email: lcarlson@d.umn.edu
Department: Information Technology Systems and Services, University of Minnesota Duluth
Document Last Modified: 2002
Page URL: http:// www.d.umn.edu /~lcarlson/testing/planning/email_12_18_02.html

The views and opinions expressed in this page are strictly those of the page author. The contents of this page have not been reviewed or approved by the University of Minnesota.