Technical Testing vs User Experience Testing

Anyone who has worked with me has been subjected to my rants on testing.

1. That time needs to be allocated for it. More than you probably planned for. (You did plan to test, didn’t you?)
2. That a good test plan includes both the “does this button work” testing AND the “can I do my job” testing.
3. That having your new users involved is just as important as your expert users.  You get different information – equally valuable.

This plan applies to small-scale single-course projects or large-scale IT solution implementations.

Technical Testing.  Do the buttons work? If I click the link, do I actually go where I am supposed to?  Do all of the functions work as documented by the vendor? Your project team and a few fool-hardy volunteers can do this. At the end of this – you want as clean an environment as possible before you give it to the next team.

What you need:

• Your project team’s time. Even if the vendor claimed to test this – do it AGAIN on YOUR side.
• An issue log. And force them to use it to document any solutions they are able to implement themselves so you can keep track of changes that have been made – on the off chance it breaks something.
• The training and communications team’s time.  If they haven’t been involved in the project to date – you need to bring them in NOW. I am including the documentation specialist with this team.
  • These folks will also provide first usability feedback. However, the primary goal of this phase is getting everything working.
• Vendor documentation (for larger-scale implementations).  This allows you to see whether the solution works as the vendor claims it does, allows the documentation specialists and trainers to identify any gaps in the existing documentation, and helps the project team really learn the tool from the end-user’s perspective.

Expert User Testing.  The expert users will do first pass of any major workflows in your organization.  Your training team will also need to be involved.  They will be looking for whether they can get through their day.  Chances are – they will identify the breakages between modules (since that tends to be what gets found in the field – even after you have gone through a clean technical test).

What you need:

• Identified expert users.  Ideally, you also want people who understand that they are testing a tool and things will NOT be perfect.  These people will be some of your champions.  If you have people who are identified complainers, you want to invite them to the next cycle of testing.
• Your project team’s time.  They will be capturing errors.  If they are doing onsite error mitigation, force them to use the issue log to track changes.
• The issue log.
• Your training and communications team. They will help with observation of how they are using the tool. Where they are getting stuck with navigation. Common questions. All of this will appear in the final training materials.
• A high-level test script with the tasks they need to execute.
  • You want to observe how they get these tasks done.
  • Document the steps they perform to do the task.
  • If you are doing this remotely, encourage THEM to document the steps they perform.  And thank them profusely when they do so 🙂

End-User Testing.  This is where you invite your new users and nervous expert users.  Whether we like it or not, the experience of the folks participating in this testing effort will impact the utilization of the solution.

What you need:

• A clean, error-free (or as close to error-free as possible) implementation of your tool.
• Your project team’s time.  They will likely be spending a lot of time with permission corrections.  They will also be capturing any errors.
• Your training and communications team.  They may be performing multiple tasks during this time.
  • Initial training on the system – also serves as a dress-rehearsal for implementation and operations training for the rest of the organization.  For a single course – this likely won’t be necessary.
  • Capturing observations of how people are using the tool, how people are navigating and common areas of difficulty.
  • Capturing common questions.
  • Translating between the end-users (“I can’t log in”) and the project team (“Can they find the page? They got into the system OK and can’t get to a module? They forgot their password?….”)
• A detailed test script for each user with specific steps for them to follow. This is also an educational opportunity for them.  The goal is to make these testers as comfortable as possible.

---

If at all possible, at least one session each of Expert User testing and of End-User testing should be done in person.

It is very difficult to track where people are having difficulty online. In person, I can see facial expressions, pauses over the mouse, where eyes are tracking, frustration, etc.

I also have not personally found a really good solution to seeing multiple screens at once remotely.  Much easier to capture people who are stuck if I can see what they are doing.