Sometimes it is handy for people to send messages over IM. It can help manage observers in observation room and the moderator can see comments.

Survey

Team (observers) liked asking multiple questions by IMing them to moderator. However, the load on the moderator is increased as if they do not have enough to deal with.

Colleague thought that IM would distract team too much during user studies.

Study:
- 11 moderators
- 15 observers

Is IM usefule in methodology? Likert scale, 1-5
- Moderators = 2.3

Negative Feedback
- Moderators worried about distraction
- Moderators missed several interactions by paying attention to IM
- Can distract the observers in the observation room. Observers too busy chatting amongst themselves
- If observers are communicating with me, they cannot take good notes
- Danger of back seat driving - Observers can dictate moderators too much. (e.g., Product manager says, "Tell him to click OK")
- Distraction to the user. User may notice they are not getting appropriate attention. (Distract users typing on laptop even if they thought I was only taking notes)
- Observers may not turn up. They can only pretend to be next store - an extreme example, but a concern brought up in the survey. Observers would be less likely to get involved early on in the research project for creating the script.

All is not doom and gloom
- IM can be used to deal with problems of the prototype quickly (e.g., rebooting the server - occainonally a buggy prototype slips through)
- Engineer can help with debugging (e.g., don't go there, bring him back to the home page)
- Can help for handling extra questions
- Useful for training (e.g., observers can help new graduates/hires conduct the study; can help deal with unruly participants)
- So I can call in from the pool - observers saw working remotely as a positive thing whereas moderators did not.

Guidelines

There is a common middle ground.

- Neticare needed - appropriate use of IM needed
- 1 way IM only - observers only IM to moderator; not back and forth (reduces load on moderator)
- One person in the observation room relays messages to moderator. Ameliorates load on moderator.
- Create dedicated secure account for IM account. So you don't have your mother's IMs popping up during tests.
- Only do IM at the end of the session. Moderator can ask for questions before wrap up.
- Don't distract the user. Turn off sound so they don't hear when an IM comes in.

IM is not for everyone, but it can be very handy. Can be particularly helpful for buggy prototypes, training new moderators, and remote observers.

Moderator should be able to decide which guidelines to use and enforce.

Questions

1. What are the advantages of this paper - if you cut down on observation room chatter where the different observers are IMing each other. Is this more or less distracting to have communication during studies?

Good to have them talking as long as they are not talking too loud. We have reasonable sound proofing, but we have used IM to tell them to be quieter. 1 person in obseration room IMing is good.

2. [Nokia] Did you find IM being very much used tool by all the people you interviewed? Did they use it frequently? For me, using IM is not very common in tests we organize. But it depends on the technology when you use web page systems and such.

It is actually very rare. Most of the people mentioned the use of it once or twice in various contacts. The moderators have tried it, but were put off by the chance of distraction and back seat driving. 1 or 2 used it regularly as well as myself.

3. Have you ever tried putting subject matter expert in the room with you?

I guess we could do that. Having more than one person talking to the user can make it quite stressful for them. That is one option.

[Xerox] We have tried it and it is helpful when you have a buggy prototype.

[Xerox] Do you think a tablet PC would be more helpful so the participant is less burdened by your attention to a PC.

Yes that is a really good idea or some type of handheld device. Good idea.

3. [Texsmith] Do you use any kind of technology to record what the users are doing? Is there a reduced need to take notes and use IM more if you are recording?

Good point. If people are taking notes, than the IM window can be the point that they capture those notes. We use video as well so people can go back and take more notes. SO it is very much up to the individual observer. So ideally we want to separate the notes and IM to take load off the moderator.

4. [Fidelity] I am surprised by your guidelines to IM at the end of the session. Is there any instance/issues where IMing during the session would be better so you don't have to back track?

Personally I recommend IMing throughout the study. At the end of the study it limits distraction and load on the moderator. Perhaps a protocol can be set up for if a serious thing happens IM can happen.

[Fidelity] They do have communication during tests.

I have done that as well previously, but that alerts the user that they are being watched through the mirror. Usually people forget they are being watched otherwise.

Thank you.

---

Influences of Personal Preference on Product Usability

Did not show up at CHI.