Wednesday, April 7, 2010

User testing with students

After musing a lot on interface design and evaluation, I've finally ended up with a thesis subject. Though slightly complex, it goes something along the lines of the following:

A lot of research takes place in Human Media Interaction, the Computer Science department and the university as a whole. But a lot of this research (at least as far as psychology and CS are considered) is done on students.

Why? Because students are a handy group to use. They're generally found near universities, have a large amount of spare time, are interested in research and are willing to take part in experiments.
In fact, they're such a handy group of students that about 50% of HMI research is done with students exclusively. And psychology has skirted up to the insane percentage of 90%.
This is -in my opinion- bad research practice. Students are a very specific subset of the population and in my research I will try to prove that there is a distinct difference in using students and non-students in HCI experiments.

Students groups differ from a theoretical "average population sample" (if there ever was such a thing) on the following features:

Students are a very specific group of people age-wise, with ages ranging from 17-19 (enrollment age in the Netherlands) when starting their studies, with most finishing their bachelor and master studies at age 22 and 25 respectively. The majority of students in the Netherlands is both older than 17 and younger than 27.

A no-brainer, bachelor and master students at universities are receiving the highest level of education available, discounting the scant few post-doc students that are around.

Though students in general are pretty much evenly distributed in male-female ratio, there are some interesting outliers, especially at the more technical studies in the Netherlands, which are unpopular with women for some reason. A high prevalence of men (Or a shortage of women, as they themselves call it) as CS students means that most research involving CS students will mainly (or only!) have men as test subjects.

Technological Aptitude
Technological aptitude is a measure of the ease and comfort the subjects of research feel when presented with a technological artefact, problem or interface. Most CS students are very adept and experienced at using computers. Moreso than other students and very much more so than the 'average population'.

I believe that CS students are a bad group to use for experimentation if this is the only group used. Good research requires good data and researchers doing user testing would do well to do so over a group varying over the 4 features named above.

Sunday, February 28, 2010

A story

And just to top it off, a little story that applies to all the things i said earlier:

Spider loved dancing. He had 8 legs, which made him one of the best dancers known, save for one: The centipede.
No matter how hard he tried, the spider could never outperform the centipede with its hundred of feet, literally!
The centipede danced and danced, as all other insects nodded in praise, but spider had a plan.
"Centipede!" Spider said, "I just looove how well you can dance."
"Thank you." The centipede replied, and danced a little jig just for spider.
"I love how you move your 22nd foot just before you tap with your 59th foot." spider said.

The centipede stopped dancing.
"And the way you moved your odd legs on your left side in tandem with the even ones on your right side. Marvellous!" spider said, smiling .
Centipede was dumbstruck, he never really thought about dancing, he just used to do it.

"Could you dance again for me?" spider asked, smiling very broadly indeed.
Centipede moved his first leg, the one one the left side and stopped. "No wait" he said, droplets of sweat on his brow. "I think i forgot how to dance!" he cried out.

Spider nodded and smiled, knowing that centipede would never dance again.

Interfaces (3)

Continuing with my education, there's another thing i've noticed about interfaces and their design and/or evaluation:

In using a device, there are always 4 actors, though some barriers might get blurry when it comes down to really simple programming:

1 - A user with an internal model of the goal, actions and preferences that the users wishes to bring into being. The user feels a need to change the data on the device. How and why depend mostly on the abstraction or the metaphor that the user has created for the usage of the device

2 - A physical interface that is handled by the user to manipulate the software interface. This is the input/output layer where actual communication with the device takes place.

3 - A software interface that translates specific actions with the physical interface into predetermined actions that can be understood by the software model.

4 - The software model, which is the full functioning of the software at the lowest level[1].


To elaborate : Most interfaces hide the inner workings of the software. A user could press play on a mediaplayer and is blissfully unaware of the fact that a filestream is opened, sound hardware is initialised and data is sent to the audio interface. Just like most users do not need to completely understand a car to drive it, the car itself will need to know.
I suspect that a lot of trouble with interfaces stems from a large gap between the complexity of the software model and the mind model of a user.

Interfaces (2)

After reading several papers on the previous subject, namely interface evaluation, a few interesting things do crop up.

Conscious evaluation
I find it interesting to note that (And i'm paraphrasing here, as i could not trace this specific though) "Good technology weaves itself in the fabric of our lives, becoming invisible but still serving its purpose." The irony here consists therein that all interface evaluation attempts to create solid, qualitative data on the usability and 'betterness' of an interface by actually drawing the attention of the user to the interface.

When performing a task either consciously or unconsciously does influence the way a task is handled. And most interface evaluation comes down to the testing of one or more new interfaces, effectively only evaluating the behaviour of users learning to use the interface. I believe this results in a bias towards easy-to-learn and simple interfaces.

But back to the conscious and unconscious tasks. Undoubtedly you've walked through a door several times this day, noting the door, but not the interface : the door handle. Stare back at your door, walk up to it and open it whilst being aware of what you are doing. By making the interface visible and experiencing it consciously, you've changed the way, timing and experience of using the door. And that's just a door!

What metrics
As explained in the previous post, in HCI we attempt to formalize ways to determine wether an interface will be a good* interface. Preferrably, we'd form some kind of repertoire of metrics that can be applied universally and still be true.
Sadly, however, this is not the case. The efficiency of an interface is something that can only be measured in relation to the underlying goals of the interface, which are generally somewhat fuzzy. It would be folly to attempt to use the same metrics for an accounting program to a game for children as the metrics for the accountants will not incorporate things such as fun, learning, experience, etc..

Next to that, most of our interface design knowledge comes from a very narrow domain:the standard interface or the use of a computer with a mouse and a keyboard, which has been the prevalent (And to a certain degree, invisible) form of physical interface to software up to this point. In the twenty or so years that this physical interface has been used, both our software interfaces and our knowledge of them in this domain have matured and grown. Outside of this domain, however, the old methodology breaks up. I believe great damage can be done to the potential of new physical interfaces (take for instance mobile platforms) as the expertise of the standard interface is wrongly applied outside of its own domain.

This worries me most of all; interface evaluation is still a very narrow field as far as metrics go. I had expected a richness of all kinds of metrics, but most research is still done using simply timing, users rapporting on their feelings and easy-to-measure input events such as mouseclicks and/or keystrokes.
Timing is the main workhorse of interface evaluation, but measures only one thing: Speed of use when performing one specific pre-set task in a new enviroment.
Users rapporting is somewhat fuzzy, even after statistical analysis, as most users in these tests are CS or HCI students, instead of aveage users
And lastly, input events do not translate well over different types of physical interface as the method of input itself differs over these platforms.

And still these are the most used metrics in interface evaluation. There is a creeping suspicion that interface evaluation metrics are used so the researchers are able to at least get some qualitative data, as opposed to truly attempting to determine the viability and effectiveness of an interface. This also shows another problem with current metrics: They measure primarily efficiency in tasks or user preference.
I find it hard to believe that all software could be correctly evaluated over just these two dimensions.

* The discussion on what makes interfaces good is actually a very broad and large one, which i'll forego here.

Tuesday, February 2, 2010

Interface evaluation

As part of a research subject at the university, I'm currently reading up on the evaluation of interfaces. You might not realize it, but a lot of thought and work goes into making the interface of a software product nowadays. Take for instance the new Office 2007 interface, which seems to be the new defacto standard for microsoft (open paint and wordpad in windows 7, if you haven't yet.)

There's some really basic ways of testing and designing your interface, but it comes down to a few simple metrics:

Thou shalt be clear and concise
When making an interface, it is good to separate different functionalities. That way, a user can focus on only that functionality that he or she wishes to use. Next to that, the switch of focus to another grouping of functionality is easier, the user can easily discern that certain functionality is not present in this group and determine the correct group more easily.
Simply put: Group functionality that is alike together, and give this functionality a way of standing out as grouped.

Thou shalt not offer too much information
People can only handle a finite amount of information. The more information you add, the more time it takes to properly use this information. Read up on the "7 plus or minus 2"-rule if you're interested. Also, this is why we want interfaces to be neat and uncluttered. It's harder to keep focus and actively perceive cluttered data.
This all boils down to one point : Information overload, which can impair or even paralyze decision-making when using your interface. For this reason it is better to use few elements in an interface

Thou shalt not make it too much work
Good interfaces are intended to support your actions and should never needlessly hinder or delay the wish to use an action. A simple metric for this in classic interfaces are clicks or actions taken to actually get something done. If I need to interact with my interface twice instead of thrice to make it do what I want it to do, my interface is better. Do this for all possible interactions and multiply this by the incidence of each interaction, and you'll get a nice weighed average that indicates the complexity of your interface. Again, low is good here.

These principles automatically take you through a few decisions that are very visible in the new office interface: Functionalities are grouped, the ribbon tabs make sure you only need to take two actions to actually reach the specific action and functionality you seek, the ribbon gives you a clear and concise view of the functionality within the group and lastly, only the 'active' functionality is visible, reducing clutter and ordering the interface for use.

Not that this is the best of all possible interfaces, but it is a good case study of basic interface design. But there's one catch; this is all classical interface design. These principles have been known for years, if not decades, but have finally grown into firm guidelines for the making of classic computer interfaces. Again, this is a good thing(tm), as initially (and we're speaking seventies and eighties here) interfaces were mostly an afterthought.

But how will we deal with other, new types of interfaces? The obvious way is to adapt current interface standards to the new interfaces, but this does pose some difficulties, as there are myriad ( Speech, haptic, BCI, multitouch, 3D or combinations of such) forms of interfaces that by their very nature offer different forms of interacting and showing information.

Each of these interfaces is different and offers different ways of interacting through them, which is probably the key word here: through.
A good interface should become invisible and disappear from view as soon as possible, as it only exists as a framework for a user to interact through, translating the abstract thoughts and wishes of users into precise and defined actions that are then translated to an abstract model inside the computer.

The direction this course and research is quickly taking is along these lines: Not quantifying merely the way we interact with an interface, but also determining how and what we think and how much cognitive work is needed to effect a wish through means of the interface towards the computer. A more pro-active approach that I'll hopefully delve deeper in as this research project advances

Wednesday, January 13, 2010


Being a professional IT-nerd, I have the nice chance to both work at many companies in a year, spending a few weeks or months wherein I interact with every layer of a company that even remotely thinks about using computers.

And though the technical knowledge I gain at these different jobs is a very good economical motivator to keep on doing this line of work, the social knowledge and experience is something I highly value for myself.

In earlier days I generally scoffed at those studying management, business and HRM, but I've learned that a company is a very complex creature and I've gained a modicum of respect for those whom manage this well or attempt to do so.

I say a modicum, since I've seen the same all too human story of failure and ineptitude played out in every company, albeit in different amounts (And in some companies the story actually ends well, bless them).

It starts out small and well-thought. A few people find a way to make money and start applying their effort and ideas. It works and soon they start repeating and expanding upon their initial idea to fill their niche. This is probably the most basic description of any business. But as the idea grows, more people are hired and employees are no longer able to discern the workings of the company as it has grown more and more complex. And why should they? They (The new ones) are just employees, whilst the original owners are moved up and lose sight of the primary process of the company. The owners reap the benefit, but the employee has a steady job and just has to 'do his job'.

Groups form. Every group starts off to do work, but parkinsons law quickly starts to apply, with every group isolating themselves. This happens both in work time spent in contact with other groups and physically in the location one occupies with a group or department.
In one way, this is the direct result of specialization, but i'd like to argue that this also encourages "I just do my job, and the rest is not my responsibility" as a work ethic.

It is at this point where communication starts to falter. Sales sells something impossible, management sets lofty but unattainable goals or key employees taking a holiday. The point is, that as a company grows, it is virtually impossible for an employee to actually know and see what is going on in the company. Heaven forbid that they actually try to do anything about this, as it is a social taboo to encroach upon the domain of other groups.
Besides, shouldn't you be doing your normal work?

In our collective wisdom, however, we have decided that someone needs to have an idea of what is going on. We have a word for people whom are supposed to make sure that we can do our jobs well whilst also keeping the company running as smoothly as possible : They're called managers. (For those in the know. the non-pointy-haired version)

I am tempted to go on a rant on what strange social mechanisms have been involved in evolving the previous job description into the simple word "Boss". (Or take the civil servant, an oxymoron nowadays.) But i've got another point i'd like to make:

We have shifted the responsibility for the company away from ourselves. It is now the domain of that elusive group of managers, whom are now the sole owners of that responsibility and like all other groups, have formed their own culture and extra work to inhibit the availability of their time.
And as this group detaches itself from the other groups, it loses that key quality that is generally found on the lower rungs of the workplace : Actually knowing what it all is about. The group of managers are no longer able to grasp the other groups. My point? A good company has managers AND leaders that are aware of these groups encroach upon the working enviroment and are knowledgable (or preferrably: experienced) in navigating and working in the other groups.