Seminar 2 - Summary

Recollection of discussed topics during seminar 2

• We have learned the importance of evaluating, but sometimes it is hard to user-test something beforehand (scale, patents, secrecy). We also discussed ways to circumvent this.
• For testing our own design, we could use testing in the wild: adding markers and prototypes to the SL stations, and have people try to accomplish tasks (getting from A to B while using our prototype app). This would be testing in a natural setting involving users.
• We could use people from Copenhagen (which has a similar system in place) to heursitically evaluate our prototype and tell us what they feel are the good parts and bad parts of our system when it is adjusted to Stockholm.
• We could also take Stockholmers to Copenhagen and let them see what they like and don’t like about that system, and see how they compare it to stockholm.
• This would be be a kind of triangulation: we see the problem both from the perspective of users who have used it for a long time, and users who are using it for a first time. Their combined heuristics would give us a better view of the actual good and bad sides of our idea.
• Also, Opportunistic Evaluation is a thing that we can actually, practically do (not assuming infinite resources etc). We could even ask the group we’re presenting to what they think, since they are almost surely users of the public transportation. We can also interview people in the subway and present our design to the, ask what they think.
• We discuss how we should evaluate the different parts of our designs, since it is quite broad. It might be good to evaluate each part on its own, ut we also need to evaluate the whole together since we need to know how well the prats cooperate.

Individual notes - Seminar 2

When designing a product it is very easy to get stuck in coming up with ideas, thinking about new things to add to the product or just improving the design as you see fit. It is easy to forget that designing should be an iterative process where we should evaluate the product every so often. An obvious reason to why evaluation is needed is because there might be bugs or other problems with the product that are hindering the usability of the product. Finding such problems early in the development cycle is important as the cost and time to fix them increase over time. Another reason to evaluate is to deal with problems that impact the users experience in a negative way. While designing you as a designer might think that your system is intuitive and easy to understand, but a new user might experience it as nonintuitive and difficult to use. I think many of us have experienced this with webpages where it feels like it's impossible to find what you are looking for, only to find out that you have been searching in the wrong place. 

On a lecture, Jan Gulliksen mentioned two types of evaluation: Empirical and Analytical evaluation. These could both be used during a formative evaluation during the design process to test prototypes. I feel that analytical evaluation is a lower budget evaluation if you can't afford setting up a test lab, if the test can't be performed with test subjects (for example: too dangerous) or if you are standing between solutions that are similar, costly to develop, and can be predicted. The empirical evaluation, I feel, will yield more interesting, qualitative data. It might be costly to use, as you need test subjects, a test environment and a way to record the data, but the information you get will be from the users, the people that will be the ones to use the product. Personally I feel that involving the users is the preferred method.  

An example of an analytical method is the KLM method where you break down a task into individual keystrokes. An important thing to have in mind is that there can be different ways to perform the task depending on if the user is experienced and knows shortcuts or if the user is inexperienced and takes the long route. When the "route" is known you can calculate the time it takes to press the keys and even move the mouse using a formula. An empirical method is the Thinking-out-loud, where the actions on the user and the voice of the user will be recorded. While testing the product the user will think out loud, hence the name, talking about how and why it does certain things. This test is very dependant on the user, as doing two things at the same time can be difficult, especially if you get stuck. Getting some biased results will also be the case, as you'll want an observer in the same room in case the user gets stuck. 

My question is: What evaluation method should we use on our design this early in the process, and why?

Individual notes - seminar 2

Today, tech industry is obsessed with metrics - every decision must be evaluated, every change is A/B tested, and hard numbers are king. I've often wondered how useful this data-driven approach is in practice, since they often only collect quantitative data, and does not focus as much on the reasons behind the numbers

Therefore, I found it interesting that the book takes up other approaches than using technology to evaluate designs on your target group quantitatively. Measuring usability using e.g. interviews and think-alouds can give additional context to the feedback from the evaluation that helps understanding

On the other hand, collecting quantitative data from the use of the product when it is actually has the upside that it more accurately reflects how the product is used in the intended setting. Collecting qualitative data could interfere with the user in its testing of the design, which may bias the results.

I found that heuristic evaluation was very interesting. My view of usability was that it can often be counter-intuitive, since elements of a design may interact with each other in ways that are not obvious. However, using heuristics such as Fitts' law seems like it could provide useful information despite this. So while it is very hard to evaluate usability without getting feedback from users, heuristics could have great use with when testing certain properties.

My question for the seminar is: in what ways can we collect qualitative without interfering with the user, to avoid introducing bias.

Individual notes, Seminar 2 - Niklas Lindqvist

Chapter 13 – Introducing Evaluation

This chapter presents an introduction on how to evaluate a design. It focuses on both the usability of the design and the user’s experience/satisfaction.

The problem with a design often occur when the producers find the product useful but don’t understand that other users don’t understand how to use it. This I find true in numerous of cases, for example webpages where it’s almost impossible to find what you are looking for. Evaluation of a product enables them to check that their design fit others as well, and should probably be used more often.

There are 3 main types of evaluation:

• Controlled settings involving users (labs, everything is measured)
• Natural setting involving users (field work)
• Any setting not involving users (models and analytics)

Doing a combination of the evaluations is something I would like to aim for in our project following something similarly to:

Chapter 14 - Evaluation studies: From controlled to natural settings

Chapter 14 describes how to evaluate in different types of environments. For example controlled lbs and natural settings. Usability testing is the focus of the chapter.

An interesting concept is the in the wild studies where the researchers are far away and monitor the user with different methods let the users act more “natural”. This is a study that probably have become much easier to implement with new technology and might be improved even further in the future.

Chapter 15 - Evaluation: Inspections, Analytics, and Models

Inspection evaluation methods are in focus of this chapter but I find the Predictive model to be the most interesting. In Fitts' Law where predicting the time it takes to complete a small task with a pointing device such as the mouse on a computer can be used to estimate the total time of a complex task in the same program. This could be develop further using time it takes to press certain keys on a keyboard or other bottoms on a special dashboard when the final design in to expensive to produce iterate to its final stat.

Question for seminar:

What sort of evaluation will be able for us to do on our design?