Individual notes - Seminar 2

When designing a product it is very easy to get stuck in coming up with ideas, thinking about new things to add to the product or just improving the design as you see fit. It is easy to forget that designing should be an iterative process where we should evaluate the product every so often. An obvious reason to why evaluation is needed is because there might be bugs or other problems with the product that are hindering the usability of the product. Finding such problems early in the development cycle is important as the cost and time to fix them increase over time. Another reason to evaluate is to deal with problems that impact the users experience in a negative way. While designing you as a designer might think that your system is intuitive and easy to understand, but a new user might experience it as nonintuitive and difficult to use. I think many of us have experienced this with webpages where it feels like it's impossible to find what you are looking for, only to find out that you have been searching in the wrong place. 

On a lecture, Jan Gulliksen mentioned two types of evaluation: Empirical and Analytical evaluation. These could both be used during a formative evaluation during the design process to test prototypes. I feel that analytical evaluation is a lower budget evaluation if you can't afford setting up a test lab, if the test can't be performed with test subjects (for example: too dangerous) or if you are standing between solutions that are similar, costly to develop, and can be predicted. The empirical evaluation, I feel, will yield more interesting, qualitative data. It might be costly to use, as you need test subjects, a test environment and a way to record the data, but the information you get will be from the users, the people that will be the ones to use the product. Personally I feel that involving the users is the preferred method.  

An example of an analytical method is the KLM method where you break down a task into individual keystrokes. An important thing to have in mind is that there can be different ways to perform the task depending on if the user is experienced and knows shortcuts or if the user is inexperienced and takes the long route. When the "route" is known you can calculate the time it takes to press the keys and even move the mouse using a formula. An empirical method is the Thinking-out-loud, where the actions on the user and the voice of the user will be recorded. While testing the product the user will think out loud, hence the name, talking about how and why it does certain things. This test is very dependant on the user, as doing two things at the same time can be difficult, especially if you get stuck. Getting some biased results will also be the case, as you'll want an observer in the same room in case the user gets stuck. 

My question is: What evaluation method should we use on our design this early in the process, and why?

Individual notes - seminar 2

Today, tech industry is obsessed with metrics - every decision must be evaluated, every change is A/B tested, and hard numbers are king. I've often wondered how useful this data-driven approach is in practice, since they often only collect quantitative data, and does not focus as much on the reasons behind the numbers

Therefore, I found it interesting that the book takes up other approaches than using technology to evaluate designs on your target group quantitatively. Measuring usability using e.g. interviews and think-alouds can give additional context to the feedback from the evaluation that helps understanding

On the other hand, collecting quantitative data from the use of the product when it is actually has the upside that it more accurately reflects how the product is used in the intended setting. Collecting qualitative data could interfere with the user in its testing of the design, which may bias the results.

I found that heuristic evaluation was very interesting. My view of usability was that it can often be counter-intuitive, since elements of a design may interact with each other in ways that are not obvious. However, using heuristics such as Fitts' law seems like it could provide useful information despite this. So while it is very hard to evaluate usability without getting feedback from users, heuristics could have great use with when testing certain properties.

My question for the seminar is: in what ways can we collect qualitative without interfering with the user, to avoid introducing bias.

Individual notes, Seminar 2 - Niklas Lindqvist

Chapter 13 – Introducing Evaluation

This chapter presents an introduction on how to evaluate a design. It focuses on both the usability of the design and the user’s experience/satisfaction.

The problem with a design often occur when the producers find the product useful but don’t understand that other users don’t understand how to use it. This I find true in numerous of cases, for example webpages where it’s almost impossible to find what you are looking for. Evaluation of a product enables them to check that their design fit others as well, and should probably be used more often.

There are 3 main types of evaluation:

• Controlled settings involving users (labs, everything is measured)
• Natural setting involving users (field work)
• Any setting not involving users (models and analytics)

Doing a combination of the evaluations is something I would like to aim for in our project following something similarly to:

Chapter 14 - Evaluation studies: From controlled to natural settings

Chapter 14 describes how to evaluate in different types of environments. For example controlled lbs and natural settings. Usability testing is the focus of the chapter.

An interesting concept is the in the wild studies where the researchers are far away and monitor the user with different methods let the users act more “natural”. This is a study that probably have become much easier to implement with new technology and might be improved even further in the future.

Chapter 15 - Evaluation: Inspections, Analytics, and Models

Inspection evaluation methods are in focus of this chapter but I find the Predictive model to be the most interesting. In Fitts' Law where predicting the time it takes to complete a small task with a pointing device such as the mouse on a computer can be used to estimate the total time of a complex task in the same program. This could be develop further using time it takes to press certain keys on a keyboard or other bottoms on a special dashboard when the final design in to expensive to produce iterate to its final stat.

Question for seminar:

What sort of evaluation will be able for us to do on our design? 

Seminar 2 - Individual notes

Evaluation

I was intrigued by how many different factors and methods that comes in play when evaluating. Surprised that it’s such a big area in interaction design. I thought on forehand that the most common way of evaluating a product today must be analytics using the data of natural setting involving users because most tech made today is connected and thus can feed user data to the designer. It became clear that’s not the case. I believe that think-a-louds, which is a controlled setting involving users, provides a lot of useful information to a designer. Unfortunately, the results can be somewhat non representative because the part of the brain that’s responsible for decision making is also used for speech, high level of cognitive awareness, meaning that when the most interesting decisions are made the user is most prone to be quiet. 

As seen in the chapter of data gathering usability testing also might provide both quantitative and qualitative data. The first might be given by keystrokes, mouse movement or the time to complete a task. Qualitative might collected semi structured interviews for example. It’s important though that when interpreting and presenting the data one should be aware of that results might be influenced, a test conducted in a lab says little about how users will interact with the design “in the wild”.

When lacking real user’s heuristic evaluation is the way to go. It seems that the set of heuristics is great guidelines to see whether the design is meeting demands of good interaction design. Doing this before releasing might save lots of time. Fitts’ law describes, in a mathematical manner, the time it takes to select objects on a screen and thus need no users at all and can still be motivating major design decisions which I does seem really useful.

Question: What evaluation method would be the best for our design nr.2?

Seminar 1 - Individual notes Niklas Lindqvist

This is a short summary of the chapters regarding data gathering, data analysis and establishing requirements

The first chapter is about Data gathering, where different techniques is explained and discussed. The main gathering techniques for interaction design is interviews, questionnaires and observations.  One of the most interesting parts from the chapter is the five issues that are broth up. These five issues should be taken in account for a data gathering to be successful. In the sub chapter about the interviews we learn the importance of how to construct an interview and how it might affect the result. To choose a Focal group that you want to make a product for and not choose a Focal group that likes your product is clearly shove in this chapter.

In the following chapter the difference between quantitative and qualitative data and analysis. Quantitative data could be summarized to a data that can easily be described in numbers, where qualitative data is data that include descriptions, quotes from interviews, images and more. Also that one data gathering techniques can result in both types of data. This got me thinking, should we maybe expand our data gathering method with questionnaires and more observations?

How to analyse the gathered data is described in different methods, where I find the sub chapter about analyses of qualitative data and the three methods: Grounded theory, Distributed cognition and Activity theory the most interesting due to my personal gathered data being of a qualitative typ.

In the last chapter, establishment of requirements, we learn that it is important to set up requirements for the project such as required gathering, required analysis and in our case required presentation in the form of who our presentation needs to be clear, and just to our ides. 

Seminar 1 - Individual notes

This first seminar will be about the early stage of the project, for instance, the data-gathering process in the form of interviews, questionnaires and observations. These are all good methods to utilise when focusing on user-centered design as they all involve the people that will eventually use the product.

When we want to establish the user requirements of the product it's a good idea to use more than one technique, which is known as "triangulation". This way we are able to work around the strengths and weaknesses of the different techniques. If possible, we want to gather our information as a part of a field study so that we can talk directly to/observe our core demographic. 

Interviews are a form of qualitative data-gathering, where we can utilise an unstructured form, where the questions are open and our goal is to explore more thoughts and insights that the interviewee might have. Or we can use a structured form where questions are not as open, almost like a questionnaire, but we where we still have connection to the user.

When processing the information we can look for recurring patterns in both the qualitative and the quantitative (questionnaires) data. At this stage we do not have a specific product in mind, rather, we are trying to establish the requirements of the user. As this project is an iterative process we can always come back to the data-gathering step with more closed question to evaluate more specific product ideas. 

My question for the seminar is about something I encountered when interviewing people, how do we avoid them giving us "the right answer" (something they think is important to our studies) when gathering data.

Notes: reflection on readings for seminar 1

The central pillar of user-centred systems design is to involve
the users of a system to a high degree. Many times, especially
in government or enterprise projects, it is the case that a system
is designed in a very rigid fashion, with some experts giving
requirements and the development team then implementing these
requirements independently of the surrounding organization.

The article (Key Principles in User-Centred Systems Design) is basically a guide to how to avoid this class
of failures - involve the user! Do not only design with the user
in mind, but make them part of the process. It suggests three
things that from my own experience is most important when trying
to achieve this goal:
- have the end user participate in the design and development process. They have the domain knowledge you may lack
- make prototypes, early and often. Agile development with many iterations is a good combo.
- usability expertise is a thing, use it. Programmers and the population use computer systems differently.

But how does one do when involving users? Gathering data from users
to apply in the design process isn't easy. During prototyping and
iterating on the prototype, direct user feedback can be gathered
and used. In the planning phase, I've found interviews to be a good
way at identifying problems to tackle. During the MVK course, we've used
rather structured interviews when collecting requirements, which
has worked well.

Another of the main techniques for gathering data, using questionnaires,
is harder when trying to identify requirements, but is good when determining if a change introduced to a system is good. However,
questionnaires must be carefully designed not to introduce biases.
There are biases both in how we answer certain question (for example
on how it is worded, where in the questionnaire the question is), and
getting a good random sample - maybe a certain group refuse to answer
questionnaires in general, skewing the results.

Automatically observing the users when utilizing the system (indirect observation), is a common technique nowadays.
We attempt to describe how the system is used in terms of small, logable actions
taken by the user. There is a big potential for biases though, and mistakes arising from optimizing the wrong thing (can an increase in the use of a UI language selector result from both a good and a bad change?).

My question for the seminar is: how do we reduce biases in the different data gathering techniques?

Summary of readings inb4 Sem1

The main points of the reading seem to be the early stages of a project, for example, data gathering, establishing requirements, making theories about users (use cases) and presenting your data.

Many techniques for gathering data are presented together with their flaws and strong sides, but the main ones were questionnaires, interviews and observation. Questionnaires are good for gathering mainly quantitative data, and might not fit best at the very beginning of a project since it's harder to formulate concrete and useful questions without first having analyzed your demographic with other tools. Interviews and observation might be a better fit since they allow for more qualitative data gathering, and interviews are also easier to perform. Interviews additionally have the advantage of not taking the interviewee out of his natural environment. There were also many detailed tips on how the interviews should be performed, but many boiled down to simple common sense. 

A technique mentioned for minimizing bias and false conclusions was triangulation, in which one is supposed to analyze the data from different perspectives or through different theoretical frameworks. If the conclusions still held true, that would be a good argument for them being representative of reality; otherwise, they might have sprung from biases or faults inherent to the methods of analysis used.

Having access to large amounts of data, for example from automated observations (indirect observation) of usage of a website or an app, a way of analyzing that data might be through the framework of Activity Theory, in which one tries to reduce the human interaction with a system to small, individual "activities". This might help to show where these "activities" distract from each other or are hard to perform, however, it might not always be clears exactly how interactions should be subdivided into these activities.

Another important topic is understanding and interpreting the given requrements for a system, but since these are not always clearly defined, it's often "part of the project" to establish the requirements. Several techniques are presented, in which generally the data gathered earlier is compiled and interpreted into various forms. For example, use cases, personas or scenarios can be used when presenting the established requirements to the interested audience (say, users or stakeholders), each having their different upsides and downsides. One reason for using personas or scenarios might be to "add life" to a requirement report and make it more relatable for the audience. 

A question I'll like to discuss is: "How do we avoid the user trying to please you with their answers?", as that is something I experienced heavily when conducting the interviews.

Seminar 1 - individual notes

I have a few years experience of working with and around interaction design, therefore — unsurprisingly and somewhat reassuringly — this week's reading did not offer any major revelations for me to reflect on. But I will endeavour with the spirit of the exercise.

The material deals with data gathering, data analysis and requirements discovery, presented in that order and suggesting a work process in that same order. This order is also reinforced by the course project (so far), wherein we first gather data about a group of persons at a "journey-location", and most noteworthy is that this group is chosen for no business or even product-developmental reason, and then we do state-of-the-art analysis of what they might or ought to interact with in relation to the chosen journey-location, still with no concern for business or product. The aspiration is to discover the needs of the group at the journey-location, and from that come up with a design which meets their need, which is interesting though somewhat lofty.

A more practical situation to become accustomed to would be that of the start-up team who knows where their development skills and domain knowledge lies, and from it they agree what product they can make, and therefore have a product-centric approach to gathering data about potential users of their product, then analysing that very relevant data to discover what their product needs to be. Another approach could be that of the business, which has investment in (or at least funding for) a specific business plan, and therefore has a business-centric approach to gather data and so on as in the previous example.

This is of course an iterative process, where better understanding of product leads to better understanding of users and vice versa. Hopefully the project will allow for at least one such repeat iteration..

The literature focuses on ways to gather data that are easy to label and put in a book — interviews, questionnaires and observations, of which the least covered and probably most available and used in software development are indirect observations of usage stats (including A/B testing), and to some degree observations in controlled environments (such as inviting users to the office to be observed when using the product, or sitting with a user at her workplace). There are no assumptions of pre-existing expertise, or of really "becoming a user". (Direct insider observation suggests observing other "real" users as a spy, rather than being a real user and focusing on whatever real users focus on.)

The literature offers well considered approaches for design, such as user-focused design

(building what the users need) and activity-focused design (building what's needed to do the activity), but it should also mention practical considerations e.g. working with what you know, as when designing tools for other developers (as in any API) or when the developers are users and develop what they want to use (as in most open source software). Or the very common "accidental design", which starts out with no design, for the sake of curiosity or development in itself. Only later, if ever, are the edges rounded off for easier use — and such has been the design approach for much of the world's technical innovations, and most if not all high art. (Cf. the KTH insignia "Vetenskap och Konst".)

I was amused by the revelation that the think-aloud protocol is an established technique, something to do consciously, rather than what inevitably happens when you draw interactions on a whiteboard or demo a product/mockup :)

My question for the seminar is how to overcome the Dunning-Kruger effect? — In re-evaluating what one has learned so far about the product/users, in feedback from users/experts, and in those with greater business authority over the design (e.g. managers, executives, shareholders…).

Seminar 1 - Individual notes

The first reading seminar concerns the first steps of an interaction design project; data gathering, data analysis and establishing requirements.  

To obtain good data there’s three main techniques; interviews, questionnaires and observations. Often more than one technique is used to validate results of some inquiry by exploring similar results, also known as triangulation.

Interviews can be unstructured (using lots of open questions and leaves room to explore insights from interviewee) or structured where questions are usually closed and has options. I’d say the best is somewhere in between collecting both quantitative data and a more what motivates those answers, qualitative data. Questionnaires was an especially interesting section; interactive web-based questionnaires have some great benefits although the main problem is the difficulty of obtaining a random sample of respondents.

To process the gathered data the method varies with the technique used for collecting. From audio and video there’s possible to extract both qualitative and quantitative data. The latter is easier to transcript and present as graphs or similar. For qualitative data on the other hand one must first identify recurring patterns or themes to be able sorting the data.

In chapter 10 the authors point out the importance of establishing requirements. There’re two main aims; understand as much as possible about the users and secondly to produce a set of stable requirements that form a sound basis to start designing. Doing a proper foundation for the project is both cost effective and professional because what a “customer” think/mean/says they want may differ.

To establish requirements, it’s a good idea to discuss both functional and nonfunctional requirements. It’s often intuitive to use scenarios, which is described as an informal narrative description in the context of use.

My question to the seminar: How can one minimize the effect that our data gathering technique has on the respondents?