The goal of this assignment is to test your prototype with two people to further streamline your app and inform your A/B test question and design.
Protocols, or "usability scripts," help keep tests consistent across testers and facilitators. Write a user testing protocol that covers:
Submit your testing protocol and signed consent form for each participant. Immediately after each test, do a quick debrief with your team and write down any reactions or thoughts that came up.
Observe at least two different people test your interactive prototype. Try to find representative testers who you would expect to use your app.
One person will facilitate the test and interact with the tester, and the rest of the team will be in charge of taking notes/photos/video/etc. This time your user will not be writing down the problems they find for you. It's your job to learn what the people testing your prototype are thinking; the feedback they provide you will be invaluable for your next iteration. Your goal is to find ways to improve your interface.
Submit a photo or sketch of each tester using your prototype. As with the needfinding assignment, these photos with captions should show breakdowns and design opportunities. Contextualize them by capturing the action, e.g. using over-the-shoulder shots, and the setting. Look for other breakdowns and pain points in your interface and try to understand what the problems are and how you might fix them. When possible, modify/update your prototype before running the next participant.
After testing, take some time with your team to reflect on your findings. Go through all the notes and other records. Try to be objective; don't write problems off. Discuss as a team and define some general patterns in people's behavior. When you identify some interesting points, talk deeply about them - ask each other questions, recreate the different tests, analyze the decisions people made, other paths they could have taken, and so on.
Submit a detailed and understandable list of changes that you will implement as a result of your testing and discussion, with justifications. Fix the bugs that are either small and easy to fix, or too severe to ignore. Make sure that you do this before moving on to the next step of this assignment.
A/B testing is a powerful web design tool that leverages random assignment and the easy-to-use chi-squared statistical test. In general, it requires measuring count, or frequency, data of some kind, e.g. number of heads in a series of coin flips, or number of users out of all visitors to visit a certain screen in your app. From your user testing, you should have identified many design breakdowns or opportunities and their potential solutions. Each solution will manipulate, or have consequences, on the user’s interaction in some way. For instance, changing the size or location of a button may increase the likelihood that a user follows an optimal navigational path. In this case, you can measure the effect of this manipulation by counting the number of users to follow this path, and those who didn't, using the chi-squared test. Identify and redesign ONE component of your prototype that resolves a breakdown or leverages an opportunity with an outcome that users can be binarily classified into, e.g. clicked or didn't click. The redesigned component needs to be noticeably different from the original design. Design is often a slow and iterated process, so select something small and manageable in scope. See the lecture videos for more information.
Creating simple paper mockups of your redesign is highly recommended. Then, electronically create and submit two separate URLs for your redesign and your original application - don't write over your old code!
Submit a description of the online test you will run for the next assignment. How will you measure your manipulation? What are the possible outcomes and their interpretations?
In studio, you will present your ideas informally to the other teams: What were some major findings? What changes did they translate to? What are you going to do moving forward? Then you will work with your studio leader to prepare for A/B testing!
Here are three student examples from last year.
By Mike Davison, Community TA and UX Project Manager
Testing your high fidelity prototype with users closes the circle. It is vital to ensure your solution meets the needs identified during the first assignment, and that the agency has not simply spent months drifting further from the problem.
It also allows you third party reflection and suggestions for tweaks to the design. Everything we learn here and correct is a problem we don’t have to live with because it has already been coded and is too costly to change....it’s a valuable phase of the process.
Remember this - feedback is not criticism, feedback is not personal. User centred design works best when pride is left aside, and the feedback of others is incorporated into your design thinking!
The following statements are common feedback given on this assignment. We call these statements 'I like' feedback because they are a way to express positive aspects of the submission. You could think of these as elements to aim for.
The rubric below contains criteria that are worth one point each and will be graded independently and in a binary fashion.