Better real-time captioning for deaf & hard of hearing students
Microsoft Research sponsored my graduate capstone aiming to improve real-time captioning for deaf and hard of hearing (DHH) students. The work was published and directly influenced Microsoft Translator.
Opportunity
Microsoft Research discovered that its language translation technology could also benefit deaf and hard of hearing users as a real-time captioning tool. We investigated how their system could better support DHH scenarios.
Qualitative Research
With no previous experience designing for the deaf, user research was especially critical. I met with university and middle school students, instructors and accessibility services coordinators.
Research methods
- Classroom observations
- Interviews
- Surveys
- Baseline usability testing
- Diary studies
-
- We met with DHH students to learn about the access technologies they use and their experiences in the classroom. This student explained her cochlear implant gives her some hearing, but not enough to follow lectures without real-time captions too.
-
- We interviewed a real-time captioner who showed us his captioning setup with stenographic keyboard.
-
- We gave students tablets with Skype Translator to try as a captioning solution. Students liked the form factor but encountered issues with setup and accuracy.
Data Analysis & User Definition
We used affinity diagrams to distill insights, define goals and design requirements, and conduct a stakeholder analysis.
-
- What’s a UX portfolio without Post-it notes?
-
- We did most affinity diagramming with cut up printouts. Not as attractive as colorful Post-its but it worked and saved time!
-
- Here I am connecting top research insights to rough sketches.
Design & Prototyping
Measuring against the requirements and research insights I iterated from the rough sketches to simple wireframes, up to mid-fidelity designs. While not pixel perfect, the mid-fi designs were great to string into a prototype to start evaluating with stakeholders.
-
- Sketching helped get our thoughts out on paper and start to organize our ideas.
-
- Low-fi wireframes began to define the architecture and interaction of our solution.
-
- We iterated up to mid-fidelity. While not pixel-perfect, these were good enough to show to participants for high-level reactions.
Co-design & Evaluation
We brought students, instructors, and technology providers together in a co-design workshop to gather more information about their experiences, brainstorm solutions, and evaluate our designs.
-
- We gathered stakeholders for a co-design session, and had participants share their thoughts and ideas.
-
- Futuristic ideas were fun for participants to dream up. But they also indicated that students wanted solutions more convenient and portable that what exists today.
-
- Participants called out issues they face with current captioning solutions, and I wrote them on the board in an ad-hoc affinity diagram.
Delivering the Design
After honing our designs at mid-fidelity, I created high-fidelity comps with Illustrator following Windows design patterns for a pixel-perfect representation of our solution.
-
- The high-fi comps present all the right sizes, colors, icons and spacing.
-
- Here’s our team showcasing Skrybe research and design in Seattle.
-
- Our work directly influenced the design of Microsoft Translator’s new feature designed for real-time conversations.
Real Life Outcomes!
The reference, research and design requirements helped to shape Microsoft Translator, and we later conducted usability studies evaluating the new product with deaf and hard of hearing students and university instructors.
A few months later, I saw the Translator team at a restaurant conversing with a deaf colleague thru the app on their phones! It’s an amazing product I that I’m proud to have had a small part in shaping.