How do screen reader users use search interfaces differently than sighted users searching visually

Graphical User Interfaces afford a visual information seeking model, which may not be very suitable for non-visual or auditory modalities when using assistive technology. Yet, there is a lack of research to understand behavior in complex web searches when using screen readers. 

NOTES:
1. This independent student research was conducted with the help of the Gavin Lew fund at DePaul University under the advisement of Dr. Cynthia Putnam. 

2. Recently, this research was selected to appear as a late breaking publication at ACM CHI 2020.

NOTE: Collaboration organized by the Steans Center at DePaul University to explore Service design for community service - Coursework for Design Ethnography, group of 3 graduate students.

People conduct complex information searches at all times, aiding decision making and problem solving in their day to day tasks. ​As search interfaces become more complex and visually directed, potential accessibility concerns hinder information seeking when using screen readers.

 

Most research in accessibility highlights usability concerns and browsing behavior within the context of "navigability of web pages". It is unclear how screen reader users interact adaptively with the search environment in an auditory interaction to maximize gain of desired information.

 need for research, and our questions 

Need for better understanding of a non-visual foraging model

Information foraging theory states that "people will modify their strategies, or structure of the environment, to maximize their rate of gaining valuable information". The decision to pursue a source of information (scent) is based on its  perceived cost of access and value

Many users who are legally blind use assistive technology such as screen readers. Screen reader users use keystroke commands to select HTML page elements to receive auditory feedback. The nature of non-visual web access is sequential and auditory.

  • There are increased costs (time, effort and decision making to access information).

  • Cognitive overload from excessive auditory information.

  • Multiple usability and accessibility issues due to missing semantic HTML tags


Prior studies place screen reader users' adaptive browsing strategies in context of navigability of websites. We sought to understand:

  • What information behaviors do screen reader users employ to maximize information access?

  • How is this information behavior different from sighted people using the visual web for search?

 research plan and methodology

We recruited ten sighted web users and ten screen-reader users 

Ten sighted and ten screen reader users (evenly distributed between the ages of  18-65) were recruited from the university, and an institute offering computer classes for people with vision impairment. All participants conducted a variety of information seeking tasks regularly as part of their work or academics, and reported their extent and frequency of internet use in a screener. Non-sighted participants were screened for less than 20/200 visual acuity, and did not use screen magnification. View some recruitment considerations in the recruitment documents below.

   

Data collection using task-based systematic observations

In a 40-minute session, participants completed a scenario based task, where they were asked to gather information on graduate schools in psychology (towards a potential discussion with their academic advisor). They recorded desired information about schools in a text document while conducting their search. Participants' information search behavior was captured in the form of screen recordings.

   

Data logging from screen recordings of participant behavior in tasks

Participants' screen recordings were transcribed in the form of logs that captured behavior around query formulation, search result exploration and link following to reach desired information. These transcription logs captured the following instances (and their attributes), as they occured in the search activity :-

Query formulations

 

  • Query terms

  • Type of query - informational, navigational or transactional

  • Any accessibility concern in query formulation

  • Any adaptive behavior in query formulation

​Search result exploration

 

  • # results explored

  • Method of exploration - headings, descriptions, url

  • Any accessibility concern in search result exploration

  • Any adaptive behavior in search result exploration 

  • Any revisit to a set of search results

​Link following

 

  • Capturing links followed from a search result 

  • Adaptive browsing within the webpage

  • Reasons for site leaving

  • Any revisits to the webpage

Figure shows screenshot of coding of participant’s activity from screen recording in an excel sheet chronologically capturing steps. The coding captures search query terms, methods, result exploration behavior, # results explored, ways of exploring, websites visited and links followed from a search result.

CAPTION: we transcribed and coded screen recordings of participant activity

Using our data, we created web behavior diagrams to capture participants' actions in context of the entire search session, and conducted quantitative analysis of querying and link following.

CAPTION: This web behavior diagram shows one participant completing the search task using a screen reader. like many other participants, tina anchored her search process on the information scent gained from the first search result exploration

sighted participants - extensive querying (5)

sighted participants - extensive link following (5)

Non-sighted participants

 findings | information seeking behaviors 

Sighted participants either demonstrated extensive querying or extensive link following

  • Extensive Querying - 5 sighted participants used a combination of multiple queries to reach desired information

  • Extensive Link following5 sighted participants followed links to reach desired information

  • Both these groups used methods that were comfortable to their browsing style to reach desired information

caption: these are web behavior diagrams of 2 sighted participants. on the left - extreme link following, on the right, extreme querying

Screen reader users demonstrated neither behaviors. Instead, they adaptively maximized gain from their initial searches by returning to that page. In order to explain this, let's discuss search result exploration behavior of initial searches. 

  • Non sighted participants employed adaptive navigation methods for exploring search results. 

  • On an initial scanning navigation using Headings, if the search results were relevant, they conducted an exhaustive exploration. 

  • Exhaustive exploration included reading relevant/ skipping irrelevant headings, reading descriptions and sometimes url for relevant search result headings. 

caption: in screen reader use, exhaustive search result exploration included one pass of heading navigation, and a second pass of skipping irrelevant results, and detailed exploration of relevant results

Why might screen reader users have demonstrated less querying or link following?

  • Search result exploration is a costly process, and perhaps, that cost of exploration wasn't justified for multiple search result explorations. Scanning navigation was common when exploring results of navigational searches.

  • They chose a careful selection of high-scent links or combined few informational and navigational queries

  • Perhaps, extensive link following led to a loss of sense of context in the browsing process

 adapted mental models | Learned behaviors  

screen reader users maximized gain from their initial searches. they returned to certain scent-heavy pages to use the scent gained to further their information search  

extensive querying or link following by sighted participants

 inferences for systems and interactions

​Through these findings and visualizations, we can argue that adaptive behavior is exercised not only in light of inaccessible content on standalone web pages to escape. Instead, screen reader users illustrated learned behaviors with respect to an understanding of how the system, and the interaction works. 

How might we aid different search result exploration methods in screen reader use?

 

  • Informational searches included both scanning + comprehensive exploration techniques 

  • Navigational searches included only scanning exploration

  • Screen reader shortcuts to aid both types of navigation 

 ​

How might we improve information scent in informational searches - varied results?

  • Transcoding solutions could explore highlighting relevant keywords in Headings, and descriptions, highlight relevant meta information(such as order of results), to allow for better scanning on a revisit.

  • Bookarking relevant results to return to on a revisit of search results page.

How might we improve information scent in navigational searches - results leading to a web address?

  • Transcoding solutions can clarify key differences between similar search results (keywords) to avoid following undesired links. 

  • Quick scanning of a gist of keywords in heading navigation