An annual rite of passage, Design Day is an opportunity for engineering students to prove that they can translate theoretical knowledge learned in classrooms and laboratories into creative, practical solutions to real-world problems. As part of this year’s event held on Tuesday, May 4, students from around WSE presented their work to sponsors and mentors from industry, faculty members, clinicians, fellow students, and other guests through virtual live and recorded poster sessions and demonstrations.
(left to right) Jonathan Edwards, David Calvo, Jada Campbell, Kavya Tumkur and Hannah Lynn.
One computer science project that attendees learned about was an artificially intelligent screen reader reads webpage content out loud to users. But unlike most screen readers currently available, “A Set of Eyes” doesn’t work mechanically to just read aloud loud whatever is hardcoded onto a webpage. Instead, it uses computer vision and a flexible user interface to create a more accessible—and natural—experience for users.
We caught up with computer science student students David Calvo, Jada Campbell, Jonathan Edwards, Hannah Lynn, and Kavya Tumkur, the team behind A Set of Eyes, to learn more about the project.
Q: What is an intelligent screen reader, and why is your design different from screen readers already available?
Edwards: The term “screen reader” is kind of a misnomer—it’s not really a piece of software that “reads” your computer screen. It actually scrapes the webpage’s underlying HTML code and reads back to you what that code is out loud. So if you’re on a popular site like Netflix, for example, a normal screen reader would look at the code that makes up Netflix and read back to you all the titles that are displayed on the screen out loud, one by one. It might even miss a few, because sometimes the code that the screen reader reads doesn’t always represent what’s actually on screen.
Our approach involves using computer vision and machine learning to detach the screen reader from the underlying code and also provide users with a flexible user interface—meaning they can practically talk with our screen reader to get information on the page. Our screen reader will be like a set of eyes looking at the computer screen for the user.
Q: What was your biggest challenge in creating “A Set of Eyes”?
Tumkur: The biggest challenge in creating ASoE has been communicating the problem and solution. Most people who we have spoken to don’t know what a screen-reader is, let alone the intricacy of how one works. One of our biggest problems has been conveying what ASoE is, why it’s a necessary piece of technology for members of the visually impaired community, and how it has the potential to revolutionize how they browse the web.
Q: What was the team’s inspiration for this project?
Calvo: The idea came from growing up watching my father, who’s been blind since birth. I’ve seen my dad struggle with completing basic tasks online on a daily basis. I couldn’t tell you how many times I’ve had to do something for him as simple as clicking a button, or typing in a number, because a developer didn’t make their app or website accessible for the visually impaired. Imagine constantly relying on other people to do things like pay your bills or schedule appointments online! ASoE is the screen reader that my dad can rely on to do all of these things himself.
Q: What impact did the need to work on the project remotely have on you and your team?
Lynn: While we would all love to be on the Homewood campus together, we have been able to adapt to the remote setting. In fact, being remote has allowed us to meet and collaborate with ease and more frequently. Additionally, we’ve been able to validate the problem we are aiming to solve and our solution with relevant individuals across the United States. Given the virtual nature of the world today, people are very willing to hop on a Zoom call and answer questions, provide feedback, or give general impressions regardless of where in the U.S. they might be located.
Q: How important has the senior design experience been to your computer science education?
Campbell: This design experience gave us a unique opportunity for us to apply the skills that we learned early on in our CS education to a cause that means a lot to us. We know that AI has incredible potential, and screen readers are an assistive technology that blind people depend on. So this project was a way for us to build the future, which, when you think about it, is why we studied CS in the first place.