Better Accessibility for Screen Readers (Voiceover/TalkBack)

[danielw97]

Hi there,
Many thanks for all of the work done on this project so far.
I have no vision and use voiceover (which is a screen-reader built into iOS).
Whilst a vast majority of the app is accessible already (as far as button labeling etc go) there are a few issues such as not being able to use explore by touch and scroll through albums/artists.
It is difficult to summarize succinctly as voiceover is a large topic, although it would be great to have some improved voiceover support in the future if possible.
There is a link on Apple's developer docs, although not sure if it is any use in this instance:
https://developer.apple.com/documentation/uikit/accessibility_for_uikit/supporting_voiceover_in_your_app
Let me know if there is any way I can assist with this as someone who isn't a developer, although I'm happy to answer any questions or test things as they are investigated.
Thanks in advance.

[Chaphasilor]

Hey, thanks for reaching out to us about this! As someone interested in user experience design, I'm actually unreasonable exited to hear from you, as I've been wondering about this topic for a while now. I've never looked into improving this kind of accessibility yet, but I would be glad to remove any pain points you might be experiencing!

I'll try to take a look at this whole topic soon, and if you are able to share any additional info about how you use the app, how you interact with the different functions, and how it's being done by similar apps, that would be immensely helpful!
Not sure how convenient it is for you do discuss this via text, so we could also look into scheduling a call or something down the road to further discuss things!

[jmshrv]

I've also been meaning to look into this, since I haven't done much beyond labeling. One issue I noticed when messing about with VoiceOver a while ago was that it would describe album covers, would it make sense for them to be ignored/marked as decorative?

[danielw97]

Hi,
Thanks very much to both of you for being willing to look into this.
I should have some time this coming week to hopefully sit down and write some detailed notes on the user experience from the perspective of a voiceover user, and am happy to either share those here or an alternative platform depending on what is preferred.
The biggest issue that sticks out to me currently is the ability to scroll through albums/artists. Typically the feature known as explore by touch is used to move through lists quickly, and also occasionally the voiceover cursor doesn't seem to be able to select certain elements.
However, I've never found that in finamp that the content of the categories such as albums/artists is visible with explore by touch, however the buttons at the top of the UI such as different categories etc are navigable in the usual way.
The navigation between previous and next elements does allow for reaching these in most cases though so they are getting detected.
There are also several buttons on the playback screen that are unlabelled, although I'll have to sit down and write some more details.
Hth a bit, and let me know if there is anything particular you would like information on.
As I mentioned above, I should have more time next week and can get something written up if that is useful.

[Chaphasilor]

Thanks for getting back to us! I only have an Android phone, but I did spend half an hour today using TalkBack to navigate Finamp. I also noticed the unlabeled player buttons and will investigate how to fix those. However, I didn't encounter any issues with scrolling through albums and artists. Both the list of albums, and the album screen itself that lists the songs within the album work just fine for me. So if you could explain a bit more what exactly isn't working for you that would be great. If you are able to record your screen as a demonstration, that would be even better, but I'm not sure if you're comfortable doing that.
Maybe Android just doesn't have a feature equivalent to explore by touch. In that case, maybe @jmshrv can help out fit debugging the issue.

And please take your time with the write up, there's no rush from our side. We're busy with the redesign anyway, but that still has a long way to go, so your feedback should be able to have a positive influence on the new layout!

[Chaphasilor]

Hi, just wanted to let you know that the redesign beta is now available via Apple Testflight. I'm sure that the initial experience using a screen ready will be slightly worse, but the design is an interim state and many things will still change to better fit with the rest.

However, if there's somthing critical that's not accessible, or you don't understand how to navigate the UI properly, I might be able to do some basic fixes for now :)

[Chaphasilor]

Also, I just stumbled across a Flutter app that has a "Screen Reader Mode". Essentially, it's a preset for various settings within the app that tries to minimize the amount of elements and added complexity.

Given that some level of customization is already planned (since many requested it), I think it would be feasible to do something similar.
I'd need your help with figuring out the necessary elements though.

[danielw97]

Hi again,
apologies I didn't end up getting you a detailed summary, the last few months have been extremely busy and it fell off the list of priorities.
I've done some testing of the new beta on testflight, and am still running into the issues of voiceover not being able to focus on elements although for what it's worth everything seems to be labeled fine.
This is difficult to explain, would it be better to send you a recording or similar to show what I'm talking about assuming none of the maintainers are able to activate voiceover on one of their devices?
It changes the interface and you have to double-tap on elements until it is disabled, so I completely understand if there is a reluctance to do this.

[Chaphasilor]

Thanks for getting back to me. I'm actually one of the maintainers and already tried interacting with the app using Android's Talk Back feature. The problem is that I'm not used to interacting with the app that way, and I'm already familiar with the layout and structure of the app. If you could provide a recording, that would certainly help!

Even better would be a list of features that you frequently use, or a step-by-step overview of how you usually interact with the app. That would help me figure out how you're using the app, and what could be improved.

[danielw97]

No problem, can do. This may very well be an ios specific problem although I'm happy to document as much as I can.

[albertotirla]

so, let's get afew things straight:

The concept of explore by touch is relatively simple, and is basically what everyone without screenreaders does to activate an item. Basically, after a while, even with screenreaders and the swiping we often do to get information, we end up creating spacial, topographic, memory, of apps we use often. So, we might know, for example, that the back button for a certain screen is on the top left corner, that the tabs to switch between various categories is towards the bottom of the screen, etc. Some of us could, though rarely, pay close enough attention to notice kinda where a specific list is, so we go to that, and even rarer, how much space is generally between items, so one can reach a more approximately correct place for the item they were looking for. How we do this? this works on both android and ios btw, normally anyway. You...just put your finger on a part of the screen, where you think a control is or near it anyway. If you were correct in your assumption, focus will be moved to that control and the screenreader will read it. To activate it, you have to touch your screen twice in quick succession, this disambiguates explore by touch from activation. The focus would be moved from your current position to the one where your finger is, only after you released the finger. What that means in practice is that, if you know circa where the item is, but you're not sure, you can keep sliding your finger in the aria till you get confirmation you're on the correct thing, then release to be moved there, but you still have to double tap, as the gesture mentioned above is described, in order to activate the item. So then, what @danielw97 describes could be either a case of having too many custom widgets which are not wrapped in semantic(more about that later), or having too small a touch area, meaning that the sensitivity rectangle for a control is far too small, in which case, even people with motor disabilities would struggle with your app, because the touch target would be too small. Also, you could have a bounding rectangles problem, because the screenreader is reading not from the actual screen of the phone, but from a virtual accessibility tree, so if you say that your rectangle is 20X20 DPI while it's on screen half that, you're gonna get what issue author described.

About accessibility information: while apple documentation is indeed slightly relevant here, what you need is flutter specific documentation, which tells you how to decorate your widgets, how you can hide certain widgets from the accessibility tree entirely at all times, move elements semantically to make them easier to access for screenreader users even as you keep their place visually, and so on. The links which should have been given in that instance, then, are these:

• https://docs.flutter.dev/ui/accessibility-and-internationalization/accessibility
• https://docs.flutter.dev/ui/widgets/accessibility
• https://medium.com/@enzoftware/flutter-accessibility-getting-started-52286746314e
• https://verygood.ventures/blog/exploring-accessibility-and-digital-inclusion-with-flutter
  that should be enough for starters. I'm not a flutter developer, in fact I stay far away from frontend stuff usually, but if there are any questions about what I said above, feel free to ask them and I'll answer to the best of my ability.

I do agree though, a recording from the issue author would also be very informative for most people here, so my explanations should in no way be interpreted as detracting from the value of such, in fact, I believe they have an aditive role, as those can be checked side by side and connections can more easily be made.

[danielw97]

albertotirla, thanks a tun for the detailed development-orientated feedback.
I've been doing some more testing, and it seems as though with version 0.9.7 there has been some improvement.
I'm still not able to navigate things with explore by touch, although I am able to swipe to the elements such as albums and artists.
The interesting thing is that after selecting an item from the main screen, playback controls are accessible and able to be navigated to with explore by touch no problem.
I'm not sure if this is any help, although as you mentioned marking albums etc as illustrations might this have something to do with the fact they can't be navigated to with explore by touch?
I've done a preliminary recording as my schedule is less busy now, is there a preferred way to get this to you?
Thanks again.

[PlasticMemmories]

Hello.

I created a very similar issue without knowing that this one existed, (#756) apologies.

There is a flutter library called semantics
which contains all of flutter's accessibility related classes.
I'm not much of a flutter developer so I can't give many tips or specifics, but I'm happy to help.

one tip I have is that buttons can be labeled by making a text widget the button's child widget.

[slannon97]

I just tried the iOS app with VoiceOver on an iPhone 11 and have to agree with what has been said. The main issues for a screen reader user are either unlabeled controls that just say things like button, controls that are probably buttons but don't indicate they are, and focus issues. A lot of this should be easy to fix. Controls need clear labels that indicate to a screen reader user what the purpose is. For example, instead of saying button when navigating to the control to play or pause media, it should say play button or pause button depending on the current state. Controls without labels are also a little difficult to understand, as a screen reader user might confuse them as text and not know they can be clicked on. The type attribute should be assigned to such areas, so the screen reader can clearly communicate to the user what they are. The tab bar should be located at the bottom of the screen for better consistency. Everything should be navigable by either dragging a finger around the screen, or swiping through the elements with a single finger left or right swipe. The screen reader focus shouldn't jump randomly while navigating the UI. I haven't tested the Android app yet, but I imagine the problems are very similar there as well. I have a Pixel 7 with Android 14, and if I find anything significantly different, I'll add another comment.

Remember that you can always test the changes yourself by enabling VoiceOver and/or TalkBack on your device. VoiceOver is in Settings>Accessibility>VoiceOver, and TalkBack can be found in Settings>Accessibility>TalkBack. You may find it convenient to enable the accessibility shortcut in both operating systems to quickly toggle the screen reader on and off. I recommend doing this before anything else, as it allows you to quickly turn the screen reader off if you need to do something and don't want to deal with the modified gestures. In iOS, it's at the bottom of the accessibility screen. Configure it to toggle VoiceOver by clicking the side or home button three times. On modern versions of Android, the shortcut can be found in TalkBack settings. Press and hold both volume keys for a couple seconds to turn it on or off.
The core gestures you'll need to know to effectively test are simple. Swipe a finger left or right on the screen to move to the previous or next item sequentially. I'm not a programmer, but I'm pretty sure this can be customized in your specific app. My understanding is that it navigates from the top left of the screen to the bottom right, following the order of UI elements in the accessibility tree. You can also put a finger on the screen and drag it around to quickly move the focus somewhere. Unlike how a sighted person would navigate, touching a particular item on the screen just moves the focus to it rather than activating it. If you want to click on things, quickly tap twice with a finger once the focus has moved there. It doesn't matter where you physically tap on the screen.
You may also find it helpful to enable VoiceOver's screen curtain with a three-finger triple tap, or go into the TalkBack menu with a three-finger tap and select hide screen. This will make the screen go completely black, so you can experience the app the same way I do. This may be helpful for you to gain a clearer understanding of what accessibility really means, since you have to rely on the same feedback I do. These features can be disabled by either disabling the screen reader, or repeating the steps used to enable them.
There are also settings to put an outline around the focused item and show what the reader is speaking visually on the screen. You may find the large cursor and caption panel options in VoiceOver settings to be particularly helpful. On Android, check under advanced>developer settings in TalkBack settings for some options. I'm not sure how these work since I can't see, but I think the display speech output setting does the same thing as the VoiceOver caption panel.
Finally, I'd be more than happy to beta test and provide feedback.

[Chaphasilor]

I've just added a few improvements, mostly button labels. This will be released with the next beta update. I still don't know why Explore by Touch isn't working / buggy, since the Android-equivalent works just fine for me. I also don't have an iOS device to test with, so if anyone could investigate this, it would be much appreciated!

[danielw97]

Thanks for your continued work on this. I'll download the next version via testflight and report back.
I also have access to both android and ios now, so am happy to provide accessibility feedback for both platforms assuming someone else doesn't pic it up prior to me getting around to it.

[Chaphasilor]

Hey, any updates regarding the new accessibility labels? Anything still missing or unclear?

[PlasticMemmories]

You did an amazing job, and, at least on my end, the explore by touch issue has been resolved. There's a label missing on the miniplayer for the pause button, but that is pretty recognizable by the way you have the app laid out. The download button on an artist's page appears twice, once as a labeled button, and right beside it is an unlabeled button that appears to do the same thing. Those are nitpicks though and the issues I've been talking about have been largely resolved.

[Chaphasilor]

That's wonderful to hear! I'm surprised that explore by touch works now, but maybe some internal component got update, which fixed the issue. Would be nice if someone else could confirm that it works now!
I've fixed the buttons you mentioned, will arrive with the next beta update. Let me know if there are any more missing labels, it's pretty quick to add them, but harder to find them.
Oh and FYI, the two download buttons are in fact the same button, it was just labeled twice. So need to worry which one to press!

[Chaphasilor]

Hey, did you find anything else so far that is still unlabeled or unclear? Planning to push a beta update soon and maybe I can get some more fixes in before releasing that!

[PlasticMemmories]

the only real bug is download buttons appear twice, (one without a label), but that's not an accessibility issue more of a ui clutter issue for screen reader users. It doesn't just show up once, there's a download button in the navigation menu, and anything that has a download button in the "download settings" menu of the settings. Hope I was clear about what I meant.

[Chaphasilor]

Yeah, that's a bug which I already fixed. Just wanted to make sure nothing new came up.

[Chaphasilor]

Got a question for you:
I'm working on re-designing the list elements for tracks in the app. They now have a button for adding that track to a playlist or favorites right from the list, without going through the menu. Should this be picked up by screen readers? Right now when going through the list, it would select the track itself first and read the title, artist and so on. Continuing through the list would then select the "add to playlist"-button. So going through long lists would be a bit slower.
I'm wondering if such a button would be of much use to you guys? Tracks can be added to a playlists or favorites from the track menu already, which can be opened by long-pressing the track. But I'm not sure how easy that is to do with screen readers, that's why I'm asking.

I also noticed one more thing I'd like to improve in the queue. In the "Playing from" section, the current progress is shown, as a fraction like 10 out of 13, with a slash in-between. This seems to be pronounced like an actual fraction by screen readers, so I'll try to re-word it. The remaining duration is also inconsistent with the other durations throughout the app.
I'll try to remember fixing that.

[PlasticMemmories]

I'm confused, do you mean the playing screen or the list of tracks in your library? If it's the player screen, the add to playlist button would avoid the extra step of clicking on the menu
Is there any way to either change that progress indicator to either a regular 0 to 100 progress bar.
This isn't necessarily an accessibility, but could there be a track position slider that would you let you do the same thing as the rewind/fast forward buttons?
While I have you, what would be needed to integrate finamp with siri? That's the only thing keeping me from using jellyfin fully.

[Chaphasilor]

It's about the list of track in the library.

On the player screen, you can already skip the menu by "long-pressing" the menu button, that will open the playlist selection right-away. However, this isn't obvious at the moment, neither for screen readers nor in the graphical user interface.

I'm not quite sure what you mean regarding the progress bar. Right now there is a progress bar that reflects the current position in the track. There currently are no rewind or fast forward buttons, just those for skipping to the next track or going back to the previous track.

Regarding Siri, that has been requested before, but there currently isn't great support for this in Flutter (the development framework we're using). The functionality for voice commands is already implemented, because that is used in Android Auto. So once Flutter's Siri support improves, it should be relatively quick and easy to add support for it.