Help for GMod to become accessible to the blind and visually impaired

Hello,

I’ve acquired GMod recently, mainly to make it accessible for the blind and visually impaired.
This is a quite huge undertaking, and after hours of browsing the official wiki (http://wiki.garrysmod.com), I am really not sure where to start.
Here are the features I’d like to implement. I’d be glad for any pointers, or comments regarding a possible way of implementing them!
Please NOTE: I do not necessarily require code, rather engine functions, or ideas, if it is not possible to realise my original idea. As I know almost nothing about the engine, specific custom or engine functions are really appreciated!

Textual information, where possible, would be sent to a Screen Reader. The simplest solution for this now is to send text to the clipboard. I’m probably going to use Global.SetClipboardText for this.

  1. Extracting game text does not seem to be hard, the console has for example hooks that return the printed text, fire on activate/deactivate events, etc. My main issue here is the UI. I couldn’t find any hook/library function that’d allow me to determine which menu item received focus, and then to get the control’s caption and current value, where applicable (i.e. for a checkbox, combobox, etc).
    Furthermore, due to the blind mainly using a keyboard, or a gamepad, some menus are impossible to use, such as the main menu.
    At this point, I am not entirely sure whether I should write a custom UI, or just forget about menus and dialogs altogether, since the console can be used to set most, if not all the game options.
    If I can replicate menus with ease, just with the added speech/extra sounds/keyboard navigation, chances are high that it’d be a lot easier to use menus rather than typing in commands.

  2. Most items on a map would need a looping sound, so that they could be located easily. This would be especially crucial for pickable items and interactables. Of course this would be hugely dependant on the mod creator, but as long as the built-in ones emit sound, constructing something that is accessible would not be a problem.
    I’ve thought of scanning the map around the player, and playing a sound where I found an object. Does EmitSound work for items, or just for entities?

  3. Most of the time, multi-level maps are quite problematic, since you have to look up/down to aim. This is unfortunately quite hard to illustrate using audio only, even if HRTF is available. Thus I think that either AutoAim, or using flat maps while locking the mouse’s Y axis would work. Providing key bindings for LookUp and LookDown seems to be doable as well.
    I think there is a hook so I can ForceCenterView when the mouse moves up/down, not sure about autoaim though.

  4. Construction. As GMod is mostly about in-game construction, I am really not sure how to approach this. Perhaps having a mode where the player can scan ahead, and would be read out the name of an object, or switch between possible targets…

Of course any modifications I make would only be useful for the blind or VI, including multiplayer gaming, thus in this case it does not really matter what’s on the screen (i.e. a minigun weapon model while someone’s using a custom-built pistol). As long as the audio signals that it is indeed a pistol, and it does not do 10k damage per second, it is fine! :slight_smile:

Thank you very much in advance!
Erion

Even though it may be hard for blind people to have the full enjoyment of the game, I applaud you trying.

It’s possible to loop through all the vgui from a set parent, luckily all vgui stem from the world panel:

vgui.GetWorldPanel( );

You can determine which panel has focus like so:
panel:HasFocus( );

Some panels have text, or will have children which have text for you to read-off. You could get the children locations and get the position of the mouse to read off locations.

in cl_panel_metatable.txt has some of the get functions and other functions which could be called in here: https://dl.dropboxusercontent.com/u/26074909/tutoring/metatables/__metatables_sorted_func_names.rar or https://dl.dropboxusercontent.com/u/26074909/tutoring/metatables/__metatables.rar

you could do a recursive function based on the input ( starting from vgui.GetWorldPanel( ) ) and loop through i = 0; i < input:ChildCount( ) - 1; i++
then use input:GetChild( i ) and if it HasChildren, recurse, otherwise do something else.

If child:GetClassName( ) is Label then you can print the child:GetText( ) or send it to your reader.

I hope some of this helps.

Way more than you think! Huge thanks!

Currently we have a rudimentary way of receiving console output via the -condebug switch, and a helper app. Now to make the panels readable…

Thanks again!
Erion

I could help you if you want, I have nothing better to do.

Hi,
Sure! The more the merrier! :slight_smile:

We need to find a way to share code. I’m thinking either Dropbox, BTSync, Wuala, or some other sharing service would work. If we get more people, then SVN or Git wouldn’t be a problem either.
Some sort of instant messenger would not hurt either. IRC, the falling MSN or anything else you’d like! Voice chat also works, Teamtalk, Teamspeak, etc.

Once we have things going, we can discuss coding, and all the fun! :slight_smile:

Let me know your preferences!

Cheers,
Erion

Sorry for sounding negative, but I don’t think blind people can get any enjoyment from playing GMod ( Or rather any VIDEO game ), especially considering it’s an FPS for most part.

You would be surprised how many blind people can, and do enjoy video games only on Steam! There are even more console owners. If the game has enough audio feedback, any game can provide enjoyment. Unreal tournament, GTA, Quake, lots of fighting games like Skullgirls or Mortal Kombat, Rhythm games like Rhythm heaven, LOTR (e.g. The Return of the King) - the list goes on.
GMod has everything to make us transform it into a playable and enjoyable game. All we need is optimism, a good knowledge of the engine and Lua knowledge.

Cheers,
Erion
PS: Try Resident Evil 6 with closed eyes, relying only on game sounds and your camera helper (yes, it does exist!). The same for Ninja Gaiden. You will be able to do a lot more than you think :slight_smile: Reply if you don’t enjoy it! :smiley:

Feel free to add me on Steam; I wrote a ton of helper-functions, and some of them may do what you’re looking to create.

Thanks! I’ve sent an invite (robjoy88).

Erion