@brandon Yeah desktop controlling agents are a good option. Could have them run remotely. I just worry about power consumption and internet connectivity requirements for that path. For Android apps it'd be great if Waydroid had some way to bridge to the GTK acessibility tree :P
@mauve I have tried to do this, but haven't managed to get anything to work yet. My two weaknesses are web sites and Android apps that are required to function in society but have no alternative implementations like Signal and Lyft. At some point we should be able to slap TUIs on top of these with an AI middleware layer to translate. These days I'm thinking forget interpreting HTML and just use something that can analyze screenshots.