Came across this cool #rust based screen reader for Linux. Their approach seems compatible with what I've been wanting.

https://odilia.app/