You think we could teach human brains about embedding spaces by turning embedding vectors into audio and giving people tools to navigate the sound and the meanings at the same time?
@mauve Was this written on a psychedelic or am I lacking some knowledge to understand this?
@fredy_pferdi Fully sober! What seems confusing? By embedding vectors/spaces I mean the ones in machine learning contexts. https://en.wikipedia.org/wiki/Embedding_(machine_learning)
E.g. inside an autoencoder.
@mauve Ahh yeah i was thinking about this as the fourier space representation in quantum physics and really asked myself whats meant by this /why/how/for what
@fredy_pferdi Oh! No more like, if you train a model with its own internal representation of concepts, could a human learn a similar structure by listening to the embedding vectors translated to something like intensities of different harmonies. e.g. given a 1d embedding, each float would be the intensity of a specific frequency.
@mauve What is meant by a similar structure in that case? This is to abstract for me to grasp TBH.
I understand the vector embedngs of an LLM as representation of human written social concepts.
In the end every word written down by a human is already an abstracted representation of the world (some so fare from reality not even worth a single thought) in some way and that gets further abstracted one layer by putting this abstract generalized data into vector embedding.
@fredy_pferdi I was imagining that after listening to the audio representations of embedding vectors for a while a person could get a grasp of the "vibe" of some data just from the sound. Then they could get information out of embeddings in a similar way to a machine trying to do the usual cosine distance between two vectors. Like, what if we could navigate our timeline with sounds before we bother converting them to speech or actually reading the text.
@mauve
@fredy_pferdi
Very interesting question. Audio representations are usually useful when the signal contains information encoded as relative positions that can be perceived as structure in time. Encoding embedding activations as intensities of frequencies would be encoding "vibe" of data as timbre, but thats challenging because timbres are the consequence of physical law and are not generic, like we can tell woodwinds from brass but we can't perceive differences in arbitrary spectral structure as well without training. However auditory prostheses that e.g. encode visual information as sound or even as patterned electrical stimulation on the tongue or skin can be learned, and there could be some interesting mapping between the activations and a spectrum that could make their structure perceivable.
@fredy_pferdi @mauve exactly, ya, "try it and see" would be my general attitude
@jonny @fredy_pferdi @seeingwithsound Oh yeah I tried using that recently! It was cool to read some blog posts of fully blind people using it to look around at buildings and descriving what they percieved
@mauve Ahh interesting, any #neuroscience or #sociology ppl here having an explanation if we could get a "vibe" of a social construct based on abstract audio? I have strong doubts and I still do not really understand how to replace the "vibe" aspect with a more scientific description but there might be something to it.