Mauve 👁💜 @mauve@mastodon.mauve.moe

@homestucklinebyline This is me checking emails in the morning

@pry Merkle Search Trees are way more complicated to deal with and implement in my experience. I wrote up a spec for the prolly tree impl I use here: https://github.com/ipld/ipld/blob/776b537e0d16dc0341f1bec13ee79bd05a0dfb9e/specs/advanced-data-layouts/prollytree/spec.md

It talks about how merging works.

The tl;dr is you traverse the merkle tree and where you see a diff, you add in keys from there, you can quickly skip places where the data is the same. You could treat it like an eventually consistent system in that way. Also could merge just subsets of the tree according to the keyspace

@pry The other cool part is that compared to other DBs like hyperbee from https://holepunch.to Prolly Tree based databases can be determenistically merged without much hassle so you can have communities merging datasets into larger and larger indexes without having to fully replicate all the data, just the parts at boundries.

@pry Right now most p2p data things use some variation on an "oplog". As your data grows it gets slower and slower for a peer to get the state of the data and it makes it hard to do "sparse replication" of just the data you want.

Using an indexed database as a building block means you can replicate just the data you need for a particular db query. This means you applications get more speedy and you use less data overall. You can still do full replication in the background.

@KatS Oh good, I think back when it first came out there was a max size limitation. Glad my knowledge is out of date on that!

@pry Oh! Have you seen my blog post on peer to peer databases by any chance? https://blog.mauve.moe/posts/peer-to-peer-databases

I'm less into realtime document sync and more into optimizing for sparse replication of views. CRDTs are cool but they often share the same issues as oplogs. I think the latest version of automerge is pretty speedy though.

@pry @yala @quinn IIRC the datamodel and execution is similar to datalog? I'm mostly interested in "local-first" daabases so you can merge data from multiple sources as you network connectivity grows with local data being available.

Datalog over #EarthStar might also be cool to do some day.

@KatS Nice. Yeah I was considering just using Neo4j but I wasn't sure if the max node limit would be something I'd run into. I think my matrix server data alone leads me to at least a few million records.

@pry @yala I think the work @quinn was doing on Rhizomatic seems relevant. Unsure what the state of it is.

https://www.youtube.com/watch?v=vkMXbk7Pn_M

@KatS Do you have any example projects building on it out there? Sounds relevant to what I'm into.

@yala @pry I looove datalog. Have you seen any good impls out there other than datomic? I also don't think I've seen any "live query" options out there yet.

I was thinking wrapping stuff in HTTP with an EvenSource endpoint would get pretty far towards easy to use interfaces.