I think what I want instead of apps is a standard datamodel in a graph database with live queries I can tie to table views.
@mauve
To the point.
LogSeq is fun here.
@mauve @yala https://github.com/tonsky/datascript
this has always looked pretty cool to me! but rlly this and datomic are all I've seen. I'm a big datalog fan.
lemme know if u know of any others!
I also kinda want to implement a datomic like db myself tbh
@pry @yala @quinn IIRC the datamodel and execution is similar to datalog? I'm mostly interested in "local-first" daabases so you can merge data from multiple sources as you network connectivity grows with local data being available.
Datalog over #EarthStar might also be cool to do some day.
@mauve @yala @quinn ooh I actually really became interested in distributed systems because of learning about local first software and CRDTs. I really want to do work making CRDTs more practical for more use cases. I think it's rlly cool to combine the eventually consistency of CRDTs with maybe slightly stronger snapshot consistency of some other sort of protocol. CRDT truncation also seems important?
@pry Oh! Have you seen my blog post on peer to peer databases by any chance? https://blog.mauve.moe/posts/peer-to-peer-databases
I'm less into realtime document sync and more into optimizing for sparse replication of views. CRDTs are cool but they often share the same issues as oplogs. I think the latest version of automerge is pretty speedy though.
@mauve ok this is a really really cool blog post. I definitely am going to read up more on prolly trees.
I think I don't fully understand how these can help in the world of p2p data management yet
@pry Right now most p2p data things use some variation on an "oplog". As your data grows it gets slower and slower for a peer to get the state of the data and it makes it hard to do "sparse replication" of just the data you want.
Using an indexed database as a building block means you can replicate just the data you need for a particular db query. This means you applications get more speedy and you use less data overall. You can still do full replication in the background.
@mauve ooh interesting. do you know any good resources for learning the merging semantics of a prolly tree? wasn't able to find too much about them. I've also been pretty interested in Merkle Search Trees and once I understand what a prolly tree is, I'll be able to compare them
@pry Merkle Search Trees are way more complicated to deal with and implement in my experience. I wrote up a spec for the prolly tree impl I use here: https://github.com/ipld/ipld/blob/776b537e0d16dc0341f1bec13ee79bd05a0dfb9e/specs/advanced-data-layouts/prollytree/spec.md
It talks about how merging works.
The tl;dr is you traverse the merkle tree and where you see a diff, you add in keys from there, you can quickly skip places where the data is the same. You could treat it like an eventually consistent system in that way. Also could merge just subsets of the tree according to the keyspace
@mauve ah thank you!! really super cool work you are doing! this is exactly the sort of work I'd one day like to be involved with
@pry The other cool part is that compared to other DBs like hyperbee from https://holepunch.to Prolly Tree based databases can be determenistically merged without much hassle so you can have communities merging datasets into larger and larger indexes without having to fully replicate all the data, just the parts at boundries.