I built something that lets you import Node modules from BitTorrent. It's weird. https://gitlab.com/EvanHahn/node-torrent-import
#Node #NodeJS #JavaScript #BitTorrent #decentralization #programming
@lutindiscret @mauve @dat_ecosystem @ninabreznik I'd love to try building a similar idea for Dat but don't know much about it. Is there a way to refer to a file by some immutable ID, like BitTorrent does?
@EvanHahn @lutindiscret @mauve @ninabreznik
The main mechanism is to refer to a file based on pubkey + version.
The pubkey refers to the "filesystem" containing the file. The version is a number between 0 and latest which refers to the exact point in time.
If you have the merkle root hash for that specific filesystem version or the specific file hash, you can later verify that the file has not been tampered with.
This is app/usecase specific
@EvanHahn @lutindiscret @mauve @ninabreznik
Dat started out to build "git for any data".
version controlled torrents authors can update over time. This is what dat is good for and was intended to do when it started as a project in 2013.
@dat_ecosystem Is there good example code for this? My dream would be something like:
let publicKey = "abc123"
let version = 456
let data = await dat.download(publicKey, version)
Is there something like that? Or would I need to build that myself (using Hyperdrive)?
@EvanHahn @dat_ecosystem yeah check out hyper-sdk. The p2p stufd can be a pain to set up for browsers tho
if you will build a project using any part of the dat-stack (=modules built by dat-ecosystem projects), you might even get your project listed as a dat-ecosystem project ♻️ 🥬
if you want to learn about the low level parts of the stack, it can be recommended to checkout https://docs.holepunch.to/
We did publish an interview with one of the core maintainers of the low level modules and will publish a recorded tech demo on the dat-ecosystem blog in the coming days.
@mauve @dat_ecosystem Alright, built a prototype! https://gitlab.com/EvanHahn/node-hyperdrive-import
Lets you do things like:
import myModule from "hyper+node:?key=abcd&discoverykey=1234&path=%2Fmodule.js";
Let me know what you think...I'm new to the Hypercore world so I'm sure I've made some mistakes. (Feel free to respond here or contact me via https://evanhahn.com/contact/)
@dat_ecosystem How would the versioning work?
And I didn't know about the discovery key derivability property...good to know!
A hyperdrive is just delegating all api interaction to
1. hyperbee (b-tree based key value store) for index
2. hyperblob to store raw file content
Both of those again just translate all their api calls to underlying hypercores (=merkleized append only logs) which are torrent like easy to replicate (range based log replication between participating peers)
Because all hyperdrive api interqction translates to appending new entries to the logs, eqch change increases the log length
A hyperdrive allows you to "check out" any version, which means pretend the underlying logs only containnentries up to that version (length).
Each log length between 0 and latest always has a single merkle roots hash signed by the authors keypair for that underlying log (hypercore).
Every change, = every append, =every new version creates a new merkle roots hash, which can be used to verify any data in the log via merkle proofs.
@dat_ecosystem This stuff is a little over my head but I expect to learn more about this soon and possibly revisit my project. Thanks for your help!
@EvanHahn That's awesome! As dat_ecosystem mentioned, we in the community usually use a different URL scheme. Have you seen my web browser that loads p2p URLs (including JS) natively? @agregore
I also have a cli environment which supports p2p web apis that might be relevant: https://github.com/AgregoreWeb/agregore-cli
@mauve I have not...this is neat! agregore-cli seems very similar to my idea, but better.
@EvanHahn @mauve @dat_ecosystem
Here is the original proposal which still makes a lot of sense.
https://github.com/hypercore-protocol/hypercore-proposals/blob/master/proposals/0002-hyper-url.md
The additions worth considering are:
1. a revision number, which increases every time a hypercore is truncated to edit history.
2. a rootsignature to make sure a specific pubkey+revision.version is actually what it is supposed to be by providing a z32 encoded to be able to verify the merkleroots using the feedpubkey after download.
@EvanHahn @mauve @dat_ecosystem
e.g. basically:
`hyper://<feedpubkey>+<revision>.<version>.<rootsignature>/<path¶ms&fragment>`
...so the rest of the url looks just like regular urls.
The rootsignature needs to include the version it was created for, which might be equal or greater than the version used in <version> but it must us the same <revision>
@EvanHahn @mauve @dat_ecosystem
another prefix could be `dat://` though :-)
@mauve @dat_ecosystem I don't need to support browsers for a demo, thankfully. Node is fine!
I need to learn more about the Hypercore ecosystem but I might try building something like my original post, but for Dat. We'll see...