Follow

Due to my hubris I am in the process of rewriting my zip file parser from scratch. :P At least now I have experience using the particular APIs and the things I need to watch out for from using Blobs.

· · Web · 2 · 0 · 0

Though TBH my initial approach would have been better for a purely streaming mode where we process and chunk the WARC file on the fly. Now I need to be able to do random access to a WARC either on the FS or via Range queries over the network.

@mauve You know about gildas-lormeau.github.io/zip.j right? Amazing zip library which works with WHATWG streams, Blobs, etc.

@gwil On interesting, I could potentially use some of their internals for zip parsing. The surface level API won't work for my use case though since I'm looking to do a close-to-zero-copy streaming read of the file as I construct an IPFS DAG from it. The `seekSignature` file seems pretty useful however.

@gwil @mauve

the other day i tried to use chatgpt to extract minimal code from this to allow creating a zip archive with files/folders uncompressed in a browser just for the sake of allowing download, but what i got wasnt working yet sadly and then gave up on it, but was only ~50 loc 🙂

@serapath @gwil yeah! Honestly the zip parsing isn't so bad. It's a bit strange since you *have* to scan from the back in order to search for the signature (inefficient AF IMO). But outside of that it's pretty straightforward.

Sign in to participate in the conversation
Mauvestodon

Escape ship from centralized social media run by Mauve.