How many files can be adressed by the muwire network?

The MuWire file sharing application
zlatinb

Re: How many files can be adressed by the muwire network?

Post by zlatinb »

zlatinb wrote: 15 Feb 2022 03:37 Other than that MuWire supports something like plugins, so most if not all of the work can be done outside of the main code. If you clone the source from git and look inside the "mwmodules" directory you will see an example plugin.
On a second thought it would be better to build an entirely separate application for this that uses the MuWire core. Then you don't force the users to run a GUI or you can build your own. The core is published as a gradle artifact so you can easily integrate it. For example see https://github.com/zlatinb/mucats
zlatinb

Re: How many files can be adressed by the muwire network?

Post by zlatinb »

zlatinb wrote: 15 Feb 2022 03:37 Maybe something more clever can be done by the webbrowser extension where a .zip with all the images gets pulled as a single file from MuWire and then unpacked in the browser?
Actually MuWire already supports something called "collections". This is a single file which describes a collection of files and they all can be downloaded in parallel. The collection already includes the publisher which is used as a source, so you only need to do a single search on to find the collection descriptor.

So a webpage hosted on this filesystem (I'll call it "MuFS") would look like this:

Code: Select all

<html>
<!-- the "hash" is the hash of the collection and "file" points to a file inside the collection -->
<mufs hash="asdf123..." file="img1.png"/>
<!-- if the hash is the same the plugin knows not to kick off a new search -->
<mufs hash="asdf123..." file="img2.png"/>
<!-- but if it's different then it needs to search for it.  That way you can split many images into fewer network searches -->
<mufs hash="qwer456..." file="img3.png"/>
</html>
If I were building this I would put most of the logic in the MuFS daemon and keep the browser plugin as minimal as possible. For example the plugin would just request a hash+file combination from the daemon which would in turn do all the network searching and serving of files.

MuWire learns about other sources from the existing ones already, so you don't need to do any changes to enable multi-source downloads.
robabok978
Posts: 36
Joined: 12 Feb 2022 02:58

Re: How many files can be adressed by the muwire network?

Post by robabok978 »

If the website uses more or less the same images for navigation etc., it could be bundled with collection into something like website_x.core. It would be hashed, so distinct and the feature of https://muwire.com/file-certificates.html could come handy in the future too.

If a webside has core + more variable content it should be handleable with a size restriction or an option to download.

But i am not really sure if smaller files are really that of a problem. we don't look at all images hosted by the webserver at once. So it is more site based. Ok one million images would be to much but smaller numbers should be ok as long as they are cached after the time. The first time will be slow and restricted to your mentioned time and number of searches. After that it should be ok. but something like downloading files smaller than x should be optional and sitebased.

I think it is important to have a seperate routine IsItDownloaded, which should be fast and seperated from the search. That would allow faster cache reaction. I am no expert maybe something like bitwise/character compare of the hashes first ok, second wrong, break, not downloaded.

If a demon like you mentioned exists, would be possible to use apt-get or wget with muwire/i2p only with slightlyy changes, to download packages for distros etc.? You need to have the right hashes for muwire ok and after download apt does its own verificationn. Would be great to download packages for my distro with i2p especially if the server runs only in i2p or do some scripting.
Post Reply