Disable pre-allocation of storage space

Issues and ideas about I2PSnark
Post Reply
egyxsdbo
Posts: 8
Joined: 25 Jan 2023 07:48

Disable pre-allocation of storage space

Post by egyxsdbo »

It seems that the complete space that the package has at the end is allocated. How can I set this so that only the space actually needed is used?
User avatar
lgillis
Posts: 144
Joined: 20 Oct 2018 12:52

Re: Disable pre-allocation of storage space

Post by lgillis »

I2PSnark is designed to be a user-friendly BitTorrent client. The program has all the necessary features and can handle Magnet-URI and the user can even determine which files from the entire torrent should be downloaded and which not. Due to its straightforward web interface, the client is well suited as a "workhorse" running in the background. Many of the gimmicks found in "bloatware" have been deliberately omitted, also to avoid distracting new users of I2P from the essentials, namely exploring the concept of personal security. If you want to create your files to be stored according to certain methods, e.g. fill them with zeros, create sparse files or as it is typical for the XFS file system, you will look for it in vain in I2PSnark. That the storage space is pre-allocated can be seen as a feature, it prevents defragmentation and at the same time determines that sufficient space is available for the download.
Spring https://www.youtube.com/playlist?list=PLF-q-IGQQb1uK7fYuaQiRpcORDSmfsY2n
egyxsdbo
Posts: 8
Joined: 25 Jan 2023 07:48

Re: Disable pre-allocation of storage space

Post by egyxsdbo »

In times of SSD, defragmentation no longer counts.

I have worked around the problem with file system compression.
unveiledwrath
Posts: 2
Joined: 28 Mar 2023 08:43

Re: Disable pre-allocation of storage space

Post by unveiledwrath »

I had the same question: Why does I2PSnark allocate the files I want to skip? Isn't that wasteful?

It actually doesn't waste any space. Let me explain.

I looked at the code and did some testing. Turns out it allocates what's called a sparse file. That's a file filled with all zeroes. Modern, and I use that definition loosely, operating systems and file systems detect this as a sparse file and don't actually write those zeroes to disk. Then as data is written to the file, only that part is written to disk. So if you're downloading a torrent that's 10GB total, and you only download a 1MB file in it, the amount of disk used is basically 1MB, even if the size is shown as 10GB. File size is not same as size on disk. A caveat here is if you then want to copy the downloaded torrent to a more primitive filesystem like FAT32, it might not support sparse files and write the whole 10GB.

Why would I2PSnark do this in the first place? Because the BitTorrent v1 protocol works with pieces. Those pieces are not aligned to files. So you may want to download only one file, but the first or last piece of that file often overlaps with another file. I2PSnark writes that tiny overlapping part to the corresponding files, the rest of which remains all zeros.

There are alternative methods of doing this. For instance, qBittorrent does not create the files you skip. Instead, it keeps the overlapping pieces in hidden .parts files in the downloads folder. I'd argue that's superior to I2PSnark because you don't have to hunt the exact file you downloaded among the sea of empty files. It just shows what you downloaded. I prefer that.

BTW the new BitTorrent v2 spec fixes this. The pieces are aligned to individual files and no longer overlap.
lgillis wrote:e.g. fill them with zeros, create sparse files
That's exactly what I2PSnark seems to be doing.
egyxsdbo wrote:I have worked around the problem with file system compression.
That's indeed a foolproof way to make sure the skipped portions are not written to disk, but it's unnecessary thanks to sparse files, as I wrote above.
Post Reply