Data Hoarder

168 readers

1 users here now

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time (tm) ). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

founded 1 year ago

MODERATORS

[email protected]

Question about YT-DLP and a mess .part-frags (alien.top)

submitted 11 months ago by [email protected] to c/[email protected]

5 comments fedilink hide all child comments

So I'll fully admit this is probably user error and not due to any bugs with the software. But I've been on a bit of a tear lately utilizing YT-DLP to download videos from YouTube. Now I'm not a stranger to this process, I'd used Youtube-DL successfully for years now. But I did recently take advantage of using it in conjunction Aria2c with the understanding that it would give me a bit more of a boost . Well...

I don't know if it's YT-DLP or Aria or my NAS or my Internet connection, but I went through to organize these downloads and found TONS and TONS of video files ending with ".part-frag###." There are thousands and thousands, and upon a cursory glance, seems to affect hundreds of video downloads. Each folder has dozens of these files and if it doesn't, it has split mp4/m4a. Rarely does a folder seem to have one clean video file like I would expect. I skimmed through and there seems to be finished files amongst the mess that seem to play fine, I can't be 100% sure though these files are as complete as they should be (surely the part-frags exist for a reason).

So firstly, what program or setting is causing this plague of fragments? Secondly, is there anything I can do to resolve theses files at this point? Yes, I can delete all the frags, but that doesn't give me any comfort in the integrity of whatever video file remains (and it won't solve the problem for future downloads). And as I archive to a txt file, I can't just re-download as YT-DLP considers them downloaded (I've tried redownloading to see it forces a re-check and cleans things up, but no dice). Is it just refusing to merge or delete for some reason?

top 5 comments

sorted by: hot top controversial new old

[–] [email protected] 1 points 11 months ago

May be the NAS is the problem.

Aria2c will get these fragments. But at the end ffmpeg or equivalent will merge everything and delete all the temporary fragments.

Somehow it is not getting deleted

Ask in yt-dlp discussions (github).

[–] [email protected] 1 points 11 months ago

have you tried checking the Aria2c settings? It might be causing the fragmentation issue. Also, try using the --fixup flag in YT-DLP to see if it can repair the fragmented files. Good luck!

[–] [email protected] 1 points 11 months ago (1 children)

For those of us unaware. Can you mention what purpose aria serves in your downloading process?

[–] [email protected] 1 points 11 months ago

it can speed up downloads on sites that limit per connection so downloading fragments 8 at a time speeds things up considerably

that's what I've used it for

[–] [email protected] 1 points 11 months ago

What I would do is create a new directory. Download a few videos within subdirectories within that new directory using only YT–DLP. Don’t use aria and upon completion compare the whole videos from the new download session using only YT-DLP with the previous one (aria). I would check by playing the files side by side if possible and check the videos’ duration. Check the file sizes and run diff in a terminal with both files provided as arguments. If diff complains of a mismatch between the binary files they indeed differ in one aspect.