Introducing Yum Fast Downloader plugin

by Hedayat, posted in Computer, Fedora, Linux

A few days ago I was thinking that it’d be nice if yum would use a download manager application like aria2c for downloading files rather than it’s own urlgrabber, which lacks many features available in such applications. In fact, I wished for such a feature almost every time I used yum and it needed to download something more than a few MBs. But this time, I decided to see if I can implement it as a yum plugin and in a few hours I had a working prototype which could download packages using aria2c. It was really interesting, so I decided to work on it more seriously, and the first version is available now. It is still experimental, but has worked fine for me up to now! 😛

Warning: this is on of my first serious Python experiences, and my biggest Python code! 😀

Anyway, as I consider the current version to be usable, I created a repository for it which is available here so that updates can be delivered automatically. All you need to do is to add its .repo file to your /etc/yum.repos.d/ directory.

The current version (0.5) provides the following features:
• Using Aria 2 as the download manager application
• Downloading repository metadata from several mirrors simultaneously
• Downloading multiple packages in parallel from several mirrors
• Tries to use fastest mirrors for downloading

Please let me know if encountered any problems.

Have fun! 🙂

100 thoughts on “Introducing Yum Fast Downloader plugin”

  1. The new aria2c (version 1.10.9) has an annoying 16 max-connection-per-server limit. Hope you found a work around for that.

    Reply

    1. No, I wasn’t aware of this limitation at all. However, I think that’s a reasonable limit. Considering the number of Fedora mirrors, this is not an issue for them. Also, we should care about the servers too! 🙂

      Reply

      1. I tried the plugin last week, and the only few things I can think of is that there should be color escapes, to make things a little more interesting, and that some of the verbosities are HASHES FLYING BY at +2.3Mb/s…these ought be truncated and or logged at configurable intervals, but not to the intent of eliding or truncation of the most readable data, NAMES which in spite of features, one must assume that LESS IS MORE in such a case, like READING SIGNS ON THE HIGHWAY. unless what might be expressed in more concise terms, be frankly relegated to a per transaction-entry log file.

        I recommend using cache by grepping /proc/meminfo at a preselected ratio, bear with me, I don’t have the source code yet…but using aria2 is a fantastic idea. Keep up the good work. The meta-ordering in renaming the plugin from y*_fd* to y*-f*-d*. fits: -A-Za-z0-9,./?%&#:_=+@~ in BaSH. The convention is yum-plugin-~ (Don’t take that literally, or it might be dead last.) I have no opinion on the name per se, but I am glad I caught the change. It seems perfectly logical. I saw this site; as a source of color escapes: [##http://www.bashguru.com/2010/01/shell-colors-colorizing-shell-scripts.html.##] cf: printenv @@ LS_COLORS. (Hope that helps some)

        Don’t hesitate to integrate with presto, it’s a great idea. Hardware acceleration on CUDA might be insane at this juncture, but very doable if one uses /usr/share/pci.ids to ascertain a given range of cards, which scalable vectors? for memory allocation might be assigned in CUDA extensions.

    1. I didn’t forget, but I were not sure if it works fine on F15 then. However, now it seems that they should work fine on F15 as is, so I added packages for F15 too.
      Thanks for your consideration 🙂

      Reply

  3. Dude, this is AWESOME. Combined with presto, it should make updating much, much quicker. I hope it gets put into the mainline distro as a plugin turned on by default.

    Reply

    2. It seems to work really well on F14 here but I am not so sure it uses presto at all. It seems to bypass it. I just updated gthumb and it seemed to download the .rpm directly rather than the .drpm.

      I think most people would want both to stack and if you just want one or the other you can enable/install them by themselves. Having one plugin silently disable or bypass another is not desirable.

      Reply

      1. Well, this is not an intended behavior. The plugin does not try to interfere with presto’s work. However, it seems that yum call’s this plugin before presto, and so this plugin starts to download full packages rather than letting presto to download delta packages instead. I’ll look into this problem to see how can I change the order in which different plugins are called.

        Thank you for reporting the issue! 🙂

      2. OK, it is fixed now. The plugin is renamed (yum-fast-downloader instead of fast_downloader) so that it will be run after presto. Not a good solution but apparently there are no other solutions without modifying yum itself. 😛

      3. Thanks! the new version works much better! I tested it updating a F15 LiveUSB.

        Presto runs and downloads the drpms and then YFD kicks in and downloads the full rpms that were not available as drpms. It would be ideal if YFD downloaded the drpms for presto but that might not be possible without integration of YFD deep in the presto plugin.

        It also may not be ultimately all that useful since the drpms are usually quite small and YFD really would only make a significant difference on bigger packages where the download has a chance to accelerate. I suspect YFD might actually be slower for small files due to the overhead. I noticed that the speed reported was marginal on small files.

        So this seems to be a good compromise.

      4. 🙂 Happy to hear that!
        I want to add support for downloading delta rpms too. Yes, for small files it won’t make any difference and it usually falls back to single connection download, but not all drpms are that small. Also, it has an additional benefit that it can download the file from the best mirror.
        Additionally, it still can provide speedups by downloading multiple drpms in parallel. However, it seems that it is not possible without modifying presto, or integrating these plugins together. I’ll try to find a good solution soon.

  4. This is nicely done. Have you talked to yum developers about merging this code or submitting it to the fedora repo?

    Reply

    1. Not yet. I was not sure if I’ll continue it when I started, so I just worked on it. On the other hand, I think that yum developers will not merge this code since they could go with this approach in the beginning, but they decided to use their own urlgrabber. But I’ll probably talk with them soon.
      I am thinking about submitting the package to Fedora repositories, but it is probably too early to do it. I’ll submit this package after a little more testing.
      Thanks 🙂

      Reply

      1. It is a great idea that definitely deserves to be in the main fedora repo. I would guess that the yum devs likely want to minimize the external dependencies for the default but this is a nice option for those that want to try and boost speed even further.

        Regardless, thanks for introducing me to aria … nice utility and knows a few more tricks than wget :]

      2. When I suggested talking to yum developers, the outcome isn’t necessarily merging of code but improvements and review and so on. For instance, being able to work with presto properly would be a major plus and I would suggest doing that as early as possible before putting anything into the Fedora repo.

        Anyway, as the maintainer of aria2 in Fedora, I am glad someone is taking advantage of it more.

      3. Thanks Rahul. 🙂 OK, I’ll certainly consult with them for further improvements.
        The problem with presto is resolved already and version 0.5.1 is in the repo. 😛
        Finally, thank you for providing aria2. It is really a great download utility. I discovered its abilities while I was developing this plugin, and its really great.

      4. It is definitely worth coordinating upstream! No doubt.

        Adding some hook mechanism (other than alphabetical order) with some flexibility to make sure plugins fire in the correct order would be a good discussion point. I can see some plugins wanting to execute before others at some stages and after others in later stages. Something that can’t be done now, it would seem.

        Another enhancement would be the abstraction of the downloader used so that this plugin could be made simpler and would simply define the downloader and gather the right parameters needed; possibly opening the doors to use wget, axel, rsync as well as aria2 … obviously not to be used all at once :]

      5. Yes, the current alphabetical ordering is sub-optimal when the complexity and the number of plugins increase. I’ll start the discussion.
        While it is possible to override urlgrabber with new functionality considering Python’s flexibility, it is possible to have a better thought download framework making things such as parallel downloading more structured. Currently, I were only able to provide parallel downloading of multiple files for rpms. Since parallel downloading is on yum’s TODO list, I might be able to work on this feature with yum developers.

  5. Fantastic work.
    Coincidentally I was playing with aria2c last night and was amazed at how much faster it was than wget.
    Now if we could only get a yum plugin that would install non-dependent packages immediately after download you’d see serious speedup of large updates!

    Reply

      1. Would changes need to be made to yum or could this indeed be handled with a plugin?

      2. Well, considering that yum is currently clearly has separated download and install parts, it will most probably needs some structural changes inside yum itself (while Python’s flexibility let you overwrite some parts of yum in a plugin, you’ll be probably end up overwriting most parts of the base code!).
        However, I’m not sure, and if you are serious about it you should discuss it with yum developers.

  6. Now that parallel downloading of Delta RPMs is supported, another way to speedup the update process would be to rebuild packages in parallel as well.

    I would love to see yum utilize multi-core-processors for that, although I don’t know if yum-fast-downloader is the right place to ask.

    Reply

    1. Notice that currently, it doesn’t support downloading delta RPMs, and they are downloaded using regular yum downloader. I’m planning to support downloading delta rpms, however it still will not download many delta rpms in parallel. I’ll probably need to talk with presto author to make it possible.

      However, AFAIK presto currently supports rebuilding packages in parallel and it seems that it is doing it already. And it makes integration of presto and yum-fast-downloader more challenging.

      Reply

      1. Thanks for sorting things out. Maybe it just doesn’t feel like it rebuilds packages in parallel then, because that process only begins after all packages have been downloaded.

        Anyway your work on the download front is highly appreciated and thank you very much!

  7. Pingback: Velocizzare gli aggiornamenti di Fedora (Yum Fast Downloader plugin) – Micene.net

  8. I was hoping for this support to come to yum for a long time now. ever since i saw Smart doing it and Arch Powerpill.

    Keep up the good work 🙂

    Reply
  10. Pingback: Fedora yum加速下载插件 | 美丽Conky配置一枚 - 笨兔兔

    1. Hi Ray,
      Thank you for the post. I’ll certainly look at your codes and try to merge them. Both for supporting other download managers and drpm download. 🙂

      Reply

  12. Hi Hedayat!

    I’ve talked to you the night you have written the fabulous plugin. I just want to remind you my suggestion and that’s it.

    Can add torrent download support to your plugin?

    Aria2c can handle the torrents and since you can use distributed way of keeping track of peers (by using “local peer discovery” and “DHT” features of aria2c), it could be a great way of improving update process in semi big local networks (for example a really big LAN in a university).

    Thanx!
    🙂

    Reply

    1. Hi Abbas!
      Yes, I think it is possible without any problems. The only thing to consider is security issues, but I think that is also assured as signatures are downloaded directly.
      Thanks 🙂

      Reply

      2. Oops, you are right. The used options doesn’t work on older aria2c versions, and so it has problems under Fedora 14. Sorry for the inconvenience. I’ll push an update to fix it soon. Please revert back to the 0.5.2-1 version for now.

        Thanks for the report

  14. Hi,

    I’ve been waiting for a plugin like this for ages. Just one question: why does it appear to download everything twice? The aria2 download works (fast), then yum takes over and does a slow download of the same package – it seems to ignore the .yfd download?

    Thanks!
    Andi

    Reply

  16. Hi Hedayat, its a great plugin I must say, speeds up a lot using aria2c and parallel downloads. I an using it for the past week and its a great help to me installing and updating packages as I am on a slow connection. However, I cant say if its a bug or not, but when downloading a single package, the plugin uses aria2c to dwonlaod, but the same RPM is again downloaded using yum primitive download. Please have a look at the terminal output pasted below:

    [srvmdk@Saurav ~]$ sudo yum install deja-dup
    Loaded plugins: blacklist, fastestmirror, remove-with-leaves, yum-fast-
    : downloader
    Loading mirror speeds from cached hostfile
    * fedora: mirror.cse.iitk.ac.in
    * rpmfusion-free: mirror.cse.iitk.ac.in
    * rpmfusion-free-updates: mirror.cse.iitk.ac.in
    * rpmfusion-nonfree: mirror.cse.iitk.ac.in
    * rpmfusion-nonfree-updates: mirror.cse.iitk.ac.in
    * updates: mirror.cse.iitk.ac.in
    Setting up Install Process
    Resolving Dependencies
    –> Running transaction check
    —> Package deja-dup.i686 0:18.2.1-1.fc15 will be installed
    –> Finished Dependency Resolution

    Dependencies Resolved

    ================================================================================
    Package Arch Version Repository Size
    ================================================================================
    Installing:
    deja-dup i686 18.2.1-1.fc15 updates 586 k

    Transaction Summary
    ================================================================================
    Install 1 Package(s)

    Total download size: 586 k
    Installed size: 2.5 M
    Is this ok [y/N]: y
    Downloading Packages:

    2011-09-06 17:59:19.925642 NOTICE – ServerStat file /home/srvmdk/Downloads/Packages/i386/15/aria2c_server_stats loaded successfully.
    [#1 SIZE:576.0KiB/586.3KiB(98%) CN:1 SPD:10.6KiBs]
    2011-09-06 18:00:12.844382 NOTICE – Download complete: /home/srvmdk/Downloads/Packages/i386/15/updates/packages/deja-dup-18.2.1-1.fc15.i686.rpm.yfd

    2011-09-06 18:00:12.845527 NOTICE – ServerStat file /home/srvmdk/Downloads/Packages/i386/15/aria2c_server_stats saved successfully.

    Download Results:
    gid|stat|avg speed |path/URI
    ===+====+===========+===========================================================
    1| OK| 11.6KiB/s|/home/srvmdk/Downloads/Packages/i386/15/updates/packages/deja-dup-18.2.1-1.fc15.i686.rpm.yfd

    Status Legend:
    (OK):download completed.
    deja-dup-18.2.1-1.fc15.i686.rpm | 586 kB 01:00
    Running rpm_check_debug
    Running Transaction Test
    Transaction Test Succeeded
    Running Transaction
    Installing : deja-dup-18.2.1-1.fc15.i686 1/1

    Installed:
    deja-dup.i686 0:18.2.1-1.fc15

    Complete!
    —–

    hope you will be able to fix it. All the bset and continue doing good things like this in your life 😀

    PS: I am on Fedora 15 and using the plugin’s latest version.

    Reply

    1. 🙂 Thanks!
      About the problem, if it happens for single package downloads, I think I know the problem. I’ll push an update soon, I hope it’ll fix the problem.

      Reply
  17. Pingback: YFD plugin update « Hedayat’s Blog

  18. Hi.
    I tried to install tomahawk with the repo file in this link:
    http://download.opensuse.org/repositories/home:/pansenmann:/tomahawk-player/Fedora_15/
    but yum gave me this error:

    http://download.opensuse.org/repositories/home%3A/pansenmann%3A/tomahawk-player/Fedora_15/repodata/7980af617770c7ad630d1aec99f40899b8fcba7ec5a950ac98bfb441f27728a0-primary.xml.gz: [Errno -1] Metadata file does not match checksum
    Connessione ad un altro mirror in corso.
    Errore: failure: repodata/7980af617770c7ad630d1aec99f40899b8fcba7ec5a950ac98bfb441f27728a0-primary.xml.gz from home_pansenmann_tomahawk-player: [Errno 256] No more mirrors to try.
    I removed yum-fast-donwloader and all work very well.
    Why?

    Reply

    1. Hi,
      I’ll check tomorrow to see if I have the same problem. However, the file could have been corrupted for some reason. Have you tried removing the metadata for that repo and try again to see if the problem persists? Also, it would be helpful if you posted the full output. Anyway, if I could reproduce the problem I’ll try to fix it. Currently I have no idea why it must happen with this plugin.
      Thanks a lot

      Reply

  19. Yes! This is the yum plug-in I have looking for for a long time. Thank you very much.

    My Internet connections are always very slow (about ~ 10kB/s). This plug-in increases download rate more than 15 times (up to ~160kB/s) specially with “yum update” to download many packages simultaneously.

    Thanks again.

    Reply

  20. got problem with apper, the gui frontend in fc16 kde. it just stalls there showing nothing while in fact the package successfully installed. so for anyone using gnome, does this works with gnome gui frontend for yum?

    Reply

    1. Yes, I’ve personally tried gnome front-end and works fine. But I’ve not tried the KDE front-end. It probably expects a specific kind of output from package kit?! It’d be nice if we can find the reason and see if it can be solved in the plugin.

      Reply

  21. Thanks,
    works for me too,

    keeping it up to date and responsive to users needs would make it one of the best plug-ins of yum.

    good luck Hedayat

    Reply

  22. I found another idea while surfing yum. Try grepping yum info!!! That is super. I will be back on tonight. I think that would be super. I am pulling the cloneurl.

    Reply

      1. There is better dialog in ‘yum info packagename‘ that actually gives the source URL for the code, and a brief description of the package contents, if you are interested.

        I figure if you needed a ready source for pertinent information for a downloader plugin, this was the icing on the cake, There are SO many things, library calls, system calls, kernel calls, (a module then) It is harder to say when one should stop, but there is none of that in linux. Typically, if something doesn’t grow, it dies, but for now, you might think what might improve yum somewhat, BUT DON’T write the thing yourself. Limit yourself to a plugin, but you can source anything you like, It’s not plagiarism to cite libraries, and macros. get some .bashrc macros while you’re at it, and take tons of notes.

        I have been on all day retooling a build-dev enviornment, mostly to make Linux code manipulation a bit easier for me, (I can be slow sometimes, but so was Saturn V.)
        I am almost done, I will keep you posted. (I think once I am in it it can’t be worse than 24 hours in Redmond’s flagship Vista kernel. If there is anything more I can do, drop me a line. RSVP kpharris32@gmail.com God Bless. Have a good night. (I have a lot of back pain, so I have to retire early tonight, but if I can at all help, I REALLY like your idea, I will keep you abreast. I’ve seen your repo on fedora. you might want to post a link to the code and force fedora to mainstream it. I like plugins better than a monsterous backend piece. But I remember how long updates took before yum-presto plugin. I thought I read it wrong, and when I saw that you were using aria2 as a file parser, I said. Gee. THAT took long enough, but I dug for the link. (Aria2 had been around from four years ago at least.) It just goes to show you that it is but time, chance, and favor that dictate the times, If you have all three, you do. I think lynx/ncurses iintegration would provide a MENU for downloadable links in a repoview, but that is a bit over my head, and I wonder if that is so applicable, but sometimes there no accounting for tastes..,

  23. Thank you! You might have increased my upgrade time from years to days, making it feasible to upgrade. The text interface might be better implemented in a GUI like ‘pirut’ (whatever they’re calling it these days), just keep that in mind if this ever gets integrated upstream.

    Reply

    1. 🙂 Happy to hear that it is useful for you. Thanks.
      Yum is a CLI program, so I doubt if it can have a GUI (maybe it can have something like that in text mode). But PackageKit could have that!

      Reply

      1. Well, what I mean is, if something like this is implemented in PackageKit. 🙂

      2. Since upstream is also working on a multi-downloader facility, probably it is time to think about a GUI counterpart in PackageKit. But I doubt if there will be enough interest there. PackageKit apparently thinks that saying that it is in “Downloading” state is all the information a user should get!

  24. Pingback: Fedora 17, Offline Updates, Parsidora, Yum Fast Downloader and Nokia N9! « Hedayat’s Blog

  25. Could you sign the package? The repo is only available over HTTP, so a GPG signature is the only guarantee of not changing in the future.

    Reply

  27. Man, you just saved my ass on Fedora 18 which tried to download with FASTEST MIRROR at 56KB/s !!!!!

    What a moron made that yum!!! (I have bandwidth of 64Mb/s and to this mirror I have FULL OPEN bandwidth – never goes below 64Mb/s!)

    Funny thing is that I always thought its the problem of “unsupported ones” – I had FC14 not long time ago – but no – with the newest crappy FC18 it’s the same!

    Your plugin accelerated my download to about 8MByte per second! Which is insane to be honest! 😀

    Reply

  28. Hello,

    Thank you for the great plugin, upgrading a clean instillation take much less time with this plugin.

    when i use the plugin with a opensuse build server repo i get a error,

    
================================================================================
Downloading (home_The-IceMan-Blog) repodata/170e335d0bc2b1d59d8b667a0932f006c69ccc92aac99467a97b12b210ee19f6-primary.xml.gz
================================================================================

2013-05-26 16:35:23.659322 NOTICE - ServerStat file /var/cache/yum/x86_64/18/aria2c_server_stats loaded successfully.

2013-05-26 16:35:23.843504 NOTICE - Download complete: /var/cache/yum/x86_64/18/home_The-IceMan-Blog/170e335d0bc2b1d59d8b667a0932f006c69ccc92aac99467a97b12b210ee19f6-primary.xml.gz.yfd

2013-05-26 16:35:24.155415 NOTICE - Download complete: /var/cache/yum/x86_64/18/home_The-IceMan-Blog/170e335d0bc2b1d59d8b667a0932f006c69ccc92aac99467a97b12b210ee19f6-primary.xml.gz

2013-05-26 16:35:24.156491 NOTICE - ServerStat file /var/cache/yum/x86_64/18/aria2c_server_stats saved successfully.

Download Results:
gid|stat|avg speed  |path/URI
===+====+===========+===========================================================
  1|  OK|        n/a|/var/cache/yum/x86_64/18/home_The-IceMan-Blog/170e335d0bc2b1d59d8b667a0932f006c69ccc92aac99467a97b12b210ee19f6-primary.xml.gz.yfd
  2|  OK|  38.9KiB/s|/var/cache/yum/x86_64/18/home_The-IceMan-Blog/170e335d0bc2b1d59d8b667a0932f006c69ccc92aac99467a97b12b210ee19f6-primary.xml.gz

Status Legend:
(OK):download completed.

================================================================================
Downloading (updates) repodata/356de58b6476f98ac586b0c0a20d04d247855a85d6651e6fbfd545c1906f4a6e-primary.sqlite.bz2
================================================================================
mv: ‘/var/cache/yum/x86_64/18/home_The-IceMan-Blog/170e335d0bc2b1d59d8b667a0932f006c69ccc92aac99467a97b12b210ee19f6-primary.xml.gz’ and ‘/var/cache/yum/x86_64/18/home_The-IceMan-Blog/170e335d0bc2b1d59d8b667a0932f006c69ccc92aac99467a97b12b210ee19f6-primary.xml.gz’ are the same file

2013-05-26 16:35:24.174521 NOTICE - ServerStat file /var/cache/yum/x86_64/18/aria2c_server_stats loaded successfully.
[#1 SIZE:8.8MiB/9.2MiB(96%) CN:1 SPD:302.9KiBs ETA:1s]                                                                                                                                                             
2013-05-26 16:35:46.201887 NOTICE - Download complete: /var/cache/yum/x86_64/18/updates/356de58b6476f98ac586b0c0a20d04d247855a85d6651e6fbfd545c1906f4a6e-primary.sqlite.bz2.yfd

2013-05-26 16:35:46.203211 NOTICE - ServerStat file /var/cache/yum/x86_64/18/aria2c_server_stats saved successfully.

Download Results:
gid|stat|avg speed  |path/URI
===+====+===========+===========================================================
  1|  OK| 431.1KiB/s|/var/cache/yum/x86_64/18/updates/356de58b6476f98ac586b0c0a20d04d247855a85d6651e6fbfd545c1906f4a6e-primary.sqlite.bz2.yfd
    Reply

      1. thank you, will you announce the fix here in the blog post?,
        or we will just get a silent update via yum ?

        =)

  29. This plugin is just f$”%”$%ing AWESOME! (I especially appreciated the management of mirrors)
    I’m eager to see when it downloads also the delta rpms … then updating will be a breeze!

    Reply

    1. Thanks. Hopefully I’ll get some time soon and update the plugin to better work for Fedora 19. 🙂 Maybe playing nicer with delta rpms too.

      Reply

  30. It would be nice to see the current/total number of packages that have been/need to be downloaded, as yum usually does without this plugin installed. In any case, this is very useful, thanks much!

    Reply

  31. It appears that the fc18 version still works on fc19, but it would be very nice to have a site update to better work with simple yum installations.

    Reply

  32. Two issues may you work on :

    First :: Yumex package manager can’t detect downloading progress when YFD used .
    Second :: YFD download packages from build.opensuse.org many many times without detect them to install !!!!!!

    Thank You For Your Attention .

    Reply

    1. Thanks for your attention. I will fix the bug with opensuse build service. Also compatibility with Fedora 19 presto feature in yum (no longer a plugin). But Yumex is not a priority and I don’t have enough free time for that now.

      Thanks again. 🙂

      Reply
  35. Pingback: Nowruz Gift: Yum Fast Downloader 0.6.0 Released! | Hedayat's Blog

    1. If using CentOS 7.0, the package available in Fedora 20 repo should also work there too. However, for centos 6.5, older packages might or might not work (depends on the version of yum and aria2 packages).

      Reply

  39. Upgrading from Fedora 20 to Fedora 21 would have taken forever without this handy plugin!

    However, it didn’t work with fedup out of the box. During the upgrade, it attempts to log the server stats in the yum cache directory of the new release, which does not exist. The error message was:

    Failed to open ServerStat file /var/cache/yum/x86_64/21/aria2c_server_stats for write

    Creating the directory manually solved the problem.

    Reply
  40. Pingback: Unix:Why yum's download speed is too slow – Unix Questions
  41. Pingback: Mejorar velocidad de descarga en YUM – CentOS 7 – El blog de Ghabriel

Leave a reply to Anonymous Cancel reply