A few days ago I was thinking that it’d be nice if yum would use a download manager application like aria2c for downloading files rather than it’s own urlgrabber, which lacks many features available in such applications. In fact, I wished for such a feature almost every time I used yum and it needed to download something more than a few MBs. But this time, I decided to see if I can implement it as a yum plugin and in a few hours I had a working prototype which could download packages using aria2c. It was really interesting, so I decided to work on it more seriously, and the first version is available now. It is still experimental, but has worked fine for me up to now! 
Warning: this is on of my first serious Python experiences, and my biggest Python code! 
Anyway, as I consider the current version to be usable, I created a repository for it which is available here so that updates can be delivered automatically. All you need to do is to add its .repo file to your /etc/yum.repos.d/ directory.
The current version (0.5) provides the following features:
• Using Aria 2 as the download manager application
• Downloading repository metadata from several mirrors simultaneously
• Downloading multiple packages in parallel from several mirrors
• Tries to use fastest mirrors for downloading
Please let me know if encountered any problems.
Have fun! 
Posted by Mohammad on May 11, 2011 at 6:52 pm
The new aria2c (version 1.10.9) has an annoying 16 max-connection-per-server limit. Hope you found a work around for that.
Posted by Hedayat on May 11, 2011 at 7:01 pm
No, I wasn’t aware of this limitation at all. However, I think that’s a reasonable limit. Considering the number of Fedora mirrors, this is not an issue for them. Also, we should care about the servers too!
Posted by Peter on January 5, 2012 at 5:28 pm
I tried the plugin last week, and the only few things I can think of is that there should be color escapes, to make things a little more interesting, and that some of the verbosities are HASHES FLYING BY at +2.3Mb/s…these ought be truncated and or logged at configurable intervals, but not to the intent of eliding or truncation of the most readable data, NAMES which in spite of features, one must assume that LESS IS MORE in such a case, like READING SIGNS ON THE HIGHWAY. unless what might be expressed in more concise terms, be frankly relegated to a per transaction-entry log file.
I recommend using cache by grepping /proc/meminfo at a preselected ratio, bear with me, I don’t have the source code yet…but using aria2 is a fantastic idea. Keep up the good work. The meta-ordering in renaming the plugin from y*_fd* to y*-f*-d*. fits: -A-Za-z0-9,./?%&#:_=+@~ in BaSH. The convention is yum-plugin-~ (Don’t take that literally, or it might be dead last.) I have no opinion on the name per se, but I am glad I caught the change. It seems perfectly logical. I saw this site; as a source of color escapes: [##http://www.bashguru.com/2010/01/shell-colors-colorizing-shell-scripts.html.##] cf: printenv @@ LS_COLORS. (Hope that helps some)
Don’t hesitate to integrate with presto, it’s a great idea. Hardware acceleration on CUDA might be insane at this juncture, but very doable if one uses /usr/share/pci.ids to ascertain a given range of cards, which scalable vectors? for memory allocation might be assigned in CUDA extensions.
Posted by pankaj on May 11, 2011 at 7:19 pm
great idea, but it seems you forgot to add the F15 packages in the repo
Posted by Hedayat on May 11, 2011 at 9:31 pm
I didn’t forget, but I were not sure if it works fine on F15 then. However, now it seems that they should work fine on F15 as is, so I added packages for F15 too.
Thanks for your consideration
Posted by Eric Mesa on May 11, 2011 at 7:41 pm
Dude, this is AWESOME. Combined with presto, it should make updating much, much quicker. I hope it gets put into the mainline distro as a plugin turned on by default.
Posted by Hedayat on May 11, 2011 at 10:03 pm
Posted by Michael Wiktowy on May 11, 2011 at 10:22 pm
It seems to work really well on F14 here but I am not so sure it uses presto at all. It seems to bypass it. I just updated gthumb and it seemed to download the .rpm directly rather than the .drpm.
I think most people would want both to stack and if you just want one or the other you can enable/install them by themselves. Having one plugin silently disable or bypass another is not desirable.
Posted by Hedayat on May 11, 2011 at 10:36 pm
Well, this is not an intended behavior. The plugin does not try to interfere with presto’s work. However, it seems that yum call’s this plugin before presto, and so this plugin starts to download full packages rather than letting presto to download delta packages instead. I’ll look into this problem to see how can I change the order in which different plugins are called.
Thank you for reporting the issue!
Posted by Hedayat on May 11, 2011 at 11:16 pm
OK, it is fixed now. The plugin is renamed (yum-fast-downloader instead of fast_downloader) so that it will be run after presto. Not a good solution but apparently there are no other solutions without modifying yum itself.
Posted by Michael Wiktowy on May 12, 2011 at 1:12 am
Thanks! the new version works much better! I tested it updating a F15 LiveUSB.
Presto runs and downloads the drpms and then YFD kicks in and downloads the full rpms that were not available as drpms. It would be ideal if YFD downloaded the drpms for presto but that might not be possible without integration of YFD deep in the presto plugin.
It also may not be ultimately all that useful since the drpms are usually quite small and YFD really would only make a significant difference on bigger packages where the download has a chance to accelerate. I suspect YFD might actually be slower for small files due to the overhead. I noticed that the speed reported was marginal on small files.
So this seems to be a good compromise.
Posted by Hedayat on May 12, 2011 at 1:45 am
I want to add support for downloading delta rpms too. Yes, for small files it won’t make any difference and it usually falls back to single connection download, but not all drpms are that small. Also, it has an additional benefit that it can download the file from the best mirror.
Additionally, it still can provide speedups by downloading multiple drpms in parallel. However, it seems that it is not possible without modifying presto, or integrating these plugins together. I’ll try to find a good solution soon.
Posted by Rahul Sundaram on May 11, 2011 at 8:12 pm
This is nicely done. Have you talked to yum developers about merging this code or submitting it to the fedora repo?
Posted by Hedayat on May 11, 2011 at 10:12 pm
Not yet. I was not sure if I’ll continue it when I started, so I just worked on it. On the other hand, I think that yum developers will not merge this code since they could go with this approach in the beginning, but they decided to use their own urlgrabber. But I’ll probably talk with them soon.
I am thinking about submitting the package to Fedora repositories, but it is probably too early to do it. I’ll submit this package after a little more testing.
Thanks
Posted by Michael Wiktowy on May 11, 2011 at 10:28 pm
It is a great idea that definitely deserves to be in the main fedora repo. I would guess that the yum devs likely want to minimize the external dependencies for the default but this is a nice option for those that want to try and boost speed even further.
Regardless, thanks for introducing me to aria … nice utility and knows a few more tricks than wget :]
Posted by Rahul Sundaram on May 11, 2011 at 11:07 pm
When I suggested talking to yum developers, the outcome isn’t necessarily merging of code but improvements and review and so on. For instance, being able to work with presto properly would be a major plus and I would suggest doing that as early as possible before putting anything into the Fedora repo.
Anyway, as the maintainer of aria2 in Fedora, I am glad someone is taking advantage of it more.
Posted by Hedayat on May 11, 2011 at 11:22 pm
Thanks Rahul.
OK, I’ll certainly consult with them for further improvements.
The problem with presto is resolved already and version 0.5.1 is in the repo.
Finally, thank you for providing aria2. It is really a great download utility. I discovered its abilities while I was developing this plugin, and its really great.
Posted by Michael Wiktowy on May 11, 2011 at 11:40 pm
It is definitely worth coordinating upstream! No doubt.
Adding some hook mechanism (other than alphabetical order) with some flexibility to make sure plugins fire in the correct order would be a good discussion point. I can see some plugins wanting to execute before others at some stages and after others in later stages. Something that can’t be done now, it would seem.
Another enhancement would be the abstraction of the downloader used so that this plugin could be made simpler and would simply define the downloader and gather the right parameters needed; possibly opening the doors to use wget, axel, rsync as well as aria2 … obviously not to be used all at once :]
Posted by Hedayat on May 12, 2011 at 1:50 am
Yes, the current alphabetical ordering is sub-optimal when the complexity and the number of plugins increase. I’ll start the discussion.
While it is possible to override urlgrabber with new functionality considering Python’s flexibility, it is possible to have a better thought download framework making things such as parallel downloading more structured. Currently, I were only able to provide parallel downloading of multiple files for rpms. Since parallel downloading is on yum’s TODO list, I might be able to work on this feature with yum developers.
Posted by liam on May 12, 2011 at 5:24 am
Fantastic work.
Coincidentally I was playing with aria2c last night and was amazed at how much faster it was than wget.
Now if we could only get a yum plugin that would install non-dependent packages immediately after download you’d see serious speedup of large updates!
Posted by Hedayat on May 12, 2011 at 1:26 pm
Yes! Currently, the problem is that there are not enough yum contributors for such optimizations.
Posted by liam on May 13, 2011 at 12:01 pm
Would changes need to be made to yum or could this indeed be handled with a plugin?
Posted by Hedayat on May 13, 2011 at 10:55 pm
Well, considering that yum is currently clearly has separated download and install parts, it will most probably needs some structural changes inside yum itself (while Python’s flexibility let you overwrite some parts of yum in a plugin, you’ll be probably end up overwriting most parts of the base code!).
However, I’m not sure, and if you are serious about it you should discuss it with yum developers.
Posted by Torben Schmidt on May 12, 2011 at 12:31 pm
Now that parallel downloading of Delta RPMs is supported, another way to speedup the update process would be to rebuild packages in parallel as well.
I would love to see yum utilize multi-core-processors for that, although I don’t know if yum-fast-downloader is the right place to ask.
Posted by Hedayat on May 12, 2011 at 1:24 pm
Notice that currently, it doesn’t support downloading delta RPMs, and they are downloaded using regular yum downloader. I’m planning to support downloading delta rpms, however it still will not download many delta rpms in parallel. I’ll probably need to talk with presto author to make it possible.
However, AFAIK presto currently supports rebuilding packages in parallel and it seems that it is doing it already. And it makes integration of presto and yum-fast-downloader more challenging.
Posted by Torben Schmidt on May 12, 2011 at 3:39 pm
Thanks for sorting things out. Maybe it just doesn’t feel like it rebuilds packages in parallel then, because that process only begins after all packages have been downloaded.
Anyway your work on the download front is highly appreciated and thank you very much!
Posted by Hedayat on May 12, 2011 at 5:36 pm
Posted by Velocizzare gli aggiornamenti di Fedora (Yum Fast Downloader plugin) – Micene.net on May 12, 2011 at 8:54 pm
[...] Post ufficiale del plugin [...]
Posted by Rabin on May 15, 2011 at 12:37 am
I was hoping for this support to come to yum for a long time now. ever since i saw Smart doing it and Arch Powerpill.
Keep up the good work
Posted by ibentu on May 19, 2011 at 12:54 pm
great,so thanks
Posted by Fedora yum加速下载插件 | 美丽Conky配置一枚 - 笨兔兔 on May 19, 2011 at 1:19 pm
[...] 感谢Hedayat同学，制作的Fedora下yum加速下载插件。 [...]
Posted by ray on May 24, 2011 at 12:21 pm
hi,hedayat
I have written another similar codes before and use axelget to download package as well as preto/drpm.
I have no time to update the codes, could you like to add axelget to your parts. Our codes can be merged into a single tools.
http://code.google.com/p/yum-axelget/
Thanks,
Ray
Posted by Hedayat on May 24, 2011 at 1:30 pm
Hi Ray,
Thank you for the post. I’ll certainly look at your codes and try to merge them. Both for supporting other download managers and drpm download.
Posted by Abbas on May 26, 2011 at 1:21 pm
Hi Hedayat!
I’ve talked to you the night you have written the fabulous plugin. I just want to remind you my suggestion and that’s it.
Can add torrent download support to your plugin?
Aria2c can handle the torrents and since you can use distributed way of keeping track of peers (by using “local peer discovery” and “DHT” features of aria2c), it could be a great way of improving update process in semi big local networks (for example a really big LAN in a university).
Thanx!
Posted by Hedayat on May 28, 2011 at 11:28 pm
Hi Abbas!
Yes, I think it is possible without any problems. The only thing to consider is security issues, but I think that is also assured as signatures are downloaded directly.
Thanks
Posted by tim on June 20, 2011 at 7:37 pm
yum update
..
..
mv: missing file operand
Try `mv --help' for more information.
..
..
is that OK ?
ver 0.5.4-2 F14
Posted by Hedayat on June 21, 2011 at 1:04 am
No, it’s not. Would you please post the full output?
Thanks
Posted by tim on June 21, 2011 at 10:34 am
yum clean all
yum update -v
output: http://fpaste.org/6e8k/
Posted by Hedayat on June 21, 2011 at 2:13 pm
Oops, you are right. The used options doesn’t work on older aria2c versions, and so it has problems under Fedora 14. Sorry for the inconvenience. I’ll push an update to fix it soon. Please revert back to the 0.5.2-1 version for now.
Thanks for the report
Posted by Andi on July 9, 2011 at 5:48 pm
Hi,
I’ve been waiting for a plugin like this for ages. Just one question: why does it appear to download everything twice? The aria2 download works (fast), then yum takes over and does a slow download of the same package – it seems to ignore the .yfd download?
Thanks!
Andi
Posted by Hedayat on July 10, 2011 at 11:30 pm
Hi Andi,
Which version of the plugin are you currently using? And also on which version of Fedora?
Posted by Hedayat on July 13, 2011 at 9:22 pm
Well, I found the problem. It is fixed in an update. Thanks!
Posted by muadib on July 22, 2011 at 5:36 pm
oh yes! this is exactly what I’ve been looking
Posted by Saurav Modak on September 6, 2011 at 4:10 pm
Hi Hedayat, its a great plugin I must say, speeds up a lot using aria2c and parallel downloads. I an using it for the past week and its a great help to me installing and updating packages as I am on a slow connection. However, I cant say if its a bug or not, but when downloading a single package, the plugin uses aria2c to dwonlaod, but the same RPM is again downloaded using yum primitive download. Please have a look at the terminal output pasted below:
[srvmdk@Saurav ~]$ sudo yum install deja-dup
Loaded plugins: blacklist, fastestmirror, remove-with-leaves, yum-fast-
: downloader
Loading mirror speeds from cached hostfile
* fedora: mirror.cse.iitk.ac.in
* rpmfusion-free: mirror.cse.iitk.ac.in
* rpmfusion-free-updates: mirror.cse.iitk.ac.in
* rpmfusion-nonfree: mirror.cse.iitk.ac.in
* rpmfusion-nonfree-updates: mirror.cse.iitk.ac.in
* updates: mirror.cse.iitk.ac.in
Setting up Install Process
Resolving Dependencies
–> Running transaction check
—> Package deja-dup.i686 0:18.2.1-1.fc15 will be installed
–> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
deja-dup i686 18.2.1-1.fc15 updates 586 k
Transaction Summary
================================================================================
Install 1 Package(s)
Total download size: 586 k
Installed size: 2.5 M
Is this ok [y/N]: y
Downloading Packages:
2011-09-06 17:59:19.925642 NOTICE – ServerStat file /home/srvmdk/Downloads/Packages/i386/15/aria2c_server_stats loaded successfully.
[#1 SIZE:576.0KiB/586.3KiB(98%) CN:1 SPD:10.6KiBs]
2011-09-06 18:00:12.844382 NOTICE – Download complete: /home/srvmdk/Downloads/Packages/i386/15/updates/packages/deja-dup-18.2.1-1.fc15.i686.rpm.yfd
2011-09-06 18:00:12.845527 NOTICE – ServerStat file /home/srvmdk/Downloads/Packages/i386/15/aria2c_server_stats saved successfully.
Download Results:
gid|stat|avg speed |path/URI
===+====+===========+===========================================================
1| OK| 11.6KiB/s|/home/srvmdk/Downloads/Packages/i386/15/updates/packages/deja-dup-18.2.1-1.fc15.i686.rpm.yfd
Status Legend:
(OK):download completed.
deja-dup-18.2.1-1.fc15.i686.rpm | 586 kB 01:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : deja-dup-18.2.1-1.fc15.i686 1/1
Installed:
deja-dup.i686 0:18.2.1-1.fc15
Complete!
—–
hope you will be able to fix it. All the bset and continue doing good things like this in your life
PS: I am on Fedora 15 and using the plugin’s latest version.
Posted by Hedayat on September 6, 2011 at 4:30 pm
About the problem, if it happens for single package downloads, I think I know the problem. I’ll push an update soon, I hope it’ll fix the problem.
Posted by Hedayat on October 8, 2011 at 2:47 pm
I think it is fixed now.
Posted by YFD plugin update « Hedayat’s Blog on October 8, 2011 at 2:53 pm
[...] October 8, 2011 by Hedayat in Fedora, Linux, Parsidora. Leave a Comment A while back I introduced Yum Fast Downloader plugin, a plugin to use a download accelerator application (Aria2) to download yum metadata and packages [...]
Posted by mpstylep on November 24, 2011 at 4:16 pm
Hi.
I tried to install tomahawk with the repo file in this link:
http://download.opensuse.org/repositories/home:/pansenmann:/tomahawk-player/Fedora_15/
but yum gave me this error:
http://download.opensuse.org/repositories/home%3A/pansenmann%3A/tomahawk-player/Fedora_15/repodata/7980af617770c7ad630d1aec99f40899b8fcba7ec5a950ac98bfb441f27728a0-primary.xml.gz: [Errno -1] Metadata file does not match checksum
Connessione ad un altro mirror in corso.
Errore: failure: repodata/7980af617770c7ad630d1aec99f40899b8fcba7ec5a950ac98bfb441f27728a0-primary.xml.gz from home_pansenmann_tomahawk-player: [Errno 256] No more mirrors to try.
I removed yum-fast-donwloader and all work very well.
Why?
Posted by Hedayat on November 25, 2011 at 2:31 am
Hi,
I’ll check tomorrow to see if I have the same problem. However, the file could have been corrupted for some reason. Have you tried removing the metadata for that repo and try again to see if the problem persists? Also, it would be helpful if you posted the full output. Anyway, if I could reproduce the problem I’ll try to fix it. Currently I have no idea why it must happen with this plugin.
Thanks a lot
Posted by Anonymous on December 19, 2011 at 6:59 pm
Yes! This is the yum plug-in I have looking for for a long time. Thank you very much.
My Internet connections are always very slow (about ~ 10kB/s). This plug-in increases download rate more than 15 times (up to ~160kB/s) specially with “yum update” to download many packages simultaneously.
Thanks again.
Posted by Hedayat on December 19, 2011 at 9:01 pm
Thanks
Posted by anonymous on December 20, 2011 at 5:01 pm
got problem with apper, the gui frontend in fc16 kde. it just stalls there showing nothing while in fact the package successfully installed. so for anyone using gnome, does this works with gnome gui frontend for yum?
Posted by Hedayat on December 20, 2011 at 6:23 pm
Yes, I’ve personally tried gnome front-end and works fine. But I’ve not tried the KDE front-end. It probably expects a specific kind of output from package kit?! It’d be nice if we can find the reason and see if it can be solved in the plugin.
Posted by saeed144 on January 19, 2012 at 7:10 pm
Thanks,
works for me too,
keeping it up to date and responsive to users needs would make it one of the best plug-ins of yum.
good luck Hedayat
Posted by Hedayat on January 19, 2012 at 10:53 pm
Thanks for your support.
BTW, looks like that something similar is being developed inside yum itself.
Posted by Peter on January 28, 2012 at 8:42 pm
I found another idea while surfing yum. Try grepping yum info!!! That is super. I will be back on tonight. I think that would be super. I am pulling the cloneurl.
Posted by Hedayat on January 28, 2012 at 8:50 pm
Sorry, would you please elaborate your comment? I didn’t get it.
Posted by Peter Harris on January 29, 2012 at 5:17 am
There is better dialog in ‘yum info packagename‘ that actually gives the source URL for the code, and a brief description of the package contents, if you are interested.
I figure if you needed a ready source for pertinent information for a downloader plugin, this was the icing on the cake, There are SO many things, library calls, system calls, kernel calls, (a module then) It is harder to say when one should stop, but there is none of that in linux. Typically, if something doesn’t grow, it dies, but for now, you might think what might improve yum somewhat, BUT DON’T write the thing yourself. Limit yourself to a plugin, but you can source anything you like, It’s not plagiarism to cite libraries, and macros. get some .bashrc macros while you’re at it, and take tons of notes.
I have been on all day retooling a build-dev enviornment, mostly to make Linux code manipulation a bit easier for me, (I can be slow sometimes, but so was Saturn V.)
I am almost done, I will keep you posted. (I think once I am in it it can’t be worse than 24 hours in Redmond’s flagship Vista kernel. If there is anything more I can do, drop me a line. RSVP kpharris32@gmail.com God Bless. Have a good night. (I have a lot of back pain, so I have to retire early tonight, but if I can at all help, I REALLY like your idea, I will keep you abreast. I’ve seen your repo on fedora. you might want to post a link to the code and force fedora to mainstream it. I like plugins better than a monsterous backend piece. But I remember how long updates took before yum-presto plugin. I thought I read it wrong, and when I saw that you were using aria2 as a file parser, I said. Gee. THAT took long enough, but I dug for the link. (Aria2 had been around from four years ago at least.) It just goes to show you that it is but time, chance, and favor that dictate the times, If you have all three, you do. I think lynx/ncurses iintegration would provide a MENU for downloadable links in a repoview, but that is a bit over my head, and I wonder if that is so applicable, but sometimes there no accounting for tastes..,
Posted by Ryan on May 11, 2012 at 9:01 am
Thank you! You might have increased my upgrade time from years to days, making it feasible to upgrade. The text interface might be better implemented in a GUI like ‘pirut’ (whatever they’re calling it these days), just keep that in mind if this ever gets integrated upstream.
Posted by Hedayat on May 11, 2012 at 9:07 pm
Yum is a CLI program, so I doubt if it can have a GUI (maybe it can have something like that in text mode). But PackageKit could have that!
Posted by Ryan on May 14, 2012 at 2:31 am
Well, what I mean is, if something like this is implemented in PackageKit.
Posted by Hedayat on May 15, 2012 at 11:37 am
Since upstream is also working on a multi-downloader facility, probably it is time to think about a GUI counterpart in PackageKit. But I doubt if there will be enough interest there. PackageKit apparently thinks that saying that it is in “Downloading” state is all the information a user should get!
Posted by Fedora 17, Offline Updates, Parsidora, Yum Fast Downloader and Nokia N9! « Hedayat’s Blog on July 9, 2012 at 11:02 am
[...] YFD Plugin: If you were using my yum-fast-downloader plugin version 0.5.9 or older on Fedora 17, you have faced an error message trying to use yum after the recent python-urlgrabber update. If you have not already done, you must update to version 0.5.10 to be able to continue using it on Fedora 17. The latest urlgrabber has support for external downloaders, it’ll help YFD to provide better support. Considering the latest updates to presto to use URLGrabber’s async download, presto support can also come to YFD more easily. [...]
Posted by Kenny on December 12, 2012 at 9:13 pm
Could you sign the package? The repo is only available over HTTP, so a GPG signature is the only guarantee of not changing in the future.
Posted by Hedayat on December 12, 2012 at 11:11 pm
OK, I’ll.
Thanks for the feedback.
Posted by Hedayat on January 24, 2013 at 2:03 am
The repository is now GPG enabled, and the packages are signed. Thanks again.
Posted by Mohsen Pahlevanzadeh on January 17, 2013 at 4:56 am
http://code.google.com/p/yum-axelget/
Posted by SumOne on February 28, 2013 at 12:22 am
Man, you just saved my ass on Fedora 18 which tried to download with FASTEST MIRROR at 56KB/s !!!!!
What a moron made that yum!!! (I have bandwidth of 64Mb/s and to this mirror I have FULL OPEN bandwidth – never goes below 64Mb/s!)
Funny thing is that I always thought its the problem of “unsupported ones” – I had FC14 not long time ago – but no – with the newest crappy FC18 it’s the same!
Your plugin accelerated my download to about 8MByte per second! Which is insane to be honest!
Posted by Hedayat on February 28, 2013 at 12:52 am
Thanks for your comment. Hopefully, new features are going to be added to YFD plugin soon…