{"id":15,"date":"2005-03-14T18:05:06","date_gmt":"2005-03-14T18:05:06","guid":{"rendered":"http:\/\/www.reactivated.net\/weblog\/archives\/2005\/03\/using-getdelta-to-reduce-size-of-distfiles-download\/"},"modified":"2005-03-14T18:05:15","modified_gmt":"2005-03-14T18:05:15","slug":"using-getdelta-to-reduce-size-of-distfiles-download","status":"publish","type":"post","link":"http:\/\/www.reactivated.net\/weblog\/archives\/2005\/03\/using-getdelta-to-reduce-size-of-distfiles-download\/","title":{"rendered":"Using getdelta to reduce size of distfiles download"},"content":{"rendered":"

Yesterday, Stefan Schweizer (a developer who I work with) brought a very cool thing to my attention: getdelta<\/a>.<\/p>\n

getdelta is based on deltup<\/a> which is a way of storing and applying differences between files. Kind of like diff and patch, but designed for binary files such as compressed tarballs.<\/p>\n

deltup is very useful in the context of upgrading distfiles, because typically very little changes between foobar-0.1 and foobar-0.2, so a deltup diff file which could upgrade foobar-0.1.tar.bz2 to foobar-0.2.tar.bz2 would be a much<\/b> smaller download than downloading the entire foobar-0.2.tar.bz2 file.<\/p>\n

The magic of getdelta is that it integrates into portage for downloading your distfiles. As an example, I have subversion-1.1.1.tar.bz2 present in \/usr\/portage\/distfiles, but I now want to upgrade to version 1.1.3.<\/p>\n

\ndsd ~ # emerge -f subversion
\nCalculating dependencies …done!
\n>>> emerge (1 of 1) dev-util\/subversion-1.1.3-r1 to \/
\n>>> Downloading http:\/\/gentoo.blueyonder.co.uk\/distfiles\/subversion-1.1.3.tar.bz2
\nSearching for a previously downloaded file in \/usr\/portage\/distfiles<\/p>\n

We have following candidates to choose from
\nsubversion-1.1.1.tar.bz2<\/p>\n

The best of all is … subversion-1.1.1.tar.bz2<\/p>\n

Checking if this file is OK.<\/p>\n

Trying to download subversion-1.1.1.tar.bz2-subversion-1.1.3.tar.bz2.dtu<\/p>\n

[…snip the wget download verbosity…]<\/p>\n

17:41:57 (490.22 KB\/s) – `subversion-1.1.1.tar.bz2-subversion-1.1.3.tar.bz2.dtu’ saved [828684]<\/p>\n

GOT subversion-1.1.1.tar.bz2-subversion-1.1.3.tar.bz2.dtu<\/p>\n

Successfully fetched the dtu-file – let’s build subversion-1.1.3.tar.bz2…<\/p>\n

subversion-1.1.1.tar.bz2 -> subversion-1.1.3.tar.bz2: OK
\ncleaning up
\nThis dtu-file saved 5 MB (87%) download size.<\/p>\n

>>> subversion-1.1.3.tar.bz2 size ;-)
\n>>> subversion-1.1.3.tar.bz2 MD5 ;-)
\n>>> md5 src_uri ;-) subversion-1.1.3.tar.bz2\n<\/p><\/blockquote>\n

The above process basically found that I had got a previous subversion tarball already downloaded, so it just downloaded the upgrade deltup patch from the getdelta server, which saved 87% of my download for subversion-1.1.3. You’ll notice that portage does its usual md5 checking independant of the deltup process so I don’t see any possible problems relating to binaries being hijacked.<\/p>\n

I did a lot of package upgrades yesterday and for every one where I had an older distfile present and I was watching, getdelta typically saved me 70-95% of the downloading, and only once refused to download the deltup patch because it was bigger than the tarball it was going to construct.<\/p>\n

The only disadvantage of this is that constructing the new file can sometimes be time consuming, especially for the big files. Here’s an example where I construct linux-2.6.11.2.tar.bz2 (~35mb) from the 2.6.10 distribution:<\/p>\n

\nlinux-2.6.10.tar.bz2 -> linux-2.6.11.2.tar.bz2: OK
\ncleaning up<\/p>\n

real 3m42.395s
\nuser 2m2.787s
\nsys 0m4.462s\n<\/p><\/blockquote>\n

Is this any quicker than downloading the whole thing? Perhaps not while I’m staying at uni, where I can get 500kb\/sec from UK mirrors, but at home where I’m on standard\/unreliable broadband, it definately would be. If you are a dialup user, this is a godsend.<\/p>\n

It seems that there are plans to at least trial this as an official Gentoo service, but this is a very<\/em> nice preview of what might be coming up. To get started, just emerge getdelta<\/code> and follow the basic instruction to set a new FETCHCOMMAND in make.conf<\/p>\n","protected":false},"excerpt":{"rendered":"

Yesterday, Stefan Schweizer (a developer who I work with) brought a very cool thing to my attention: getdelta. getdelta is based on deltup which is a way of storing and applying differences between files. Kind of like diff and patch, but designed for binary files such as compressed tarballs. deltup is very useful in the […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-15","post","type-post","status-publish","format-standard","hentry","category-gentoo"],"_links":{"self":[{"href":"http:\/\/www.reactivated.net\/weblog\/wp-json\/wp\/v2\/posts\/15","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.reactivated.net\/weblog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.reactivated.net\/weblog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.reactivated.net\/weblog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.reactivated.net\/weblog\/wp-json\/wp\/v2\/comments?post=15"}],"version-history":[{"count":0,"href":"http:\/\/www.reactivated.net\/weblog\/wp-json\/wp\/v2\/posts\/15\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.reactivated.net\/weblog\/wp-json\/wp\/v2\/media?parent=15"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.reactivated.net\/weblog\/wp-json\/wp\/v2\/categories?post=15"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.reactivated.net\/weblog\/wp-json\/wp\/v2\/tags?post=15"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}