## page was renamed from DebmirrorHowto ## page was renamed from https:/help.ubuntu.com/community/DebmirrorHowto ## page was renamed from KarlGoetz/DebmirrorHowto ## LP Bug: #1266031 ## Content needs updating to at least 12.04 Apach2 levels. ## It's a good article, just needs refreshed to reflect current LTS and possibly 14.04 ## I did the update to 14.04 pablodav. ||<>|| ''For information on Ubuntu mirroring, see [[https://wiki.ubuntu.com/Mirrors|Mirrors]]'' == To build a mirror with debmirror follow these steps == This is not covering all use cases but it will allow you experiment. To do this safely 100 gigs of space are needed to mirror main source and binaries. This number grows a lot as time progresses. === Alternative Consideration === 1. If you want to don't download all the apt binaries for all boxes onto one box on a network then maybe <
>1.1. Apt-cache, [[https://help.ubuntu.com/community/Apt-Cacher-Server|Apt-Cacher-Server]] <
>1.2. Apt-Cacher-Ng [[https://wiki.debian.org/AptCacherNg|apt-cacher-ng]] <
>1.3. Transparent proxy [[http://help.ubuntu.com/community/HttpReplicator|http-replicator]] 2. rsync. The [[http://help.ubuntu.com/community/Rsyncmirror|Rsyncmirror]] page goes into greater detail on this method. Rsync can create a full mirror of a site and selects files indiscriminately. Debmirror, however, does allow for better control over which packages are mirrored and can be easier on the amount of hard drive space required. === Start The Mirror Build Process === Open gnome-terminal by clicking Applications -> Accessories -> Terminal. <
>If your sources list has universe enabled, skip the next step. <
>Open your sources list ({{{sudo sensible-editor /etc/apt/sources.list}}}), and add the following line: {{{ deb http://archive.ubuntu.com/ubuntu/ focal universe }}} <
>Then save and return to a prompt. '''Update the package lists and then install debmirror:''' {{{ sudo apt-get update sudo apt-get install debmirror gnupg xz-utils }}} Gnupg is need for keyring. Xz is need for decompression. <
> Now we have to create a directory to save the files to. The approximate size of the components of the mirror are: # Changes over time. || feisty || ~27G || main, restricted, universe, multiverse (i386 only) || || || dapper || 15G || main, restricted, universe, multiverse (i386 only) || || || trusty || 60G || main, restricted, universe, multiverse (i386 only) || || || Bionic || 192G || main, restricted, universe, multiverse (amd64 ) || bionic bionic-security bionica-updates|| || focal || 142G || main, restricted, universe, multiverse (amd64 ) || focal focal-security focal-updates|| Please ensure where ever you make the mirror has room for the packages you want to download! <
>(If you are more advanced you can make different parts of the mirror on different volumes, but that is not something for this howto). <
>The tree structur we will end up with: {{{ /opt/UbuntuMirror/ ├── mirrorkeyring │ ├── trustedkeys.kbx ├── ubuntu-mirror │ ├── dists │ ├── pool │ └── project ├── mirrorbuilder.sh }}} <
>We will make our mirror on /opt/ with {{{ sudo mkdir /opt/UbuntuMirror }}} == Create the file `mirrorbuild.sh` == Now to create `mirrorbuild.sh` <
>Open your prefered editor thus {{{ sudo sensible-editor /opt/UbuntuMirror/mirrorbuild.sh }}} Copy the text from the box below into the editor. <
>You may omit the lines starting with # (EXCEPTION: the line that starts #!/bin/bash must be left in.) === Contents of the mirror script, please adapt it to your needs === {{{ #### Start script to automate building of Ubuntu mirror ##### ## THE NEXT LINE IS NEEDED THE REST OF THE LINES STARTING WITH A # CAN BE DELETED #!/bin/bash ## Setting variables with explanations. # # Don't touch the user's keyring, have our own instead # export GNUPGHOME=/opt/UbuntuMirror/mirrorkeyring # Arch= -a # Architecture. For Ubuntu can be i386, powerpc or amd64. # sparc, only starts in dapper, it is only the later models of sparc. # For multiple architecture, use ",". like "i386,amd64" arch=amd64 # Minimum Ubuntu system requires main, restricted # Section= -s # Section (One of the following - main/restricted/universe/multiverse). # You can add extra file with $Section/debian-installer. ex: main/debian-installer,universe/debian-installer,multiverse/debian-installer,restricted/debian-installer # section=main,restricted,universe,multiverse # Release= -d # Release of the system (, focal ), and the -updates and -security ( -backports can be added if desired) # List of updated releases in: https://wiki.ubuntu.com/Releases # List of sort codenames used: http://archive.ubuntu.com/ubuntu/dists/ release=focal,focal-security,focal-updates # Server= -h # Server name, minus the protocol and the path at the end # CHANGE "*" to equal the mirror you want to create your mirror from. au. in Australia ca. in Canada. # This can be found in your own /etc/apt/sources.list file, assuming you have Ubuntu installed. # server=archive.ubuntu.com # Dir= -r # Path from the main server, so http://my.web.server/$dir, Server dependant # inPath=/ubuntu # Proto= --method= # Protocol to use for transfer (http, ftp, hftp, rsync) # Choose one - http is most usual the service, and the service must be available on the server you point at. # For some "rsync" may be faster. proto=http # Outpath= # Directory to store the mirror in # Make this a full path to where you want to mirror the material. # outPath=/opt/UbuntuMirror/ubuntu-mirror # The --nosource option only downloads debs and not deb-src's # The --progress option shows files as they are downloaded # --source \ in the place of --no-source \ if you want sources also. # --nocleanup Do not clean up the local mirror after mirroring is complete. Use this option to keep older repository # Start script # debmirror -a $arch \ --no-source \ -s $section \ -h $server \ -d $release \ -r $inPath \ --progress \ --method=$proto \ $outPath #### End script to automate building of Ubuntu mirror #### }}} === How to use the file === Save the file and exit. Give it execute permissions {{{ sudo chmod +x /opt/UbuntuMirror/mirrorbuild.sh }}} === GPG keyring file === Depending on what version of gpg that is used, different filenames are used for the keyring. "pubring.gpg", "trustedkeys.gpg" {{{ sudo mkdir /opt/UbuntuMirror/mirrorkeyring gpg --no-default-keyring --keyring /opt/UbuntuMirror/mirrorkeyring/trustedkeys.gpg --import /usr/share/keyrings/ubuntu-archive-keyring.gpg }}} '''For a mirror host running older releases (Unknown change date):''' {{{ sudo mkdir /opt/UbuntuMirror/mirrorkeyring gpg --no-default-keyring --keyring /opt/UbuntuMirror/mirrorkeyring/pubring.gpg --import /usr/share/keyrings/ubuntu-archive-keyring.gpg }}} '''Now change the permissions and ownership on the mirror.''' The group name is your username: {{{ sudo chown -R root:username /opt/UbuntuMirror/mirrorkeyring sudo chmod -R 571 /opt/UbuntuMirror/mirrorkeyring }}} == Run the program == And the final part of setting up the mirror is to execute the files: {{{ /opt/UbuntuMirror/mirrorbuild.sh }}} Now walk away. Your machine has a lot of downloading to do! Seriously... it'll take hours to download > 140 Gigabyte of data. == Set up the remote access to the mirror via http == We need to install Apache2. You could choose Apache version 1.3 but that is beyond the scope of this document. <
> We can do this with `sudo apt-get install apache2` <
> <
> Then (for simplicity sake) lets make a link from /home/UbuntuMirror to /var/www/ubuntu. <
> Replace"${mirrorbox-hostname}" with your DNS or IP-address. {{{ ln -s /opt/UbuntuMirror/ubuntu-mirror /var/www/html/ubuntu }}} This means when you go to download from your mirror, you will visit `http://${mirrorbox-hostname}/ubuntu/`. to test from commandline: {{{ wget http://localhost/ubuntu/dists/focal-security/Release.gpg }}} <
> <
> Right! So we have all the packages, and a working server. Now we need to set up the clients. == Add a cronjob to keep the mirror updated == If you want to keep your mirror fresh, you will want to stick your mirror script in cron. To edit root's crontab, do: {{{ sudo crontab -e }}} And now add the following line: {{{ 2 */12 * * * /opt/UbuntuMirror/mirrorbuild.sh }}} Which will run two minutes after the hour every twelve hours. MAKE SURE that there is a blank line after the line you added, otherwise it will never run. If you want to be paranoid, restart cron: {{{ sudo /etc/init.d/cron restart }}} == Set up a client system == Open a terminal and enter {{{ cd /etc/apt sudo mv sources.list sources.list.orig sudo sensible-editor sources.list }}} Replace"${mirrorbox-hostname}" with your DNS or IP-address. <
> Now in your editor, put the following lines: {{{ # Local network mirror sources. deb http://${mirrorbox-hostname}/ubuntu focal main restricted universe multiverse deb http://${mirrorbox-hostname}/ubuntu focal-security main restricted universe multiverse deb http://${mirrorbox-hostname}/ubuntu focal-updates main restricted universe multiverse }}} Then save and exit <
> If you then run {{{ sudo apt-get update sudo apt-get dist-upgrade }}} you should be updating from your new server! == advanced == === Local translations === By default language spesific packages will be download, examples are: . "*language-pack-*" . "*locale-*" . "*l10n-*" . "*myspell-*_" . "manpages-*" and if [[http://manpages.ubuntu.com/manpages/trusty/man1/debmirror.1.html|--i18n]] is use Additionally download Translation-.bz2. To filter out this package you need to use ''''--exclude='''' and ''''--include=''''. Example with norwegian: {{{ --exclude='/Translation-.*' --include='/Translation-(no|en).*' --exclude='/language-pack-.*' --include='/language-pack-?(gnome|kde)-(no|en).*' --exclude='(thunderbird|firefox)-locale-.*' --include='(thunderbird|firefox)-locale-(nb|nn).*' --exclude='/libreoffice-l10n.*' --include='/libreoffice-l10n-(nb|nn).*' --exclude='/manpages-(\w{2}|\w{2}-extra|\w{2}-dev|\w{2}-extra-dev)_.*' --exclude='/myspell-(\w{2}|\w{2}-\w{2,3}|)[-_].*' --include='/myspell-(nb|nn).*' }}} === Exclude deb section === [[https://www.debian.org/doc/debian-policy/ch-archive.html#s-subsections|deb section]] For a industral setting not all is needed: {{{ --exclude-deb-section=games \ --exclude-deb-section=news \ --exclude-deb-section=science \ --exclude-deb-section=xfce \ --exclude-deb-section=mail \ --exclude-deb-section=hamradio \ --exclude-deb-section=haskell \ --exclude-deb-section=education \ }}} == Debmirror using config file == Debmirror has support for using config files, making it easyer to automate with tools like Ansible: The script: {{{ #!/bin/bash export GNUPGHOME=/etc/debmirror #### Start script ####### debmirror --config-file=/etc/debmirror/ubuntu.conf #### End script to automate building of Ubuntu mirror #### }}} Example of config: {{{ $mirrordir="/volume1/debmirror/Ubuntu"; $progress=1; $host="no.archive.ubuntu.com"; $remoteroot="/ubuntu"; $download_method="http"; @dists=( 'focal', 'focal-security', 'focal-updates',); @sections=( 'main', 'restricted', 'universe', 'multiverse',); @arches=( 'amd64',); @excludes=( '/Translation-.*', '/Translation-en.gz', '/language-pack-.*', '(thunderbird|firefox)-locale-.*', '/libreoffice-l10n.*', '/manpages-(\w{2}|\w{2}-extra|\w{2}-dev|\w{2}-extra-dev)_.*', '/myspell-(\w{2}|\w{2}-\w{2,3}|)[-_].*', '/unidic-mecab.*', '/libreoffice-dev-doc.*', '/thunderbird-dbg.*', '/firefox-dbg.*', '/texlive-fonts-extra.*',); @includes=( '/myspell-(nb|nn).*', '/libreoffice-l10n-(nb|nn).*', '(thunderbird|firefox)-locale-(nb|nn).*', '/language-pack-?(gnome|kde)-(no|en).*', '/Translation-(no|en\.|en_GB|en_US).*',); $i18n=1; $do_source=0; $state_cache_days=1; $timeout=1; $rsync_batch=200; $rsync_options="-aIL --partial"; $diff_mode="use"; # The config file must return true or perl complains. # Always copy this. 1; }}} ---- CategoryNetworking CategoryPackageManagement