johker's blog » rsync http://johker.ibutho.de stories about me, my life and my trips Sun, 18 Dec 2011 11:53:38 +0000 en hourly 1 http://wordpress.org/?v=3.3 Der neue Server: Teil 9 Backup http://johker.ibutho.de/2009/08/02/der-neue-server-teil-9-backup/ http://johker.ibutho.de/2009/08/02/der-neue-server-teil-9-backup/#comments Sun, 02 Aug 2009 15:05:25 +0000 johker http://johker.ibutho.de/?p=261 Continue reading →]]> Der Server ist zwar nicht mehr neu, aber dieser Beitrag ist evtl. dennoch für einige hilfreich. Deshalb: Viel Spaß!

Heute widme ich mich dem Thema Backup. Das habe ich viel zu stark vernachlässigt und will das jetzt nachholen. Das Tool der Wahl ist duplicity mit dem c’t Wrapper-Skript ftplicity. Duplicity benutzt unter der Decke rsync zur Erkennung der Deltas. Das Skript wird so konfiguriert, dass es täglich ein inkrementelles Backup macht und monatlich ein volles Backup und dabei die alten Backups löscht. Während des Backup-Prozesses werden auch Backups der MySQL- und postgreSQL-Datenbanken mit den mitgelieferten Tools erstellt (nicht einfach durch Kopieren der Datenbankdateien, da man dabei einen inkonsistenten Zustand erwischen könnte und die Datenbank unbrauchbar ist).

Installation

Zuerst die Installation der benötigten Pakete:

1
2
3
4
5
6

aptitude install duplicity ncftp
wget http://downloads.sourceforge.net/project/ftplicity/ftplicity/1.5.0.2/ftplicity_1.5.0.2.tgz?use_mirror=switch
tar xvfz ftplicity_1.5.0.2.tgz
mv ftplicity_1.5.0.2/ftplicity /usr/local/bin/
chown root: /usr/local/bin/ftplicity
chmod 755 /usr/local/bin/ftplicity

Konfiguration

Erstellung eines GPG-Schlüssel für Backups

Da ftplicity Backups automatisch verschlüsselt ablegt, ist es nötig einen GPG-Schlüssel zu erstellen:

1

gpg --gen-key


root@domain.tld:~# gpg --gen-key
gpg (GnuPG) 1.4.9; Copyright (C) 2008 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

gpg: keyring `/root/.gnupg/secring.gpg' created
Please select what kind of key you want:
(1) DSA and Elgamal (default)
(2) DSA (sign only)
(5) RSA (sign only)
Your selection? <-- ENTER
DSA keypair will have 1024 bits.
ELG-E keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) <-- ENTER
Requested keysize is 2048 bits
Please specify how long the key should be valid.
0 = key does not expire
= key expires in n days
w = key expires in n weeks
m = key expires in n months
y = key expires in n years
Key is valid for? (0) <-- ENTER
Key does not expire at all
Is this correct? (y/N) <-- y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
"Heinrich Heine (Der Dichter) "

Real name: Musterserver Backup
Email address: backup@domain.tld
Comment: Key for System Backups on domain.tld
You selected this USER-ID:
"domain.tld Backup (Key for System Backups on domain.tld) "

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? <-- O
You need a Passphrase to protect your secret key.

Enter passphrase: <-- Passwort eingeben
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++.+++++++++++++++.+++++++++++++++.++++++++++++++++++++.+++++..+++++++++++++++>++++++++++>.+++++...........<+++++>+++++..<.+++++.....................................................>.+++++.....+++++

gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key 123ABC45 marked as ultimately trusted
public and secret key created and signed.

...

Die achtstellige ID (im Beispiel 123ABC45) bitte merken, da diese später benötigt wird. Ebenso das Passwort.

Danach legen exportieren wir noch unsere Schlüssel (public/private), damit wir diese an einem sicheren Ort speichern können, damit unsere Daten nicht verloren gehen, sollte die Festplatte sich verabschieden:

1
2

root@domain.tld:~/gpg-key# gpg --output backup_pub.gpg --armor --export 123ABC45
root@domain.tld:~/gpg-key# gpg --output backup_sec.gpg --armor --export-secret-key 123ABC45

ftplicity

Nun starten wir ftplicity einmal, damit eine Standardkonfiguration angelegt wird, die wir an unsere Wünsche anpassen können:

1

ftplicity system create

Nun editieren wir die Datei “/root/.ftplicity/system/conf“:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86

# gpg key data (for symmetric encryption comment out GPG_KEY)
GPG_KEY='123ABC45'
GPG_PW='PASSWORD'
# gpg options passed from duplicity to gpg process (default='')
# e.g. "--trust-model pgp|classic|direct|always" 
#   or "--compress-algo=bzip2 --bzip2-compress-level=9"
#GPG_OPTS=''
 
# credentials & server address of the backup target (URL-Format)
# syntax is
#   scheme://user[:password]@host[:port]/[/]path
# probably one out of
#   file:///some_dir
#   ftp://user[:password]@other.host[:port]/some_dir
#   hsi://user[:password]@other.host/some_dir
#   cf+http://container_name
#   imap://user[:password]@host.com[/from_address_prefix]
#   imaps://user[:password]@host.com[/from_address_prefix]
#   rsync://user[:password]@other.host[:port]::/module/some_dir
#   rsync://user[:password]@other.host[:port]/relative_path
#   rsync://user[:password]@other.host[:port]//absolute_path
#   s3://host/bucket_name[/prefix]
#   s3+http://bucket_name[/prefix]
#   scp://user[:password]@other.host[:port]/some_dir
#   ssh://user[:password]@other.host[:port]/some_dir
#   tahoe://alias/directory
#   webdav://user[:password]@other.host/some_dir
#   webdavs://user[:password]@other.host/some_dir 
###
# TARGET='scheme://user[:password]@host[:port]/[/]path'
TARGET='ftp://<USER>:<PASS>@<HOST>/<DIR>'
# optionally the password can be defined as extra variable
# if password is set already in TARGET, this setting replaces it
#TARGET_PW='_backend_password_'
 
# base directory to backup
SOURCE='/'
 
# Time frame for old backups to keep, Used for the "purge" command.  
# see duplicity man page, chapter TIME_FORMATS)
# defaults to 1M, if not set
#MAX_AGE=1M
 
# Number of full backups to keep. Used for the "purge-full" command. 
# See duplicity man page, action "remove-all-but-n-full".
# defaults to 1, if not set 
#MAX_FULL_BACKUPS=1
 
# verbosity of output (5 for gpg errors, 9 for bug fixing)
# default is 4, if not set
#VERBOSITY=5
 
# temporary file space. at least the size of the biggest file in backup
# for a successful restoration process. (default is '/tmp', if not set)
#TEMP_DIR=/tmp
 
# sets duplicity --time-separator option (since v0.4.4.RC2) to allow users 
# to change the time separator from ':' to another character that will work 
# on their system.  HINT: For Windows SMB shares, use --time-separator='_'.
# NOTE: '-' is not valid as it conflicts with date separator.
# ATTENTION: only use this with duplicity < 0.5.10, since then default file 
#            naming is compatible and this option is pending depreciation 
#DUPL_PARAMS="$DUPL_PARAMS --time-separator _ "
 
# activates duplicity --short-filenames option, when uploading to a file
# system that can't have filenames longer than 30 characters (e.g. Mac OS 8)
# or have problems with ':' as part of the filename (e.g. Microsoft Windows)
# ATTENTION: only use this with duplicity < 0.5.10, since then default file 
#            naming is compatible and this option is pending depreciation  
#DUPL_PARAMS="$DUPL_PARAMS --short-filenames "
 
# activates duplicity --full-if-older-than option (since duplicity v0.4.4.RC3) 
# forces a full backup if last full backup reaches a specified age, for the 
# format of MAX_FULLBKP_AGE see duplicity man page, chapter TIME_FORMATS
#MAX_FULLBKP_AGE=1M
#DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE " 
 
# sets duplicity --volsize option (available since v0.4.3.RC7)
# set the size of backup chunks to VOLSIZE MB instead of the default 5MB.
# VOLSIZE must be number of MB's to set the volume size to. 
#VOLSIZE=50
#DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE "
 
# more duplicity command line options can be added in the following way
# don't forget to leave a separating space char at the end
#DUPL_PARAMS="$DUPL_PARAMS --put_your_options_here "

Und nun noch die Datei “/root/.ftplicity/system/exclude“:

1
2
3
4
5
6
7

/dev
/proc
/sys
/tmp
/var/cache
/var/tmp
/var/run

Danach erstellen wir unter “/usr/local/sbin/” die Datei db_backup mit Rechten “700″:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

#!/bin/bash
 
# Script to backup MySQL and postgreSQL databases
# author: johker
 
############### CHANGE THESE VARIABLE IF NECESSARY ####################
 
MYSQL_USER="MYSQL_USER"
MYSQL_PW="MYSQL_PASSWORD"
POSTGRES_USER="POSTGRES_USER"
POSTGRES_PW="POSTGRES_PASSWORD"
TMP="/tmp/"
FINAL_LOC="/var/backups/"
 
################ DON'T CHANGE ANYTHING BELOW THIS LINE ################
 
echo "Creating backup of MySQL databases"
mysqldump -u$MYSQL_USER -p$MYSQL_PW --all-databases > $FINAL_LOC"mysql.sql"
gzip -f $FINAL_LOC"mysql.sql"
 
echo "Creating backup of postgreSQL databases"
su -c "cd $TMP && pg_dumpall > $TMP'postgres.out'" $POSTGRES_USER
gzip -f $TMP'postgres.out' 
mv -f $TMP'postgres.out.gz' $FINAL_LOC
 
chmod 640 $FINAL_LOC"mysql.sql.gz" $FINAL_LOC"postgres.out.gz"
chown root: $FINAL_LOC"mysql.sql.gz" $FINAL_LOC"postgres.out.gz"
 
exit 0

Und in die Datei “/root/.ftplicity/system/pre” fügen wir folgende Zeile ein:

1
2
3

#!/bin/bash
 
/usr/local/sbin/db_backup

Testen

Mit folgendem Befehl kann man die ftplicity-Konfiguration testen:

ftplicity system status

Wenn alles o.k. ist, kann man mit folgedem Befehl ein initiales Backup anlegen:

ftplicity system backup

Erstellen eines Cronjobs

Da wir das Backup nicht jedes mal händisch anstoßen wollen, hier einträge für die crontab von root (editierbar per “crontab -e” als Benutzer root):

1
2
3
4

# run the (incremental) backup each night at 03:23h
23 3 * * * /usr/local/bin/ftplicity system backup
# do a full backup once per month & delete old backups at 04:47h
47 4 1 * * /usr/local/bin/ftplicity system purge --force && /usr/local/bin/ftplicity system purge-full --force && /usr/local/bin/ftplicity system full

Quellen

http://www.howtoforge.com/ftp-backups-with-duplicity-ftplicity-debian-etch

http://maff.ailoo.net/2009/07/backup-virtual-machines-lvm-snapshots-ftplicity-duplicity/

http://robert.penz.name/161/howto-backup-your-dedicated-server-to-a-foreign-ftp-server/

]]> http://johker.ibutho.de/2009/08/02/der-neue-server-teil-9-backup/feed/ 0