-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathTODO
148 lines (147 loc) · 7.9 KB
/
TODO
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
* ASAP uncapitalise all variables, because CAPS_VARS are for
environmental variables.
* When btrfs-borg is feature-complete, tag 1.0
* Goal: allow separation of configuration from script
- start with upcoming CSV support
- provide example script for populating the CSV?
* Add SSH support (requires borgbackup on server)
- Write HOWTO for server-initiated backup?
> limit available commands for root on $HOST to btrfs-borg
+ Would it be better to configure passwordless sudo
for the btrfs-borg command for $HOST's btrfs-borg user?
> create btrfs-borg user on $SERVER with mount permissions
> limit btrfs-borg commands to borg-server
> link to passwordless ssh and borg key-creation HOWTOs
= $SERVER sshes into $HOST, runs btrfs-borg,
and this runs backup and sshes into $SERVER
* More thoroughly test SSH support.
* Store the partition and subvolume layout, ARCHIVEIDS,
and also strong checksums of the generated files as:
`hostname`_partitions_subvolume-layout_ARCHIVEIDS
as a tiny borg-archive?
* Add local backup disk mount/umount support? (or provide example
wrapper script)
- Is udisk really the best way to safely logically disconnect USB disks?
- allow borg-btrfs to manage mount/umount via udisk by defining mount point
!! Leaving it blank means borg-btrfs will not manage the mount !!
!! MUST correspond to a fstab entry with adequate permissions !!
- udisk encrypted device support?
- get $DEVICE: df -h /home/media/usb/$REPO | tail -n1 | \
awk '{print $1}' \
|| mount HOW_DO_I_FIND_THE_DEVICE/DIR_TO_MOUNT_IF_IT'S_NOT_MOUNTED?
- sync; sleep 1; sync; udisksctl unmount -b $DEVICE
- sync; sleep 1; sync; udisksctl power-off -b /dev/sdd1
? Optional generation of comprehensive manifest. eg:
for i in `borg list $BORG_REPO | awk '{print $1}'`; do
borg info $BORG_REPO::$i >> $BORG_REPO/$REPO/btrfs-borg/borg_info_output
done
- Alternatively, submit a request for this functionality upstream
- Is this an information leak? Hmm....probably...
* Shift lots of configuration into new btrfs-borg.csv; adapt the script to read
from this file; allow the location of this config file to be defined
- Include example btrfs-borg.csv
- tail -n +2 ~/btrfs-borg.csv
- while IFS=, read X Y Z rest; do echo "X is $X" "Y is $Y" "Z is $Z";\
done < ~/btrfs-borg.csv
!! PROBLEM !! -> need to somehow strip row 1, eg:
ROOT,SNAPS,SUBS,CONTAINERS
- easiest thing to do is tail -n +2 $CSV-CONFIG > \
$REPO/btrfs-borg/config.csv.tmp
then < REPO/btrfs-borg/config.csv.tmp
!! Don't forget to remove config.csv.tmp in the cleanup !!
* Work for README
- Shift many long inline comments into README.md
- Break things into sections (eg: Introduction, Recommendations,
Configuration)
- Section on backing up LX Containers: a) unprivileged containers are
assumed to not be on their own subvolumes, therefore they are assumed to
be included in a snapshot of a subvolume such as @home. b)
* Clean up LXC snapshots by default
# TODO: We now clean up snapshots by default...
# Make this configurable in the future
# TODO ASAP!!: remove LXC snapshot generated by $0
# something like:
#
#for i in $CONTAINERS; do
# lxc-snapshot -L -n $i
# EXPIRE="`lxc-snapshot -L -n $i | tail -1 | awk '{print $1}'`"
# echo "Removing $EXPIRE"
# lxc-snapshot -n get-transmissions -d $EXPIRE
#done
* Create knob for btrfs snapshot of LXC containers, without using
lxc-snapshot, which shuts them down. This is desirable for all but
database workloads:
https://www.flockport.com/supercharge-lxc-with-btrfs/
- Alternatively, treat LXC containers as regular subvols by default
* This means using btrfs sub list to automatically find them
* If an absolute path is given, then check to see if it's an LXC
and if it is a container, then run lxc-stop and lxc-snapshot
* Add optional support for backing up multiple hosts to the same repo
...which means more efficient deduplication!
- Gabor's suggestion :-)
- Needs to automatically uniquely name backups
Is `hostname` sufficient?
- Needs to pass the unique identifier to borg prune, using the
prefix parameter, so that the backups of each host are pruned
(backups for host) instead of (backups for host1 and host2 and
etc.)
* Make location of ARCHIVEIDS configurable
- alternatively, use the "--prefix" feature to store ARCHIVEIDS
within the borg REPO and prevent them from being pruned... This prevents
information leaks, but I'm not sure if it's sufficiently secure...
* Related to the comma separated list and --prefix:
- maybe I should do it as follows...
1) automatically prefix hostname 2) comma separated list for
"backup unit" which becomes a particular "borg archive" 3)
The backup unit must have the following elements:
- ROOT volume
- Directory: check to see if it's a btrfs subvol, otherwise make
it an "OTHERS"
- This SIMPLIFIES things for the user, because he/she no longer
has to worry about SRC data types! ;-)
4) Pruning needs to be adapted to operate on these new backup
5) Iterate through all of the backup units
6) Accept a backup unit as the sole command line argument?
- This idea enables multiple ageing tiers for backups, and multiple
btrfs volumes. This functionality might come later...but it
will immediately allow the sysadmin to recover hard drive space
by "early retirement" of a backup unit that does not require as fine
grained history as another backup unit. eg: @work-in-progress
needs to always have an up-to-date backup, and X days of backups,
but probably does not need months or years...it just takes up
extra space, and @work-portfolio is what is important in the long-term.
> eg: BACKUP-UNIT="/btrfs-admin/Red,@home var/lib/lxc/my_container"
- test for data type with "btrfs sub show $ROOT/$SUBVOL" and check
value of $? for each unit of the second element. If true
append to $SUBS, otherwise append to $OTHERS
- Backup units are important...OS data from multiple hosts might
not matter that much, so multiple hosts can share the same
repository. It is also advantageous to centralise backups of
containers from multiple container servers. On the other hand,
I think that it is probably just going to slow things down if
deduplication of user data from host A is attempted against user
data from host B.
> Or should "backup units" include target disk?
eg: BACKUP-UNIT="/Red,@home var/lib/lxc/my_container,[user@host:]/dest/mountpoint,[user@host:]/borg/repo"
* Make /borg/repo variable relative to /dest/mountpoint! (see section
on udisk mounting) By default, if BACKUP-UNIT is passed as a
command line argument, do not unmount without explicit
authorisation...needs another command line argument.
The config-file.csv method makes one borg archive per line, and
unmounts after the backup of the last line completes.
* A cleanup is due!
* Centralise documentation to an .asciidoc and use this to generate a manpage.
* Support backing up LX containers with wildcards?
eg: find the rootfs of all /var/lib/lxc/* /home/webdev0/* /home/webdev1/*
get basename for each container
snapshot them under /$ROOT/snapshots/LXC-$BN-rootfs
come up with a special archive naming scheme?
- /var/lib/lxc/* -> "$ARCHIVE_NAME lib_LXCs"
- /home/webdev0/* -> "$ARCHIVE_NAME webdev0_LXCs"
* Alternatively, provide a script to populate a .CSV with all
containers found under a path.
* Allow BACKUP-SRCS, or BACKUP-UNITS to point to a .csv file.
The idea is to make basic configuration as easy as possible.
Pre-populated value should be two lines: one for @ and one for
@home. Write an example config for OpenSuSE users?
* BORG-REPO needs to be part of BACKUP-UNIT