-
Notifications
You must be signed in to change notification settings - Fork 0
/
Changelog.GNU
6629 lines (3901 loc) · 193 KB
/
Changelog.GNU
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
2014-10-23 Kenneth Loafman <[email protected]>
* Prep for 0.7.0
* Merged in lp:~ed.so/duplicity/manpage.blocksize
- add --max_blocksize doc
- reorder 'a note on filename prefixes' into alphabetical order
2014-10-18 Kenneth Loafman <[email protected]>
* Merged in lp:~johnleach/duplicity/1315437-swift-container-create
- Check to see if the swift container exists before trying to create it,
in case we don't have permissions to create containers. Fixes #1315437
* Merged in lp:~moritzm/duplicity/duplicity
- Use lftp for both FTP and FTPS
* Adjust unit tests to expect single FTP backend
2014-10-16 Kenneth Loafman <[email protected]>
* Merged in lp:~ed.so/duplicity/0.7-dpbx.importfix
- fix this showstopper with the dropbox backend
"NameError: global name 'rest' is not defined"
2014-10-12 Kenneth Loafman <[email protected]>
* Merged in lp:~hooloovoo/duplicity/updated-README-REPO
- Changes to README-REPO to reflect the restructuring of the directories.
* Merged in lp:~jflaker/duplicity/BugFix1325215
- The reference to "--progress_rate" in the man page as a parameter is
incorrect. Should be "--progress-rate".
2014-09-30 Kenneth Loafman <[email protected]>
* Fixed bug 1375304 with patch supplied by Aleksandar Ivanovic
2014-08-17 Kenneth Loafman <[email protected]>
* Added sxbacked.py, Skylable backend. Waiting on man page updates.
* Merged in lp:~ed.so/duplicity/manpage.verify
- Clarify verify's functionality as wished for by a user surprised with a big
bandwidth bill from rackspace.
* Merged in lp:~jeffreydavidrogers/duplicity/duplicity
- This change fixes two small typos in the duplicity man page.
2014-06-28 Kenneth Loafman <[email protected]>
* Merged in lp:~3v1n0/duplicity/copy.com-backend
- I've added a backend for Copy.com cloud storage, this supports all the
required operations and works as it should from my tests.
- You can use it by calling duplicity with something like:
copy://[email protected]:[email protected]/duplicity
- The only thing I've concerns with is the optimized support for _delete_list
which can't be enabled here because the test_delete_list tries also to
delete a not-existing files, and it requires the backend not to raise an
exception in that case (is this somewhat wanted or could we do the same as
for _delete or _query?)
* Merged in lp:~ed.so/duplicity/webdav200fix-0.7
- webdav backend fix "BackendException: Bad status code 200 reason OK. " when
restarting an interrupted backup and overwriting partially uploaded volumes.
* Merged in lp:~mterry/duplicity/webdav-fixes
- This branch fixes two issues I saw when testing the webdav backend:
- 1) Errors like the following: "Attempt 1 failed. BackendException: File
/tmp/duplicity-LQ1a0i-tempdir/mktemp-u2aiyX-2 not found locally after get
from backend". These were caused by the _get() method not calling setdata()
on the local path object, so the rest of the code thought it didn't exist.
- 2) Some odd issues from stale responses/data. We have a couple places in
webdavbackend.py where we close the connection before making a request
because of this problem. But I've changed it to do it every time, more
reliably, by putting a _close() call inside the request() method.
- With this, the webdav backend seems fine to me.
* Merged in lp:~antmak/duplicity/0.7-par2-fix
- Useful fix for verbatim par2cmdline options (like "-t" in par2-tbb version)
* Fixed bug 1327550: OverflowError: signed integer is greater than maximum
- Major and minor device numbers are supposed to be one byte each. Someone
has crafted a special system image using OpenVZ where the major and minor
device numbers are much larger (ploop devices). We treat them as (0,0).
2014-05-11 Kenneth Loafman <[email protected]>
* Merged in lp:~mterry/duplicity/py2.6.0
- Support python 2.6.0.
- Without this branch, we only support python >= 2.6.5 because that's when
python's urlparse.py module became its more modern incarnation. (I won't
get into the wisdom of them making such a change in the middle of the
2.6 lifecycle.)
- Also, the version of lockfile that I have (0.8) doesn't work with python
2.6.0 or 2.6.1 due to their implementation of
threading.current_thread().ident returning None unexpectedly. So this
branch tells lockfile not to worry about adding the current thread's
identifier to the lock filename (we don't need a separate lock per thread,
since our locking is per process).
- I've tested with 2.6.0 and 2.7.6 (both extremes of our current support).
* Update shebang line to python2 instead of python to avoid confusion.
2014-05-07 Kenneth Loafman <[email protected]>
* Applied expat fix from edso. See answer #12 in
https://answers.launchpad.net/duplicity/+question/248020
2014-04-30 Kenneth Loafman <[email protected]>
* Merged in lp:~mterry/duplicity/encode-exceptions
- Because exceptions often contain file paths, they have the same problem
with Python 2.x's implicit decoding using the 'ascii' encoding that we've
experienced before. So I added a new util.uexc() method that uses the
util.ufn() method to convert an exception to a unicode string and used it
around the place.
- Bugs fixed: 1289288, 1311176, 1313966
2014-04-29 Kenneth Loafman <[email protected]>
* Merged in lp:~mterry/duplicity/backend-unification
- Reorganize and simplify backend code. Specifically:
- Formalize the expected API between backends and duplicity. See the new
file duplicity/backends/README for the instructions I've given authors.
- Add some tests for our backend wrapper class as well as some tests for
individual backends. For several backends that have some commands do all
the heavy lifting (hsi, tahoe, ftp), I've added fake little mock commands
so that we can test them locally. This doesn't truly test our integration
with those commands, but at least lets us test the backend glue code.
- Removed a lot of duplicate and unused code which backends were using (or
not using). This branch drops 700 lines of code (~20%)
in duplicity/backends!
- Simplified expectations of backends. Our wrapper code now does all the
retrying, and all the exception handling. Backends can 'fire and forget'
trusting our wrappers to give the user a reasonable error message.
Obviously, backends can also add more details and make nicer error
messages. But they don't *have* to.
- Separate out the backend classes from our wrapper class. Now there is no
possibility of namespace collision. All our API methods use one
underscore. Anything else (zero or two underscores) are for the backend
class's use.
- Added the concept of a 'backend prefix' which is used by par2 and gio
backends to provide generic support for "schema+" in urls -- like par2+
or gio+. I've since marked the '--gio' flag as deprecated, in favor of
'gio+'. Now you can even nest such backends like
par2+gio+file://blah/blah.
- The switch to control which cloudfiles backend had a typo. I fixed this,
but I'm not sure I should have? If we haven't had complaints, maybe we
can just drop the old backend.
- I manually tested all the backends we have (except hsi and tahoe -- but
those are simple wrappers around commands and I did test those via mocks
per above). I also added a bunch more manual backend tests to
./testing/manual/backendtest.py, which can now be run like the above to
test all the files you have configured in config.py or you can pass it a
URL which it will use for testing (useful for backend authors).
2014-04-26 Kenneth Loafman <[email protected]>
* Merged in lp:~mterry/duplicity/py3-map-filter
- In py3, map and filter return iterable objects, not lists. So in each case
we use them, I've either imported the future version or switched to a list
comprehension if we really wanted a list.
2014-04-25 Kenneth Loafman <[email protected]>
* Fixed bug #1312328 WebDAV backend can't understand 200 OK response to DELETE
- Allow both 200 and 204 as valid response to delete
2014-04-20 Kenneth Loafman <[email protected]>
* Merged in lp:~mterry/duplicity/more-test-reorg
- Here's another test reorganization / modernization branch. It does the
following things:
- Drop duplicity/misc.py. It is confusing to have both misc.py and util.py,
and most of the code in misc.py was no longer used. I moved the one
function that was still used into util.py.
- Consolidated the various ways to run tests into just one. I made tox runs
go through ./setup.py test, rather than nosetests. And I made the
./testing/run-tests scripts just call tox. Now we no longer need nosetests
as a test dependency (although you can still use it if you want).
- Added two more code quality automated tests: a pep8 one and a pylint one.
I disabled almost all checks in each program that gave a warning. These
tests just establish a baseline for future improvement.
- Moved the test helper code into TestCase subclasses that all tests can
use. And used more code sharing and setUp/tearDown cleverness to remove
duplicated code.
- Reorganized the tests in ./testing/tests into ./testing/functional and
./testing/unit -- for whether they drive duplicity as a subprocess or
whether they import and test code directly. Each dir can have specialized
TestCase subclasses now.
- Renamed the files in ./testing/unit to more clearly indicate which file
in ./duplicity they are unit testing.
- Added some helper methods for tests to set environment and globals.*
parameters more safely (i.e. without affecting other tests) by
automatically cleaning up any such changes during test tearDown.
- Removed test_unicode.py, since it is kind of dumb. It used to be more
useful, but now with py2.6, we are just testing that one line of code
in it is actually there.
2014-04-19 Kenneth Loafman <[email protected]>
* Merged in lp:~mterry/duplicity/2.6isms
- Here's a whole stack of minor syntax modernizations that will become
necessary in python3. They all work in python2.6.
- I've added a new test to keep us honest and prevent backsliding on these
modernizations. It runs 2to3 and will fail the test if 2to3 finds anything
that needs fixing (with a specific set of exceptions carved out).
- This branch has most of the easy 2to3 fixes, the ones with obvious and
safe syntax changes.
- We could just let 2to3 do them for us, but ideally we use 2to3 as little
as possible, since it doesn't always know how to solve a given problem.
I will propose a branch later that actually does use 2to3 to generate
python3 versions of duplicity if they are requested. But this is a first
step to clean up the code base.
* Merged in lp:~mterry/duplicity/drop-static
- Drop static.py.
- This is some of the oldest code in duplicity! A bzr blame says it is
unmodified (except for whitespace / comment changes) since revision 1.
- But it's not needed anymore. Not really even since we updated to python2.4,
which introduced the @staticmethod decorator. So this branch drops it and
its test file.
* Merged in lp:~mterry/duplicity/encode-for-print
- Encode translated strings before passing them to 'print'.
- The print command can only apparently handle bytes. So when we pass it
unicode, it freaks out. There were only four instances I saw where we used
print, so I figured it was easiest to just convert them to use the log
framework too.
- That way all user-visible strings go through that framework and are subject
to the same encoding rules.
2014-04-17 Kenneth Loafman <[email protected]>
* Merged in lp:~fredrik-loch/duplicity/duplicity-S3-SSE
- Adds support for server side encryption as requested in Bug #996660
* Merged in lp:~mterry/duplicity/consolidate-tests
- Consolidate all the duplicity-running code in the test framework.
- This takes the four test files [1] that run duplicity itself (i.e. the
high-level functional test files) and consolidates their code. Before, it
was madness. Each one had its own run_duplicity method, with its own
slightly tweaked features. This should reduce a lot of duplication.
- [1] cleanuptest.py, finaltest.py, restarttest.py, and badupload.py
* Merged in lp:~mterry/duplicity/modern-testing
- Enable/use more modern testing tools like nosetests and tox as well as more
common setup.py hooks like test and sdist.
- Specifically:
* move setup.py to toplevel where most tools and users expect it
* Move and adjust test files to work when running "nosetests" in toplevel
directory. Specifically, do a lot more of the test setup in
tests/__init__.py rather than the run-tests scripts
* Add small tox.ini file for using tox, which is a wrapper for using
virtualenv. Only enable for py26 and py27 right now, since modern
setuptools dropped support for <2.6 (and tox 1.7 recently dropped <2.6)
* Add setup.py hooks for test and sdist which are both standard targets
(sdist just outsources to dist/makedist right now)
* Merged in lp:~mterry/duplicity/require-2.6
- Require at least Python 2.6.
- Our code base already requires 2.6, because 2.6-isms have crept in. Usually
because we or a contributor didn't think to test with 2.4. And frankly,
I'm not even sure how to test with 2.4 on a modern system.
* Merged in lp:~mterry/duplicity/drop-u1
- Ubuntu One is closing shop. So no need to support a u1 backend anymore.
* Merged in lp:~mterry/duplicity/fix-drop-u1
- Looks like when the drop-u1 branch got merged, its conflict got resolved
badly. Here is the right version of backend.py to use (and also drops
u1backend.py from POTFILES).
* Merged in lp:~mterry/duplicity/drop-pexpect
- Drop our local copy of pexpect in favor of a system version.
- It's only used by the pexpect ssh backend (and if you're opting into that,
you probably can expect that you will need pexpect) and the tests.
- I've done a quick smoketest (backed up and restored using
--ssh-backend=pexpect) and it seemed to work fine with a modern version
of pexpect.
2014-03-10 Kenneth Loafman <[email protected]>
* Merged in lp:~germer/duplicity/par2
- This branch adds Par2 recovery files to duplicity. It is a wrapper backend
which will create the recovery files and upload them all together with the
wrapped backend. Corrupt archives will be detected and repaired (if
possible) on the fly during restore.
- It can be used with url-string par2+webdavs://USER@HOST/PATH
- Fixes https://bugs.launchpad.net/duplicity/+bug/426282
2014-03-06 Kenneth Loafman <[email protected]>
* Merged in lp:~ed.so/duplicity/fix.dpbx
- Fix dpbx backend "NameError: global name 'rest' is not defined"
* Merged in lp:~prateek/duplicity/botoimportfix
- Switches the boto backend back to using lazy imports so there are no
complaints during the importing of backends.
2014-02-26 Kenneth Loafman <[email protected]>
* Merged in lp:~prateek/duplicity/s3-glacier
- Fixes https://bugs.launchpad.net/duplicity/+bug/1039511
- Adds support to detect when a file is on Glacier and initiates a restore
to S3. Also merges overlapping code in the boto backends
- Fixes https://bugs.launchpad.net/duplicity/+bug/1243246
- Adds a --s3_multipart_max_timeout input option to limit the max execution
time of a chunked upload to S3. Also adds debug message to calculate
upload speed.
2014-02-24 Kenneth Loafman <[email protected]>
* Merged in lp:~mterry/duplicity/pexpect-fix
- duplicity has its own copy of pexpect. Use that instead of requiring one
from the system.
2014-02-05 Kenneth Loafman <[email protected]>
* Merged in lp:~mterry/duplicity/gpg-encode
- getpass.getpass(prompt) eventually calls str(prompt). Which is a no go,
if the prompt contains unicode. Here's a patch to always pass getpass() a
byte string.
- Our tests didn't catch this because they always set PASSPHRASE. I've added
a test that passes the passphrase via stdin.
2014-01-31 Kenneth Loafman <[email protected]>
* Applied two patches from mailing list message at:
https://lists.nongnu.org/archive/html/duplicity-talk/2014-01/msg00030.html
"Added command line options to use different prefixes for manifest/sig/archive files"
This resolves https://bugs.launchpad.net/duplicity/+bug/1170161 and provides
a workaround for https://bugs.launchpad.net/duplicity/+bug/1170113
2014-01-24 Kenneth Loafman <[email protected]>
* Prep for 0.6.23 release.
2014-01-17 Kenneth Loafman <[email protected]>
* Merged in lp:~louis-bouchard/duplicity/add-allow-concurrency
- Implement locking mechanism to avoid concurrent execution under the same
cache directory. This is the default behavior.
- Also implement --alllow-concurrency option to disable the locking
if required.
- This functionality adds a dependency to python-lockfile
2014-01-13 Kenneth Loafman <[email protected]>
* Restored patch of gdocsbackend.py from original author (thanks ede)
* Applied patch from bug 1266753: Boto backend removes local cache if
connection cannot be made
2014-01-02 Kenneth Loafman <[email protected]>
* Restored missing line from patch of gdocsbackend.py
* Reverted changes to gdocsbackend.py
2013-12-30 Kenneth Loafman <[email protected]>
* Merged in lp:~mterry/duplicity/encoding
- This branch hopefully fixes two filename encoding issues:
- Users in bug 989496 were noticing a UnicodeEncodeError exception which
happens (as far as I can tell) because some backends (like webdav) are
returning unicode filenames from list(). When these filenames are combined
with the utf8 translations of log messages, either (A) the default ascii
encoding can't handle promoting the utf8 bytes or -- if there aren't any
utf8 bytes in the translation -- (B) the resulting unicode string raises
an error later when log.py tries to upgrade the string again to unicode
for printing.
- This fix is largely implemented by adding a wrapper for backend list()
implementations. This wrapper ensures that duplicity internals always see
a byte string. (I'd like to eventually use this same wrapping strategy to
implement generic retry support without backends having to add any logic,
but that's just a thought for the future.)
- That is, the fix for issue #1 is completely inside backend.py and the
changes to backends/*.py.
- The rest of the invasive changes deal with filenames that may not be valid
utf8. This is much rarer, but possible. For proper handling of this, we
need to print using unicode, and convert filenames from the system filename
encoding to unicode, gracefully handling conversion errors. Some of the
filenames we print are remote names. Who knows what encoding they are in;
it could be different than the system filename encoding. 99% of the time,
everything will be utf8 and we're fine. If we do get conversion errors,
the only effect should be some question mark characters in duplicity
logging output.
- I tried to convert as much of the actual codebase to use unicode for
printing. But I stopped short of adding an assert in log.py to enforce
unicode, because I didn't want to go through all the backend code and
manually adjust those bits without being able to test each one.
2013-12-28 Kenneth Loafman <[email protected]>
* Merged in lp:~verb/duplicity/boto-min-version
- Update documentation and error messages to match the current actual version
requirements of boto backend.
* Merged in changes from main trunk.
* Merged in lp:~ed.so/duplicity/debian.paramiko.log
- upstream debian patch "paramiko logging"
http://patch-tracker.debian.org/package/duplicity/0.6.22-2
* Merged in lp:~ed.so/duplicity/debian.dav.mkdir
- upstream debian patch "webdav create folder recursively"
http://patch-tracker.debian.org/package/duplicity/0.6.22-2
* Nuke tabs
2013-11-24 Kenneth Loafman <[email protected]>
* Merged in lp:~jkrauss/duplicity/pyrax
- Rackspace has deprecated python-cloudfiles in favor of their pyrax
library, which consolidates all Rackspace Cloud API functionality into
a single library. Tested it with Duplicity 0.6.21 on both Arch Linux
and FreeBSD 8.3.0.
* Changed to default to pyrax backend rather than cloudfiles backend.
To revert to the cloudfiles backend use '--cf-backend=cloudfiles'
2013-11-19 Kenneth Loafman <[email protected]>
* Applied patch to fix "Access GDrive through gdocs backend failing"
- see https://lists.nongnu.org/archive/html/duplicity-talk/2013-07/msg00007.html
2013-11-17 Kenneth Loafman <[email protected]>
* Merged in lp:~verb/duplicity/bucket_root_fix
- Fix bug that prevents backing up to the root of a bucket with boto backend.
* Merged in lp:~gliptak/duplicity/415619
- Better error message when chown fails
* Merged in lp:~mterry/duplicity/log-path-type
- Any backup browser built on top of duplicity will need to indicate which
files in the backup are folders and which are files. The current logging
information doesn't provide this detail. So I've added a field to the
log.InfoCode.file_list output that includes the path type.
* Merged in lp:~mterry/duplicity/manifest-oddities
- We may accidentally end up with an oddly inconsistent manifest like so:
Volume 1
Volume 2
Volume 3
Volume 2
As did get reported recently on the mailing list:
http://lists.nongnu.org/archive/html/duplicity-talk/2013-11/msg00009.html
- One way this can happen (the only way?) is if you back up, then duplicity
gets interrupted between writing the manifest and uploading the volume.
Then, when restarted, there is no longer enough data to create as many
volumes as existed previously.
- This situation can cause an exception when trying to restart the backup.
- This branch fixes it by deleting any excess volume information encountered
when loading in the manifest. We discard volume with higher numbers
than the last one read.
* Merged in lp:~mterry/duplicity/disappearing-source
- When restarting a backup, we may accidentally skip the first chunk of one of
the source files. To reproduce this,:
1) interrupt a backup
2) delete the source file it was in the middle of
3) restart the backup
- When replaying the source iterator to find where to resume from, we can't
notice that the file is gone until we've already iterated past where it
would be!
- The solution I came up with is to just let duplicity stuff the data we
accidentally read back into the source iterator.
- This is actually a data loss bug, because it's possible to back up
corrupted files (that are missing their first chunk).
* Merged in lp:~mterry/duplicity/normalize-before-using
- Avoid throwing an exception due to a None element in a patch sequence.
- None elements in a (non-normalized) patch sequence are perfectly normal.
With the current code in the patched function, it is certainly possible to
hit a crash due a None.
See http://lists.nongnu.org/archive/html/duplicity-talk/2013-11/msg00005.html
- This branch fixes that by normalizing the sequence before using it in the
logging code. It's acceptable to bring the normalize_ps() call outside the
try/except block because normalize_ps is not expected to throw. It's
relatively simple and doesn't really use its objects besides checking if
they are None.
2013-09-23 Kenneth Loafman <[email protected]>
* Merged in lp:~mterry/duplicity/catch-seq-copy-error
- Any* exception when running patch_seq2ropath should be ignored (though
logged) and duplicity should move on. This covers the two asserts in that
function (bug 1155345 and bug 720525) as well as errors that happen during
file copying (bug 662442).
* Merged in lp:~mterry/duplicity/argv
- Fix use of argv when calling os.execve
2013-09-14 Kenneth Loafman <[email protected]>
* Merged lp:~mterry/duplicity/ignore-missing to fix patch below.
2013-09-13 Kenneth Loafman <[email protected]>
* Applied patch from bug 1216921 to fix ignore_missing().
2013-08-17 Kenneth Loafman <[email protected]>
* Merged in lp:~scowcron/duplicity/ftp_password_pexpect
- Use common backend.Backend get_password() rather than _ssh_pexpect.py specific code.
* Merged in lp:~mhu-s/duplicity/swiftbackend
- This branch adds support for Swift, the OpenStack Object Storage service. See
https://blueprints.launchpad.net/duplicity/+spec/swiftbackend
* Merged in lp:~verb/duplicity/boto-gcs
- These patches add support for Google Cloud Storage via the boto backend.
- boto has supported GCS in interoperability mode for a few years now. This change
adds support by taking advantage of boto's storage_uri abstraction layer.
* Merged in lp:~ckornacker/duplicity/megacloud
- Add support for Mega (mega.co.nz) backend.
* Applied patch from Eric S Raymond to man page to fix markup problems.
* Merged in lp:~ed.so/duplicity/man.page
- update paramiko links
- add command parameters to synopsis
- add --compare-data
- some polishing and several improvements
2013-04-27 Kenneth Loafman <[email protected]>
* Merged in lp:~tblue/duplicity/paramiko-1.10.0
- This fixes bug #1156746, making the Paramiko backend compatible with
Paramiko 1.10.0. It keeps compatibility with older Paramiko versions.
* Merged in lp:~townsend/duplicity/fix-1161599-2
- The fix in revno. 912 didn't take into account that the parameter "body"
passed into request() is overloaded, so when it was NULL or of a type other
than file, it would fail. This checks if "body" is of type "file" before
actually seek()'ing back to the beginning of the file.
* Merged in lp:~tblue/duplicity/paramiko-fix-delete-retry
- This fixes bug #1115715, which is really annoying. Basically it makes
using the Paramiko backend with the default settings impossible.
* Merged in lp:~juan-f/duplicity/progress
- From time ago, there are people asking for a progress bar estimation in duplicity.
There is even a script that circumvents the issue, getting info from the log so as
to estimate the progress status ( https://github.com/quentin/Duplicity-progress )
but does not give enough feedback and the estimation is rather plain.
- I have developed a set of heuristics that gather information from the deltas and
the transfer ratios of the backend so as to forecast % of progress, estimation of
remaining time and average speed, for both full and incremental backup uploads.
- The current implementation works for boto backend, but to port the other backends
to use this feature would be quite easy (we can discuss the details if interested).
- The algorithm is activated by the --progress command line flag, and will perform a
first-pass dry-run to collect evidence for all the deltas. Next it will trigger the
real upload, while a thread statistically estimates the ratio of changes and
compression for the data in/out, and uses these ratios to forecast time remaining
and % of completion.
- The progress data will be logged each 3 seconds, or the --progress-rate flag.
* Merged in lp:~jnoster/duplicity/dpbx-added
- The application key was approved as "production" one after some changes to the code
to suit the requirements of Dropbox team (the keys are now obfuscated, for instance).
* Applied blocksize.patch from https://bugs.launchpad.net/duplicity/+bug/897423
- New option --max-blocksize (default 2048) to allow increasing delta blocksize.
* Applied duplicity-ftps.patch from https://bugs.launchpad.net/duplicity/+bug/1104069
- Don't try to delete an empty file list.
2013-02-17 Kenneth Loafman <[email protected]>
* Applied patches from Laszlo Ersek to rdiffdir to "consume a chain of sigtar
files in rdiffdir delta mode" which supports incremental sigtar files.
* Merged in lp:~jnoster/duplicity/dpbx-added
- Add Dropbox backend
- NB! In order to use the backend one must:
1. Install Dropbox Python SDK first.
2. Run the duplicity with Dropbox backend (dpbx://) first time
*interactively* to catch and follow the oAuth URL.
* Merged in lp:~ed.so/duplicity/verify.data
- add switch --compare-data, to selectively enable formerly always disabled
data comparison on verify runs
2013-01-18 Kenneth Loafman <[email protected]>
* Merged in lp:~duplicity-team/duplicity/po-updates
- Updated translations
* Prep for 0.6.21 release
2013-01-18 Kenneth Loafman <[email protected]>
* Merged in lp:~ed.so/duplicity/webdav.fix-retry
- added ssl certificate verification (see man page)
- more robust retry routine to survive ssl errors, broken pipe errors
- added http redirect support
* Merged in lp:~ed.so/duplicity/webdav.manpage
- explanation of webdav changes above
* Merged in lp:~mterry/duplicity/pygi
- Python bindings for the gobject stack (used in the gio backend) have changed
from static to dynamically-generated bindings. The old static bindings are
deprecated. So here's a branch to change the gio backend from old to new ones.
* Merged in lp:~mterry/duplicity/py3rsync
- This branch lets one build the _librsync module with Python 3. You can't
really do anything useful with it, but it's a nicely-isolated piece to add
Python 3 support for.
- The changes are a mix of modernization and #ifdef logic.
- All tests still pass in Python 2.7 and 2.4. I tested manually that the module
worked as expected in Python 3.
2013-01-07 Kenneth Loafman <[email protected]>
* Merged in lp:~mterry/duplicity/static-corruption
- This branch fixes three possible ways a backup could get data-corrupted.
Inspired by bug 1091269.
A) If resuming after a volume that ended in a one-block file, we would
skip the first block of the next file.
B) If resuming after a volume that ended in a multi-block file, we would
skip the first block of the next file.
C) If resuming after a volume that spanned a multi-block file, we would
skip some data inside the file.
- A and B are because when finding the right place in the source files to
restart the backup, the iteration loop didn't handle None block numbers
very well (which are used to indicate the end of a file).
- C is what bug 1091269 talks about. This was because data block sizes would
get smaller as the difftar file got closer and closer to the volsize.
Standard block sizes were 64 * 1024. But say we were close to the end of
the difftar... When resuming, duplicity doesn't know the custom block sizes
used by the previous run, so it uses standard block sizes. And it doesn't
always match up, as you can imagine. So we would leave chunks of data out
of the backed up file.
- Tests added for these cases.
- This branch is called 'static-corruption' because all these issues occur
even when the source data doesn't change. I still think there are some
corruption issues when a file changes in between duplicity runs. I haven't
started looking into that yet, but that's next on my list.
- C only happened without encryption (because the gpg writer function already
happened to force a constant data block size). A and B happened with or
without encryption.
2013-01-02 Kenneth Loafman <[email protected]>
* Fixed 1091269 Data corruption when resuming with --no-encryption
- Patches from Pascual Abellan that make block size consistent and
that add no-encryption option to manual-ctrl-c-test.sh.
- Modified gpg.py patch to use 64k block size so unit test passes.
2013-01-01 Kenneth Loafman <[email protected]>
* Merged in lp:~ed.so/duplicity/u1_and_manpage
- Manpage
- document Ubuntu One required python libs
- added continuous contributors and backend author notes
- U1backend
- lazily import non standard python libs, fixes
http://article.gmane.org/gmane.comp.sysutils.backup.duplicity.general/5753
- fix "not bytearray" prevents PUT with python 2.6
- don't hang after putting in credentials (cause it silently retries in background)
but go through with backup
2012-12-22 Kenneth Loafman <[email protected]>
* Merged in lp:~ed.so/duplicity/webdav.fix-retry
- bugfix: webdav retrying broke on ERRORS like "error: [Errno 32] Broken pipe" in
socket.pyas reported here https://answers.launchpad.net/duplicity/+question/212966
added a more generalized 'retry_fatal' decorator which makes retrying backend
methods even easier
* Merged in lp:~ed.so/duplicity/manpage
- Clear up PASSPHRASE reusage as sign passphrase. Minor fixes.
2012-12-17 Kenneth Loafman <[email protected]>
* Merged in lp:~lenharo-h/duplicity/duplicity
- Generate encrypted backups without revealing the user's key id
via option --hidden-encrypt-key
* Merged in lp:~mterry/duplicity/u1-utf8
- Make sure u1backend returns filenames as utf8
* Merged in lp:~carlos-abalde/duplicity/gdocs-backend-gdata-2.0.16.-upgrade
- Upgrade of GoogleDocs backend to python gdata lib >= 2.0.15:
Stop using get_everything method.
2012-11-19 Kenneth Loafman <[email protected]>
* Merged in lp:~ed.so/duplicity/lftp.netrc
- Allow .netrc auth for lftp backend
* Merged in lp:~mterry/duplicity/946988
- This fixes bug 946988 by not duplicating the checks for when we should ask
for the password (those same checks are done more correctly inside
get_passphrase). And add a test to reproduce the bug.
2012-11-10 Kenneth Loafman <[email protected]>
* Merged in lp:~satwell/duplicity/caching
- Add a cache for password and group lookups. This significantly improves
runtime with very large password and group configurations.
* Merged in lp:~ed.so/duplicity/manpage
- more formatting fixes, clarifications in sections EXAMPLES, FILE SELECTION
2012-11-03 Kenneth Loafman <[email protected]>
* Merged in lp:~mterry/duplicity/u1-oauthlib
- As the Ubuntu packager for duplicity, I would prefer u1backend.py
used oauthlib instead of oauth. oauthlib is well maintained upstream
(unlike oauth), has a python3 port (for the future), and is in Ubuntu
main (so is oauth right now, but hopefully in the future we can drop
it to universe, in which case duplicity can't use it anymore).
* Merged in lp:~mterry/duplicity/delete-new-sig-in-cache
- In duplicity 0.6.20, we fixed bug 1031269. This means that we no longer
leave sig files on the remote location. Leaving sig files on the remote
location also caused a bug with deleting cache files. Code used to leave
remote new-sig but delete the locale cache new-sig; this meant that we would
keep downloadoing the new-sig all the time from remote. We had worked around
that by just not deleting the new-sig in the cache, which was sort of the
wrong side of that problem to tackle. Now that we handle the remote
new-sigs better (by deleting them), I don't think we need this code anymore.
Patch by [email protected].
* Merged in lp:~mterry/duplicity/u1-ascii-error
- Fix for u1backend unicode error. Patch by Paul Barker.
2012-10-29 Kenneth Loafman <[email protected]>
* Merged in lp:~ed.so/duplicity/24syntaxfix
- fix python 2.4 vs 2.5 syntax error
2012-10-28 Kenneth Loafman <[email protected]>
* Remove dist/mkGNUchangelog script.
* Prep files for 0.6.20 release.
2012-10-27 Kenneth Loafman <[email protected]>
* Applied patch from az for bug #1066625 u1backend
- add delay between retries
2012-10-25 Kenneth Loafman <[email protected]>
* Merged in lp:~mterry/duplicity/u1-402
- Switch the code we check for out-of-space in u1backend.
2012-10-25 Michael Terry <[email protected]>
u1backend: interpret http code 402 as no-space-left rather than 507
2012-10-17 Kenneth Loafman <[email protected]>
Update CHANGELOG and Changelog.GNU
2012-10-17 Kenneth Loafman <[email protected]>
1039001 --exclude-if-present and --exclude-other-filesystems causes crash with inaccessible other fs
2012-10-17 Kenneth Loafman <[email protected]>
995851 doc improvement for --encrypt-key, --sign-key
2012-10-17 Kenneth Loafman <[email protected]>
Update Changelog.GNU
2012-10-17 Kenneth Loafman <[email protected]>
1066625 ubuntu one backend does not work without gnome/dbus/x11 session
2012-10-16 Kenneth Loafman <[email protected]>
Updated Changelog.GNU
2012-10-16 Kenneth Loafman <[email protected]>
* Merged in lp:~ed.so/duplicity/ssh.manpage
- added gdocs and rsync REQUIREMENTS
- added cloudfiles documentation
2012-10-16 ede
added gdocs, rsync REQUIREMENTS
2012-10-13 ede
some refinements
add cloudfiles documentation
2012-10-03 ede
sort urls alphabetically
add abs vs. relative file urls
2012-10-03 ede
typo
2012-10-03 ede
minor fix
2012-10-03 ede
some clarifications mostly for ssh pexpect backend
2012-10-16 Kenneth Loafman <[email protected]>
Update Changelog.GNU
2012-10-16 Kenneth Loafman <[email protected]>
* Merged in lp:~ed.so/duplicity/gpginterface
- refactor GnuPGInterface to gpginterface.py
reasoning can be found in README
2012-10-15 ede
some clarifications in README
2012-10-15 ede
refactor GnuPGInterface to gpginterface.py
reasoning can be found in README
2012-10-07 Kenneth Loafman <[email protected]>
Update Changelog.GNU
2012-10-07 Kenneth Loafman <[email protected]>
* Merged in lp:~ed.so/duplicity/gpg.tmp
- place gpg.py tempfiles in duplicity's tmp subfolder which is cleaned
whatever happens
2012-10-07 ede
place gpg.py tempfiles in duplicity's tmp subfolder which is cleaned whatever happens
2012-10-02 Michael Terry <[email protected]>
tests: apparently on hardy chroots, /bin can be smaller than 3MB compressed, so instead of using /bin in test_multi_volume_failure, use largefiles
2012-10-02 Michael Terry <[email protected]>
tests: whoops, I had accidentally disabled one of the new tests for ignoring double entries in a tarball
2012-10-02 Michael Terry <[email protected]>
tests: don't use subprocess.check_output, which was only added in Python 2.7
2012-09-30 Michael Terry <[email protected]>
don't use unittest.TestCase.assertSetEqual, which isn't supported in older Python versions
2012-09-30 Kenneth Loafman <[email protected]>
Update Changelog.GNU
2012-09-30 Kenneth Loafman <[email protected]>
* Merged in lp:~ed.so/duplicity/ssh-pexpect-msgbug
- Fixes 'UnboundLocalError: local variable 'msg' referenced before assignment'
in _ssh_pexpect.py
2012-09-30 ede
probably fix
File "/usr/local/lib/python2.7/dist-packages/duplicity/backends/_ssh_pexpect.py", line 223, in run_sftp_command
log.Warn("Running '%s' with commands:\n %s\n failed (attempt #%d): %s" % (commandline, "\n ".join(commands), n, msg))
UnboundLocalError: local variable 'msg' referenced before assignment
2012-09-29 Kenneth Loafman <[email protected]>
Wrap CHANGELOG to col 80.
2012-09-29 Kenneth Loafman <[email protected]>
Update Changelog.GNU
2012-09-29 Kenneth Loafman <[email protected]>
- Merged in lp:~mterry/duplicity/ropath.index
+ This branch does two main things:
1) Skips base dir entries when compiling the list of deleted delta iters.
(this gracefully recovers from the sort of situations that lead to bug
929067). I'm reasonably confident this is an uninvasive change, but
please confirm.
2) Overwrites the sigtar file on backup-restart. This is because AFAICT,
duplicity will rewrite the entire sigtar each restart. But we were
opening the sigtar file as "ab", so we'd just dump the contents on top
of the previous contents. Which was causing some confusion in bug 929067.
If I'm wrong that we don't always rewrite the entire sigtar each time,
this needs some rethink. Please also confirm that.
+ In addition, I added two tests for the above two changes and make some
improvements elsewhere in the restarttest.py file while I was at it.
2012-09-28 Michael Terry <[email protected]>
gracefully handle multiple duplicate base dir entries in the sigtar; avoid writing such entries out
2012-09-29 Kenneth Loafman <[email protected]>
- Merged in lp:~gregretkowski/duplicity/cf-retry-delete
+ This will retry cloudfile delete commands. With large numbers of archive
files over mediocre links transient network errors will occasionally cause
deletes to fail and these should be retried.
2012-09-27 Greg Retkowski <[email protected]>
Retry cloudfiles deletes
This will retry cloudfile delete commands with large numbers of
archive files over mediocre links deletes occasionally fail
and should be retried.
2012-09-29 Kenneth Loafman <[email protected]>
- Merged in lp:~ed.so/duplicity/duplicity.manpage
+ disabled hyphenation and block justification for better readablility of command line examples.
+ reformatted REQUIREMENTS section for hopefully better online rendering
+ minor clarifications
2012-09-21 ede
missing space
2012-09-21 ede
- disabled hyphenation and block justification for better readablility of command line examples.
- reformatted REQUIREMENTS section for hopefully better online rendering
- minor clarifications
2012-09-29 Kenneth Loafman <[email protected]>
- Merged in lp:~mterry/duplicity/leftover-sigtar
+ So currently, duplicity does not delete signature files when doing a remove-all-but-n operation. Seems wrong, since those signature files are now useless and take up space.
+ This branch does several things:
1) Make remove-all-but-n operate on chains. In practice it did before, since the sets it operated on always came from complete chains (i.e. it never used only some of the sets from a chain)
2) Add a new method to get all signature chains before a certain time.
3) Use this new method to also delete signature chains during remove-all-but operations.
+ And it cleans up the cleanuptest.py file:
1) Removes crufty, unused code
2) Disallows changing the destination folder for the test, which no one would ever want to do and isn't really supported anyway
3) Add some additional checks to the existing test
4) Adds two new methods to test remove-all-but-n and remove-all-inc-of-but-n-full
2012-09-19 Michael Terry <[email protected]>
delete signature files when doing remove-all-but
2012-09-29 Kenneth Loafman <[email protected]>
- Merged in lp:~mterry/duplicity/1031277
+ ssh: actually delete all the requested files, not just the first one
2012-09-19 Michael Terry <[email protected]>
ssh: actually delete all the requested files, not just the first one
2012-09-13 Kenneth Loafman <[email protected]>
Update Changelog.GNU and CHANGELOG.
2012-09-13 Kenneth Loafman <[email protected]>
Merged in lp:~mterry/duplicity/utf8-po.
2012-09-13 Michael Terry <[email protected]>
make sure translations are in utf-8
2012-09-12 Kenneth Loafman <[email protected]>
Fix dates.
2012-09-12 Kenneth Loafman <[email protected]>
Update CHANGELOG and Changelog.GNU to reflect recent changes.
Update location path in mkGNUChangelog.sh.
2012-09-11 Kenneth Loafman <[email protected]>
Merged in lp:~ed.so/duplicity/duplicity.tmpspacefix
2012-09-11 edso
use tempfile.TemporaryFile() so unused temp files are deleted automagically
2012-09-10 edso
propbably solve bug 'Out of space error while restoring a file'
see bug tracker/mailing list
https://bugs.launchpad.net/duplicity/+bug/1005901
http://lists.gnu.org/archive/html/duplicity-talk/2012-09/msg00000.html
2012-09-11 Kenneth Loafman <[email protected]>
Merged in lp:~ed.so/duplicity/duplicity.helpfix
2012-09-10 edso
fix rare 'TypeError: encode() argument 1 must be string, not None'
read here
http://lists.nongnu.org/archive/html/duplicity-talk/2012-09/msg00016.html
2012-06-21 Michael Terry <[email protected]>
log.py: add a couple comments to reserve error codes 126 and 127 because they conflict with running duplicity under pkexec (very similar to how 255 is reserved because gksu uses it)
2012-05-18 Kenneth Loafman <[email protected]>
Add note on GnuPGInterface and multiple GPG processes.
2012-05-16 Kenneth Loafman <[email protected]>
Merged in lp:~ed.so/duplicity/backend_fixes
- fixed ssh/gio backend import warnings
+ ssh paramiko backend imports paramiko lazily now
+ gio backend is not imported automatically but on request when --gio option is used
- added a warning when --ssh-backend is used with an incorrect value
2012-05-16 edso
- fixed ssh/gio backend import warnings
+ ssh paramiko backend imports paramiko lazily now
+ gio backend is not imported automatically but on request when --gio option is used
- added a warning when --ssh-backend is used with an incorrect value
2012-05-15 Kenneth Loafman <[email protected]>
Update changelogs, remove misleading date.
2012-05-15 Kenneth Loafman <[email protected]>
Merged in lp:~ed.so/duplicity/ssh-fixes
- ssh paramiko backend respects --num-retries now
- set retry delay for ssh backends to 10s
- ssh pexpect backend
+ sftp part does not claim 'Invalid SSH password' although it's only 'Permission denied' now
+ sftp errors are now more talkative
- gpg.py
+ commented assert which broke otherwise working verify run
2012-05-15 edso