Déjà Dup Backups Roadmap: Restic Edition

Applications
1

Déjà Dup has used Duplicity since its beginning, over 15 years ago. Though it’s never been intended as a Duplicity frontend, but rather a backup app that used Duplicity behind the scenes.

And for the past four years, folks have been able to use an experimental Restic backend instead of the usual Duplicity one.

Over those years, the feedback has been positive, bugs have been fixed, and feature parity has been achieved.

I think it’s time to promote the Restic backend out of experimental mode and into real world usage.

Benefits of Restic

i.e. Why even consider Restic when we have Duplicity?

First, a couple words about how each works. (extremely simplifiedd)

  • Duplicity takes a “rolling incremental” approach. Each snapshot builds upon the previous ones, in a continuous chain. Déjà Dup occasionally creates a new full snapshot so that older chains can be deleted.
  • Restic takes a “repository” approach where chunks of files are stored in a folder tree with associated metadata connecting the chunks to files and snapshots.

Now back to the benefits of Restic:

  • Faster
    • About twice as fast in my testing
  • More robust
    • If anything goes wrong in Duplicity’s chain of incrementals, it can have a difficult time recovering, since all the future snapshots depend on the previous one. Whereas restic has some tooling around fixing problematic repositories
    • Restic also seems to handle interruptions more gracefully
  • More flexible, enabling possible future Déjà Dup features:
    • Changing your backup password
    • Searching the repository directly, using less memory
    • Snapshot management (e.g. deleting snapshots at will)
    • Limiting upload speed

Current situation

  • Déjà Dup defaults to using Duplicity and puts the files directly in the storage folder.
  • If Déjà Dup was built with --Denable_restic=true, you can opt into using Restic instead in the Preferences dialog. Déjà Dup will put Restic files in a restic/ subfolder of the storage folder.

Coming plans

  1. Add code to detect whether the storage folder holds Duplicity or
    Restic files (including looking for a restic/ subfolder).
  2. If the storage folder holds Restic files, use Restic. If Duplicity files, use Duplicity.
  3. Add a new build flag -Ddefault_tool=duplicity/restic that will determine if Déjà Dup will use Duplicity or Restic for fresh backups into empty storage folders. This will default to duplicity for continuity.
  4. But the official Déjà Dup flatpak builds on Flathub will instead use restic as the default tool for new backups, to begin getting broader testing.

Down the Road

Eventually if feedback continues to be positive, we may remove the build flag and unconditionally default to Restic, even for non-Flathub builds.

But existing Duplicity backups would continue to be supported. And any change there would be well advertised in advance and/or we’d offer some clever migration path. I’m not in a rush here and want to handle this cautiously.

Thoughts?

Please chime in with your experiences with Restic or concerns or happiness with this approach. Thanks!

6 Likes
2

Hi Michael! I remember exchanging a few mails when you were doing this initially. I’ve been rolling with the restic backend ever since and I’ve done multiple restores and backups from all of my gear. Nothing to report so far!

The backend support for restic is the killer feature, it’s nice to have the flexiblity to use almost any bucket. I keep a set locally and a few on remote cloud buckets, it’s nice.

May I recommend a large and more obvious donation button on the homepage? My life would be a disaster without good backups I want to pay.

2 Likes
3

I remember! Thank you and I’m glad it’s been doing OK!

Hah, I struggle with self promotion. But thank you. I’ll think about how to more prominently feature the liberapay link.

1 Like
4

New to Deja Dup so I read the FAQ because I use Restic icw BackBlaze.

I understand (IIRC) that support for BackBlaze is not something to develop, as the intended audience are casual users. I configured Restic to backup to BackBlaze B2, environment and parameters in place and all … but I really would appreciate a Gnome native frontend. So maybe I am kind in the middle.

My question is:
If Restic uses B2, can Deja Dup be a frontend to it, without direct support for it?

If its out of scope it would be understandable tho.

2 Likes
5

This is maybe a different issue than whether we use Restic or duplicity.

But both the Restic and Duplicity backends now use rclone for cloud support. It seems plausible that I could find a way (even a hacky way without a UI maybe) to let folks specify a custom rclone target. But that often means environment variables or special setup. I’d have to sit down and think about it, and I haven’t done that yet.

But speaking of rclone, I think it can mount a cloud target as a local folder. So maybe that could be a stopgap - just point Deja Dup at the mounted path?

2 Likes
6

Oh … interesting, I have used RClone to mount B2 …

7

Just FYI, this “restic by default” change landed in the nightly flatpak release - any testing would be appreciated.

Specifically the following things are worth testing:

  • Deja Dup should make restic repositories when given an empty storage folder.
  • Deja Dup should detect whether a non-empty storage folder has duplicity or restic files and use the right tool, adding a new snapshot to either one or restoring from it as appropriate.
  • Restic support in general.