Pluggable backends

Since Deja Dup is not going to support certain backends (like B2), would you consider pluggable backends as an alternative? I’ve peeked around the backends in libdeja and it looks fairly modular. I don’t know if Vala allows this TBH, or if they would have to be scripted in Python.

I think it would be ok if they use json files or something like that for configuration, as long as they let us use deja dup normally like the other backends.

2 Likes

Hello! I’m the maintainer, and I do have opinions on this. But I’m a bit busy and will try to respond more completely later.

3 Likes

Background Vibes

As table setting, I don’t think of Deja Dup as a duplicity frontend but rather a backup program that just happens to use duplicity. Tomorrow it might switch to borg or dumping everything to a single file if those meet our needs more.

So my initial gut reaction to requests for new duplicity backends is that they are an abstraction break we have to maintain, rather than the natural feature gap the requester is probably thinking of.

Deja Dup also has a mission statement that aims squarely at ease of use and converting people that don’t have a backup to people that do. So I tend to consider that if a user needs something more complex than Deja Dup, they should use one of our alternatives rather than Deja Dup growing that complexity.

But let’s think about how we might expose things a bit if we wanted to support folks that miss a particular backend.

Rough Needs for a Backend

  • Ability to get free & total size on backend - needed so we know if we have to make room for new backups
  • Ability to login and save credentials, if needed. If it’s not just a simple username/password setup, we need to be able to give duplicity something it can use to authenticate itself. Each duplicity backend varies in what it will need from us and may even try to do console interactions.
  • Passing arguments, environment variables, and a url to duplicity.

Do we need a plugin?

Ideally we could get away with something lighter weight than bundled code. Depending on the duplicity backend, we could get away with just allowing users to pass arguments and such to duplicity via gsettings keys or similar.

Some downsides though:

  • it only works for some duplicity backends
  • won’t let us manage backup space (and thus will eventually give some error messages)
  • we can’t make the experience nicer at all (like interpreting backend-specific error messages better or the like)
  • no ability to “maintain” it, as new Deja Dup features appear (like maybe “keep backup size less than $maximum” or maybe we migrate to borg and have to stop supporting this method)
  • “secret” methods like gsettings keys gets a little annoying these days because flatpaks keep them in a keyfile out of the normal way. And snaps don’t install the gschema in a global spot. So normal tools even an advanced user would try might not be able to tweak the settings
  • this approach is only really useful for people comfortable with the duplicity command line, in which case, they can just make a cron job and call it a day anyway

Plugins

Plugins could be implemented via libpeas. Which makes the loading part easy anyway.

On my end, this means I’d probably need to put a fair bit more thought into the design and extension points of the backend api. And maybe create a way for the plugin to extend the preferences UI (or just do the json file approach you mention). Probably need to have plugins declare a minimum version of duplicity too.

On the plugin developer’s end, they’d need a way to distribute & update the plugin. Maybe just have folks download a Python file and forget about updating it?

And on the user’s end, they’d need to find and install it. And any dependencies the duplicity backend has (like the module for talking to B2 or whatever).

This gets a little weirder in the snap & flatpak worlds. While Deja Dup can likely load the files from the system even confined (since it needs wide read permissions anyway), getting the dependencies for the duplicity backend playing nice with the confined Python might be tricky. I’m thinking version mismatches I guess. Path issues could be solved with the right environment variables.

GVFS

This is a lot more work, but this is already Deja Dup’s supported “extendable” backend.

If someone writes a gvfs backend for a service or protocol, then anyone can just use that directly as a Network Server.

And those can be plugins too I think?

Summary

So a plugin system seems possible. But would need a lot of thought. And distribution is a concern.

GVFS might be a good option? But that is non-trivial for sure. Though it does mean that everyone benefits, not just Deja Dup.

Thoughts on any of that?

1 Like

Hey, thanks for the detail reply! My notes below.

This got me thinking on the exact reason I want to use B2 as a backend for deja dup:

  • It’s not Google
  • It’s cheap (I literally paid one dollar last year for my backups)
  • I don’t have to manage a server or care about infrastructure (as they have a pretty good track record)

I think most users wanting this or other advanced backends like S3 would have similar reasons. It’s not as much specific about features but rather cost, maintainability and reliability (and not giving more data to Google is a plus too).

Considering the downsides, this would probably bring more problems than solutions.

This could work considering it would be an advanced user feature. There should be some warning about outdated plugins so users know they have to update.

Good point. I forgot about flatpak. Maybe there’s some way to install dependencies inside the sandbox? Blender bundles pip so you can install libraries for its Python interpreter.

This could be an option. TBH I don’t know much about the GVFS to say, but it sounds like it would be quite complicated. And bugs would have to handled “upstream” from deja dup.

1 Like

I know this may not be any more comfort than Google, but the next two consumer-friendly backends I was planning on adding were Amazon Drive and Microsoft OneDrive.

Tech giants all, but at least you can encrypt the backup and then they don’t get any data from ya.

Just have to get off my duff and fix up those duplicity backends’ oauth code.

I think of that as a feature not a bug, in the sense that there would be more eyes on it in gvfs. But it is likely more complicated.

I looked at Amazon Drive and it seems like:

  • It’s only available in a few countries
  • The Paid plan is only for photos and videos, while Brazil and China get a free 5GB storage plan
  • Seems like they don’t want third parties or are looking to discontinue their API at some point

Wikipedia
Thread about the API in the duplicati forums

Tech giants all, but at least you can encrypt the backup and then they don’t get any data from ya.

True, data mining shouldn’t be much of a problem with encryption.
Microsoft and Google are going to be a tough sell for Linux users anyways :smile:

3 Likes

Ah thanks for the intel, I had not even gotten that far into looking at them. A discouraged API and low global availability are both problems for sure.

3 Likes

This “harrowing tale” (linked as such from @januz’s duplicati-forum link) is admittedly three years old, but it also links to this Amazon Drive Developer Guide which apparently used to contain as a “guideline”:

  • Don’t build apps that encrypt customer data

I guess they’ve since relaxed their position on that somewhat, because the quoted phrase no longer appears in the doc. The current guideline that replaced it may not be particularly reassuring, though:

Applications that perform client-side encryption will prevent customers from viewing data in any other application, impacting the overall customer experience.

Which, you know, could obviously be used as an excuse not to approve any data-encrypting app they choose, considering that entire document is positioned as a guide to appeasing the gatekeepers of their particular walled garden. The page opens with:

Being successful on the platform

With our updated RESTful API and SDKs for Android and iOS, Amazon Drive is moving to an invite-only developer offering to ensure we can provide a consistently viable service available for supported use-cases. We want you to be successful when developing apps on the Amazon Drive platform. We have created a guide to help you on your way from getting started to having your app reviewed by our team before releasing it to the public. Please also refer to our Brand Guidelines and the App Distribution and Services Agreement before developing your app.

I know my peril-sensitive sunglasses just went solid black, anyone else’s?

2 Likes

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.