Backup to NFS share fails because free space is calculated based on host system instead of remote NFS share

amadeus · May 31, 2024, 6:48am

I want to use Déja Dup (Flatpak) to backup my machines to an NFS share.

For this I mounted the share to /var/nfs/share-name/ and set Déja Dup to backup to a Local Folder with that same Path (/var/nfs/share-name/).

Initially it looks as if the backup is performed normally, only to receive an error message after a while that the storage location does not have enough free space.

Data to be backed up: around 600GB
Available space on the NFS share: around 4TB
Available space on host system: around 400GB

I therefore assume that Déja Dup uses the free disk space of the host system as the basis for the abort.

Do I have to define the NFS share differently in the backup settings?

This is my system:

OS: Manjaro Linux Wynsdey 24.0.0 x86_64
Kernel: 6.9.0-1-MANJARO
Packages: 1446 (pacman)[stable], 189 (flatpak)
Shell: zsh 5.9
Display (VP32UQ): 3840x2160 @ 60Hz
DE: Gnome 46.1
WM: Mutter (Wayland)
Terminal: GNOME Console 46.0
CPU: AMD Ryzen 9 3900X (24) @ 4.67 GHz
GPU 1: AMD Radeon Pro WX 7100 @ 0.82 GHz [Discrete]
GPU 2: AMD Radeon Pro WX 7100 @ 1.21 GHz [Discrete]
Memory: 4.29 GiB / 62.71 GiB (7%)

Many thanks for your help in advance!

amadeus · May 31, 2024, 6:58am

I tried to set the backup destination to a Network Server but could not figure out how to make this work using NFS.

amadeus · May 31, 2024, 7:03am

I was able to use the same destination via SMB and Network Server as a location but I would prefer to not have to use SMB.

mterry · May 31, 2024, 12:23pm

Ah OK good catch - I’ll look into fixing that. Thanks!

I don’t think there’s a possible workaround in the meantime, sorry.

amadeus · May 31, 2024, 2:04pm

Thanks! Let me know if I can be of any help.

mterry · June 1, 2024, 1:53pm

OK I just tested and we do correctly get the size of the nfs server mount - so it might be another similar issue, like us running out of /tmp space on the host or maybe a place where we accidentally look up host space instead of mount space.

Can you provide more info on the exact error you see? A screenshot maybe

amadeus · June 2, 2024, 3:38am

Thanks for looking into this. I am happy to provide some screen grabs.

Settings

Error message

I was only allowed to upload two screen grabs. All 4 I made are available for download here:
https://send.internxt.com/download/8627a8d6-523c-4d44-a177-458651b99c45?code=d333e9ed064e8d331ad6b1c5a9cddbba515bf2e356f943f0765dac17fa8b171f

mterry · June 2, 2024, 1:39pm

Hmph OK… This is all confusing

Here’s what Deja Dup is checking here

Because duplicity isn’t the greatest at handling running out of space, we are a little conservative and check that we have plenty of it.

So we take the uncompressed size of the backup files, and multiply by two (two full backups worth of space) and then see if the total space at the destination is at least that big.

(That’s total space, not just free space. There’s also some checks against free space, but you’re hitting the total space check.)

Your error message

So what that error message is saying is that the detected total space of the target is less than 829GB. Which… is weird because your local hard drive has 989GB and the NFS drive has 12TB…

So what total size are we grabbing?

Easier testing

You should be able to see the same numbers that Deja Dup is by running gio info -f /var/nfs/vault/backup/amadeus-4/

What do you see when you run that?

amadeus · June 2, 2024, 2:08pm

This is what I get when running gio info -f /var/nfs/vault/backup/amadeus-4/:

  filesystem::size: 23800304238592
  filesystem::free: 12927738314752
  filesystem::type: nfs
  filesystem::readonly: FALSE
  filesystem::used: 10872444289024
  filesystem::remote: TRUE

mterry · June 3, 2024, 8:31pm

Hmph… the mystery deepens.

So that shows 24TB total, 13TB free. So again, Deja Dup should be just fine.

I’ve tried editing the code locally to use your numbers instead of my drive’s numbers, and I still couldn’t get an error state.

Next step might be creating a flatpak bundle that includes a lot of debugging output that you could run, so we can see where the fault lies.

mterry · June 3, 2024, 9:57pm

OK if you’re willing, try this:

Download https://gitlab.gnome.org/World/deja-dup/-/jobs/3944941/artifacts/raw/org.gnome.DejaDupDevel.flatpak
Install it like flatpak install --user ~/Downloads/org.gnome.DejaDupDevel.flatpak
Run it like flatpak run org.gnome.DejaDupDevel
Start a backup, look for some warnings on the console about detected space.

(This is based off the unreleased 46.0 - so there may be some small UI differences - that’s expected, but everything should still work fine.)

amadeus · June 4, 2024, 12:15am

Thank you so much, dear Michael.

This is what I get:

(org.gnome.DejaDupDevel:2): deja-dup-WARNING **: 07:11:51.506: DuplicityJob.vala:479: TEST: total space detected: 8.0 GB

(org.gnome.DejaDupDevel:2): deja-dup-WARNING **: 07:11:51.506: DuplicityJob.vala:480: TEST: free space detected: 8.0 GB

(org.gnome.DejaDupDevel:2): deja-dup-WARNING **: 07:11:51.507: DuplicityJob.vala:481: TEST: backup size detected: 414.4 GB

(org.gnome.DejaDupDevel:2): deja-dup-WARNING **: 07:11:51.507: DuplicityJob.vala:482: TEST: required total space: 828.8 GB

I have no idea where these 8GB come from.

mterry · June 4, 2024, 1:08am

Oh oh oh - I think this is because /var is a weirdo inside a flatpak. Where parts of it get re-mounted instead of the host’s /var. That 8GB might be a tmpfs mounted over /var inside the flatpak and the flatpak might not be seeing the real nfs mount at all (at that location anyway).

Let me retry my nfs testing and mirror your /var setup. There should be a way to work around that.

mterry · June 4, 2024, 1:31am

OK - after some quick testing, there’s no way to reliably avoid this problem.

If you use the file chooser dialog and end up with a folder like /run/user/1000/doc/1f7832c1/amadeus-4 it won’t persist after a reboot (because flatpak-portals does not persist folder access for flatpaks with host permissions - which… might be a thing they’d be willing to change, but that’s not a today fix).

And if you manually enter the /var/nfs/... location in the preferences, it won’t actually be usable, because /var/ is a fake mount inside the flatpak.

So the only workaround I can think of for you right now is to use a different prefix - i.e. nothing under the following:
"/lib", "/lib32", "/lib64", "/bin", "/sbin", "/usr", "/boot", "/root", "/tmp", "/etc", "/app", "/run", "/proc", "/sys", "/dev", "/var"

I need to add some UI to warn the user about this problem…

amadeus · June 4, 2024, 1:37am

Oh, I see. Interesting. Let me know if there is anything else I can do to help you. Like adding special permissions to the Flatpak or mount the shares outside of /var/.

amadeus · June 4, 2024, 1:39am

Good to know! I will try that.

amadeus · June 4, 2024, 6:00am

I hope I did not use another forbidden path when mounting my NFS shares to /mnt/nfs/ but it seems to be working with that path as the backup is currently in progress.

Thanks again for your help!

mterry · June 8, 2024, 5:17pm

I’ve added some warnings for this use case now:

Gonna also add a clear error if you try to back up or restore too. But at least now there’s some clear feedback about this workflow.

Thanks for flagging it! Hope things have been working since.

amadeus · June 8, 2024, 10:54pm

Thanks a lot and it has been working like a charm ever since.

system · July 23, 2024, 10:55pm

This topic was automatically closed 45 days after the last reply. New replies are no longer allowed.