Add backup and restore-backup scripts

[Exairnous]

What?

Adds scripts to backup the data from your Hubs instance to your local hard drive and restore a backup to your instance.

Why?

This will allow you to keep one or more local copies of your data, restore your data to your instance if needed, and migrate all your data from one instance to another, e.g. when moving from one hosting company to another hosting company.

Examples

Backup folder structure

• data_backups
  • data_backup_1749037375184
    • reticulum_storage_data
      • cached
      • expiring
      • owned
    • pg_dump.sql
  • data_backup_1749058020987
  • etc.

Note: additional folders may be present in the reticulum_storage_data_folder and/or some may be omitted. This depends on the individual instance.

How to test

1. Run the backup script.
2. Make a change, e.g. create a small scene.
3. Run the backup script again.
4. Run the restore-backup script with the name of the first backup (see the readme instructions for details).
5. Check that your change has been reverted.
6. Run the restore-backup script with no backup specified (this will use the latest backup).
7. See that your change is back.

Documentation of functionality

Instructions have been added to the readme. A PR for further documentation is planned for the Hubs docs repository.

Limitations

This requires the pods to be running, so in order to prevent people from using the instance while backing up/restoring you'd need to remove the load balancer IP from your DNS A records (unless there's some other way to bar people from the instance while keeping the pods running that I'm unaware of). UPDATE: This has been addressed for the restore script, by introducing a crude maintenance mode. Thanks to the review comments for pointing me toward this solution.

Alternatives considered

• Using Node directly to transfer the data, but this is easier and doesn't reinvent the wheel.
• Using something like rclone, but that would require another dependency to be installed.

Open questions

• What happens if someone is saving something on the instance while a backup is running? Will the backup be corrupted?
• What happens if someone is saving something on the instance while a backup is being restored? Will the instance get corrupted? UPDATE: No longer applicable after updates from the review - a crude maintenance mode has been introduced to prevent people being connected to the instance during the restore.
• Is there a way to bar people from the instance, while keeping the pods running, that doesn't require you to edit your DNS A records? UPDATE: Yes. See the update for the previous question.

Additional details or related context

This will be needed to migrate the data from the persistent volumes on your node, to persistent volumes that are completely separate from the node (PR #363).

[DougReeder]

If no other app is using the load balancer used by Hubs, changing the DNS address is fine (though it doesn't take effect immediately nor can be reversed immediately, due to DNS cacheing).

A more focussed approach is to add this annotation to the Ingress:

    haproxy.org/allow-list: 11.22.33.44

where 11.22.33.44 is the external IP address of your development machine. It takes effect immediately, can be reversed immediately, and allows you access to your Hubs instance while denying all others.

Editing the ingress can be done with the command

kub edit ingress reticulum -n hcce

and similarly for dialog and nearspark.

To restore access, it's probably best to re-apply hcce.yaml

[Exairnous]

If no other app is using the load balancer used by Hubs...

True, I didn't think about other apps using the load balancer. So yes, disabling the DNS is not a great solution :(

A more focussed approach is to add this annotation to the Ingress:

I tried adding haproxy.org/allow-list: 66.66.66.66 to the annotations section of the three ingresses to test the blocking, but my instance still loaded fine for me.

These are the commands I used to edit the ingresses (I'm guessing kub is your alias for kubectl? Also, it appears the ingress name for reticulum is ret, which we should update to reticulum at some point):

EDITOR=kwrite kubectl edit ingress ret -n hcce
EDITOR=kwrite kubectl edit ingress dialog -n hcce
EDITOR=kwrite kubectl edit ingress nearspark -n hcce

Also, reapplying hcce.yaml didn't seem to remove the edits (running those three commands again showed the additions still there).

Am I misunderstanding the procedure you're suggesting or is there something more I need to do to apply the edits (I closed the editor and saw this in the terminal: ingress.networking.k8s.io/ret edited)?

[DougReeder]

You're correct that reapplying the template after manually editing the ingress doesn't reset it. The correct procedure is to add the annotation to the template file and apply it. To revert, one then comments out the annotation and re-applys the template.

It's possible that that annotation wasn't supported for the beta version of HAProxy ingress controller that Hubs normally uses. (I'm using the latest).

You might try

     haproxy.org/request-redirect: example.com

[DougReeder]

As I'm using an external database, I can't fully test these. I have to comment out the pgsql bits, and I'm hesitant to restore only the reticulum files.

That said, the code looks fine and the backup script did create a copy of the reticulum files for me.

[Exairnous]

You might try
haproxy.org/request-redirect: example.com

Thanks. That looks like it'll work. I've updated the PR.

As I'm using an external database, I can't fully test these. I have to comment out the pgsql bits, and I'm hesitant to restore only the reticulum files.

Good point. If you think it would be an easy change to support backing up/restoring an external database as well, then it would be good to add that in. If not, then we should probably wait until we add official support for creating external database setups and add backup/restore support then.

[DougReeder]

Let's leave external DB support for another PR - it won't be trivial to add.

[Exairnous]

Let's leave external DB support for another PR - it won't be trivial to add.

Okay. Sounds good.

[Exairnous]

@hobbs-Hobbler tested the restore-backup script on Windows with an old backup from a previous version of the scripts (but using the latest version of the restore-backup script) and everything appeared to work well (it was a non-standard setup, so the results should probably be interpreted with some reservation, but I'm optimistic that at least the restore-backup script should work correctly on Windows).

[DougReeder]

restoring Reticulum '._cached' folder
restoring Reticulum '._expiring' folder
restoring Reticulum '._owned' folder
restoring Reticulum '._storage' folder

The restore-backup script should probably not copy back folders created by the OS, like these.

[Exairnous]

....
restoring Reticulum '._storage' folder
The restore-backup script should probably not copy back folders created by the OS, like these.

I'm not familiar with these folders. Are they a Mac thing? The reason I'm looping over them is because we can't know exactly what folders will be present for a backup/restore, but if those will never be needed and can be reliably detected by the leading period and underscore then I could add a guard to filter them out.

[DougReeder]

A key source of kubectl cp failures appears to be that, under the hood, it's using tar, which is not designed to deal gracefully with network failures. So, having the backup script retry kubectl cp is okay. I recommend not attempting more than a dozen retries at either level, though.

[Exairnous]

Updates:

I have updated this to integrate the "junk" npm package and remove all the OS files from the backup before uploading. I realized that the OS files could potentially be present in any of the subfolders as well as the main folder, so since kubectl cp doesn't have any option for ignoring files/folders I thought this was an easier solution than rebuilding the structure piece by piece on the pod storage.

I found that disabling the websockets did work reliably for me, so I think this is a much better solution than auto-retrying, and it should hopefully just automatically get phased out when kubectl gets fixed (@DougReeder you were right at the dev meetup last week that there was a better way). I still think we want to keep it on infinite retries for kubectl cp itself, though; so far there have been no issues reported with doing that and it should ensure that the backup will succeed no matter how large it is, even when there is an unstable network connection. But yes, kubectl using tar under the hood is very far from ideal; I hope they redo it to be much more reliable.

I have als updated the restore script to clear the Reticulum pod storage before uploading the backup in order to ensure a return to the exact state of the backup.

I think these updates should make this about ready to merge (assuming no one finds any issues when reviewing/testing and I didn't miss any review comments), but it would be good to try and get as many people to test as we can (I'll see if I can get this tested at the documentation meetup this week).

[Exairnous]

Oh, and documentation for the Hubs docs for the backup/restore scripts has been written, but hasn't been put up as a PR yet.

[Exairnous]

Also, sorry about the completely mangled diff for the Filter out OS created helper files/folders commit, it's because of the indentation change. If you're looking at it, you'll want to use GitHub's Hide whitespace option (which actually shows the changes really well).

[DougReeder]

working fine for me