Modern file backup options: cloud? / HD?

In summary, the best strategy for backing up important files is to have a multi-format backup system, including external drives, cloud storage, and possibly CDs. It is important to have backups stored in different locations and to periodically check for integrity. Incremental backups and a Tower of Hanoi strategy can help maximize retention across multiple tapes or media. It is also recommended to use a RAID system for extra security. Cloud storage maintained by a reputable company can also be a safe option. For programming projects, using version control software like Git and storing files on GitHub can provide an additional level of security. Additionally, creating a NAS with a Raspberry Pi and external HDD can be a cost-effective option for backing up files.
  • #1
DaveC426913
Gold Member
22,986
6,661
TL;DR Summary
I want to stop losing my precious projects. I need a reliable way to backup files for me, and friends & fam
I thought I had a long-term backup of my important stuff when I bought a 1Tb Seagate External Drive. It died and took years of projects and memories with it.

Running Win10 on a laptop.

I think there are three options:
  1. Another (better) external drive
  2. Some newfangled solid-state external drive
  3. Cloud storage
Some considerations:
  • I want to ensure it covers mine and my wife's files - (and, in an ideal world, daughter-in-law-at-separate-residence). Whether that means one, two or three devices/services.
  • I think I'll need to set up an auto-backup service for my wife. I personally didn't need one, but I'm more conscientious about securing my important files. But maybe I'll let a computer do something for me that I can do myself. (Big step there.)
  • I don't want to spend a half hour every month watching progress bars push my files around. (I guess that's auto backup during off hours)
  • I do have my own webhost and domain, if that makes any diff. There is storage there. I'm just not sure it's a serious, robust consideration (I've had bad experiences with webhosts).
Anyway, I'm looking for advice and personal opinions about what to go with.
 
  • Like
Likes sreerajt
Computer science news on Phys.org
  • #2
The best strategy is multi format backup. I save stuff to sticks, cds/dvds, external drives and the cloud.

if you were paranoid then you might use multiple cloud backup as well as multi drive raid backup with backups saved in bank vaults at multiple locations.
 
  • Like
Likes russ_watters and sysprog
  • #3
Nope, not that paranoid. Have to have a balance between robustness and effort/convenience.(Forgot to mention CDs. They're pretty robust, long-term, but the routine burning process is prohibitive.)
 
  • Like
Likes sysprog
  • #4
Here's what I do.

I have a central file server. Format is ZFS, which does "snapshots". I take one every 15 minutes, keeping 4, one every hour, keeping 48, one every day, keeping 31, one every week, keeping 7, and one every month, keeping 60. Also, ZFS mirrors every disk, and every two weeks I check every byte for integrity.

Weekly, I back up my desktop PC to the server.

Daily, I backup the central server to an external hard disk using rsync. This typically takes 30 seconds, unless I have recently done a weekly backup, in which case it takes 20 minutes. I keep the external drive in ext4 format. The idea is that if I bork my ZFS pool, I can mount the backup while I figure out went wrong.

My biggest risk is correlated by having everything in one place. Pre-Covid, I kept an external drive at work and another at home and swapped them weekly.

The downside? Cost. Every byte is stored no fewer than 3 times. That costs money. On the plus side, I have never lost a byte.
 
Last edited:
  • Like
Likes Keith_McClary, russ_watters and sysprog
  • #5
Searching on tape backup strategy I found this article:

https://www.techrepublic.com/article/tech-tip-determine-your-tape-backup-strategy/

I remember years ago there was a work strategy of doing incremental backups and periodically a full backup and there was a scheme to use a limited number of tapes in a clever way to maximize retention across multiple tapes which could apply today with any kind of media. Sadly, I can't remember / didn't actually know the full strategy being a lowly programmer at the time.

However, this Tower of Hanoi strategy looks like the one my company used:

https://www.recordnations.com/articles/tape-rotation-2/
 
Last edited:
  • #6
You can never be too paranoid about data loss. I don't trust 'cloud' backup.

My main personal work stations are surplus diskless HP ProLiant DL360 G7 1U 12 core 64-bit Servers running 64-bit Linux Debian remote boot over the network to a main 12 core DL360 G7 with a 8 SAS drive raid and 4 port 1Gbit ethernet for remote NFS mounts to the workstations.

A typical surplus under $200 system: HP Proliant DL360 G7 12 Core Server [ 96G PC3 RAM - 4TB HD - 2x 2.40GHz ]
https://h20195.www2.hpe.com/V2/Getdocument.aspx?docname=c04284501

A physically (workshop on the house lot) separate computer system uses Bacula to backup the main server and other computers to internal 10TB mirrored drives and external 10TB USB-C raid storage pools. It automatically selects incremental and full backups.
 
  • Like
Likes russ_watters
  • #7
Here’s a Smarter Everyday video on how he improved his storage system

 
  • #8
DaveC426913 said:
Summary:: I want to stop losing my precious projects. I need a reliable way to backup files for me, and friends & fam

I thought I had a long-term backup of my important stuff when I bought a 1Tb Seagate External Drive. It died and took years of projects and memories with it.
If that drive was a backup, then you must still have your work on the working drive. If that is the only drive where the files existed, it is not a backup. I recommend periodic backups of your work on a separate drive, which is stored in a separate location. If you want more security, you could set up a RAID system where two drives mirror each other and notify you if one of them has problems.
Running Win10 on a laptop.

I think there are three options:
Another (better) external drive
Being external does not make it a backup drive unless you run backups periodically and keep it in a different location.
Some newfangled solid-state external drive
Same
Cloud storage
Cloud storage maintained by a reputable company will tend to be much safer than the typical system run by an amateur. But I do not know what it costs or how much data you have.
 
  • Like
Likes lomidrevo
  • #9
You can use more than one external drive for storing files that you don't want in your PC's hard disk. Two or more external drives are less probable to fail at the same time, so you will always have a copy of your files on at least one drive.

For my programming projects, I always initialize Git and push them to GitHub (in a private repo where necessary). I can pull the repo even if I lose my local folder.

You can use a Raspberry Pi to create a NAS with an external HDD. There are a lot of tutorials to guide you on the process.
 
  • Like
Likes FactChecker
  • #10
One backup is never enough if you really don't want to lose anything. Anything physical has the potential to fail, especially external HDD's so you need a multi backup approach. You also need online and offline backups to guard against online and offline threats.

I would use a cloud service for online backups and probably an external SSD for occasional offline backups.

Using an online cloud service prevents you loosing data by suffering a hardare / disk failure or worse, like say your house gets struck by lightning that blows all your electronics or your house burning down. Don't count on keeping all your data just on your premisis.

It does not however help of you get malware which then encrypts all your data no matter where it's kept, this is where offline backups come in that are not connected to anything and only connect while the backup is occurring. This can be an external HDD / SSD you plug into backup your data once per week.
 
  • Like
Likes FactChecker
  • #11
Microsoft 365 Family. You and your wife (not sure about licensing for daughter-in-law at different address) each get 1TB of cloud storage that synchronizes in the background transparently. Also other benefits.

Not sure what kind of 'projects' you are talking about: if it is software projects then that doesn't work well with cloud backup (large number of file changes every time you switch branches, transient build artifacts, 3rd party libraries) so GitHub/GitLab/Bitbucket is the answer.
 
  • #12
If you want your backups to last a lifetime, don't forget that they must be migrated to new technology as it becomes available. You can't count on Windows, or Unix variants, or file formats like JPG or PDF or DOC being around forever.

Library science deals with those problems. They found that digital data storage can be more difficult to archive than ancient scrolls, painting, and writings.

I think back to the first backups I made. They were paper tape. The tape might still be in good condition, but try finding a paper tape reader today, and try to find a computer that can run those executable files and source programs (with their dialects and OS dependencies).

We also had magnetic tapes in those days, but they had the same problem. Lots of luck finding a way to read those tapes today.

Migration might include changing file formats and rewriting source programs.

1613567236577.png
1613567290027.png
 
  • Like
Likes FactChecker and Wrichik Basu
  • #13
nsaspook said:
My main personal work stations

Aren't they loud? Anway, back on topic:

One thing to consider with cloud backup is the time to back up and the time to restore. If you have 10 Mb/s ethernet and want to backup or restore 8 TB, that's 80 days up and 80 days down. Even at 100 Mb/s, it's more than a week.

Some commercial clouds will FedEx you an external USB drive if you need to get your data back. Of course you pay for this.
 
  • Informative
Likes FactChecker
  • #14
FactChecker said:
If that drive was a backup, then you must still have your work on the working drive. If that is the only drive where the files existed, it is not a backup.
Well of course, I didn't go looking on my backup for filesvuntil my laptop crashed. That's when I found the external drive had stopped working

But you do make a point, I was using it as an archive as well, to make room on my computer.

I hadn't really been making that distinction until you pointed it out
 
Last edited:
  • Like
Likes FactChecker
  • #15
pbuk said:
Not sure what kind of 'projects' you are talking
photos, Photoshop files, documemts , not code
 
  • #16
anorlunda said:
We also had magnetic tapes in those days, but they had the same problem. Lots of luck finding a way to read those tapes today.

There are companies that will transfer 6250 tapes to LTO. Maybe even a USB stick. Paper tape, well, you're on your own, I think.
 
  • #17
Question: given the choice of cloud storage and one external drive, which would you trust more, long-term?
 
  • #18
DaveC426913 said:
Question: given the choice of cloud storage and one external drive, which would you trust more, long-term?
Define long term. 2 years? 20 years? 200 years?
 
  • Like
Likes Vanadium 50
  • #19
DaveC426913 said:
photos, Photoshop files, documemts , not code
I keep a large collection (~150 GB) of music and photos on multiple external USB drives. I don't store new stuff to them except once in a while and consider them as long-term archival rather than regular backups. It is a problem to keep them up to date and identical, so I accept some losses if the worst happens. I do regular backups to a RAID system and to another USB drive.
 
  • #20
Yes this is just personal use stuff for me, So I accept a certain amount of rot.

My wife's stuff is business stuff though mostly documents.

Email is another concern for her
 
  • #21
DaveC426913 said:
Well of course, I didn't go looking on my backup for files until my laptop crashed. That's when I found the external drive had stopped working.
I understand. You are not alone there. I have horror stories. I occasionally do spot checks to see if it looks like files are on the backup drive. I hope that your data can be recovered. Not all broken backup drives are completely broken.
PS. My horror stories are the reason that I have so many backup drives, RAID systems, etc., and I still don't feel completely safe. Good luck.
 
  • #22
DaveC426913 said:
photos, Photoshop files, documemts , not code
Ok, these are fine for cloud backup.

DaveC426913 said:
Question: given the choice of cloud storage and one external drive, which would you trust more, long-term?
Cloud of course. If your external drive breaks then you are the only one interested in fixing it. If OneDrive goes down there are resources of USDbns (and GBPbns, and EURbns) interested in fixing it.
 
  • Like
Likes FactChecker
  • #23
Vanadium 50 said:
Aren't they loud? Anway, back on topic:

The disk-less server class workstations with upgraded video cards are very quiet at idle with a proper Linux fan-control daemon. The main servers, online UPS and house networking on are an old DEC rack in the garage so noise is not a problem there.
50953366728_5a2cc93e58_b.jpg

I've mainly used physically redundant hard-drive based backup systems with limited offsite backup to sites like github for sharing ongoing projects.
 
  • #24
I got so pissed off at myself about 20 years ago for losing an important file that I wrote myself a backup program and I've improved on it some since then. I won't lose any files even if the house burns down (well, I might lose a couple of days work).

I just plug in my external drive, click on one icon and I'm done (the computer isn't done but who cares). When it finishes I swap that drive out with the same kind of drive that I keep in the trunk of my car (I used to take one to work with me but I'm retired now so "off-site" is the trunk of my car. If I'm adding a lot of files I might do that as often as every day or two but usually it's about once a week.

Every year or so I swap one of them out with a third that my son keeps (WAY off-site).

The program only backs up new files and files that have changed. Optionally, it will delete files that are no longer on the internal hard drives (it backs up various files from several drives).

None of that is much help to you, Dave, I know, but my point is that I think that if you want a reliable backup system you need at least 2 external drives (I don't use the cloud) and my preference is for 3.
 
  • Like
Likes jack action and FactChecker
  • #25
I think you can get some secure cloud backup space essentially for free. That is certainly the most secure. I don't know how much data they allow before you have to pay for more storage space. Being an old-timer, I want my backups to be clutched in my sweaty little hand. ;-)
 
  • Like
Likes nsaspook
  • #26
You can backup a lot for free. If you want to restore...that'll cost you. :wink:
 
  • Haha
  • Like
Likes Wrichik Basu and FactChecker
  • #27
I back up about about 500,000 gigs so even if it were free storage the download for a backup would be a killer.

I've written a "delta site mirror" program for my wood ID site (200,000+ JPG and HTML files) that I could expand to my whole backup but I would not be happy w/ the amount of time it would take so I use the external drives.
 
  • #28
phinds said:
I back up about about 500,000 gigs so even if
Did you really mean a half a petabyte? :))
 
  • Like
Likes russ_watters
  • #29
DaveC426913 said:
Did you really mean a half a petabyte?

Wood is serious business.
 
  • Like
  • Love
Likes DaveC426913 and nsaspook
  • #30
DaveC426913 said:
Did you really mean a half a petabyte? :))
No, I meant what I said. Half a terabyte.
 
  • #31
I think there are some extra zeros there. Half a terabyte is only 500 GB.
 
  • Like
Likes DaveC426913 and jedishrfu
  • #32
phinds said:
I back up about about 500,000 gigs so even if it were free storage the download for a backup would be a killer.

I've written a "delta site mirror" program for my wood ID site (200,000+ JPG and HTML files) that I could expand to my whole backup but I would not be happy w/ the amount of time it would take so I use the external drives.
I think that the cloud system would have to be considered the mass storage and only download the data that is in use currently. That is usually a very small percentage. I would leave the mass backups and recoveries to the cloud data administrators.
 
  • Like
Likes pbuk
  • #33
phinds said:
No, I meant what I said. Half a terabyte.
Er Nope. :wink:
 
  • #34
Vanadium 50 said:
I think there are some extra zeros there. Half a terabyte is only 500 GB.
DaveC426913 said:
Er Nope. :wink:
 

Attachments

  • egg_small.jpg
    egg_small.jpg
    1.5 KB · Views: 121
  • Like
Likes DaveC426913, FactChecker and Vanadium 50
  • #35
500GB is not too much for cloud backup, and one of the advantages of some systems is that you can restore incrementally and selectively.

If you need to rebuild your local copy (perhaps when changing main computers), select the files you want to work on right now and they will appear almost immediately. Leave it running overnight and when you get up, all or most of it will already be done - anything you need that is not there yet you can pull forward at the click of a mouse.

Of course if your internet connection is limited you will need another solution.
 
Back
Top