Data management: backups and storage

timp · 2023-11-05T21:06:26.404Z

For anyone that worries multiple HD backups or a NAS are too expensive, these are a very cheap option: Unitek docking station. (I’ve been using one with 4TB hard drives for offline storage for many years, along with NeoFinder OSX app to scan & make a searchable database of offline HDs)

US$40-us$50 plus HDs

UNITEK

HDD & SSD Docking Station

Our HDD & SSD docking station provides protection and even built-in fan on premium models. Additional functionality such as offline clone and sleep mode provides convenience to users.

hundreadP · 2023-11-05T21:23:44.540Z

Did you know that:

CERN manages the largest archive of scientific data in the high-energy physics domain and that it is constantly increasing? 
the energy consumption of tape storage is negligible compared to disk technology consumption?
the tapes are between one to three orders of magnitude more reliable than the most reliable disk drives?

For all these reasons and many more, CERN is using magnetic tapes for long-term data preservation.

The CERN Tape Archive (CTA) is the tape back-end to EOS disk. CTA went into production in June 2020 and currently stores around 400 Petabytes of physics data. During 2022, CTA will ramp up to full production data-taking volumes with the start of Run-3 of the Large Hadron Collider (LHC). CTA is an open-source system which is being evaluated and adopted by a number of scientific institutes besides CERN. This presentation will cover the outlook for archival storage and give an overview of how tape storage fits into CERN’s integrated storage strategy and the suite of storage and data transfer products/services provided by CERN’s IT Department.

Obolus · 2023-11-05T22:55:24.194Z

hundreadP:

For all these reasons and many more, CERN is using magnetic tapes for long-term data preservation.

Crazy talk, right there. Why would they even bother doing this if digital storage is so reliable?

hundreadP · 2023-11-05T23:04:47.092Z

Obolus:

even bother doing this ?

Space and time

Still, the majority of storage present comes in the form of tape archiving, albeit on a grand scale. This robotic tape housing can rapidly pick out the tape that holds specific pieces of data, and bring it to one of a small collection of read/write hardware housed in the same structure.

Cannot find that cool video when the robot picks the tape put into the machine locate file by the astrophysicist 40TB each. zipped compressed of course. But its like this images in the article.

Starthief · 2023-11-05T23:54:34.350Z

Digital data stored on tape has a lot of fault tolerance for minor errors/noise.

monopole · 2023-11-06T00:48:26.099Z

I thought we were talking at a consumer/prosumer level and cassettes.

The article and tech involved is VERY interesting, but there’s no relation between the issues CERN faces when storing data with the ones we face, specially the hardware available.

Moreover, CERN has been around since 1953, I can imagine the data center was there long before modern digital storage came into place. After a bit of research, It looks SHIFT was developed and operated in the 90s, then CASTOR I (1998 to 2007), CASTOR II (2005 to 2022) then CTA (2020) being the latest sucesor. All these projects are just the continuity from the previous ones, not full replacements:

The CTA Tape Server is an evolution from the CASTOR Tape Server and the underlying data format of files on tape is exactly the same. This means that migration from CASTOR to EOSCTA is a pure metadata operation without physical data movement.

They also use disk storage since 2011 that had 140PB vs 20PB stored on CASTOR (tape) at some point (there wasn’t any date on the paper weirdly).

But there’s this as well:

As of the end of May 2021, we had 380 PB of data on tapes.

As of the end of May 2021, 271 PB were stored on disks for a total capacity of 487 PB on disks. However, as disks are less reliable than tapes (a thousand times less, 30 disks fail each week in the data centre), we always copy twice the data on disk so that if a disk fails we do not lose data => so ‘only’ 135 PB of ‘real’ data is on disks at the moment.

It looks indeed that they took a change by gradually increasing the tape storage capacity due to costs (up to x5 less for the same capacity) and reliability agains disks. Also a crucial thing to consider for the disks failure rate is that the disk are taking extremely intensive use:

Within one year, when the LHC is running, more than one exabyte (the equivalent to 1000 petabytes) of data is being accessed (read or written).

The CERN storage system, EOS (disk), was created for the extreme LHC computing requirements. EOS in 2020 has served 2.5 exabyte of physics data to the experiments and at the end of 2019, EOS instances at CERN exceeded five billions files, matching the exceptional performances of the LHC machine and experiments.

That being true, their tape system still doesn’t compare to what we have available for tape write/read/store hardware at consumer level, so my view still holds that disks/cloud are a better option.

Happyanimal1 · 2023-11-06T05:23:49.170Z

Spent too many hours crawling through Sun storagtek tape libraries if that’s what you’re referring to.

Haven’t seen any of those for 10+ years now though.

Should be understood that cassette tape is a very far distant, 5th cousin away from how data is stored on tape in data center with LTO. Not only big differences in the physical media but the drives that RW the tape cartridges as well.

Chaos215bar2 · 2023-11-07T20:42:44.231Z

I’m seeing a lot of confusion in this thread about digital vs. analog as opposed to local vs. remote storage. These are two different discussions. (As is “physical” media vs. … something else, which isn’t really a thing since all storage is ultimately on physical media, just not necessarily in a location you control and with varying tradeoffs in terms of reliability and error tolerance/recoverability.)

CERN is most assuredly using a digital tape archive system, which is not uncommon for long-term storage.

Obolus · 2023-11-08T20:29:56.731Z

Chaos215bar2:

CERN is most assuredly using a digital tape archive system, which is not uncommon for long-term storage.

Okay. But there is a hierarchy in play here. While I would certainly prefer the easy access of the cloud, I would rank a digital archive on tape above that of the cloud (whether it’s common or not). Even if neither of them count as “analog”, tape still seems more durable and less volatile an option. There have been points made to the contrary above, and I will admit that I don’t find them very compelling.

Am I insisting that tape is “not volatile”? No. Am I ignoring the potential of those tapes being destroyed in a flood or fire? No.

Similarly, I would rank cassette tapes above compact discs. Simply for the reason that a scratch on a compact disc can corrupt all of the data. Even if the tape in a cassette brakes, gets chewed up, or warps, it can still be repaired and played back. Even if the recording is no longer (or never was) up to someone’s snuff.

Multi-head machines are still available to play them, and can be repaired without much difficulty. I come by such machines at thrift stores every few months. There are still a lot of them out there.

Despite their rumored obsolescence, cassettes continue to be relevant to subcultural music genres, and cutting edge electronic music-making. Yes, of course they are all wearing away, and facing imminent doom. *Shrug, so is our species.

Again, I’m not asserting anything about archival quality. It is simply my opinion that more tapes will outlast more digital storage devices. Digital disc and hard drive storage faces many more threats, including the vulnerability of electric grids, user neglect, disposable consumerism, company irresponsibility, world wars, and data corruption by hackers who routinely use data as a hostage.

Even the issue of theft of physical media brought up before does not dissuade me from this opinion, because that physical media will simply be sold or kept in new collections.

Data corruption on a hard drive is not comparable to theft because the loss of the art is total.

Despite all of these caustic and irreparable differences between us, I offer the following short film as my final statement on the matter:

Chaos215bar2 · 2023-11-09T03:55:47.845Z

I don’t think there’s any particular disagreement here, it’s just that there’s no “best” format. Everything is a tradeoff, it just depends what your goals are.

If we’re talking about what’s most likely to survive a hypothetical society-destroying disaster, then simpler is better. Records (e.g. vinyl) easily wins here, since playback requires only a mechanical device. You don’t even need an understanding of electromagnetism to design one. Analog tape is a safe next best option, since playback still relies on relatively simple physical principals.

Anything digital is at a disadvantage in this scenario simply because playback requires some knowledge of the format of the data. A PCM audio format encoded directly in a simple linear physical form (no error correction, no checksums, etc.) will be pretty easy for anyone capable of building a playback device to figure out. But something like an MP3 on a USB drive involves so many layers of protocols and formats that it’s going to be really difficult for anyone without that knowledge to interpret.

If we’re not assuming some mass loss of technical knowledge, digital becomes a lot more attractive. However, there are still tradeoffs.

A single copy on a CD is easy to store and play back, but it’s a terrible archival format. As you point out, CDs are easily damaged and they were never designed to be all that resilient. 3 copies of the same thing on 3 CDs, one at your house, one at a friend’s house, and one in the seed vault at Svalbard is a totally different scenario, especially if you check the CDs every 3 months and replace any one that’s beginning to degrade. You do have to actively maintain your archive now, though, which is another tradeoff.

We can still do much better. Archival tape formats support levels of redundancy that allow data to be fully recovered even if significant portions of a tape have been lost entirely due to damage. And again, we can keep multiple copies, in multiple locations. It’s worth repeating that any long term archive requires intentional management. However, if you neglect that, you’re still going to be much better off with a highly redundant digital tape format than something like a CD, which was simply never intended for this purpose.

This is where online services come into play. As an individual, you don’t have physical access to anything akin to the Global Seed Bank. But you do have access to services like Amazon Glacier, which are designed for redundant archival storage, with options to have your data housed in multiple locations distributed around the globe. The chances of Amazon unintentionally losing your data are as close to zero as a consumer could reasonably care about, and they’ll do most of the management for you.

However, any form of online storage comes with the downside that any entity with access to your account can, with a single action, cause all of your data to be deleted. This is why local, offline storage still matters. Keep a copy on a drive at home that you check once a year, plus a copy at a friend’s house if you really want to be safe, and now you have a backup scheme which admittedly requires a society with deep technical knowledge to maintain, but which will easily outlast anything you can manage with analog storage. If that’s your aim at all.

abalone · 2023-11-10T00:23:14.136Z

maybe uxn files on 3 cassettes

IMG_86051284×2778 129 KB

en.m.wikipedia.org

List of cassette tape data storage formats

Many early microcomputer and home computer systems used cassette tapes as an inexpensive magnetic tape data storage system. This article lists some of the historically notable formats. As interoperability between platforms was difficult, there was little purpose to, or effort expended on, using standardized formats. The main exception to this rule was the Kansas City standard, which was supported by most S-100 bus based computers and was later adopted by a few other vendors like the BBC Computer...

idioteque · 2023-11-18T09:08:52.884Z

As others already suggested, I also highly recommend Carbon Copy Cloner for those who are on MacOS.

I personally came to this (very modest) setup over the years:

I store everything I need to access for my work-in-progress projects in my 2Tb laptop internal SSD
I keep a copy of that SSD in an external 5Tb classic LaCie Drive. The backup is automated with Carbon Copy Cloner on a daily base - it’s a “setup and forget” thing.
I have two other identical LaCie external 10Tb drives where I store on each the same copy of my archives (past projects, my full samples personal collection, photos…).
I store on iCloud Drive all my “administrative paperwork”, and in general all my administrative stuff - personal and professional.

I also tend to clean up my stuff from times to times, to free space by periodically deleting boring / dusty / unnecessary / uninteresting old stuff.

It works pretty well for me.

monopole · 2024-05-22T09:53:55.909Z

I’m ready to take my data management more seriously and I think this setup could work:
10TB HDD for long term storage, 2TB M.2 NVMe for more accesible storage and side loading of apps and libraries and a dock for both. If I want to expand on the future, I could use the same drives and get another dock so it looks like the better choice.
I’m aware that the read/write speeds for the M.2 will be around 1GB/s through USB, that’s why I’m going for the 3500MB/s version and not the 5000MB/s.

I’m sharing in case someone thinks that another approach could be worth considering.

screencap1920×1365 138 KB

mnm207 · 2024-05-22T14:06:50.285Z

I do something similar but use Samsung T5 and T7 drives for my working files and then archive projects to a pair of hard drives that are accessed through an OWC dock which mounts two hard drives simultaneously. I prefer standalone SSDs and have a dozen or so of the Samsungs which are small and convenient and have proven reliable. Keep in mind one HDD isn’t a secure long term storage solution because drives fail. Also, note that a dock or drive enclosure is another point of failure (this is why I prefer the standalone SSDs). I’ve had both HDDs and enclosures/docks fail. So far so good on SSDs (fingers crossed). My method is not the platonic ideal, I know, but it’s a workable solution for me.

Chaos215bar2 · 2024-05-22T15:17:39.701Z

mnm207:

Keep in mind one HDD isn’t a secure long term storage solution because drives fail.

This. These are good solutions, but if you care about your data, you need a backup plan or it could all be gone tomorrow.

If you really want to take this seriously, keep three copies. One easily accessible to work on. One at home, but “offline”. And one remote; could be cold storage in a datacenter somewhere, or just a drive at a friend’s house. Depends how much you care about your data, of course.

illumina · 2024-05-22T22:59:10.148Z

monopole:

10TB HDD for long term storage

A single platter storage drive is not really good for long-term storage for a lot of reasons.

Now, if you use something like https://bvckup2.com/ to sync the data to a google workspace account with drive space, that’d be a small amount each month but more reliable in case of emergency!

monopole · 2024-05-23T09:13:29.983Z

This is interesting, unfortunately it’s only for Windows. I believe Carbon Copy will be able to do the same, ie, periodic backups.

I don’t discard to get more disks later on or use cloud services as many have suggested, however the above it’s an addition to an existing 1TB external SSD + keeping important stuff in the work computer itself, so I’ll have 4 physical copies soon.

robbie · 2024-09-13T09:54:05.432Z

Mixonline – 19 Aug 24

Inside Iron Mountain: It’s Time to Talk About Hard Drives - Mixonline

By Steve Harvey. Iron Mountain sounds the alarm: Aging tracks created through an all-digital workflow aren't guaranteed to play back.

Est. reading time: 8 minutes

Of the thousands and thousands of archived hard disk drives from the 1990s that clients ask the company to work on, around one-fifth are unreadable. Iron Mountain has a broad customer base, but if you focus strictly on the music business, says Robert Koszela, Global Director Studio Growth and Strategic Initiatives, “That means there are historic sessions from the early to mid-’90s that are dying.”

The regular public service announcement: If you care about it, back it up – multiple locations, multiple mediums.

Jendrik · 2024-09-13T10:43:32.173Z

Even if you have multiple drives your bios can fail or an update can break it. Friend of mine is still in shambles but its also an IDE drive no idea how I could back this up. Probably dead anyway.

Obolus · 2024-09-13T15:35:56.462Z

Print, or perish.

(Twenty characters remix)