Anyone that's never had the (dis)pleasure of working with tapes, there's some gotchas to know about before archiving and forgetting about it.
When you create an archive, do not skip the step of reading it back. I'm not talking about just having the system confirm it is readable. I mean actually restoring the data from the tape. I've had tapes fail to retrieve data off of the tape even though the writing app confirmed it was readable.
Drives can have issues reading data written by another drive. Drives can have issues reading a tape it wrote itself.
Why write one when you can write two at twice the time/cost? In all of my experience with tape, I've never written just one tape. In the early days of DVDs, the images were written to LTO to send to have the discs pressed. Two were always sent on the chance one wasn't readable and schedules were so tight that the loss of a couple of days to send a new tape would wreck delivery dates. Another facility wrote three tapes: one kept on site, one sent to IronMountain, one sent to the client directly. Plenty of occasions saw the local copy unreadable and the IronMountain copy was retrieved. These were managed by large robotic tape libraries so each tape was written in a different drive at the same time so it was 3x cost while only 1x time.
Use the drive cleaning cassette as recommended. Your dirty heads can cause the data to not be written/read correctly depending on which head is dirty. If you're having issues reading, clean the heads before giving up on the tape. If you're having issues reading a tape just written, do not think someone else will be able to read it.
Ensure whoever might be using a tape you've written understands what was used to write the tape. Not all archivers are compatible. If you use the tar command to write to tape, people using a GUI app won't be able to read it.
I have worked with 5 different LTO tapes versions, 5 different other cassette formats (DAT, DDS, etc), and open spools… never, ever, ever in the 15 years had ANY problem (I’ve probably used 1000 of them). We did cycle them very often, we did have proper maintenance of the drives, through. Now that, compared with RAID Disk arrays, where we had monthly failures, was much better. Also I would not trust the „semi-permanent“ NVMEs, where after 1 year without power they get basically reset
Is there some reason that the data is not written to a section of tape using one head, and then read off of that same tape 1 inch (hundreds of milliseconds? tens?) later using a separate head for verification, before it's tucked back into the spool?
> Since the analog nature of recording the digital information to tape is subject to recording errors, LTO Tape drives also rely on error correction circuitry to constantly monitor the accuracy of the data put on tape. One clever feature of LTO heads is that they verify data as it's being written. Remember those 32 write elements and 32 read elements? The read elements are positioned just behind the write elements. As soon as data gets written to the tape, the read elements immediately check it.
I think the issue arises if the tape is out of alignment - ie. too high or too low. Then it writes in the wrong spot, reads in the wrong spot, so you think all is good...
But when you try to read it back with a different, correctly aligned machine, it is unreadable.
The big cloud providers use something like erasure coding across a collection of tapes (or hard drives) to preserve information. If you've ever downloaded something from usenet and restored it with quickpar, this is basically the same idea. Every physical cartridge gets a chunk of the file. Depending on parity settings you would have to lose many tapes distributed across an arbitrary # of locations to actually lose any data.
Note that the hardware the big cloud providers use for tape archives is very different from the devices the GP poster appears to have used, and usually have robust verification systems.
In addition the need for cloud providers to comply with userdata erasure means they have to frequently test their ability to read and write their backups - which should increase your confidence in their correctness. Yes problems can and do still happen, but IME far less often than the more traditional tape setups GP describes.
Erasure can be done much more efficiently with crypto erasure, where you encrypt the user data and just delete the key when users request deletion. This way you don't need to spend expensive tape vault / library time rewriting a tape.
Symantec Backup Exec was genuinely one of the worst pieces of software I have ever had to use. Later in life I got to use the IBM Storage Archive setup with much larger tape libraries which still sucked but my god the level of competency difference between the two applications was insane.
The cost of tape is a big deal for for some bigger science experiments.
They often cost billions to build and they take loads of data, but most of it ends up heavily distilled by the time it goes to the final analysis pipeline, so keeping the raw data on tape is pretty common.
There was a time when tape was more specialized and we were worried about loosing our limited suppliers. The explosion in the ML that powers search and more recently the AI boom makes the technology look a lot more stable these days, which is a relief for us.
Tape drives are really picky compared to HDDs: they don't like it when the environment is too hot, or too cold, or too humid, or too dry, or too dusty, or with too many vibrations, etc. They're really only meant for datacenters with controlled environments. A prosumer version would need additional engineering for environmental tolerance that the enterprise ones don't have, and end up costing more, not less, especially with the tiny markets involved.
One thing worth noting is that the tape cartridges themselves aren't necessarily the biggest expense. Standalone LTO-10 drives are currently listed at around $25,000.
> One thing worth noting is that the tape cartridges themselves aren't necessarily the biggest expense. Standalone LTO-10 drives are currently listed at around $25,000.
The drives are a fairly fixed CapEx: once they're purchased they don't cost must/anything to run. You may replace them every few years with newer tech for higher capacities of course.
Libraries/robotics last even longer, and can stick around for a decade or more once purchased.
You don't need a $25,000 LTO-10 drive, $100,000 library, and $300 tapes. The drives are frequently backwards compatible multiple generations, so your archives are safe. You can get LTO-7, LTO-8, and LTO-9 libraries for cheap (right now there are lots of LTO-7 standalone drives for under $1000 on ebay), as they've aged out of enterprise leases.
You got me briefly excited, but the only LTO-7 drives on eBay under $1000 are listed as parts-only. The price jump from LTO-6 to LTO-7 has remained steep for years; either industry is holding on to them, or they're being resold or released successfully.
I got a brand new LTO-7 standalone drive off eBay for under $1000 around 6 or 8 months ago. I must have just made it before more people started moving to tape.
That's got worse recently. LTO used to be guaranteed compatible two generations back; an LTO-7 drive can read LTO-6 and LTO-5 tape. With newer standards, they've cut that to one generation. Your point stands, but folks should be aware the treadmill has been sped up in the past few years.
Yes, drives are very expensive and that's why there is a business for renting them. Even Amazon Snowball offered that [1]. There are other rental services such as [2] but many of them are not LTO-10 yet (don't know if it really matters right now).
Tape is a fascinating technology, I recall reading the account of the SONY cofounder https://en.wikipedia.org/wiki/Made_in_Japan:_Akio_Morita_and...
and the crazy things they were trying to make their own tape media because they had no supply because of the war. They were ultimately successful in creating a compact tape machine but they had to build up their expertise in tape media from literally nothing.
There might be a huge potential for an AI which scrapes your (and my!) data and concludes what I can safely delete, saving the world tons of resources.
The millions of pictures I never got to sort, the multiple versions of a file that I wrote 2 years ago, etc.
But in the view of an AI-company that's probably a paradox. All this data is charged and could be used to learn more about me, it should never be deleted! Instead, some hole should be carved out of a mountain to store it, billing me every month!
> There might be a huge potential for an AI which scrapes your (and my!) data and concludes what I can safely delete, saving the world tons of resources.
Thinking...
User has 15 pictures of his dead mother, deleting 14.
Why use AI when duplicate finders have been a thing for decades?
For the pictures, I set up a batch script the one time that used exiftool to read all of my photo's exif data (for creation date), rename the file to that date / time, and put it in a year/month folder structure. Once that's normalized it was pretty straightforward to identify and delete duplicates (assuming the exif data on duplicates has the same date/time in it).
"Fujifilm highlights the ability to air-gap tape cartridges as a last defense for user data: “Given the sophistication of cyberattacks, magnetic tape is attracting increasing attention as a means of protecting data from threats such as ransomware.”"
I mean, it's plausible, yet I find it hard to believe this solely drives demand 5 times. Because if it does then what, companies were not using cold storage backups before?
> I mean, it's plausible, yet I find it hard to believe this solely drives demand 5 times. Because if it does then what, companies were not using cold storage backups before?
As someone in HPC: people never delete stuff. It can stick around forever, and so if more bits are generated, there are more bits that people don't want to delete "just in case". Depending on the organization, quota enforcement can be tighter or looser.
HSM is fairly popular in the HPC space for this reason:
Your newest/hottest bits are on NVMe, anything older than (e.g.) 3 months gets move to SSD, anything older than that to spinning rust HDDs, and then to tape.
There are several file systems where this happens in the background: the file still shows up at /path/for/stuff/foo.txt but the contents are shuffled about behind the scenes.
At least they have a process. That gives you a chance. Often the process itself is good - you didn't test it but the vendors all did and so the data is there - it might be hard to restore, but it can be done by experts after the fact. The drill means you can recover in a "short time" instead of a "long time" (short time can range from minutes to days, while long time days to years)
I've ran drills with companies which concluded that the incremental backups couldn't be restored due to some data corruption issue (no readback test was ever done).
Luckily the company also did a full backup every 6 months, so a disaster would have only thrown them back 6 months at max! /s ;)
But yes, some issues were even more simple: I had a case where performing a restore required twice the storage size of the uncompressed data, and the company actually didn't have the infrastructure ready to do that in case of a disaster...
I'll bet that if it really was important you could pay someone (LLMs make this easier) to dig through the data byte by byte and get everything back despite the corruption. It would take years, and be very expensive, but typically there is enough redundancy to figure it out. Often you only really need a small subset of the data and can live without the rest.
Don't get me wrong, drills are important! You should do them. However backups without a restore drill better than no backups at all. Even if you only get the last full backup, the data from the last 6 months will be the easiest to recreate from scratch if you must.
it's fascinating how the general direction became to never delete data. All the attempts of taking a photo, all the downloads of bank-statements, all the useless emails.
At some point in the future the amount of data being stored indiscriminately must consume such a huge amount of resources and cost, that it becomes a hidden fixed cost within a country's GDP, without any tangible benefit...
Historians actually would love this. Currently many historians try to reconstruct how life was like centuries ago from accidentally preserved artifacts. Future historians would have no trouble understanding how our daily life is like.
don't we inflate this problem away? storage should get more abundant and cheaper and the old media looks comparitevly small compared to higher fidelity and higher definition formats that come later. so it doesn't matter if your gmail stores 20 years of useless emails.
That continues to be the assumption, yes. Disregarding all the waste being created in the process.
What's fascinating is the fact that the production of TAPE sees a 5x boost. That's the format with the slowest retrieval rate, only making sense for long-term storage, usually 1+ years.
So is there really 5x more valuable data now, or is it just not economic to actually judge the need for preserving data...?
Fujifilm is pretty clearly trying to sell shovels in this market bubble. I haven't read the latest, but I would suspect their clientele has moved from cloud backups to AI training data, and they need to make a big show and dance of ramping production to join the bubble.
Alas, M-Disks are hugely more expensive than magnetic tapes, even when not taking into account their much greater volume.
The biggest M-Disks are of 100 GB and the cheapest that I see on Amazon (in 25-piece sets) are at about $9 without taxes and shipping. At that price, I can buy not 0.1 TB, but 2 TB of magnetic tapes, which also have a much lower volume, at 18 TB per tape for the older and cheaper LTO-9 cartridges. I can easily carry with me a suitcase with 360 TB of data (20 LTO-9 cartridges), or store it in a safe place. Carrying 3600 M-Disks or storing them in my home would be impossible.
M-Disks may be used only to archive some small amounts of more valuable data, e.g. documents or programs that you have written yourself, a few scanned legal documents, a few more important books, a small amount of music that you prefer, a few family photographs.
For bulk data, if you cannot afford the cost of tape drives, the only alternative is HDDs, but with HDDs you must take care to always use both duplicate HDDs and archive files with redundancy, which allow the regeneration of their content when any not too large part of the file becomes corrupted. You must also take care to migrate the data periodically to new HDDs, after a few years.
Due to their small size I have not tried to use M-Disks until now, so I do not know if my older Blu-ray recorder can write them. There is also the problem that the availability in the future of Blu-ray recorders and readers is uncertain.
> but with HDDs you must take care to always use both duplicate HDDs and archive files with redundancy
I'm a noob when it comes to tape backups. Is copying the same stuff to multiple tapes standard practice?
> There is also the problem that the availability in the future of Blu-ray recorders and readers is uncertain.
LTO drives used to read tapes from 2 generations back, then just 1. LTO 10 isn't backwards compatible. Future availability of compatible tape drives is also uncertain.
Maybe. For the average person here though they will already have and want a blu-ray drive for "reasons", and so using it for backup is very cheap. LTO-5 might be cheap, but they still need to buy a drive that they won't use for anything else.
If you are a business of course tape is cheap. Even then, a few usb harddrives are cheap as well and much faster.
M-Disks are not cheaper than tape for bulk data storage. Once you're doing enough of this that you're not limited by the cost of the drives, it's about a factor of 25: $150/TB for M-disk vs $6/TB for tape.
Which LTO are you comparing with? Locally (Spain) it seems I get similar EUR/GB for BlueRay M-Disks as LTO-8, but LTO-7 and LTO-6 is way cheaper than BD. What exactly are you comparing here?
When you create an archive, do not skip the step of reading it back. I'm not talking about just having the system confirm it is readable. I mean actually restoring the data from the tape. I've had tapes fail to retrieve data off of the tape even though the writing app confirmed it was readable.
Drives can have issues reading data written by another drive. Drives can have issues reading a tape it wrote itself.
Why write one when you can write two at twice the time/cost? In all of my experience with tape, I've never written just one tape. In the early days of DVDs, the images were written to LTO to send to have the discs pressed. Two were always sent on the chance one wasn't readable and schedules were so tight that the loss of a couple of days to send a new tape would wreck delivery dates. Another facility wrote three tapes: one kept on site, one sent to IronMountain, one sent to the client directly. Plenty of occasions saw the local copy unreadable and the IronMountain copy was retrieved. These were managed by large robotic tape libraries so each tape was written in a different drive at the same time so it was 3x cost while only 1x time.
Use the drive cleaning cassette as recommended. Your dirty heads can cause the data to not be written/read correctly depending on which head is dirty. If you're having issues reading, clean the heads before giving up on the tape. If you're having issues reading a tape just written, do not think someone else will be able to read it.
Ensure whoever might be using a tape you've written understands what was used to write the tape. Not all archivers are compatible. If you use the tar command to write to tape, people using a GUI app won't be able to read it.
* https://magstor.com/blogs/magstor-faqs-page/how-does-an-lto-...
But when you try to read it back with a different, correctly aligned machine, it is unreadable.
In addition the need for cloud providers to comply with userdata erasure means they have to frequently test their ability to read and write their backups - which should increase your confidence in their correctness. Yes problems can and do still happen, but IME far less often than the more traditional tape setups GP describes.
https://news.ycombinator.com/newsguidelines.html
They often cost billions to build and they take loads of data, but most of it ends up heavily distilled by the time it goes to the final analysis pipeline, so keeping the raw data on tape is pretty common.
There was a time when tape was more specialized and we were worried about loosing our limited suppliers. The explosion in the ML that powers search and more recently the AI boom makes the technology look a lot more stable these days, which is a relief for us.
LTO-10 30TB: $259–$300 per cartridge (~$8.63–$10/TB) LTO-10 40TB: $491–$555 per cartridge (~$12.28–$13.88/TB) LTO-9 18TB: $92–$129 per cartridge (~$5.14–$7.17/TB)
https://tapebackup.org/lto-tape-price-trend/lto10-price
One thing worth noting is that the tape cartridges themselves aren't necessarily the biggest expense. Standalone LTO-10 drives are currently listed at around $25,000.
The drives are a fairly fixed CapEx: once they're purchased they don't cost must/anything to run. You may replace them every few years with newer tech for higher capacities of course.
Libraries/robotics last even longer, and can stick around for a decade or more once purchased.
But the fact that -ALL- LTO drive-mechanisms are build by IBM maybe also doesn't help the pricing...
[1] https://docs.aws.amazon.com/snowball/latest/developer-guide/...
[2] https://mediastoragegroup.com/
The millions of pictures I never got to sort, the multiple versions of a file that I wrote 2 years ago, etc.
But in the view of an AI-company that's probably a paradox. All this data is charged and could be used to learn more about me, it should never be deleted! Instead, some hole should be carved out of a mountain to store it, billing me every month!
Thinking...
User has 15 pictures of his dead mother, deleting 14.
Saved 20 MB! :)
For the pictures, I set up a batch script the one time that used exiftool to read all of my photo's exif data (for creation date), rename the file to that date / time, and put it in a year/month folder structure. Once that's normalized it was pretty straightforward to identify and delete duplicates (assuming the exif data on duplicates has the same date/time in it).
Duplicates are easy, that gets sorted quite fast. Picking 1-2 out of those 5 slightly different pictures of the same occasion is another task.
Shipping your drives to Sam Altman. If everybody did that, he could then deduplicate the pictures of the same thing that multiple people took.
I mean, it's plausible, yet I find it hard to believe this solely drives demand 5 times. Because if it does then what, companies were not using cold storage backups before?
As someone in HPC: people never delete stuff. It can stick around forever, and so if more bits are generated, there are more bits that people don't want to delete "just in case". Depending on the organization, quota enforcement can be tighter or looser.
HSM is fairly popular in the HPC space for this reason:
* https://en.wikipedia.org/wiki/Hierarchical_storage_managemen...
Your newest/hottest bits are on NVMe, anything older than (e.g.) 3 months gets move to SSD, anything older than that to spinning rust HDDs, and then to tape.
There are several file systems where this happens in the background: the file still shows up at /path/for/stuff/foo.txt but the contents are shuffled about behind the scenes.
What I observed in the rising cybersecurity attacks happening around me:
(i) There are organizations thinking that they have [complete] backups and being aware they don't once the cyberattack if effective.
(ii) Even in organizations with complete backups they don't have a [good] process to recover them, and the operations suffered a lot.
Corollary: backups are, in general, not battle tested.
Even within the same company. Every division has backup processes, only few actually try to recover the data once in a while...
I've ran drills with companies which concluded that the incremental backups couldn't be restored due to some data corruption issue (no readback test was ever done).
Luckily the company also did a full backup every 6 months, so a disaster would have only thrown them back 6 months at max! /s ;)
But yes, some issues were even more simple: I had a case where performing a restore required twice the storage size of the uncompressed data, and the company actually didn't have the infrastructure ready to do that in case of a disaster...
Don't get me wrong, drills are important! You should do them. However backups without a restore drill better than no backups at all. Even if you only get the last full backup, the data from the last 6 months will be the easiest to recreate from scratch if you must.
I'm suspecting the AI companies simply save everything (even if you said "no"), just in case they need it later for training.
Why? Because there is no moral anymore. They blatantly ignored copyright after all.
As time goes by, people are sending more and more. It started with just prompts, then the local source code, then calendar, emails, local files...
At some point in the future the amount of data being stored indiscriminately must consume such a huge amount of resources and cost, that it becomes a hidden fixed cost within a country's GDP, without any tangible benefit...
That continues to be the assumption, yes. Disregarding all the waste being created in the process.
What's fascinating is the fact that the production of TAPE sees a 5x boost. That's the format with the slowest retrieval rate, only making sense for long-term storage, usually 1+ years.
So is there really 5x more valuable data now, or is it just not economic to actually judge the need for preserving data...?
Alas, M-Disks are hugely more expensive than magnetic tapes, even when not taking into account their much greater volume.
The biggest M-Disks are of 100 GB and the cheapest that I see on Amazon (in 25-piece sets) are at about $9 without taxes and shipping. At that price, I can buy not 0.1 TB, but 2 TB of magnetic tapes, which also have a much lower volume, at 18 TB per tape for the older and cheaper LTO-9 cartridges. I can easily carry with me a suitcase with 360 TB of data (20 LTO-9 cartridges), or store it in a safe place. Carrying 3600 M-Disks or storing them in my home would be impossible.
M-Disks may be used only to archive some small amounts of more valuable data, e.g. documents or programs that you have written yourself, a few scanned legal documents, a few more important books, a small amount of music that you prefer, a few family photographs.
For bulk data, if you cannot afford the cost of tape drives, the only alternative is HDDs, but with HDDs you must take care to always use both duplicate HDDs and archive files with redundancy, which allow the regeneration of their content when any not too large part of the file becomes corrupted. You must also take care to migrate the data periodically to new HDDs, after a few years.
Due to their small size I have not tried to use M-Disks until now, so I do not know if my older Blu-ray recorder can write them. There is also the problem that the availability in the future of Blu-ray recorders and readers is uncertain.
I'm a noob when it comes to tape backups. Is copying the same stuff to multiple tapes standard practice?
> There is also the problem that the availability in the future of Blu-ray recorders and readers is uncertain.
LTO drives used to read tapes from 2 generations back, then just 1. LTO 10 isn't backwards compatible. Future availability of compatible tape drives is also uncertain.
If you are a business of course tape is cheap. Even then, a few usb harddrives are cheap as well and much faster.
https://blogs.intellique.com/tech/2022/01/27#TapeCLI
That's what they keep telling everyone else for years now...
That's where all my random pictures, duplicate bank statements and savegames will go. For the benefit of mankind! (and a small monthly fee)