LTO ARCHIVE
The below article is reproduced in this blog for my readers. I don’t have any copyright for this post and for wider reach reproduced this article. Thanks Creativecow.net & Bob Zelin for this wonderful article. Original article link https://library.creativecow.net/zelin_bob/LTO-LTFS-Archiving/1
 |
Bob Zelin Orlando Florida USA©2015 CreativeCOW.net. All rights reserved. |
Now that videotape and audio tape are going away, the media storage system being used by most audio and video professional is disk drives. Unfortunately, as most of us know, spinning disk drives are not very stable, and ultimately fail after a certain period of time. Once a disk drive no longer is willing to mount on our desktops, we can no longer get back the critical audio or video media that we need, usually sending us into a panic.
So most people’s concern today is, “How do I safely back up my media?”
The cheapest solution is to purchase a cheap drive dock, where you can slide in inexpensive single disk drives, and back up your critical media. Move your media from one set of drives to the other every few years for maximum safety.
Another more expensive solution is to buy a redundant drive array. If you have an 8 bay or 16 bay RAID array, you buy another one, and use a cloning program (Carbon Copy Cloner, Super Duper, Chrono Sync, Shot Put Pro, etc.) to back up your media.
An even MORE expensive solution is to build what is called “near line storage,” which is another entire server/disk drive system that backs up everything from your system, and can be called back to your main system. It’s a little slower, which is why it’s called “near” line instead of o line, but everything is available across your network when you need to recall it.
No matter what solution you choose, these choices are all based on DRIVES, and drives ultimately fail. It’s not a matter of “if” – it’s a matter of when. Sometimes drives can last for 5 – 6 years, other times, they will fail within the year. That does not give you a lot of confidence in the long term.
When they were first being introduced, Blu-ray disks sounded promising. With the huge amount of data storage that video professionals use, this proven to not be practical – and those too will fail eventually.
And of course, there is always “The Cloud” (Dropbox, Google Drive, Apple iCloud, etc.). But for those of us dealing with terabytes of storage, this is not very practical, due to the very limited speed of the internet today.
LTO (LINEAR TAPE-OPEN)
So, we are stuck, once again, with tape. Not video tape, or audio tape, but LTO tape (Linear Tape-Open) which is used for backing up – or more accurately stated – archiving your critical media. You simply copy your disk drive media onto the LTO tape, and now you no longer have to worry if someone drops your disk drive on the floor, because tapes won’t break when they are dropped or mishandled. Typical LTO tapes are expected to last 15 years.
The big question is, how much space are we talking about on a single tape?
LTO 4 holds 800 GB of data.
LTO 5 holds 1.5 TB of data.
LTO 6 holds 2.5 TB of data.
LTO 7 will be released in 2016, and will hold 6.4 TB of data.
Of course, the “good for 30 years” is overstated. The tapes themselves will last that long, but most modern drives will only read LTO 5 and LTO 6 tapes. You’re still going to need to pay attention to migrating your data, or plan on keeping old drives around to read your older tapes.
LTO ARCHIVING
LTO is true archiving. It’s safe. Many people have experienced the nightmare of losing the data that is on their disk drives at some point, and the safety that LTO offers is very appealing.
But the emphasis is on safety, not speed – and working with LTO slow. It’s too slow of a process to say “Quick! Grab that LTO tape, and pop it into the LTO reader! We have a client coming in 10 minutes!” LTO is an archive medium. It is not fast like a drive, and it’s not readily accessible like near line storage, or a redundant RAID array.
Once people started using more and more media with fewer safe places to keep it, everyone started to ask, “What LTO solution should I buy?” And there were lots of choices. But most were expensive, and many were complicated to use.
To compound this, many solutions were proprietary, which means that it would work wonderfully on that manufacturer’s system, but if you had to send your LTO tape to a client, or a TV station that wanted the LTO tape, they couldn’t read it, because it was in a format that was not compatible with their LTO system.
And so the demand for a standard for LTO tapes was created, and this is called LTFS, or Linear Tape File System.
WHAT IS LTFS?
 |
LTFS is a standard developed by IBM, HP, and Quantum, and is an open source standard that has some workflow advantages over previous solutions. Rather than having to copy archived data back on to your computer, LTFS archives can be mounted as disk drives like any others. AND you can simply click on the folder structure, to see your files with an LTFS formatted LTO tape, just like a hard drive.
But it is CRAZY SLOW. If you actually double click on one of the files to play it, well – it will try to play it, and you get a spinning beach ball, and 5 minutes later you see a head shot of the file because you just can’t “play” video off an LTO tape.
So it gives the impression of a drive, and you can “drag and drop” files, just like a drive, but of course, it’s all in SLOW MOTION, because when you copy a file from your desktop to a USB flash drive, it takes 10 seconds. With LTFS LTO, the same process takes 10 minutes or longer.
But imagine being able to say “I have this videotape, and I want it to mount in a 3/4″ deck, Beta VTR, DVCPro VTR, DVCam VTR, HDCam VTR, Digi Beta VTR – I don’t care, I just want it to mount no matter what” – that is LTFS.
(For more information about the Ultrium consortium of LTDA developers and partners, go to http://www.lto.org.)
WAIT: WHAT ARE LTO AND LTFS AGAIN?
To keep this straight: LTO is a tape. Just like SATA is a drive or SCSI is a drive. You can format a drive for HTFS+ or NTFS. So you can say “lets take this SATA drive and format it for Mac as HTFS+”.
Same with tape. You have an LTO tape. How do you want to format it? The only universal format that lets you pop in an LTO tape into anyone’s system (that supports the universal format) is LTFS.
TOLIS GROUP BRU
TOLIS Group has been around for 30 years, and their BRU Backup & Restore Technology was among the first widely available archiving software. Tolis also sells integrated hardware-software solutions that have millions of clients. TOLIS Group products are rock solid, and very well supported – including here at Creative COW, where Tolis is very active.
TOLIS Group’s ArGest Studio Rack and ArGest Mobile Cube
Because TOLIS Group was in business 24 years before LTFS technology was first demonstrated at NAB 2009, the solution they developed is proprietary. BRU archives can only be written and read by Tolis products.
Note that TOLIS Group BRU provides some basic LTFS functions for customers who rely on it, but it is by no means the core of what Tolis has been doing for 30 years. By starting as open source and trying to work with as many systems as possible, LTFS has some disadvantages, including slow speeds relative to Tolis solutions, and the inability to span volumes, among others. Tolis outlines those here.
That’s the thing. “Proprietary” gets treated as a bad word sometimes, but probably everything that matters most to you is proprietary. Apple Pro Res, Avid DNxHR, .r3d, Panasonic AVC-Intra, Adobe After Effects project files, your favorite iOS and Android apps – all of these and many more: proprietary.
And they all work great. That is because those proprietary pieces are specifically designed to work together. Tolis has simply developed effective LTO solutions that predate LTFS.
CHOICES
To say it one more time: you are not choosing between LTO and LTFS. All of the companies in this article write data onto LTO tapes. Tolis writes to its own format, and the others write to LTFS.
So which would I choose? The answer: I would not choose either. I don’t care. My clients do the choosing.
It’s the same way that I don’t care about RED, ARRI Alexa, or Sony F55. I don’t care about Apple Pro Res or Avid DNxHR. They all do the job. It only matters what the client wants.
If you are looking for a rock-solid, long-proven technology for yourself and do not need to share archives with clients, Tolis BRU is fantastic. If you work with multiple clients, and they are all using Tolis products, you are all set.
But if you work with multiple clients who have chosen a variety of systems from a variety of manufacturers, and they ASK you to look at LTFS solutions, then you should be looking at LTFS solutions. It’s all up to the client.
My clients asked me to look into the cheapest-possible LTFS options from other companies, so I that is what I have done.
CHEAP LTO ARCHIVING?
Don’t get sucked into the “LTFS is cheap because the software is free and open source” hype. The fact is that no LTFS software that works, and is well-supported, is cheap. The software that works starts at around $499.
The only one that is free is Hewlett Packard StoreOpen, and that package is a nightmare. It is ridiculously slow, even by LTO standards. It is painful to use, and has no support. Don’t use it. You will try to make it work, and you will fail, and you will come to Creative COW for advice, and the advice will be, stop trying to use unsupported software for mission-critical work.
HP StoreOpen and Linear Tape File System (LTFS) Software
However, my clients were very clear to me. They did not want a rundown of everything out there. They asked me to find the CHEAPEST systems that actually work, that writes LTO tapes in the LTFS format, that have capable hardware and fully-supported software.
Here is what I found.
THUNDERBOLT AND LTO
With the release of Apple’s new Thunderbolt computers, everyone wanted Thunderbolt everything. Of course that meant Thunderbolt disk drives, but it also meant Thunderbolt LTO tape drives. And this demand led to the release of themLogic mTape, which became the first commercially available LTO drive that had a Thunderbolt interface.
mLogic mTape Desktop Thunderbolt LTO6
There were lots of software programs that worked with the mLogic mTape, but “everyone” wanted the LTFS standard.
Then along came Imagine Products “PreRollPost”, which was an easy to use LTO program that ran on Apple computers, and that used the LTFS standard to create the LTO tapes. So it became the perfect combination of Apple, LTO, and LTFS.
Preview of the upcoming Windows release of Imagine Products PreRollPost
(Note that Thunderbolt is not exclusive to LTFS. Don’t forget: LTO is the kind of tape. While I have been talking about manufacturers who format LTO tapes as LTFS, TOLIS Group does in fact have the TOLIS ArGest Thunderbolt product line that writes LTO tapes to their own format.)
LTO TAPE LIBRARIES
Most people don’t want to spend the money for a tape library, so they will make single tapes – no different than you would have Sony Beta tapes in the past. Many clients have one LTO tape per show, or one tape per project. This is what I have dealt with mostly in the past, particularly with Imagine Products PreRoll Post, and the mLogic mTape.
And before this, the same single tape process with Cache-A (bought by ProMAX and renamed Pro-Cache) or Tolis BRU and a single LTO tape from from Hewlett Packard. No different than you would have Sony Beta tapes in the past. One LTO tape per show, or one tape per project. This is what I have dealt with mostly in the past, particularly with Imagine Products PreRoll Post, and the mLogic mTape.
As storage grows, and drives continue to fail as time goes on, more and more people are getting nervous, and are asking about larger LTO tape solutions.
In the past, these solutions have been very expensive. But as the entire video industry changed, and low cost video products came out, the exact same thing is happening to “enterprise level computer solutions.” LTO tape libraries are one such solution.
LTO Tape Libraries are large enclosures that hold multiple LTO tapes at one time, just like large RAID arrays hold multiple disk drives at one time. Hewlett Packard, and Quantum, and Tanberg made these large LTO Tape Drive Libraries, and most of the established LTO software companies had solutions that incorporated these products, but they were expensive.
Even if somebody was had the budget to go the LTO tape library route, they were complicated to use, and required a lot of training.
Wasn’t there an easy solution?
HIGHPOINT TECHNOLOGY
At NAB 2015, Highpoint Technology continued to release their ever expanding line of Thunderbolt to anything that you can think of adaptors. One of their little products was a new box called the RocketStor 6328, which was an adaptor that converted Thunderbolt to MiniSAS. Highpoint had been aggressively working on getting their product to be compatible with LTO tape drive products.
Highpoint Technology RocketStor 6328
But I was already happy with the mLogic mTape, which was an LTO drive that already had a built-in Thunderbolt port, so no need for any adaptor boxes. But of course, there was no LTO Tape Library product like this.
Highpoint suggested that I try their product with an LTO tape library. The problem was that most LTO libraries were expensive, and that most “solutions” that existed for LTO libraries were expensive. But thanks to wonderful trade shows like the NAB show, you can wander around and find companies that you may have never heard of, that have these amazing products that do exactly what you want.
QUALSTAR
I found two such companies. The first was Qualstar. They made LTO libraries, for a fraction of the price of other options. Qualstar was actually a very large established company, that had the “big expensive stuff, but the Q24 was an “entry level” product that looked like it would suit my requirements.
Qualstar Q24
So the Highpoint 6328 would plug into a Mac or Windows computer, and the other side of the Highpoint would plug into the miniSAS connector on the Qualstar Q24. Now, I needed software to get all this to work.
I contacted Imagine Products again, but their wonderful PreRoll Post software would not support an LTO tape library.
YOYOTTA
As I walked through the aisles of NAB, I remember seeing a little company called YoYotta, who made LTO software that supported LTFS. They informed me about their YoYotta ID Automation software, which not only supported LTFS, but was designed to operate with an LTO Tape Library. And they offered a free trial download and training videos on their website, to see how to use their product.
I had my combination of products. My little Thunderbolt Mac Mini, my Highpoint 6328 Thunderbolt to miniSAS adaptor, my Qualstar Q24 LTO tape library, and the YoYotta ID Automation software.
YoYotta software ID Automation
There was not much to the installation. Load the drivers for the Highpoint on your Mac, load the YoYotta ID software, and jump in.
I was pleasantly surprised that the Qualstar Q24 offered a simple web GUI interface, which was accessed simply by typing in an IP address in your browser to see everything about the product, including how many drives were loaded into the tape libraries, and the ability to eject the LTO tape magazines right from the web GUI interface. No need to learn the buttons on the front panel of the Qualstar – everything was right there on the web interface.
I am always intimidated by learning new complex software, but the YoYotta software was not only easy to use, they had step by step training videos on their website, on how to use their software. I literally ran these videos while I had their software up on my computer, and followed step by step on the process of formatting a tape to the LTFS format, dragging media from my disk drives into the LTO software menu, mounting the LTO tape (any one you want, simply by clicking on it), and hitting the big green PLAY button to get the LTO tape archive process started.
It was really easy, and it showed a progress bar of how long the archive would take. When it was done, it showed little green checkmarks to indicate that everything was finished, and a little chime went off indicating that everything was done.
The YoYotta software also offered the ability to email me when the archive was written, in case I wanted to walk away from the archive process, to let me know that everything was done. This software, like most of the professional LTO software products, also offers the MD5 checksum option, to insure that every drop of data that has been archived to tape is absolutely accurate.
The process could not have been simpler.
EFFECTIVE, AFFORDABLE, EASY SOLUTIONS
All of my clients are using Macs, so I did not try to be as thorough researching products that would work for other platforms.
There are of course wonderful Windows LTO products. I have already mentioned Qualstar, mLogic, and Highpoint, and Tolis Group. Others include StorageDNA, XenData, Archiware, Crossroads, and Pro-Cache. Many of these also support Linux.
Some of these are not necessarily as simple as the Mac solutions I found. For example, some of these are combined hardware-software solutions, and can be expensive. Tolis Group BRU Server supports only a subset of their Mac features on Windows. Imagine Group says only that their Windows solutions are coming soon.
However, it is simple enough for a facility that is otherwise primarily, or entirely, based on Windows, to simply add a Mac Mini and inexpensive, well-supported software applications like the ones I have mentioned.
Even with the expense of adding a Mac Mini (which most Mac shops will not need to do of course), these make for a very inexpensive solution to a complex problem, that in the past has been wildly cost prohibitive.
The realization that a complex task like this was now, not only easy to do, but cost effective, shows that everything about the audio and video industry is going out to the masses, and there are no longer boundaries as to what anyone can, and cannot do, no matter how complex a task it may appear to be.
