Archiving Film

Jul 8, 2016gbalajiUncategorizedArchive, Archiving Film, Cloud Storage, Digital Storage, Hard disk, LTO Tape, Solid State Drive

As Digital Cinema Designer / Editor and recently turned colorist, I have been observing industry from year 2000 from my first film as an Asst Editor for Tamil film Gummalam, the evolution of Film from Negative to Digital Editing to Digital Color Correction to Digital Exhibition to Digital Acquisition in the right order.

Those days when films are telecined for digital video editing and back to Cut List for splicing negative to scanning to Digital Grading, Films are always back to Negative which is mother of all film prints and when Digital Exhibition came the process of exporting digital grades move to file based, instead of going back to Negatives and thus the beginning of not back to Non-Negative storage begins. What it leads to is there is no physical form a film, which was there for 120 years or more. Storing of Film Negatives and maintaining it is an another long story which involves money and more than 95% of Film Labs in India or closed and most Labs sent out notices to producers to take their Negatives from their Labs since they are closing it down but except very few none taken it down and even if taken it down not maintained properly.

With the advent of Digital Exhibition & Digital Camera based Acquisition the Negative loose their existence, which leads to digital storage of feature films as DPX frames stored in Hard Disk and this is only possible method of storing final film outputs from Color Correction stage.

But the real problem with storing final film outputs in Hard disks are, you never know when the harddisk can fail because of physical damage or because of non use of Harddisk which also leads to failure of Hard disk (that means if you are not using an Hard disk for more than 2 or more years it can fail).

Read 6 Worst Reasons of Hard Disk Failure and Data Recovery Solutions for more info.

Then came the multiple backups of data in two or more Hard disk storages but this too can leads to failure of above mentioned reason.

Whats the solution?

Solutions practised worldwide not just for Feature films and for all financial & non-financial important datas are being archived back to LTO Tapes which are supposed to be retrievable upto 30 or more years if we keep the tape drive along with tapes and computer and is accessories in safe and secure place. The main reason for keeping the LTO Tape Drives along with this is if you are moving for archiving for more 18months or so is, LTO Tapes are evolving and upgrading their storage in capacity from LTO-1 which can store upto 100GB to LTO-7 Tape which now can store upto 6TB uncompressed. Given below evolution of LTO Tapes…

LTO Tape

LTO Tape Storage Capacity

LINEAR TAPE OPEN: (Via Wikipedia)

Linear Tape-Open (LTO) is a magnetic tape data storage technology originally developed in the late 1990s as an open standards alternative to the proprietary magnetic tape formats that were available at the time. Hewlett-Packard, IBM and Quantum control the LTO Consortium, which directs development and manages licensing and certification of media and mechanism manufacturers. The standard form-factor of LTO technology goes by the name Ultrium, the original version of which was released in 2000 and can hold 100 GB of data in a cartridge. The seventh generation of LTO Ultrium was released in 2015 and can hold 6.0 TB in a cartridge of the same size. Upon introduction, LTO Ultrium rapidly defined the super tape market segment and has consistently been the best-selling super tape format. LTO is widely used with small and large computer systems, especially for backup.

SOLID STATE DRIVE:

Then came Solid State Drive based storages which are used primarily for faster Read/Write access to data, since it doesn’t have Physical moving disk, prone to failure is less. SSD got new challenges for data recovery, since it stores data in non-linear and much more complex than existing hard disk drive methods. TRIM command allows an operating system to inform a solid-state drive (SSD) which blocks of data are no longer considered in use and can be wiped internally varies with each operating system.Data integrity of SSDs are relied upon capacitor or some form of battery when power is lost or you disconnect SSD externally or power of your system if fixed internally and it needs to hold power long enough to maintain data in the cache until power is resumed. Most consumer-class SSDs do not have built-in batteries or capacitors.

Reliability of SSDs are not 100% for one simple reason the technology is evolving every day.

Solid State Drive: (Via Wikipedia)

A solid-state drive (SSD, also known as a solid-state disk although it contains neither an actual disk nor a drive motor to spin a disk) is a solid-state storage device that uses integrated circuit assemblies as memory to store data persistently. SSD technology primarily uses electronic interfaces compatible with traditional block input/output (I/O) hard disk drives, which permit simple replacements in common applications. Additionally, new I/O interfaces, like SATA Express, have been designed to address specific requirements of the SSD technology. SSDs have no moving mechanical components. This distinguishes them from traditionalelectromechanical magnetic disks such as hard disk drives (HDDs) or floppy disks, which contain spinning disks and movable read/write heads. Compared with electromechanical disks, SSDs are typically more resistant to physical shock, run silently, have lower access time, and lower latency.However, while the price of SSDs has continued to decline over time,^[7] consumer-grade SSDs are (as of 2016) still roughly four times more expensive per unit of storage than consumer-grade HDDs.

ONLINE STORAGE or CLOUD STORAGE:

Other option is to keep your data in Cloud storage which are cheaper day by day but it involves multiple pricing. Certain company like Google Drive or Apple iCloud charge Storage per GB per year, whereas companies like Amazon charge for Storage as well as for bandwidth usage charges for uploading and downloading of data.

Google Drive Pricing

Apple iCloud Pricing

Amazon S3 Pricing

So finally it all depends on how you are going to store your digital data of feature films depends purely based on your budget and how long you want to store. But definitely Hard disk based storage alone is not an ideal solution.

As Digital Cinema Designer we are archiving to LTO-5 / LTO-6 / LTO-7 tapes the following films which we worked till date.

Nenokkadine | Madras | O Kadhal Kanmani | Varutha Padatha Valibar Sangam
Ithu Kathirvelan Kadhal | On A Quest | Nanbenda | Rajni Murugan 
Ithu Namma Aalu | Pasanga 2 | Lingaa | Brahmotsavam | Kathakali 
Kumari 21F | Orange Mittai | DAVID

And we are also in process of moving to Solid State Drive & Cloud Storage Archiving in coming days.

LTO ARCHIVE

Oct 28, 2015gbalajiArchive, LTFS, LTOArchive, ltfs, lto

The below article is reproduced in this blog for my readers. I don’t have any copyright for this post and for wider reach reproduced this article. Thanks Creativecow.net & Bob Zelin for this wonderful article. Original article link https://library.creativecow.net/zelin_bob/LTO-LTFS-Archiving/1

Bob Zelin
Orlando Florida USA©2015 CreativeCOW.net. All rights reserved.

Now that videotape and audio tape are going away, the media storage system being used by most audio and video professional is disk drives. Unfortunately, as most of us know, spinning disk drives are not very stable, and ultimately fail after a certain period of time. Once a disk drive no longer is willing to mount on our desktops, we can no longer get back the critical audio or video media that we need, usually sending us into a panic.

So most people’s concern today is, “How do I safely back up my media?”

The cheapest solution is to purchase a cheap drive dock, where you can slide in inexpensive single disk drives, and back up your critical media. Move your media from one set of drives to the other every few years for maximum safety.

Another more expensive solution is to buy a redundant drive array. If you have an 8 bay or 16 bay RAID array, you buy another one, and use a cloning program (Carbon Copy Cloner, Super Duper, Chrono Sync, Shot Put Pro, etc.) to back up your media.

An even MORE expensive solution is to build what is called “near line storage,” which is another entire server/disk drive system that backs up everything from your system, and can be called back to your main system. It’s a little slower, which is why it’s called “near” line instead of o line, but everything is available across your network when you need to recall it.

No matter what solution you choose, these choices are all based on DRIVES, and drives ultimately fail. It’s not a matter of “if” – it’s a matter of when. Sometimes drives can last for 5 – 6 years, other times, they will fail within the year. That does not give you a lot of confidence in the long term.

When they were first being introduced, Blu-ray disks sounded promising. With the huge amount of data storage that video professionals use, this proven to not be practical – and those too will fail eventually.

And of course, there is always “The Cloud” (Dropbox, Google Drive, Apple iCloud, etc.). But for those of us dealing with terabytes of storage, this is not very practical, due to the very limited speed of the internet today.

LTO (LINEAR TAPE-OPEN)

So, we are stuck, once again, with tape. Not video tape, or audio tape, but LTO tape (Linear Tape-Open) which is used for backing up – or more accurately stated – archiving your critical media. You simply copy your disk drive media onto the LTO tape, and now you no longer have to worry if someone drops your disk drive on the floor, because tapes won’t break when they are dropped or mishandled. Typical LTO tapes are expected to last 15 years.

The big question is, how much space are we talking about on a single tape?

LTO 4 holds 800 GB of data.

LTO 5 holds 1.5 TB of data.

LTO 6 holds 2.5 TB of data.

LTO 7 will be released in 2016, and will hold 6.4 TB of data.

Of course, the “good for 30 years” is overstated. The tapes themselves will last that long, but most modern drives will only read LTO 5 and LTO 6 tapes. You’re still going to need to pay attention to migrating your data, or plan on keeping old drives around to read your older tapes.

LTO ARCHIVING

LTO is true archiving. It’s safe. Many people have experienced the nightmare of losing the data that is on their disk drives at some point, and the safety that LTO offers is very appealing.

But the emphasis is on safety, not speed – and working with LTO slow. It’s too slow of a process to say “Quick! Grab that LTO tape, and pop it into the LTO reader! We have a client coming in 10 minutes!” LTO is an archive medium. It is not fast like a drive, and it’s not readily accessible like near line storage, or a redundant RAID array.

Once people started using more and more media with fewer safe places to keep it, everyone started to ask, “What LTO solution should I buy?” And there were lots of choices. But most were expensive, and many were complicated to use.

To compound this, many solutions were proprietary, which means that it would work wonderfully on that manufacturer’s system, but if you had to send your LTO tape to a client, or a TV station that wanted the LTO tape, they couldn’t read it, because it was in a format that was not compatible with their LTO system.

And so the demand for a standard for LTO tapes was created, and this is called LTFS, or Linear Tape File System.

WHAT IS LTFS?

LTFS is a standard developed by IBM, HP, and Quantum, and is an open source standard that has some workflow advantages over previous solutions. Rather than having to copy archived data back on to your computer, LTFS archives can be mounted as disk drives like any others. AND you can simply click on the folder structure, to see your files with an LTFS formatted LTO tape, just like a hard drive.

But it is CRAZY SLOW. If you actually double click on one of the files to play it, well – it will try to play it, and you get a spinning beach ball, and 5 minutes later you see a head shot of the file because you just can’t “play” video off an LTO tape.

So it gives the impression of a drive, and you can “drag and drop” files, just like a drive, but of course, it’s all in SLOW MOTION, because when you copy a file from your desktop to a USB flash drive, it takes 10 seconds. With LTFS LTO, the same process takes 10 minutes or longer.

But imagine being able to say “I have this videotape, and I want it to mount in a 3/4″ deck, Beta VTR, DVCPro VTR, DVCam VTR, HDCam VTR, Digi Beta VTR – I don’t care, I just want it to mount no matter what” – that is LTFS.

(For more information about the Ultrium consortium of LTDA developers and partners, go to http://www.lto.org.)

WAIT: WHAT ARE LTO AND LTFS AGAIN?

To keep this straight: LTO is a tape. Just like SATA is a drive or SCSI is a drive. You can format a drive for HTFS+ or NTFS. So you can say “lets take this SATA drive and format it for Mac as HTFS+”.

Same with tape. You have an LTO tape. How do you want to format it? The only universal format that lets you pop in an LTO tape into anyone’s system (that supports the universal format) is LTFS.

TOLIS GROUP BRU

TOLIS Group has been around for 30 years, and their BRU Backup & Restore Technology was among the first widely available archiving software. Tolis also sells integrated hardware-software solutions that have millions of clients. TOLIS Group products are rock solid, and very well supported – including here at Creative COW, where Tolis is very active.

TOLIS Group’s ArGest Studio Rack and ArGest Mobile Cube

Because TOLIS Group was in business 24 years before LTFS technology was first demonstrated at NAB 2009, the solution they developed is proprietary. BRU archives can only be written and read by Tolis products.

Note that TOLIS Group BRU provides some basic LTFS functions for customers who rely on it, but it is by no means the core of what Tolis has been doing for 30 years. By starting as open source and trying to work with as many systems as possible, LTFS has some disadvantages, including slow speeds relative to Tolis solutions, and the inability to span volumes, among others. Tolis outlines those here.

That’s the thing. “Proprietary” gets treated as a bad word sometimes, but probably everything that matters most to you is proprietary. Apple Pro Res, Avid DNxHR, .r3d, Panasonic AVC-Intra, Adobe After Effects project files, your favorite iOS and Android apps – all of these and many more: proprietary.

And they all work great. That is because those proprietary pieces are specifically designed to work together. Tolis has simply developed effective LTO solutions that predate LTFS.

CHOICES

To say it one more time: you are not choosing between LTO and LTFS. All of the companies in this article write data onto LTO tapes. Tolis writes to its own format, and the others write to LTFS.

So which would I choose? The answer: I would not choose either. I don’t care. My clients do the choosing.

It’s the same way that I don’t care about RED, ARRI Alexa, or Sony F55. I don’t care about Apple Pro Res or Avid DNxHR. They all do the job. It only matters what the client wants.

If you are looking for a rock-solid, long-proven technology for yourself and do not need to share archives with clients, Tolis BRU is fantastic. If you work with multiple clients, and they are all using Tolis products, you are all set.

But if you work with multiple clients who have chosen a variety of systems from a variety of manufacturers, and they ASK you to look at LTFS solutions, then you should be looking at LTFS solutions. It’s all up to the client.

My clients asked me to look into the cheapest-possible LTFS options from other companies, so I that is what I have done.

CHEAP LTO ARCHIVING?

Don’t get sucked into the “LTFS is cheap because the software is free and open source” hype. The fact is that no LTFS software that works, and is well-supported, is cheap. The software that works starts at around $499.

The only one that is free is Hewlett Packard StoreOpen, and that package is a nightmare. It is ridiculously slow, even by LTO standards. It is painful to use, and has no support. Don’t use it. You will try to make it work, and you will fail, and you will come to Creative COW for advice, and the advice will be, stop trying to use unsupported software for mission-critical work.

HP StoreOpen and Linear Tape File System (LTFS) Software

However, my clients were very clear to me. They did not want a rundown of everything out there. They asked me to find the CHEAPEST systems that actually work, that writes LTO tapes in the LTFS format, that have capable hardware and fully-supported software.

Here is what I found.

THUNDERBOLT AND LTO

With the release of Apple’s new Thunderbolt computers, everyone wanted Thunderbolt everything. Of course that meant Thunderbolt disk drives, but it also meant Thunderbolt LTO tape drives. And this demand led to the release of themLogic mTape, which became the first commercially available LTO drive that had a Thunderbolt interface.

mLogic mTape Desktop Thunderbolt LTO6

There were lots of software programs that worked with the mLogic mTape, but “everyone” wanted the LTFS standard.

Then along came Imagine Products “PreRollPost”, which was an easy to use LTO program that ran on Apple computers, and that used the LTFS standard to create the LTO tapes. So it became the perfect combination of Apple, LTO, and LTFS.

Preview of the upcoming Windows release of Imagine Products PreRollPost

(Note that Thunderbolt is not exclusive to LTFS. Don’t forget: LTO is the kind of tape. While I have been talking about manufacturers who format LTO tapes as LTFS, TOLIS Group does in fact have the TOLIS ArGest Thunderbolt product line that writes LTO tapes to their own format.)

LTO TAPE LIBRARIES

Most people don’t want to spend the money for a tape library, so they will make single tapes – no different than you would have Sony Beta tapes in the past. Many clients have one LTO tape per show, or one tape per project. This is what I have dealt with mostly in the past, particularly with Imagine Products PreRoll Post, and the mLogic mTape.

And before this, the same single tape process with Cache-A (bought by ProMAX and renamed Pro-Cache) or Tolis BRU and a single LTO tape from from Hewlett Packard. No different than you would have Sony Beta tapes in the past. One LTO tape per show, or one tape per project. This is what I have dealt with mostly in the past, particularly with Imagine Products PreRoll Post, and the mLogic mTape.

As storage grows, and drives continue to fail as time goes on, more and more people are getting nervous, and are asking about larger LTO tape solutions.

In the past, these solutions have been very expensive. But as the entire video industry changed, and low cost video products came out, the exact same thing is happening to “enterprise level computer solutions.” LTO tape libraries are one such solution.

LTO Tape Libraries are large enclosures that hold multiple LTO tapes at one time, just like large RAID arrays hold multiple disk drives at one time. Hewlett Packard, and Quantum, and Tanberg made these large LTO Tape Drive Libraries, and most of the established LTO software companies had solutions that incorporated these products, but they were expensive.

Even if somebody was had the budget to go the LTO tape library route, they were complicated to use, and required a lot of training.

Wasn’t there an easy solution?

HIGHPOINT TECHNOLOGY

At NAB 2015, Highpoint Technology continued to release their ever expanding line of Thunderbolt to anything that you can think of adaptors. One of their little products was a new box called the RocketStor 6328, which was an adaptor that converted Thunderbolt to MiniSAS. Highpoint had been aggressively working on getting their product to be compatible with LTO tape drive products.

Highpoint Technology RocketStor 6328

But I was already happy with the mLogic mTape, which was an LTO drive that already had a built-in Thunderbolt port, so no need for any adaptor boxes. But of course, there was no LTO Tape Library product like this.

Highpoint suggested that I try their product with an LTO tape library. The problem was that most LTO libraries were expensive, and that most “solutions” that existed for LTO libraries were expensive. But thanks to wonderful trade shows like the NAB show, you can wander around and find companies that you may have never heard of, that have these amazing products that do exactly what you want.

QUALSTAR

I found two such companies. The first was Qualstar. They made LTO libraries, for a fraction of the price of other options. Qualstar was actually a very large established company, that had the “big expensive stuff, but the Q24 was an “entry level” product that looked like it would suit my requirements.

Qualstar Q24

So the Highpoint 6328 would plug into a Mac or Windows computer, and the other side of the Highpoint would plug into the miniSAS connector on the Qualstar Q24. Now, I needed software to get all this to work.

I contacted Imagine Products again, but their wonderful PreRoll Post software would not support an LTO tape library.

YOYOTTA

As I walked through the aisles of NAB, I remember seeing a little company called YoYotta, who made LTO software that supported LTFS. They informed me about their YoYotta ID Automation software, which not only supported LTFS, but was designed to operate with an LTO Tape Library. And they offered a free trial download and training videos on their website, to see how to use their product.

I had my combination of products. My little Thunderbolt Mac Mini, my Highpoint 6328 Thunderbolt to miniSAS adaptor, my Qualstar Q24 LTO tape library, and the YoYotta ID Automation software.

YoYotta software ID Automation

There was not much to the installation. Load the drivers for the Highpoint on your Mac, load the YoYotta ID software, and jump in.

I was pleasantly surprised that the Qualstar Q24 offered a simple web GUI interface, which was accessed simply by typing in an IP address in your browser to see everything about the product, including how many drives were loaded into the tape libraries, and the ability to eject the LTO tape magazines right from the web GUI interface. No need to learn the buttons on the front panel of the Qualstar – everything was right there on the web interface.

I am always intimidated by learning new complex software, but the YoYotta software was not only easy to use, they had step by step training videos on their website, on how to use their software. I literally ran these videos while I had their software up on my computer, and followed step by step on the process of formatting a tape to the LTFS format, dragging media from my disk drives into the LTO software menu, mounting the LTO tape (any one you want, simply by clicking on it), and hitting the big green PLAY button to get the LTO tape archive process started.

It was really easy, and it showed a progress bar of how long the archive would take. When it was done, it showed little green checkmarks to indicate that everything was finished, and a little chime went off indicating that everything was done.

The YoYotta software also offered the ability to email me when the archive was written, in case I wanted to walk away from the archive process, to let me know that everything was done. This software, like most of the professional LTO software products, also offers the MD5 checksum option, to insure that every drop of data that has been archived to tape is absolutely accurate.

The process could not have been simpler.

EFFECTIVE, AFFORDABLE, EASY SOLUTIONS

All of my clients are using Macs, so I did not try to be as thorough researching products that would work for other platforms.

There are of course wonderful Windows LTO products. I have already mentioned Qualstar, mLogic, and Highpoint, and Tolis Group. Others include StorageDNA, XenData, Archiware, Crossroads, and Pro-Cache. Many of these also support Linux.

Some of these are not necessarily as simple as the Mac solutions I found. For example, some of these are combined hardware-software solutions, and can be expensive. Tolis Group BRU Server supports only a subset of their Mac features on Windows. Imagine Group says only that their Windows solutions are coming soon.

However, it is simple enough for a facility that is otherwise primarily, or entirely, based on Windows, to simply add a Mac Mini and inexpensive, well-supported software applications like the ones I have mentioned.

Even with the expense of adding a Mac Mini (which most Mac shops will not need to do of course), these make for a very inexpensive solution to a complex problem, that in the past has been wildly cost prohibitive.

The realization that a complex task like this was now, not only easy to do, but cost effective, shows that everything about the audio and video industry is going out to the masses, and there are no longer boundaries as to what anyone can, and cannot do, no matter how complex a task it may appear to be.

BACKUP VS ARCHIVE – WHAT DIFFERENCE DOES IT MAKE?

Jul 12, 2015gbalaji2 CommentsUncategorizedArchive, Backup

Via (lto.org)

Some words have a variety of meanings, and we can sometimes get confused with their purpose and usage. For instance, what is the difference between continually and continuously? They are often used interchangeably but there are key variances. According to Dictionary.com continually means “very often; at regular or frequent intervals,” and continuously means “unceasingly; constantly; without interruption.” For example: the executive continually reads the stock reports at noon each day versus the executive reads the stock reports continuously, all day long without stopping. The latter would be quite tiresome and nonproductive. Similarly, backup and archive are sometimes used interchangeably but important distinctions exist, especially in practice. Let’s take a closer look.

Backup and Archive – It Makes a Difference

Backup (often called Backup and Restore) is making a copy of current data so that it can be utilized to restore the data in the unforeseen circumstance that the original data was corrupted, deleted or destroyed by unintentional or intentional means. We discussed these data destroyers in a previous BlogBytes called the “Backup Blacklist.” The backup is often done incrementally and kept for certain lengths of time (seven days, two weeks, etc.) based on user set policies that consider the data value, as well as internal and external requirements.

Archive stores a single instance of the data or data sets, a historical collection, explicitly chosen for potential long term future usage. A key distinction here is that archived data is the original or single copy and is typically no longer in current usage.

Some treat multiple backups as their data archive and are unnecessarily creating a mountain of information that can consume space and costs. Once data is no longer in the ‘active’ category but a single copy is still needed in the chance that it may be recalled, it should be moved to the archive for long term retention. This practice can free up primary storage and backup space, lessen backup management overhead, and provide easier classification and retrieval of the information from the archive, rather than trying to sift through multiple copies of backed up information.

Mine that Data

Most archive management software provides capabilities not typically found in backup processes such as cataloging and metadata search. Information Lifecycle Governance (ILG) enables that cataloging, eDiscovery, defensible deletion and data retention on lowest cost data storage infrastructures. The metadata could include names, labels, data types, owners, dates and more. This can provide the means to mine the archive to address a variety of important requests such as an internal audit, customer inquiry or regulatory requirement. Tape storage data can be mined and with use of the Linear Tape File System the mining of metadata on LTO tape becomes even easier. Especially when used with archive management software offered by a variety of LTFS supporting providers.

Where Do Tape and Disk Fit?

Enterprise disk and flash are used for primary data storage and can be backed up to other storage devices

including deduplicated disk and tape. Deduplication has provided the means to reduce the amount of space typically needed to store data thereby stretching the storage investment dollar. Bear in mind though, deduplication needs to see multiple copies of the same data set to do its dedupe magic. When the original file is put in the archive for long term storage, it is the only copy. Therefore, deduplication will have little effect on this single version of the original data that is now in the archive. That and other benefits make tape the prime choice for archive storage to contain costs while providing protection for original content. This can put money back in to the enterprise piggy bank.

One last note: since the archive contains original and single copy data, make a second copy on tape and move it offsite for the ultimate protection. Essentially, it is a backup of the archive. LTO Tape is low cost…do it now.

To sum up, with carefully chosen descriptors, backup and archive processes need to occur continually to help protect data, contain costs, and provide management information to keep the enterprise running continuously.