Data tape archive - anyone using SDLT, LTO, DLT?

Warwickt,
Thanks much for this great rundown on LTO4. I have a Mac Pro and have been thinking about getting an LTO system as well (but didn't know LTO 5 was due in 2009....hmmm, wonder if I should wait). Your advice really helps, so thanks for the effort...
 
Last edited:
Warwickt,
One last question (or maybe 2!):
When you restore from tape, do you have to restore the whole tape, or can you somehow look at a list of folders and files and restore those? For instance, let's say I archive 3 short films on one tape. Each film is in a different folder, with that folder containing the final cut project file and all the raw media. Can I use the Tolisgroup's software to look at a catalog of the tape, and just choose the particilar project folder I want to unarchive?

Also, is the Tolisgroup software somehow using its own proprietary software to catalog and un-archive the files from the tape? If so, what happens in 10 years if I don't have access to that same software....will I be able to remove the files from the tape?

Many thanks!
 
k2director said:
Warwickt,
Thanks much for this great rundown on LTO4. I have a Mac Pro and have been thinking about getting an LTO system as well (but didn't know LTO 5 was due in 2009....hmmm, wonder if I should wait). Your advice really helps, so thanks for the effort...
Click to expand...

HI K2D!... regarding Ultrium LTO5.. no I would not wait for Ultrium LTO5 several reasons:
  • ifyou have a business need then dont wait for this
  • your content is critical to your business and if you lose it you lose it to el crappo cheapo external disks its all over with tears!
  • ULTRIUM LTO5 Roadmaps from vendors sporting Ultrium LTO5 say announcement later this year (Dec 2009?) so this means you wont get tape drives these ready for the consumer end before at east May2010 or much later. So availability at the local computer shop is way out from this point in time 2009JAnuary.
  • there's also the higher prices for this new LTO5 Ultrium data tape media before it becomes main stream. My guess is that for a 1.6TB Ultrium LTO5 data tape (uncompressed) will run out at least $USD100 to start with.. they will come down in time just like all the LTO ULTRIUM data cartridges have over the first 12 months to sit around $USD30-$USD45 but thats my opinion..:beer: It is worth noting that Ultrium LTO4 *current) became quite mainstream in Dec 2007 or early 2008 and the prices of the media have dropped considerably for that time.
  • LTO5 Utrium tape media: also consider the prices are usually high on initial release as vendors like Imation , FUJI etc may get the first bite on the first suppliers and they will find their way out to others as the medium gets popular - same old story... who knows?? .. just like Video tape eh?
  • several vendors still have to implement Ultrium LTO5 in their labs and THEN they need to get them to all the ISV's and then to the hardware guys and their cards before they release and then it will be the big I.T.companies that always get them first
  • usual teething issues - especially using say the SAS 6Gbs (and FC and mayebe Infiniband??) interface that is really new. ALso the likes of data tape drive vendors that may not quite implement the LTO5 specification (a la Quantum) and so on ..
  • wait until LTO4 Ultrium is dead stable and then buy. When you get you new Ultrium LTO5 tape drive in late 2010 it will read you Ultrium LTO4 tape media no worries!.
  • like all these leaps in storage technologies, the idea is to cram more into less with faster access and lower prices. There are many compromises as we can see in things like el cheap high capacity disk drives (SATA) and el cheapo cameras.. you know the deal.It will be the same with new data tape technologies and they cram more tracks (serpentine linear usually) into the same physical tape media in the same package taht is 800M's long amd probably wot go any longer unless some miracle material that is 100x stronger than kevlar™ that is a vew microns thick appears on teh scene atht doesn't permit magnetic crossover. (I'm out of my depth here). I think OPTICAL data date or some kind of non HOLOGRAPHIC storage that is super super fast is the future.. it remains to be seen.
  • I'd say of you have no need at all (but you do because you are interested). I would go buy the config that's been mentioned. It's like hanging out for the next gen technology .. you will never be satisfied
  • The current Ultrium LTO4 will be good for at least 10 years as Ultrium LTO1 (announced/shipped in 1999) can still be used today (for what reaso I dont know). TApe transport compataibility with Utrium LTO seems to be current -1.. so you will be save. In any case if you wait around long enough 1PB storage devices made with 5th dmension materials will be around.. say in the year 2020 .. so who knows. HOwever in 2020 (an arbitary date is only 11 years out .. what were you doing in 1998??) you will most certainly be able to read the DATA BLOCKS from your 2009 Ultrium LTO4 tape media. Transforming the data structures back to DVCPROHD or R3D is another story of course.. You need to be on top of this regardless of the technology. (I'll will be making everything LMJP2K [loss motion JPEG2000] in a few years when the encoding and especially the payback decoding improves. And some AUDIO format to boot. or at least some open source standard.

Simply (like many of us) if you need it go get it. You can do this economically and it is many years way from being a legacy technology. Do your business sums ofcourse. I'd bank on a safe Utrium LTO5 (say from HP or IBM) around the 2Q2010 .. but thats an educated guess. Ots like waiting for the MAC PRO to have 64cores... is this realistic for your business? nup!:huh:

I mentioned my nasty experiences on many posts losing over 1.8TB of P2 footage my trusting it to those el cheap lousy Le CIE FW800 BIG DISKS.. never again.

Summary: if you need it go get it. If you lose your content tat is CMF based (contemporary media format such as file based P2) you will tearfully regret it.

hth

w
 
k2director said:
Warwickt,
One last question (or maybe 2!):
When you restore from tape, do you have to restore the whole tape, or can you somehow look at a list of folders and files and restore those? For instance, let's say I archive 3 short films on one tape. Each film is in a different folder, with that folder containing the final cut project file and all the raw media. Can I use the Tolisgroup's software to look at a catalog of the tape, and just choose the particilar project folder I want to unarchive?

Also, is the Tolisgroup software somehow using its own proprietary software to catalog and un-archive the files from the tape? If so, what happens in 10 years if I don't have access to that same software....will I be able to remove the files from the tape?

Many thanks!
Click to expand...

sure!...

Answer#1: this is totally dependent on the software that you want to utilise on  MAC OSX . Good archive s/w (and its I.T. backup counterpart) usually provide a UI of sorts and even maintain a database (state info) on what is archived. Of course these vary . (stay away from HSM's).

Yes with most you can SEE the file system entities (file elements and directories) that have been archived.

RECALL/RESTORE: in more sophisticated software that permits the above (Like TolisGroups BRU for example and several others), you can recall / restore the smallest element on its own to either the same place (its orign) or somewhere else as you would expect.

Further data tape drives technologes since the early 1992+ have implemented what is stupidly/misleadingly referred to as "FAST FIND". It is actually implemented as a LOCATE | LOCATE BLOCK | SPACE command. Similar to use as TCR and INDEX marks on a video tape for example but far more rudamentary. This locate block | space directive will direct the tape drive transport to a SPEcific data block to the tape media. [All written data blocks have a unique identfier that the tape controller knos about when it encounters the DATA BLOCK structure on the tape medium under the read heads).

This has the excellent function of moving the data tape media to a specific place on the tape for the next operation in a very fast way. Very much like purpose of a disk SEEK (etc) or LOCATE SECTOR directive does for a disk controller. In short this is used so that the old FORWARD SPACE FILE or BACKSPACE file operations (locate tape mark etc) that are a hangover from the 1960's and are processed at a slower read speed of say 2M/sec or less are not used. So you dont have to READ the intermediatry content on the data tape.. the stuff you dont want to look at.:beer:

The data tape transport will use a HIGHSPEED movement at 10m/sec to 14m/sec (depending on the complexity and type of tape drive and price!) , generally 4-6 times faster than READING or WRITING data. This means that given the time it takes you to put the tape in the tape drive (called a mount) , you can get to stuff and have it recalled/restored within minutes at data rates of up to 110MBytes/sec (LTO4 SAS with large objects using BRU for example).

So as an example, 1 x hour of DVCPROHD 1080/50 (110Mbs+ encoded) is around 49.5GB +/- (or its P2 encapsuated .MXF object and its elements) and you have archived it on a Ultrium LTO4 DATA tape.

Recalling/restoration: : Excluding the time it takes you to put the data tape in the drive and also excluding (for discussion purposes here) likely LOCATE BLOCK | SPACE time (arbitary but real), it only takes say 7.5-8mins minutes to recall that 1 hour of DVCPROHD 1080/50 to a disk storage system (as long as the disk storage system can sustain/accomodate the 110MB/sec write data rate). (data tape performance, like disk, depends on so many things such as the content , the paths, the target and source, block sizes, the OS, the alignment of moon and planets too :) )).... another topic. Now try that on your FW800 or lesser USB2 external el cheapo disk!

Thus if you have three objects (shorts productions or otherwise) and they are archived as either complex objects (such as you have encapulated them in MXF), R3D mags , or whether they are simple objects such as a QUICKTIME (encapsulation) etc etc, you can see them. If they are requested all together for a restore, most archive software (but not all) will process them sequentially in the order they were archived on the tape media volume(s) in the first place. Each objects has had its start and end BLOCKID recorded in the database or state information by the archive/recall software. The software uses this to quickly JUMP TO the start of the next or initial archive object so it can restore it.

Depending on the software you are using for archive/backup its up to you if you maintain your material as
  • a single archive with many directories and files or
  • a single archive with a single objects (segmented or split archves) or
  • a single archive with just those selected objects (films etc) that (liek al of the above) can be appended too over many tape volumes. (a multivolumed archive) or a
  • multiple set of archives on many volumes that are all of the above - it really depends on your own workflow needs.

Its up to you.

Tolisgroup's BRU keeps its archives based on the tape volumes you have used. So yes you can. I do it a lot to recall material for repurposing (the main point of an archive I think)

It has a basic and uncomplex UI that is easy to use. (go d/l the demo of their existing BRU-LE and have a play else contact Tolisgroup to be on their BRU-PE Beta programme .. its going to be out soon anyway).

OUTAGES?: If you lose ths catalogue you can easily IMPORT the data tapes set and its all reinstated simply and DOES NOT REQUIRE YOU READ the entire line of all the tape drive content as something archive like EMC's retrospects™ probably still does. BRU is rather quick.

This is also for a recipient ( your client or post house r distributor etc) as a content distribution carrier. They can IMPORT the contents of the data tape.. (See below for details with using a free BRU-xE)

ANSWER #2: Proprietary software: Well! this is an age old argument and also can be likened to the disk file system wars that still persist (open source vs windah's).. You will always be at risk of being unable to access or worse TRNASFORM you content over time.:zombie_smiley:

Sure! if any ISV dissapears of the map as it happens all year around (just got to a NAB then IBC and see who's not there each 6 months) , the acquiring company supports the stuff for a whie and then forces you off it. Take a comapny like AVID as an example with their acquisitions and then moves every one of that customer base onto te new base. The examples are quite a lot.

However, by NOT using a prorietary format you wont get the efficiences you need to make most thngs work. For example, many systems used a TAR format for tape. THis is fine however there are features that are very difficult to implement. For example inthe higher end arcive enablers used for media and entertanment, ISV FPDI, MASSTEch and others use a propritary format on their data tapes. For FPDI this permits them amongst other things to implement a TCR based partial object recall. SGL do it another way. THese companies service many of the world eading broadcats and media and tertainet copanies.. no whinging from these global media companies it seems.

Oh and additionally its common knowledge that our mates at Apple  have not included any native SCSI tape drivers in MAC/OSX Leopard 10.5.x as far as I know. Maybe they excluded them in Tiger (10.4) .. I dunno.. so using TAR for tape archiving for  MACOSX Leopard 10.5 out of the box wont work. I guess that reason alobne you want to buy anISVproduct or at least a supported OPenSource verison of soething that not only runs flawlessly on you MACPRO under OSx but further supports the current tape drive vendors devices.. somehing that all the best intentions in the OPenSource world take time to do.

Additionally for example with TOLISGROUP's BRU, you can even use it for distribution because the reciipient (say of your distribution on LTO4 Utrium tape) can simply download the FREE (demo) unlicenced verison of BRU-PE/LE and use it to RECALL/RESTORE the media without havong t purchase it. THis is official from Tolisgroup because I asked the same question.

Like all technologies including that of data tape media , disk , memory and their related structures, you just need to be a little vigilant about its usefulness over time.

I ask you, do you still keep your data files on 120mm plastic floppy disks? I would think not. You moved them to something else more contemporary. THe same is happening over time to the el cheapo DVD-5's.. just watch.

ISV Longevity: BTW I see that Tolisgroup have been around since the early 1980's in some form so I guess they will be around a while longer. cant say teh same for other sowftare vendors that did data tape archiving such as OTC, Avalon, LEgato to name just a few. THey and their products have been literally swallowed up and in some cased completely discontinued by EMC.... who knows.. (I will stand corrected on this :) )

Also EMC is around to stay as are many others in some from at least with their large customer bases.

I for one will be watching this parcular aspect and should we keep in touch , we can revisit this in 5 years then 10 years :Drogar-Dum(DBG):. I'm quite confortable with this as I have spent a load of time thinking and researching about it before I committed to this archive workflow using data tape. Pity I didn't think the QUANTUM tape drive thing a bt more.. lol!

PM me if you have any specific questions else keep posting your Q's for others.

there's many out there that have or would like to have a stable archive workflow that does not involve cheap or even expesive disk systems.

HTH
W
HK. (where it is a sunny 18C outside in winter :dankk2:
 
Thanks much for the further answers, Warwickt! This is a lot of digest, but is much appreciated. I'm looking forward to having an LTO system set up sometime in the next few months...
 
cool!. post your result when you get your kit. ALso check out the reduser.net forums. Plenty of action on his subject over there as well regarding this setup. POst or PM any q's you have here and there as well.

Gong Si Fa Cai | Gong Hey Fat Choy in year of ox.

w
 
BRU Producers edition for MAC OSX and Linux is available now -

BRU Producers edition for MAC OSX and Linux is available now -

Anyone interest to know that Tolisgroups BRU PRoduction edition is available. works great on OSX

http://www.tolisgroup.com or

the BRU-PE site is at http://www.productionbackup.com/

Best configuration for MAC PRO is simpley:
• BRU-PE archive single user OSx based archiver
• HP Lto4 Ultrium SAS tape drives desktop models FULl Height 1840 or Half height 1760
• ATTOTEch Express H308 or Express SAS R380 SAS HBA's (new EXPRESS SAS 7Gbs out in March 09)

stay away from LSI and quantum as these vendors do not support MAC OSX and will send you on your way to a mr softee outlet.

hth

w
 
Hi Warwickt,
Do you know of any ATTO cards that support Raid 6 *and* also have a spare connector for an LTO4 device. The blasted Mac Pro only has 3 internal slots, which fill up quick. Slot for Raid card, slot for p2 5 card reader, slot for Kona card, that's it! So I would hopefully find a RAID card that could also connect to an LTO drive.
 
You must log in or register to reply here.
Back
Top