Home | Storage | A Tale of Two Archives – BlackPearl and Cleversafe
A Tale of Two Archives – BlackPearl and Cleversafe

A Tale of Two Archives – BlackPearl and Cleversafe

0 Flares Twitter 0 Facebook 0 Google+ 0 StumbleUpon 0 Buffer 0 LinkedIn 0 Filament.io 0 Flares ×

It’s been a tiring but interesting few weeks of travel lately, with the Spectralogic Summit, VMworld EMEA and IP Expo all happening within a few weeks.  One interesting observation coming from these events is the approach being taken by vendors to manage large scale archives.

Spectralogic BlackPearl

I have to say I didn’t know much about Spectralogic before attending their recent summit event in Denver and so I was somewhat skeptical on what would form the basis of their “big announcement”.  The answer has been widely reported already and it is BlackPearl, an appliance that sits in front of a tape library effectively turning it into an AWS S3 compatible object store.

Take a step back for a moment and look at a previous article I wrote after last year’s IP Expo in London during which I attended the jointly held Tape Summit.  At the time I made the following comments:

  • The tape industry is in denial
  • Tape still has a place
  • Usability is all wrong
  • Backup software vendors are holding us back
Probably the most poignant comment of the above statements is the one concerning usability.  Despite the development of LTFS, tape is still a difficult medium to access in a modern random access world.  In this recent post I replied to a comment from Mr Backup Himself (W Curtis Preston), with what at the time looked like a bit of forward thinking on tape addressability.  Little did I know that Spectralogic had been working on a solution to exactly this issue and this is what we now know as BlackPearl.
BlackPearl (all one word) is a storage caching appliance that sits in front of a Spectralogic tape library and provides an AWS S3 compatible interface that enables the storage and retrieval of objects to and from tape using a REST API.  In fact the BlackPearl API, known as DS3 is a superset of S3 as it adds additional commands to manage bulk archive and retrieval functions that deal with some of the vagaries of tape access.
BlackPearl is designed to provide cold storage, that is a long term, infrequently accessed but highly scalable data store.  We’re not talking here of stuff you’re likely to be accessing all the time (as time to first data would be slow) but rather data that does still need to be stored somewhere.  Having a generic API to store and retrieve data makes much more sense and answers points 3 and 4 above (although BlackPearl is not intended as a backup target).  When it comes to pricing, Spectralogic shared some numbers that indicate the cost could be as low as $0.09/GB, depending on scale, over a three year lifetime.  This is much cheaper than Amazon’s Glacier platform, which would be around $0.36/GB over the same timescale.
I intend to cover DS3 and BlackPearl in another more detailed post, including details of the developer portal, which provides access to a BlackPearl simulator.



Tape is one way of building a large-scale archive, however as discussed there are some issues with tape access times.  An alternative solution is to build a large-scale disk archive and keep everything on spinning media.  Disk archive solutions have been tried before; Copan Systems being a notable example with their spin down technology.  Unfortunately there are a number of big issues with implementing disk archive systems that have any sort of scale:

  • Disaster Recovery – if the archive is the only system of record for the data, then a copy or a backup has to be taken.  But as an archive scales, that becomes an increasingly complex task due to the processes involved in backing up changed or new content.  In addition, extra data copies or backups introduce significant additional cost.
  • Design Flaws – todays storage arrays aren’t designed to be large-scale archives, but rather are constructed to serve primary storage.  This results in very specific design decisions and features and is different from the requirements of an object store.
As a result of the need to provide for disk archiving, we’ve seen the rise of object or content stores, of which Cleversafe is a great example.  Again I intend to do another post on this that goes into more technical detail, however I’d like to point out two important features of the Cleversafe system.  Firstly data is encoded using Reed-Solomon (erasure coding) algorithms which physically divides and distributes data in a form that allows it to be recovered without having all of the constituent parts and without needing huge additional capacity for things like parity as we see in RAID implementations.  Second, the algorithm has the side effect that as we disperse the data more, the protection algorithm becomes more rather than less reliable, which is intuitively the reverse of the way we see RAID systems today.

Although other  solutions exist that also use erasure coding, I haven’t seen any that talk about the scale to which Cleversafe claim to operate.  In addition, some of the issues and opportunities that came up in my recent discussion with Chris Gladwin (Cleversafe founder) point to some intruiging use cases for the technology.  Again, I will be doing a more thorough review of what Cleversafe has to offer.

The Architect’s View

Whether or not you archive to disk or tape, it’s a case of choosing the right technology for you and evaluating all the costs to develop a true TCO.  It could well be that tape continues to have relevance in the future.  But that’s not really a surprise, is it?

Related Links


Comments are always welcome; please indicate if you work for a vendor as it’s only fair.  If you have any related links of interest, please feel free to add them as a comment for consideration.

Subscribe to the newsletter! – simply follow this link and enter your basic details (email addresses not shared with any other site).

Copyright (c) 2013 – Brookend Ltd, first published on http://architecting.it, do not reproduce without permission.

About Chris M Evans

Chris M Evans has worked in the technology industry since 1987, starting as a systems programmer on the IBM mainframe platform, while retaining an interest in storage. After working abroad, he co-founded an Internet-based music distribution company during the .com era, returning to consultancy in the new millennium. In 2009 Chris co-founded Langton Blue Ltd (www.langtonblue.com), a boutique consultancy firm focused on delivering business benefit through efficient technology deployments. Chris writes a popular blog at http://blog.architecting.it, attends many conferences and invitation-only events and can be found providing regular industry contributions through Twitter (@chrismevans) and other social media outlets.
  • http://www.storage-switzerland.com/ Eric Slack

    Great piece Chris, object storage certainly has emerged as the architecture of choice for large data sets and long term retention
    And Spectra’s DS3 technology now brings tape into the object storage party
    One thing you could add to this post is that a BlackPearl appliance can connect tape to an object storage system like Cleversafe, giving you a tape tier for a disk-based object store
    – it doesn’t have to be an either disk or tape situation
    You’d still have to figure out the placement of data on disk or tape but the connectivity is there today

0 Flares Twitter 0 Facebook 0 Google+ 0 StumbleUpon 0 Buffer 0 LinkedIn 0 Filament.io 0 Flares ×