In this post I’ll discuss how you can optimize performance, cost, flexibility, and resiliency when building a Windows 8 storage solution. Windows 8 provides two new storage capabilities that enable optimized storage solutions based on these considerations: Storage Spaces and File History. I won’t get into the details of File History in this post, but you can read my complete write-up here. In this post I’ll use my recent video editing PC overhaul as a case study for how to build a cost-efficient high performance storage solution on Windows 8.
The 8 drives used in “VIDEO2” a PC that I recently rebuilt
As a creative professional, I depend on high-performance hardware and apps to get my work done. When working with apps like Adobe After Effects and Adobe Premiere Pro, the storage demands are staggering. A typical 1080p uncompressed After Effects animation render can easily occupy 100GB of disk space or more. Working with video is also very demanding for storage. For some projects, I’ll store 10 or more video clips at several Gigabytes each, and then output several renders in different formats. These apps also use “Scratch Disks” to store project data that can’t be stored in RAM when previewing, rendering, and for other similar tasks. Scratch disks need to have super-fast I/O to keep up with the demands of these dynamic tasks. Of course, all of my project, media, documents and other files need to be backed up. I want to minimize the risk of losing data and I want my system to have excellent performance fundamentals.
Before I discuss how I implemented my Windows 8 storage solution, let’s first take a look at the factors that influenced this design.
Here are the considerations that I have for a complete storage solution on my PC:
To frame the design discussion, I’ll use an app that represents an “extreme scenario” for storage as a use-case. Let’s consider how Adobe Premiere Pro CS6 uses storage when launching, editing, and rendering files:
Adobe documentation provides helpful guidelines for system hardware configuration when running programs like Premiere Pro CS6 and After Effects CS6. From these guidelines and with knowledge about Windows storage technology, I created the following storage plan for my high performance PC:
Video editing file interactions shown at a high level using Premiere Pro CS6 as an example
After landing on this high-level storage layout plan, I did some thinking considering the following:
Here are the simplified high-level requirements and parameters for each logical storage device:
Logical storage devices (rows) and corresponding storage requirements (columns)
From this high-level planning exercise, I felt prepared for hardware selection and storage configuration in Windows 8. I’ll summarize here how I implemented each logical storage device.
System StorageAs I evaluated the Windows/apps logical storage device, the primary concern was performance. This storage device needed to handle both random reads/writes and sequential reads/writes really fast. Resiliency was not a primary concern since I knew I could easily reinstall Windows and my apps in the event of a hardware issue. The solution was relatively simple: this logical storage device would be a straight-up SSD drive.
A 240GB SSD was selected for OS and apps
ScratchThe app scratch drive would need to offer high performance for both random and sequential reads/writes. This scratch storage would not need any form of resiliency since the files stored would all be temp files. An SSD would be a great option for this logical storage device, but I didn’t have an extra SSD on hand. What I ended up using was the two Western Digital 10,000 RPM drives that I had been running in RAID configuration in the Windows 7 incarnation of the PC known as “VIDEO2”. This time I would use these drives differently. Windows 8 Storage Spaces would offer RAID-like performance, but would be much easier to setup. I created the scratch Storage Space, and did not use resiliency so as to optimize write performance.
With just a few clicks the scratch Storage Space was configured and ready for use
The creation of this scratch Storage Space would be a test of sorts, and some simple tests with Adobe Premiere Pro CS6 would tell me quickly whether this scratch Space was going to offer adequate performance when previewing and rendering. If there were any issues, I knew it would be super-easy to delete this Storage Space, and I could then repurpose these drives and obtain an SSD for scratch storage. You have to read the rest of the article to find out what happens. Naming this Storage Space “Scratch Space” made it easy to find from other apps when configuring storage options.
Project Files (Bulk Storage)For project files there were a few considerations. First, I needed a large extensible space to store a lot of really big files. Second, I absolutely needed a form of resiliency to ensure these files would never be vulnerable to hardware failure. Third, this space would need to keep up with the sequential read demands from various apps (Premiere Pro CS6 for the use case presented in this post). The solution was to create a Storage Space with two-way mirrored resiliency. This would offer RAID-like performance, resiliency, and extensibility. With Storage Spaces you can easily add storage at any time, so it’s a great way to implement a single storage repository for large quantities of data. I was excited about the idea that I wouldn’t have to split up my files between drives, all of my important data could live on this single extensible Storage Space.
Creating the Storage Space for bulk file storage
With this bulk Storage Space, I would always have redundant copies of my data. This would change my strategy for backups. Bulk backups would now be a mitigation for a fire, flood, and other natural disasters. Using File History to protect my primary documents would give me a history for my data and offer another form of protection. Finally, SkyDrive would add another form of redundancy for my files. I was feeling good knowing that my data would be safer than ever.
I chose to use a total of four inexpensive Seagate Barracuda 7200 RPM drives for larger logical storage devices
Just like with my scratch Storage Space, I would need to perform some simple tests to confirm that sequential reads wouldn’t be a bottleneck for video editing performance with this bulk storage Space. With RAID-like read performance for this multi-drive setup, I wasn’t too worried.
Utility and RendersThe final logical storage device was targeted at utility and storage of output renders. Since these files would be stored temporarily, this logical storage device would not need any kind of resiliency. The main concern for this space would be sequential write performance. I chose to go with the simple solution of dedicating a single Seagate Barracuda 2TB drive for this space.
Here’s an updated diagram that shows the entire storage solution as implemented on VIDEO2:
This storage implementation was quick and easy to setup, but I was really wondering how it would perform!
Here’s how I defined my “minimum bar” when using Adobe Premiere Pro CS6:
Output render performance was also a consideration, but this was second to preview performance since it has no impact on the editing experience.
What I found was that the entire system (hardware and software) met these criteria with no issues whatsoever. I used FRAPS to check my preview performance both with 1080p HD files, and with 4K REDCODE RAW files. In both cases I had some effects applied that I use frequently: brightness/contrast adjustment, sharpening, and the “fast color corrector”.
Premiere Pro CS6 previewing a 30fps AVCHD file at 100% magnification and full quality
Premiere Pro CS6 previewing a 24fps 4K REDCODE RAW file at 25% magnification and 1/4 quality
These quick tests confirmed that I would have more than acceptable editing workflow for both 1080p files and 4K files on VIDEO2 wen running Adobe Premiere Pro CS6.
This was great, but I was curious to know if this setup was going to perform better from a storage perspective than an all-on-one-SSD storage setup that would represent a simple “default” storage scenario.
As a storage test, I rendered a 4-minute uncompressed AVI clip with two storage configurations:
What I found was that my optimized storage solution performed better than an all-on-SSD configuration. Here are the results of my quick test:
SUCCESS! My storage overhaul for VIDEO2 is a huge improvement over the storage setup I had been running before the overhaul. Now I don’t need to worry about losing data at any time- I have mirrored resiliency. When I need more storage (it’s not if, but when) I can easily add more drives to my bulk Storage Space. With File History I can selectively store a history for files that are important to me. This system also boots super-fast, and launches apps super-fast. Finally, this storage setup didn’t break the bank.
Want to know more about Storage Spaces in Windows 8? Here’s a great Storage Spaces FAQ that provides a lot of great insights and information.
Stay up to date, follow me on Twitter!
ibpnyc - The test I ran was just to compare a single SSD storage solution with my optimized solution. The reason for the test was to see what effect moving scratch off primary storage and moving project files and output renders to dedicated logical storage devices. The improvement was likely mostly due to offloading the scratch drive from the SSD to the dual drive scratch space. Your setup looks good- I'd just look into adding one or two more drives for project files.
Hey Gavin nice post. I am in the same situation you were. I use my workstation for day to day operations as well as running Adobe CS6 Master Suite. I recently did a massive investment and built a system without properly researching the HDD expectations of Premiere and AE.
My system now is running:
OS & Apps: Win8 pro / CS6 Master
GPUs: NVidia Quadro 4000 + Tesla C2075.
CPUs: dual Xeons (E5-2670).
Primary: 500GB SSD
Scratch: 250GB SSD
Project Files: 2x 3TB mirrored
I did not think to setup drives for rendering but i have 3 x 75GB 10K raptors that I may setup to use for this.
My question is after this is all said and done with your drives and using Storage Spaces to setup your different scenarios I see that you noticed an increase in your 4K test. Is this test based on you older drive setup? Also what were you settings within the applications to take full advantage of your hardware?
thanks again for taking the time to address us users who are in a the same need of hardware to match these process hungry apps.
Good read. Are there any hardware sata controller cards you've found work best, or have had major issues, when using Storage Spaces? Particularly ones that aren't expensive RAID cards?
Short story is my HighPoint Rocket 640L (the nonRAID version) starts resetting the controller when I try to set up and write to a Storage Space on it. It may be a driver issue, as HighPoint doesn't list Windows 8/Server 2012 drivers for it (only the RAID version), so I might be in the market for a new card.
It is an issue of trust. Just like MS took away Previous Versions/Shadow Copy citing unexplained performance reason and deprecated Disk Management GUI, Backup etc, it can take away any storage related feature like Storage Spaces. So I choose to use DriveBender. There are other alternatives too like FlexRAID, Drive Pool etc. Who knows suddenly this feature may be discontinued in some later version of Windows? Also Storage Spaces has number of issues as it's a v1 product right now, mainly there is no rebalancing of data when new drive is added.