The City of Sioux Falls, South Dakota has much to brag about. In addition to its success in attracting new business to the area, Sioux Falls has been consistently ranked one of the most tax‐friendly U.S. cities to live in and retire to. Forbes Magazine also designated it one of the top cities in which to find a new job. Such accolades have led to major growth in the city's population, now over 154,000 people, as well as in the city services and infrastructure needed to support them. By default, the City of Sioux Falls Central Services Department—and its 20‐person IT staff—was forced to find more budget‐conscious, effi‐cient ways to support the growing data and application needs of the 22 departments and 1,100 city employees now under its care. "We're like a whole bunch of different com‐panies all in one," said Monte Watembach, the city's data center supervisor and one of the small IT staff that supports departments as diverse as the city clerk's office, library, health department, city parks, public works, and police and fire departments.
Applications on growing standalone VMware environment were at risk of single‐point failure
VMware server virtualization technology held great promise to help the city save money and grow its applications. With that in mind, Watembach and his team began putting a handful of virtual machines on first one and then two standalone ESX Servers. But when the number of VMs reached between 16 and 20, it became apparent that a more reliable environment was needed before the city could grow its virtual environment further. "We were happy with VMware, but started to worry that the failure of a single ESX Server would bring down lots of applications," said Watembach. "We just got to that point where we realized we couldn't put all of our eggs in one basket anymore. If we wanted to continue, we had to go to shared storage and [VMware] VMotion®."
At the same time, Central Services had begun to experience a host of other IT management issues that Watembach hoped a new fault‐tolerant, shared storage platform could resolve. As with many organizations, tape backup times for the city's terabytes of data had increased and now threatened nightly backup windows. Restores were often an arduous, multihour process that bogged down the IT staff. In the police department, tape headaches of a different kind were also in evidence. Police video management, tracking, and storage had become an onerous activity that the department wanted to replace with a digital video recording system based on Insight Video Net (IVN) for better retention and regulatory compliance. Could the video system be supported on virtual machines? Watembach thought so. He also suspected that the same shared storage platform would work better than the all‐in‐one solution being considered. It would also allow Central Services to take on more of the system's ongoing management and storage, thereby freeing up police resources.
Given the growth expected in virtual servers, Watembach hoped that a shared storage environment would also make the task of adding storage capacity more efficient. On the city's standalone servers and legacy storage, the process required off‐hours time, and plenty of it. "With our old storage solution, when a volume ran out of space, I had to find new space somewhere else, robocopy all of the data to it, then remap everybody to that new location. It would have taken about 6 hours to robocopy this stuff over the first time. Then it had to be after hours to get everything copied over the final time and make changes, along with hours of planning and configuring."
Pair VMware with NetApp for better reliability, faster data protection, and rapid restores
For help in upgrading the VMware server/storage infrastructure, Watembach and his team turned to John Conlin at Northland Systems, a NetApp partner and provider of IT solutions. Northland Systems outlined a game plan that would help the city achieve two of its main goals: growing its ESX server farm and mirroring the data off site for extra protection. After Watembach and his team evaluated EMC and NetApp, a short demo of NetApp Snapshot™ functionality tipped the scales in NetApp's favor.
"As we were looking at the different products out in the market, NetApp Snapshot really struck us as a lot more intelligent than other vendor solutions. The design makes it more efficient and makes it perform better," said Watembach, who compared it to less efficient copy‐on‐write snapshot and mirroring examples he'd seen from other vendors. He thought that coupling local Snapshot copies with NetApp SnapMirror® would also make it easy to replicate Snapshot data off site to one of the city's fire stations. He liked the fact that SnapMirror transmitted only the changes, or deltas, since the previous Snapshot copy.
For help in implementing the overall VMware and NetApp solution, Watembach and his team turned to a strong support triangle of individuals at Northland Systems, VMware, and NetApp, including a VMware authorized consultant and a NetApp audio visual specialist, for help in supporting the police department's new IVN video system.
Today, a NetApp FAS3050C uses the iSCSI Protocol to support 30+ VMs installed across three ESX Server nodes. An underlying IBM Blade Server connects the ESX nodes to the NetApp storage. NetApp's multiprotocol sup‐port has helped with the IVN video system especially, allowing it to simultaneously sup‐port the system's virtual IVN portal server and virtual SQL Server® database storage (via iSCSI), while also supporting fast video file storage and access (via CIFS). SnapMirror now replicates city data, including about 30GB of daily police video, to a second NetApp FAS3020 system at the fire station.
Reduced backup and restore times from multiple hours down to minutes, or even seconds
Even Watembach was surprised by how well NetApp Snapshot has worked since imple‐mentation. "NetApp Snapshot has been fantastic compared to what we used to do," he said, describing the change from the previous tape backups. "NetApp has really been a burden off our shoulders. It takes less time for us to manage. We now back up all our data almost instantly and reliably and replicate it remotely. Restores are also easier and quicker from a point in time." He is looking forward to implementing other NetApp SnapManager® solutions for Microsoft® Exchange and VMware as well.
How the City of Sioux Falls uses NetApp storage.
The City of Sioux Falls supports its growing VMware environment with a NetApp FAS3050C storage system connected via iSCSI to an IBM Blade Server environment. About 30 virtual machines are supported across three ESX Server nodes, with support for applications, now virtualized, like the police department's IVN digital video system and more traditional applications like Microsoft Exchange, Microsoft SQL Server, and Blackberry Enterprise Server. A second NetApp FAS3020 system resides at the city's remote DR site, with NetApp SnapMirror used to replicate data between the two storage devices. The city credits use of NetApp technologies like Snapshot, SnapMirror, and FlexVol with significant time and space savings, increases in IT efficiency, performance, and overall system reliability.
Implemented reliable, turnkey remote disaster recovery already happy with the automated remote replication he'd seen with NetApp SnapMirror, Watembach was even more impressed with the performance of SnapMirror after a WAN disruption caused by water damage to the dark fiber connection. Moving to the city's backup wireless connection, the reliable processes of SnapMirror and reduced WAN footprint allowed it to continue working flawlessly, even when forced to transmit point‐to‐point wirelessly, bouncing from city hall, off the city's water tower signal, to the fire station. "It was impressive to see that when we had problems with the fiber or other things, we never did have problems with SnapMirror. It's been pretty bulletproof," Watembach said.
NetApp's performance with iSCSI has also exceeded expectations. "Performance has been good with NetApp on iSCSI," Watembach confirmed. "Every time I push the envelope in our virtual environment, I have some reluctance from someone in the IT department who asks, 'Is this going to perform how we want?' But we just keep throwing things at the NetApp system and have yet to find a bottleneck that tells us to stop."
Reduced time to provision storage from 6+ hours to 5 to 10 minutes
Like the success he's experienced with NetApp Snapshot, Watembach is similarly impressed by how little time it takes to expand NetApp's flexible volumes and provision new storage, thanks to NetApp FlexVol® technology. "We've also used NetApp FlexVol to add more space to a volume in minutes instead of the 6+ hours it used to take with our old solution," he said.
For a small IT staff like theirs, gaining back this kind of time is golden. "We have to be very efficient with the staff we have. That's why we have VMware. It's also why we have a SAN from NetApp that's easier to manage. We just don't have a lot of staff to manage all those different components otherwise." The combination of NetApp technologies like Snapshot, SnapMirror, and FlexVol has cut IT time by several hours and continues to help the team work more efficiently.
When asked what he would tell others about the city's experience in pairing VMware with NetApp, Watembach responded, "I would probably say they need to take a look at both of those combinations together, VMware and NetApp. They will make their life much easier."