Building an enterprise storage area networking (SAN) infrastructure involves a great deal of relatively complex technology. A SAN has three major components: the interconnecting fabric, the management software, and the storage system. To ensure optimal utilization and flawless application support, planning and integration are paramount.

The Interconnecting Fabric

In an extended definition of the fabric, I will include host bus adapters, the interconnecting cables and Fibre Channel protocols, bridges, hubs, and switches. Figure 1 shows a simplified diagram of the interconnecting fabric components

A Fibre Channel SAN consists of a number of servers and storage subsystems connected through a high-speed hub or switch. Fibre Channel offers several important benefits, breaking the SCSI barriers in both speed and distance. Fibre Channel, designed from the outset for low-latency SANs, offers superior performance and scalability, paired with network manageability and flexibility. It scales well by allowing users to add storage capacity without needing to reconfigure servers, and it is also manageable as a separate element within the overall network infrastructure, promoting quicker fault recognition and error correction. Fibre Channel uses very large block sizes to transfer data, with delivery guaranteed at the hardware level. Today, you can transmit data at one gigabit per second (1Gbps), and that number is expected to double over the next two years.

Fibre Channel Topology: Two Fibre Channel topologies support a SAN infrastructure. The first is an arbitrated loop (FC-AL) in which data is passed from node to node, and all nodes share the same bandwidth. Up to 126 physical host or storage devices can theoretically be connected to the loop, but the response time sags noticeably when a loop exceeds 6 to 8 arrays (in fact, as additional devices are added to an arbitrated loop, some throughput is lost simply to arbitration overhead). FC-AL offers flexibility and scalability in terms of device count and requires limited changes to existing structural implementations at an affordable cost. However, the loop topology has lower aggregate bandwidth than a switched topology, and it only allows a single connection at a time. Although FC-AL has limited protection mechanisms, it is quite adequate in many situations. Its great advantage is that it is readily available and requires minimal installation effort. The majority of SAN installations in 1998 used the FC-AL topology.

The second topology, "switched topology," uses switches to establish many concurrent connections between different nodes, each connection having its own dedicated bandwidth. A switched fabric enables the movement of large blocks of data, is scalable from small to very large enterprise environments, and can support high levels of availability. A switched fabric is more costly to purchase, but generally has a lower cost of ownership than hubs in any mission-critical environment. Switched fabrics require some infrastructure implementation and management software, and have suffered until now from interoperability problems.

SCSI Switch Topology: There is limited implementation of SCSI switches, which I believe is very valuable for an interim period transitioning from the legacy SCSI to Fibre Channel. Gigalabs champions the SCSI switch approach, and CNT offers it in two products.

Adapters, Bridges, and Routers: The addition to the Fibre Channel fabric is a host bus adapter (HBA). An HBA allows you to connect a server system bus with an external device. HBA vendors include Emulex, Q-Logic, and jni (Jaycor). Bridging is a one-to-one conversion that takes place between SCSI or ESCON and Fibre Channel. Routing converts between multiple SCSI buses and one or many Fibre Channel ports. Bridges and routers help extend the life of existing storage infrastructures, enabling a progressive migration to Fibre Channel. CrossRoads and CNT are the leading vendors of bridges and routers.

Hubs: In a SAN configuration, a server will gain access to the storage network via a connection to one or more Fibre Channel hubs or switches. In a storage network, these network interconnect devices act much the same as they do in a traditional network.

A hub is a device that attaches multiple nodes to an FC-AL with all nodes sharing the 100 megabyte-per-second (100MBps) bandwidth. Managed hubs provide a limited fault isolation otherwise unavailable in an FC-AL topology due to their ability to automatically bypass a failed port. Hubs typically allow the connection of 7 to 12 nodes; you can cascade them for greater connectivity. The major limitation of hubs is that their bandwidth is shared; the failure of a single element can disrupt a critical application. Also, only one communication between two nodes can take place at one time. Hubs are very helpful in small configurations. Their price per port is about one quarter the price per port of a switch.

Managed hubs offer techniques for providing limited statistical information about port status and activity, usually through a separate management station. The management station often uses Simple Network Management Protocol (SNMP) Management Information Bases (MIBs) as a standard protocol for providing this information. The market leaders for hubs are Gadzoox and Vixel. Other suppliers include StorageTek, G2 Networks, and Emulex.

Switches: A Fibre Channel switch routes the data. It can shift circuit connections as needed to support data-transmission requests. You can expand this type of connection to include more than one switch and multiple nodes. In this category of switch, any port on any switch can provide-subject to bandwidth availability-full-speed access to any other port on the network. The network consists of a fabric of linked switches.

Switches allow simultaneous conversations across their ports at full bandwidth, essentially multiplying bandwidth by the number of connected nodes. You can use switches, alone or in combination with hubs, to build Fibre Channel fabrics, or switched fabrics of Fibre Channel connections, to provide the maximum in connectivity and bandwidth. Figure 2 shows the migration path from a single hub to a switched fabric with connectivity extending to a WAN. The leading suppliers of switches are Brocade for Fibre Channel and McData for ESCON. Other suppliers include Ancor Communications and Vixel.

A number of parameters measure hub and switch performance, the most obvious of which are the number of ports and the price. Complexity increases exponentially with the number of ports, but so does the device's usefulness in a large enterprise environment. Cascading several switches to obtain the same result can potentially congest the links between switches; careful topology planning is key to avoiding this situation. But many other issues and parameters are equally important such as the classes of service supported, frame latency, online discovery, zoning or partitioning, multicasting (fanning the traffic to multiple outgoing ports simultaneously), and different availability, serviceability, maintainability, and manageability features.

In an effort to compromise between functionality and price, some vendors have recently introduced Switching Hubs, which allow remote switching or routing of the port. Switching Hubs provide the ease of plug-and-play operation with existing SAN servers, storage subsystems, and hubs without needing the OS, firmware, or hardware changes that backbone fabric switches may require. The price for a Switching Hub is halfway between those of a hub and a switch. Suppliers include Gadzoox and StorageTek. Most vendors offer hubs and switches with an optional management software.

By: Farid Neema

This Paper was written by:
PERIPHERAL CONCEPTS, INC.
351 Hitchcock Way, Suite #B-200
Santa Barbara, California, 93105
Tel: (805) 563-9491
fneema@periconcepts.com

and published in the May 1999 issue of Windows NT Magazine