Storage Area Networks (SANs) have emerged as a very efficient
method of creating large storage systems. They allow storage
arrays to be shared between multiple servers. This enables an
economy of scale that is not possible when each server has its own
directly attached storage device.
In recent years, there has been an increasing emphasis on building
disaster-tolerant computing systems. These systems employ
SANs that are located at geographically dispersed sites. In order to
accomplish this, it is necessary to bridge the storage networks
(typically based on the Fibre Channel protocol) across the wide
area. There are many technologies available for doing this
(SONET/SDH, IP, DWDM, ATM, etc). I was one of the co-authors of
the industry specifications for carrying Fibre Channel over the wide
area. Fibre Channel Specifications are regulated by the INCITS/T11
technical committee (International Committee for Information
Technology Standards). I was also one of the co-authors of the
Fibre Channel over TCP/IP (RFC 3821) specification. Really, it's a
Request for Comment (RFC), but that's as close as the IETF
(Internet Engineering Task Force) gets to specifications.
While at LightSand, we designed and orchestrated a major
demonstration with several vendors that provided a very high
performance distributed storage cluster. We demonstrated
sustained throughput of 400 MBytes per second at a distance of
more than 50 miles. The industry participants included the Marconi,
SGI, Brocade, and the Naval Research Laboratory.
Measuring Performance of Distributed Networks
When moving data over long distances, latency can become a
significant factor. This is because most networking protocols were
designed to be operated in environments with very small latency.
Since there seems to be quite a bit of confusion about bandwidth
and latency, I wrote an educational piece on the subject for InfoStor
magazine. An excellent way to quantify the performance of a
distributed network is by simultaneously looking at the rate of data
movement (bits per second) and the distance over which the data is
being moved. The resulting figure-of-merit can be expressed in
bit-meters per second (bmps). LightSand's first generation of
product provided more than 4700 Tera-bmps (73 MBytes per
second at a distance up to 8000 Km). This was very close to the
world record at the time (late 2002). Since then, the world record
has moved considerably higher. The current Internet2 Land-Speed
Record is 216,000 Tera-bmps.
Differences between Transport Technologies
Wide area network technologies can be divided into two main
categories: Channels and routed networks. The channels cover
such technologies as SONET/SDH, ATM, DWDM, and even
dedicated fiber. Routed networks are based on the IP protocol
which is in turn carried over the channels listed above (primarily
SONET/SDH and ATM). Each method has benefits and drawbacks.
IP networks are more scalable but cannot provide deterministic
performance guarantees. Channels tend to have very deterministic
performance but cannot scale because they require dedicated links.
I wrote a brief white paper that discussed Latency Variance in Wide
Area Transport. It discussed some of the behavioral differences
between moving data using SONET (deterministic latency with
minimal data loss) and routed IP networks (non-deterministic
latency and prone to large data loss). It is true that TCP can restore
reliability to the transmission but it does so at the price of
predictable latency. For many applications, this is not a problem.
However, there are many applications (such as high performance
storage) that can be very sensitive to unpredictable latency.
I also wrote a white paper that addresses the Economics of Large
Scale Data Transfer. It compares these technologies from an
economic point-of-view. The performance analysis in this paper is
based on the TCP Reno algorithm. This is the most commonly
used algorithm to manage congestion in TCP/IP networks. Many
new TCP control algorithms have been developed. In particular, the
TCP FAST algorithm developed at the CalTech NETLab shows
tremendous promise for the future of routed IP networks.
If you have a large amount of data to move, dedicated channels will
always be more cost-effective. If you want to share the link, routed
IP is still the best answer.
There are many subtle but important aspects to designing
distributed storage arrays. If you have any questions, please feel
free to contact me.
method of creating large storage systems. They allow storage
arrays to be shared between multiple servers. This enables an
economy of scale that is not possible when each server has its own
directly attached storage device.
In recent years, there has been an increasing emphasis on building
disaster-tolerant computing systems. These systems employ
SANs that are located at geographically dispersed sites. In order to
accomplish this, it is necessary to bridge the storage networks
(typically based on the Fibre Channel protocol) across the wide
area. There are many technologies available for doing this
(SONET/SDH, IP, DWDM, ATM, etc). I was one of the co-authors of
the industry specifications for carrying Fibre Channel over the wide
area. Fibre Channel Specifications are regulated by the INCITS/T11
technical committee (International Committee for Information
Technology Standards). I was also one of the co-authors of the
Fibre Channel over TCP/IP (RFC 3821) specification. Really, it's a
Request for Comment (RFC), but that's as close as the IETF
(Internet Engineering Task Force) gets to specifications.
While at LightSand, we designed and orchestrated a major
demonstration with several vendors that provided a very high
performance distributed storage cluster. We demonstrated
sustained throughput of 400 MBytes per second at a distance of
more than 50 miles. The industry participants included the Marconi,
SGI, Brocade, and the Naval Research Laboratory.
Measuring Performance of Distributed Networks
When moving data over long distances, latency can become a
significant factor. This is because most networking protocols were
designed to be operated in environments with very small latency.
Since there seems to be quite a bit of confusion about bandwidth
and latency, I wrote an educational piece on the subject for InfoStor
magazine. An excellent way to quantify the performance of a
distributed network is by simultaneously looking at the rate of data
movement (bits per second) and the distance over which the data is
being moved. The resulting figure-of-merit can be expressed in
bit-meters per second (bmps). LightSand's first generation of
product provided more than 4700 Tera-bmps (73 MBytes per
second at a distance up to 8000 Km). This was very close to the
world record at the time (late 2002). Since then, the world record
has moved considerably higher. The current Internet2 Land-Speed
Record is 216,000 Tera-bmps.
Differences between Transport Technologies
Wide area network technologies can be divided into two main
categories: Channels and routed networks. The channels cover
such technologies as SONET/SDH, ATM, DWDM, and even
dedicated fiber. Routed networks are based on the IP protocol
which is in turn carried over the channels listed above (primarily
SONET/SDH and ATM). Each method has benefits and drawbacks.
IP networks are more scalable but cannot provide deterministic
performance guarantees. Channels tend to have very deterministic
performance but cannot scale because they require dedicated links.
I wrote a brief white paper that discussed Latency Variance in Wide
Area Transport. It discussed some of the behavioral differences
between moving data using SONET (deterministic latency with
minimal data loss) and routed IP networks (non-deterministic
latency and prone to large data loss). It is true that TCP can restore
reliability to the transmission but it does so at the price of
predictable latency. For many applications, this is not a problem.
However, there are many applications (such as high performance
storage) that can be very sensitive to unpredictable latency.
I also wrote a white paper that addresses the Economics of Large
Scale Data Transfer. It compares these technologies from an
economic point-of-view. The performance analysis in this paper is
based on the TCP Reno algorithm. This is the most commonly
used algorithm to manage congestion in TCP/IP networks. Many
new TCP control algorithms have been developed. In particular, the
TCP FAST algorithm developed at the CalTech NETLab shows
tremendous promise for the future of routed IP networks.
If you have a large amount of data to move, dedicated channels will
always be more cost-effective. If you want to share the link, routed
IP is still the best answer.
There are many subtle but important aspects to designing
distributed storage arrays. If you have any questions, please feel
free to contact me.
Distributed Storage
Networks
Networks
While at LightSand, I
was very involved in the
design of distributed
storage area networks.
These are "back-end"
networks that allow
servers in geographically
separated locations to
share the same storage
arrays.
was very involved in the
design of distributed
storage area networks.
These are "back-end"
networks that allow
servers in geographically
separated locations to
share the same storage
arrays.
Andy Helland
 | |  | |  | |  | |  | |  |
| | | | | | | | | | |