PDF of this tutorialResilient Packet Ring Technology
As service providers struggle to keep up with the demands of their customer base in the MAN, packet-based technologies are migrating from the traditional LAN to the MAN. Enterprise application growth is driving the increased bandwidth requirements and exceeding the existing capacity limits of the transport architectures in most provider networks. Until now, providers have generally deployed TDM technologies such as SONET/Synchronous Digital Hierarchy (SONET/SDH) for their offerings in this space. Inherent to Time Division Multiplexing (TDM) architectures, bandwidth is allocated in fixed amounts on point-to-point style circuits. This technology was successful for transporting traditional voice, circuit-switched links, but the growth of pure data transport has exceeded the capacity in many networks. In addition, providers are finding it very difficult to provision new services quickly and expensive to upgrade to meet the demands of their customers. Therefore, providers have been forced to look to packet-based technologies to scale these needs.
When reviewing the options available to scale this need, one immediately thinks of Ethernet as the leader for the inexpensive and flexible transport of packet-based topologies. However, Ethernet relies on the Spanning-Tree Protocol (802.1d) to provide for loop detection and elimination, generally recovering from a fault in 5 to 30 seconds.
SONET offers the ability to provide protection from physical and logical failures in the ring in 50 ms based on the automatic protection switching (APS) standard. Whereas some proprietary technologies exist for recovery in shorter periods, the 50 ms recovery time was needed for many of the voice services carried over these networks. In addition to recovery issues, Ethernet is based on point-to-point, non-meshed physical layouts not conducive to deployment over the existing ring-based architectures of SONET. These two keys issues left providers with few options for solving the bandwidth needs of their customers.
Enter packet-ring technologies. Cisco introduced Dynamic Packet Transport (DPT) in early 1999 based on a new concept called Spatial Reuse Protocol (SRP). This protocol takes advantage of both the ring-based architecture of SONET and the packet characteristics of Ethernet. DPT emerged as a new standard for deployment of these services for many providers seeking a solution without requiring the replacement of their existing fiber infrastructures. The Institute of Electrical and Electronics Engineers (IEEE) standards body quickly embraced this technology; subsequently, a new group was formed to advance the standardization of the technology. Many participate in this group, and they all work to make Resilient Packet Ring (RPR) technology more robust and interoperable.
The working group rapidly adopted numerous objectives set forth to drive the use of RPR technology into the service-provider market:
- Distributed Access—There is no "master node" required in an RPR ring, allowing for the loss of any node in the ring without affecting ring operation.
- Destination Stripping for Unicast Frames—In other packet-ring technologies, such as token ring, unicast frames must transit the entire ring to be removed only by the sending node after the intended receiver (802.5, Frame Copy Indication [FCI]) copies them. In RPR, the destination node removes unicast frames as they arrive, thereby freeing the bandwidth for downstream nodes.
- "Plug-and-Play" Support—This operation allows new nodes to be easily added to a ring without manual configuration or reconfiguration of other nodes in that ring.
- Bandwidth Distribution—Bandwidth is dynamically allocated to competing flows on demand as they enter the ring. Theories specific to this area abound. Currently, Cisco's DPT technology protects traffic already on the ring and queues traffic bound for the ring. The standards body has not reached an agreement on how this area should be handled in the draft specification.
- Dual-Ring Topology—Unlike SONET, which uses one ring to transmit live traffic and the other for protection, RPR utilizes both directions. By using both rings, transmitting data in opposite directions at the same time, RPR substantially increases fiber utilization.
- Rapid Protection Switching—RPR offers restoration from ring interruption in less than 50 ms. This restoration feature is very important for providers looking to maintain ring stability if a fiber is cut. In many voice networks, it is critical to the support of circuit-switched traffic. Also, as more IP traffic is deployed carrying loss-sensitive data, protection switching will help guarantee that traffic is not lost.
- Multicast Traffic Support—Multicast traffic travels around the entire ring one time. This topology is very different from mesh-based topologies where multicast traffic must be replicated by each device in the network in order to reach all destinations.
- Universal Physical-Layer (PHY) Support—RPR is a Media Access Control (MAC)–layer specification and, therefore, allows the use of existing PHYs already available.
- Multi-Gigabit Ethernet Transport—RPR can carry 10/100-Mbps Ethernet as well as 1 Gbps and 10 Gbps.
- QoS Support—The goal of RPR is to deliver TDM–like QoS offerings to enable service providers to offer many varying services while maintaining customer SLA requirements for jitter and guaranteed-bandwidth services.
Structured Design and Architecture for Cisco COMET Metro Ethernet Networks
Regardless of whether a pure, switched Layer-2 network or an Ethernet-over-multiprotocol label switching (EoMPLS) network is utilized, careful consideration of numerous parameters must be made. The following sections provide some general guidelines that must be considered before designing and deploying a metro network.
Failure Domain
A Layer-2 switched domain is considered to be a failure domain because a misconfigured or malfunctioning workstation can introduce errors that impact or disable the entire domain. For example, a jabbering network interface card (NIC) might flood the entire domain with broadcasts or undesirable frames at a very high rate. A protocol malfunction (for example, spanning-tree error or misconfiguration) can inhibit a large part of the network. Problems of this nature can be very difficult to localize in a flat, switched Ethernet environment. Therefore, care must be taken in terms of how this type of network is deployed.
In this model, it is strongly recommended that each enterprise customer be mapped to a virtual LAN (VLAN). This set-up affords the service provider the ability to segment the network by customer. Although it could be possible to have multiple enterprise customers per VLAN, this set-up is considered undesirable for numerous reasons. First, an unexpected broadcast storm in one customer's network could affect the performance of the other customers on that VLAN. Second, and perhaps more important, the customers will have the ability to "sniff" the other customers' traffic, providing for massive security breaches. Finally, because of the inherent ability to sniff Ethernet traffic on the wire, a malicious individual could cause significant damage to multiple customers' networks. This scenario could potentially leave the service provider open to violations in its SLAs to its customers, to say nothing of a poor customer-service situation.
Service providers can take many steps to limit the failure domain per VLAN. First, service providers can limit the number of switches that are participating in that VLAN. Cisco's VLAN trunking protocol (VTP) can enable every switch in the network to be aware of a new VLAN in the network and to autoconfigure trunk ports and spanning trees. In an enterprise network, this feature can be very helpful, but it can be highly detrimental in a service provider's Layer-2 network. Therefore, VTP should be disabled and VLANs manually configured as needed per switch. Secondly, Cisco technology can specify VLANs that are enabled on the 802.1Q trunk links. Only the VLANs of interest should be configured on a trunk link. Finally, the topology of the network should be well known and mapped out, both generally and specifically, per VLAN. This scenario allows the service provider to better isolate potential network faults.
Topology and Spanning-Tree Protocol Considerations
The topology of the network refers to the way in which the network is physically connected. It is very important for the service provider to understand the layout of the network and how to plan for its interconnection. If a service provider is implementing an Ethernet service over a SONET or Wavelength Division Multiplexing (WDM) network, then the Ethernet topology is more straightforward. In this case, the Ethernet network does not need to account for redundancy because redundancy can (and should) be accounted for at the transport layer by SONET or WDM. After all, SONET and WDM have significantly faster convergence times than spanning tree (50 milliseconds versus 50 seconds), meaning that spanning tree may not be necessary and Ethernet can be run in a simple point-to-point configuration with no loops.
However, if the service provider is building the network based on pure Ethernet transport, then the Ethernet topology becomes critically important. The first thing to account for is summarized by the rule: "If some redundancy is good, more redundancy is not better!" This mistake is one of the major ones made by network architects utilizing spanning tree and Ethernet. Spanning tree requires control packets (called bridge protocol data units, or BPDUs) to be sent out and processed by each switch in the broadcast domain to stabilize the topology and reroute around failures. The more complicated the network, the more time it will take for the network to converge. In addition, a large Layer-2 switched network may enter a state in which the central processing unit (CPU) is so busy processing BPDUs that some are missed, preventing the spanning tree from ever recovering. It was not uncommon in the early days of VLANs to have a network in such a state that all redundancy had to be removed just to stabilize the network.
The network should be designed in such a way that the primary and secondary root bridges of the spanning tree can be easily and readily identified. These switches should be located in a central point of presence (POP).
Virtual LANs
A VLAN is essentially an extended Layer-2 switched domain—that is, a broadcast domain that extends as far as the VLAN reaches. If several VLANs coexist across a set of Layer-2 switches, each individual VLAN has the same characteristics of a failure domain, broadcast domain, and spanning-tree domain, as described previously. Therefore, although customers can use VLANs to segment the metro network, deploying pervasive VLANs throughout the metro introduces complexity and reduces the deterministic behavior of the network. Avoiding loops and restricting VLANs to the specific Layer-2 switch where they have a presence minimizes the complexity.
802.1Q Encapsulation
802.1Q encapsulation, often referred to as QinQ, provides a VLAN tunneling mechanism by encapsulating a frame tagged with an 802.1Q header with another 802.1Q header. (Extreme Networks has similar functionality; its solution is called Virtual MAN, or VMAN.) This means that an enterprise could transport multiple VLANs across the service provider's network without interfering with the service provider's VLAN identifications. For example, Acme, Inc. could send 10 VLANs, with IDs 1 through 10, into the service-provider cloud. The service provider would encapsulate those frames with another 802.1Q header with its own VLAN, let's say VLAN 100. The service provider would transport VLAN 100 across its own network and, wherever VLAN 100 terminated, out would come the 10 customer VLANs. In theory, this means that a service provider could support 4,096 VLANs, with each VLAN containing 4,096 customer VLANs.
Some practical issues must be considered when implementing a QinQ encapsulation. First, QinQ assumes that customers want to transport their VLANs and spanning trees across a MAN or WAN, and, as discussed previously, this assumption is not true for most enterprise customers. This assumption has led to the failure of many previous transparent LAN services (TLSs). Second, one of the main benefits of QinQ is that, in theory, it scales the number of VLANs in the network from 4,096 to 40,962, or 16,777,216 VLANs. In reality, however, it is unlikely that an enterprise customer would be willing to put its VLANs in a service provider's "super-VLAN" with numerous other customers. The risk for security breaches and spanning-tree events damaging their network is far too great. Keep in mind that the service provider will still switch frames based on only the "outer" tag, not both tags.
Ethernet over Multiprotocol Label Switching
Many service providers are looking to expand their metro networks to very large scales or perhaps to inter-metro areas. A pure Layer-2 solution is limited by the IEEE 802.1Q specification to 4,096 VLANs. Therefore, a service provider would be able to support only 4,096 customers within the metropolitan area. To scale beyond the 4,096 VLAN limit, EoMPLS can be utilized.
Using this technology, a particular VLAN can be mapped to an EoMPLS tunnel. The provider-edge router will then transport that tunnel through the MPLS network. It is important to note a few points here regarding MPLS and its Ethernet type. First, MPLS resides on top of an IP network backbone. This set-up inherently allows the network to scale as well as provides the mapping between the MPLS label and some underlying intelligence. Therefore, it is strongly recommended that the network architect understand the best practices and guidelines for IP routing deployment on such protocols as OSPF, Intermediate System-to-Intermediate System (IS-IS) or Border Gateway Protocol (BGP).
Secondly, EoMPLS is a point-to-point solution only (based on the Martini draft; there is a follow-up draft that has not been readily adopted called Kompella, which allows for an Ethernet point-to-multipoint broadcast domain). For example, if the service provider wants to transport VLAN 100 across an EoMPLS network, the other side of the MPLS cloud can have only a single exit point. This issue, however, is not important in network design, because the enterprise will utilize multiple VLANs, one per destination site exiting the MPLS cloud. Examples of how to set up this scenario are discussed later in this tutorial.
Cisco COMET UCP Protocols
Perhaps more important than the technology differences between each protocol is how customers intend to apply each of the protocols. The way service providers apply unified control plane (UCP) protocols is important later in this section, where both the specific feature requirements and platforms and relevant protocols for different opportunity areas are discussed.
Figure 5. O–ONI and GMPLS Protocols
Figure 5 illustrates the basic aspects of both Generalized Multiprotocol Label Switching (GMPLS) and Optical User-to-Network Interface (O–UNI) protocols and how the protocols apply to routing both within a domain and between a user- (or client)-side interface and that domain. O–UNI is illustrated on the right side of the network, emphasizing that there is a clearly delineated boundary between which a client-side UNI device (O–UNI–C) communicates with a network-side UNI device (O–UNI–N). The differences between the function of the O–UNI–C and the O–UNI–N is that the O–UNI–C provides a signaling termination function, whereas O–UNI–N provides a signaling pass-through and interworking function, circuit routing, and reachability information. Between the nodes on this interface, information about light paths available on the network is presented. Light paths are illustrated in Figure 5 by the lines above the O–UNI interface. If O–UNI is employed, the optical transport network (OTN) can run any kind of routing, including GMPLS, other standards such as Private Network-to-Network Interface (PNNI) or OSPF, or even a proprietary routing protocol can be employed.
The interface between an edge GMPLS node and a GMPLS label-switched router (LSR) on the network side can also be referred to as a User Network Interface (UNI), whereas the interface between two network-side LSRs may be referred to as a Network-to-Network Interface (NNI). Nonetheless, GMPLS does not specify separately a UNI and a NNI protocol, an important point to understand when looking at the requirements. In GMPLS, edge nodes are simply connected to LSRs on the network side, and these LSRs are in turn connected between them. There is no delineated boundary over which a distinct protocol function is introduced such as with O–UNI. Of course, the lack of defined boundary and distinct protocol set does not mean the behavior of an edge node needs to be exactly the same as the behavior of an LSR on the network side. Specifically, in the aforementioned case, the edge node might be responsible for signaling paths across the network. If GMPLS is used, however, the edge node needs to communicate as a peer to the network-side device. Specifically, the two network elements will share topology, addressing, and other types of routing information.
In fact, the boundaries between devices are not only divided along protocol layers, but they are also divided between different operations management groups. Indeed, service providers typically have two or more distinct operations groups specific to either data service or transport layers. Typically one group owns the provisioning, operations, and management of the transport and another is responsible for the functions for data services. Communications between the data-services and transport organizations are defined by a workflow process whereby orders are submitted by the data-services group and then subsequently filled by the transport group. This group distinction has a major impact on the choice of protocols. Service providers with distinct groups where one supports data services and the other supports transport services are more inclined to desire strict adherence to a non-routing–enabled boundary between the two administrative functions of these groups. Here O–UNI is a preferred method because it separates the roles of each operations department.
