Symmetric DSL (SDSL) can provide up to 8Mbps both ways, and may provide a cost-effective approach to delivering the bandwidth currently required.
Some carriers provide lower cost contended 2Mbps services based on SDSL and these may be a viable option during a transition to using leased lines. However, when congested, if such a service is contended at 20:1 it may prove to be no better than a conventional ISDN2 line at 128Kbps. Where available, a service with a lower contention ratio, say at 5:1, may provide a more viable interim solution.
Inexpensive telephone grade copper pairs, providing symmetrical circuits called EPS8 or EPS9 may be purchased directly from BT only. These options require a high degree of technical knowledge to implement, as customers have to supply their own circuit termination devices to create a broadband link, and as a result are not recommended.
Some ISPs have announced that customers in parts of London can purchase an uncontended ADSL service. This service provides a download speed of 2 Mbps and an upload speed of 250 Kbps, but has the advantage that the bandwidth is not shared between customers. The maximum distance from the exchange is 3.5 Km. In 2003, about 20 exchanges in London were enabled for the service and the roll out across the UK will depend on sufficient interest being shown.
Leased Lines
Leased lines are widely available from all telecommunications providers and are often the only choice for bandwidths greater than 2Mbps. They are available in rural and many remote locations.
Under an agreement with OFTEL (now OFCOM), there is a special BT tariff for Megastream (2Mbps) leased lines available to schools, called Learning Stream. This is currently widely deployed in the schools sector. Learning Stream is available essentially at bandwidths of 2 Mbps and 34 Mbps (High bandwidth Learning Stream), the 8 Mbps has been discontinued. Some local authorities implement clusters of schools which share a Learning Stream link.
An overview of the Digital Hierarchy technology, which underpins these leased line products, is provided in section 2.1.9.
Short Distance Ethernet Services
Recent advances in fibre optic technology and switching electronics, as well a greatly reduced bandwidth costs, have meant that it is now both possible and cost effective to extend Ethernet services on a regional level.
LAN Extension Services products are offered by a number of telecommunication carriers under various brand names (e.g. BT’s LES2, LES10, LES100; Thus’s City Ethernet; etc) for short haul point-to-point distances up to 25Km. Flexible interfaces allow customers to select service speeds and connections that best meet their individual needs.
Link status of LES circuits can be very hard to determine as line faults or errors are not propagated through the network. The Ethernet Switch devices are owned and operated by the Service Provider. The customer’s equipment that connects to the Service Provider’s switch sees no change of state because the Ethernet switch is still up and active. These failure modes that cannot be detected by the customer rely on layer 3 protocols to detect faults/ errors. This can be problematic when it comes to network management.
The low costs associated with these products present an attractive value proposition for network planners but the limitations discussed need to be considered especially with respect to connections where reliability is critical. Customers opting for LAN Extension products should fully understand the technical specification of the solution being offered and the possible drawbacks.
Long Distance Ethernet Services
These are relatively newer services available from a number of telecommunications providers, where the distance limitations are removed. They are typically priced so that bandwidth can be subscribed to in steps, below the physical capacity of the link, and therefore may be more cost effective than leased lines. Examples include BT’s Megastream Ethernet and Thus’s National Ethernet.
VPN Services
These are available from a number of suppliers under a variety of names (e.g.: BT’s Metro VPN, IP Clear.). These are managed services, often viewed as “cloud” network services. These are potentially useful where there are many sites to connect together, but the tariffs can be complex and the costs may be high. Such a VPN solution is likely to limit the available IP features over the connection, whereas other solutions do not have this limitation.
Satellite Technologies
One-way satellite systems provide a down link only and interaction therefore necessitates an uplink via a dedicated telephone line. Two-way satellite links provide asymmetric services similar to premium ADSL, with a contention ratio up to 20:1.
Two-way satellite systems (e.g. Gilat 360 and Astra BBI) are available from a number of suppliers, and can be deployed virtually anywhere in the UK at a fixed cost. The monthly costs compare with those of a premium ADSL service, although there are also significant initial installation costs. Most suppliers will provide options to buy or lease the equipment. In some regions, e.g. Wales, there are two-way satellite subsidy schemes in operation for those in remote areas. A dedicated 2 Mbps link would cost about ten times that of a typical leased line connection.
While the high latency associated with satellite networking precludes the use of real-time applications such as videoconferencing and Voice over IP, satellite services are of significant value in rural and remote areas.
Reference:
Wireless
In rural or remote locations there may be few alternatives available other than to use licensed or license-exempt radio frequency (RF) technology to provide communication links.
The deployment of a wireless network requires significant technical and operational expertise if a reliable service is to be achieved. Furthermore, any wireless transmission solution has to be well designed and managed; it is never a substitute for a wired equivalent when one is available. The greater susceptibility to interference means that contracts with service providers must be tight, guaranteeing fast detection of failing, or failed, wireless links. When such situations occur, speedy resolution of the issue should be enforced. The service level agreement (SLA) with a supplier should be checked in detail to ensure that suppliers adequately monitor and manage them.
However, there may be further commercially available options appearing as a result of the recent opening up of new frequencies (e.g. Band C - 5.8GHz) for such services.
The maturity of this technology has permitted the use of microwave links as the major trunk channel for long distance communication over ‘fixed wireless’. Bandwidth capacities range from single E1 to STM-1 (155Mbps). Wireless (or "free-space") communication technologies are, however, susceptible to interference from the weather, particularly rain. Microwave systems provide point to point links which are generally used to link together networking infrastructure devices in the same way as a leased circuit might be used. Point-to-point microwave links are terrain independent so long as there is line of sight between the sites at each end of the link. Where there is no direct line of sight, it may be still be possible to implement a link, via one or more intermediate stations located on radio masts or tall buildings.
Wireless Ethernet technology (license-exempt “WiFi”) provides multiple access networking over the air to many clients using devices known as wireless access points. These systems are commonly used to provide LAN connectivity, but can also be used to provider wider-area network coverage. With appropriate antennae, it is possible to provide large areas with wireless Ethernet. Further access points can be added within range of each other to extend the coverage over larger and larger areas. This might be used with just two access points, linking two schools located close to each other - in which case both schools could make use of a single broadband connection. Other applications featuring several access points with overlapping coverage might allow a small community to have access to the broadband connection at the local school. These are often known as "wireless mesh" networks. Wireless Ethernet is in general more prone to, and susceptible to, interference effects. Wireless may not be the optimum solution for point-to-point links, but it is relatively easy to deploy at low cost.
Wireless networks of any kind pose a greater security risk than wired networks. Microwave links are relatively difficult to intercept, as they operate over a tight signal beam; however wireless Ethernet and satellite signals are easily intercepted. When using wireless solutions it is critical that traffic is encrypted for transmission over the air. Care is needed when selecting the encryption mechanism, to ensure that it is not easy to crack. For example, the wired equivalent privacy (WEP) system of 802.11 standard wireless Ethernet systems has been shown to be relatively straightforward to crack. Many organisations therefore operate additional security measures over their wireless Ethernet systems. Further discussion of wireless security issues is presented in the Network Security document.
Infra red and laser optical links can be used to provide connectivity between buildings; however, a clear line of sight is required and the service can be affected by certain weather conditions. These links are practical only over short distances, less than 1.2Km, and are sometimes used in conjunction with an unlicensed radio backup.
Overview of Digital Hierarchy Technology
PDH (Plesiochronous Digital Hierarchy) is a technology used in telecommunications networks to transport large quantities of data over digital transport equipment such as fibre optics, copper and microwave radio systems. It is a widely deployed transmission system, traditionally used for low speed leased line data circuits from 2Mbps (E1) to 140Mbps (E4). However, it lacks the fault detection, performance monitoring capabilities and recovery mechanisms offered by the newer and more efficient SDH (Synchronous Digital Hierarchy).
SDH is bandwidth-flexible and, although based on multiples of 155Mbps (STM-1) up to 40Gbps (STM-256), it permits networking at the 2Mbps, 34Mbps and 140Mbps levels, thus accommodating the existing PDH signals.
SDH differs from PDH in that the exact rates that are used to transport the data are tightly synchronised to network based clocks. Therefore the entire network operates synchronously enabling the use of extremely high transmission rates. Unlike its predecessor, SDH is an intelligent system that provides advanced network management and a standard optical interface. SDH devices provide extensive mechanisms for fault detection, notification and recovery. Fault detection and the appropriate path protection switchover are achieved within milliseconds; therefore circuit failures can go unnoticed by the end user. Path protection is achieved by employing a self-healing ring architecture that is able to reroute traffic over backup transmission paths in the event one fails.
These capabilities are only made possible by the use of complex and advanced technologies which drive up the cost of associated equipment. However, SDH does provide supreme resilience and reliability levels and is expected to provide the transport infrastructure for worldwide telecommunications for the foreseeable future.
IP Addressing
Overview
To connect to the Internet, a globally visible IP address is required. As these are a scarce resource, it has become an arduous process to obtain large numbers of these "public" addresses. Other than the perceived barrier of a complex application process, public address space is available to meet any size of requirement that can be documented and justified, according to Internet Registry guidelines.
The most difficult of these guidelines requires that an application for a sizeable number of public IP addresses provide a detailed breakdown of proposed address use in an IP addressing plan. No provision of spare addressing is permitted for administrative ease, which would entail the use of different allocation and subnet sizes depending upon the size of school. No single IP addressing scheme could be applied to every school on the RBC/LA network.
The Internet Registry that serves Europe is the RIPE NCC; further information on public IP address application requirements is published on their web pages.
Because of these tight restrictions on assignment of public IP addresses, many networks choose to use private IP addressing. Private IP addresses are reserved by the Internet Assigned Numbers Authority (IANA), and will never be routed on the public Internet. Any part of this reserved address space can be used by any number of organisations without any prior applications or registrations; these address ranges are set out in IETF RFC1918.
Private IP address ranges:
The IANA:
RIPE NCC:
Public and Private Addressing Schemes
At first sight, using private address space for internal connectivity may seem ideal, as it removes the constraints on planning imposed by public address allocation. However, there are some considerations when choosing to use private address space.
Private address space is, by definition, private. Organisations connected to the same network will be able to interconnect using their private IP addresses; however it will be impossible to make connections to other networks, particularly the Internet. Another interesting issue is what happens when two organisations merge. If both are using the same part of private address space, work (most likely renumbering) will have to take place to enable both networks to interconnect.
To connect to external networks, a privately addressed network will generally use NAT (see below), or some form of proxy service - for example, a web proxy. By default, a publicly addressed network gives full local Internet connectivity. This in itself may, of course, be undesirable. As discussed in the Network Security document, a "default deny" policy is often the best policy; public IP addresses provide the opposite (“default allow”).
With private address space, connections to internal hosts cannot be originated from outside the network, unless specifically enabled in the NAT configuration. With public address space, all connections are possible, unless specifically disabled by means of packet filters or a other firewalling techniques.
Provider Aggregateable and Provider Independent Addressing
Public IP address space is divided up into types; provider aggregateable (PA) addresses and provider independent (PI) address space. Before the early 1990s, all address space assigned was PI - the IP addresses used on a network bore no relation to any other organisation other than that using them.
As a result, the global Internet IP routing table had to maintain an entry for each and every IP network in use around the world. As the Internet grew, it was clear that this was not going to scale.
Address space is now almost always assigned as PA. Service providers are assigned blocks of address space for onward assignment to their customers. This allows the provider to use a single route in the global table to cover many customers, instead of needing to add a route for each specific customer network.
A major consideration in choosing between PA and PI addressing is what happens when a network changes service provider. To retain the routing economies of PA space, routes from one provider's address blocks should not be visible via a different service provider (although with Internet multihoming it is often the case that there is no way of avoiding this).
This implies that when changing provider, an organisation must renumber the whole of its network to new PA space. Where NAT is used, this is not too great an issue, as it is simply a matter of changing the NAT configuration to use a new pool of PA addresses.
However, where public addresses are used to number an entire network directly, moving to the new PA space is a significant issue.
(Note that the holding of PI space often involves the payment of service and/or membership fees to an Internet IP number registry.)
Network Address Translation
Network Address Translation (NAT) is a mechanism which permits hosts on a locally numbered IP network to appear to be using different addresses beyond an external border; typically a connection to a service provider. NAT is often used with a network numbered with private address space to permit external connectivity from local hosts. NAT is usually configured at the border of the privately address network, and essentially rewrites the internal IP address to a public IP address.
This may be a single external IP address, where many internal addresses are all translated to that one address. It is also possible to configure a pool of external addresses, allowing mapping between many external address. The mapping can be configured to be dynamic, on a connection by connection basis, or may be configured to map specific internal hosts to one or more specific external IP addresses.
For connections made outwards from the internal network, dynamic mapping is most likely to be sufficient. Static mapping is mostly used where it is necessary to have external access to an internal host – for example, a web server running on a machine on the internal network. In such cases, it is almost always essential to have a one-to-one mapping of public address to private address. For example, if two web servers map to the same external IP address, how would the NAT configuration know which internal box to forward the request to? Unless one server was to operate on a different port number, there would be no method of distinguishing between connections at the NAT level.
Configuring direct inward access to a network immediately decreases the security of the network. In almost all cases, external access requirements can be met using more secure technologies, such as proxy servers and virtual private networks (VPN).
Originally, NAT only changed information in the IP packet header; however some services such as H.323 videoconferencing embed IP addresses in the data portion of the packet. To enable the service to function, these IP addresses also need translating.
Most NAT equipment now has the functionality to detect data flows from services such as H.323, and will translate IP addresses in the packet data. Any organisation operating a NAT device on the RBC, LA or school network must ensure that the device provides support for any services such as H.323 that schools might be using.
Care must be taken to prevent overloading on a NAT device, or devices. The resource requirements (e.g. CPU load and memory) for NAT are proportionate to the number of hosts on the network, and not the bandwidth of the network. A network of many thousands of hosts will have the same demand for NAT resources regardless of whether the external connection is 2Mbps or 100Mbps.
NAT is often referred to as a security tool, which it is not. Elements of NAT do certainly contribute towards some level of security, by essentially rendering internal hosts unreachable from external hosts by default. However, it is no substitute for a well thought out and implemented security framework.
Wide Area Network Topologies
IP networks typically consist of two major components:
- a backbone infrastructure, consisting of points of presence ("PoPs") interconnected by high speed lines
- an access infrastructure used to connect customers and external networks to the backbone (also referred to as "aggregation")
Access infrastructures typically connect several sites to a common local access PoP using short distance links; the PoP is then itself connected to the backbone. In some topologies, access PoPs and backbone PoPs may well be at the same location; in others, access PoPs may be used for economic reasons, to allow several sites to share a single long distance (and therefore often expensive) link to the backbone.
Internet backbone infrastructures are generally constructed using a ring topology, making the network more resilient to a single backbone link or equipment failure. Should one link or switch/router fail in the ring, traffic will re-route the longer way around the ring (assuming correct configuring of the routing technology being used, of course).
Some use star based topologies, where aggregation points are directly connected to a central location. This is a more straightforward topology, but can be less resilient. If multiple aggregation points use the same single link to the backbone, failure of that link will affect many sites. Worse, if the equipment at the central location is not sufficiently redundantly provisioned, a failure here could disable the entire network.
Perhaps the worst topology is a chain of routers providing access to the core location. This provides progressive aggregation of access links, but failures along the chain have disastrous consequences.
Appendix A discusses these network topologies and resilience issues in further detail.
In addition to backbone links and equipment, many networks choose to operate an access device, commonly an IP router, at each customer site, extending the hand off (or demarcation) point to the customer beyond the telecoms supplier's equipment. This access device is typically located next to the supplier's equipment.
This allows for easy monitoring of the far end of the site link as it does not rely on any customer equipment, and also aids in troubleshooting problems. If the site access device is reachable from the wide area network without any problems, then problems are very likely to be within the local site network infrastructure.
Individual access devices at each site also enable site-specific configuration to be made locally - for example packet filters, or quality of service configurations.
Capacity planning is often a difficult issue, particularly if planning an entirely new network. Backbone capacity is seldom provided on any network on the assumption that customer access links will be near 100% loaded, so a 1:1 ratio of access bandwidth to backbone bandwidth is almost never seen.
Access links on school's networks are often heavily loaded; experience in some networks shows that an access to backbone bandwidth ratio of around 4:1 is likely to be required. It may be able to improve on this ratio by co-ordinating closely with schools to determine the mix of traffic on their access links. In many cases there will be a sizeable proportion of traffic at peak times that is not directly educationally related - managing this mix by disabling less important traffic at busy times may improve performance without the additional cost of capacity upgrades.
Routed or Switched Backbone
With the physical backbone established, the method of providing IP interconnectivity between the schools, network services, the national interconnect (via JANET) and other external networks should be selected.
Many networks operate a routed IP network, which generally makes the most efficient use of bandwidth. An IP routing protocol (such as OSPF or IS-IS) carries connectivity information for all IP networks connected to the RBC/LA backbone. Traffic is routed via the best available path to its destination (assuming optimal configuration of the routing protocol).
Other networks may choose to extend VLANs from central locations to individual schools, so that regardless of the physical topology of the network (see Appendix A); all traffic is brought into a central location before being forwarded to its destination. This arrangement will not usually deliver efficient use of capacity on the network, as much of the traffic on the network will cross the same link twice: once on the way in to the central location, and again on the way out.
IP routed networks will generally deliver more efficient use of bandwidth but can be more complex to configure than VLAN based networks. VLAN networks, on the other hand, may assist in the implementation of network security, by bringing all traffic to central locations.
Schools' Local Network Considerations
Physical Issues
In most cases, two pieces of equipment will require housing on-site at the school: the termination device from the telecommunications supplier on which the wide area link is delivered, and the access router to which the link is connected.
These devices are typically relatively small, and have no special environmental demands other than power and adequate ventilation, as for any other type of electrical device that produces heat. The access router may have internal fans that generate significant amounts of noise, and so may not be suitable for location in a general office environment.
The RBC/LA should inform schools of the space needed to house on-site equipment, and any special considerations for its location well in advance of delivery and connection to the network.
School LAN Infrastructure Issues
A well designed and provisioned local network will enable the school to benefit as fully as possible from its broadband connection. The network infrastructure at the school must be able to comply with the IP addressing plan, DNS architecture and other requirements of the RBC/LA network.
Suitable equipment will be required to connect to the hand-off point for the wide area network; this hand-off will most often be a full-duplex Ethernet connection (although the actual upstream connection off-site may be of a lower speed).
To successfully exploit the RBC/LA network and the Internet beyond, adequate capacity must be provisioned on-site, particularly if higher bandwidth applications such as video conferencing are expected to be used. Experience so far has shown problems associated with the local network at the school to be the cause of connectivity problems, and not lack of capacity in the wide area network.
The on-site networking technology will almost certainly be Ethernet based; many schools operate using Ethernet hubs, which are now an outdated technology and undesirable. An Ethernet hub is a simple shared medium, where all traffic on the hub is replicated onto each port, regardless of whether it is of interest to the device connected to that port.
This effect is compounded in many Ethernet hub designs when hubs are interconnected. For example, two eight port hubs connected together will cause 16 ports worth of traffic to be replicated on each hub port. Hubs with slightly more intelligence may not replicate traffic to this extent, but in a large network of hubs, problems with the protocol used to prevent unnecessarily replication can cause unexpected, and often intermittent and hard to trace, network problems.
It is recommended that schools should operate a full-duplex switched Ethernet network, at a speed of at least 100Mbps. Ethernet switches are in general more feature rich devices, allowing specific management of individual ports and facilities such as virtual LANs (VLAN). VLANs can be useful for separating out logical IP networks operating over the same physical infrastructure - one such use could be to separate administrative and teaching traffic (see later section).
A single Ethernet network operating with many hundreds of hosts will present performance problems. Ethernet uses a protocol called CSMA/CD (Carrier Sense Multiple Access / Collision Detect). It is the protocol that allows multiple devices to access a shared Ethernet or Fast Ethernet network. These devices form what is known as a collision domain in which only one device may transmit at any one time. The "Carrier Sense" function checks the wire to see if any other node is already sending something. If the LAN appears to be idle, then the node can begin to send data. However two nodes can begin to send data at the same time, and their signals will "collide" nanoseconds later resulting in a collision. When such a collision occurs, the two nodes stop transmitting, "back off", and try again later after a randomly chosen delay period. While the two devices involved in the collision are waiting to resend, it's possible for another device to send a packet, which may also be involved in another collision.
An Ethernet network based on switching overcomes this effect. Ethernet switches separate the network into microsegments, which should be single host segments. This creates collision-free domains which operate separately from each other. Further performance increases can by gained by using layer 3 enabled switching, or by splitting a large network into several VLANs. Using multiple VLANs requires local IP routing functionality to interconnect the VLANs, which is typically provided by a separate IP router.
A switch can also provide improvements in performance and security - notably a PC connected to an Ethernet hub has the ability to snoop on all traffic on the hub, if not the entire local network. A switch passes only traffic relevant to the connected device, including broadcast and multicast traffic relevant to the Ethernet or VLAN configured on that port.
Operational Issues
Unless maintenance or repair work is being undertaken or as instructed by the RBC/LA, the RBC/LA network equipment based in a school must never be switched off, as this may cause an alarm on the network monitoring equipment and may prevent overnight updates or backups over the network.
From time to time it may be necessary for the school to permit third party staff access to the on-site RBC/LA network equipment. For example, if there is a fault on the telecommunications termination device or failure of the access router, staff from the supplier or a maintenance contractor will need access to repair or replace the unit.
In other circumstances, the RBC/LA may need to call on the assistance of school staff to reset the equipment, or otherwise assist with troubleshooting of network problems. In such cases, it is essential that school staff can easily and safely gain access to the telecommunications equipment and network access device. The RBC/LA should ensure that at least one member of school staff is shown how to make these checks. If possible this staff member should be present at the installation of the equipment.
The school must ensure that at least one member of staff is able to perform these checks, and that the RBC/LA is notified should the staff member responsible change. The RBC/LA must undertake to visit the school to familiarise any new designated staff members with the equipment and procedures.
Separation of Administrative and Teaching Traffic
Some schools may feel it desirable to segregate administrative traffic from teaching and other education traffic; which could be achieved by the use of a separate physical infrastructure, or a VLAN configuration.
The DfES Standards Fund Guidance clearly states that the delivery of ‘whole school networks’ is a priority. Schools should make optimum use of their hardware and software through ensuring that they have an integrated local area network (LAN) to provide easy and timely access to ICT tools for the whole school workforce and that this is capable of providing both curriculum content and management information. Therefore, schools should not expect that the RBC/LA equipment will be capable of providing direct connectivity to more than one separate physical LAN.
Where a school chooses to implement two or more VLANs, the responsibility for providing interconnectivity between the VLANs lies with the school. This will involve the provision of an IP routing device; some Ethernet switches may be capable of performing this function. On the other hand, some local authorities provide access to administrative services only via VLANs. The hand-off point from the RBC/LA network should be capable of accepting VLANs from the school, even if the traffic is combined at that point.
DfES Standards Fund Guidance, Grant 31a: Infrastructure etc Paragraphs 9 and 10:
Network Security
Network security is, without question, a vital part of any network design, and as such is detailed separately in the Network Security document.
Router Management
Edge Equipment
Each site will require an access device, which will be the demarcation point between their LAN and the WAN and in many cases the management boundary between the school and local authority.
It is recommended that a layer 3 router is used, rather than a layer 2 switch with IP routing capabilities. This is based on experience that shows performance and capability issues with the latter especially relating to advanced features e.g. packet filtering. The router should also use dedicated hardware, rather than router software on a PC.
This device will support a WAN interface that will terminate the circuit from the RBC/LA. Taking budgetary constraints into consideration, it may be appropriate to use a device that supports modular interfaces that can be swapped out if the access circuit is upgraded rather than having to replace the whole chassis.
It will also support a local interface(s) which will connect the school’s LAN(s). The most widely used LAN technology used is Ethernet of which there are various standards which can run at 10/100 or 1000Mbs.
The use of ‘Static routes’ to route traffic to and from the schools network should be adequate and will simplify configuration of the devices. However, if there is a need to multihome sites then advanced dynamic routing protocols like Border Gateway Protocol (BGP) may need to be supported (see Appendix A).
Router Security Policies
Being on the management boundary the ‘edge device’ would be the ideal place to implement security policies for that site. This is unless it is agreed that a common security policy will be applied to all schools in which case the policies can be set further into back into the Service Providers network.
Security policies are dealt with in detail in the Network Security standards document.
Firewall Features
Experience has shown that many security control features required by network administrators are in fact bundled into certain versions of operating systems provided by the major routing equipment vendors. This can save on cost and can reduce management complexity associated with advanced security products like firewalls. The implementation by schools of their own firewalls independently of the RBC/LA central firewall service is likely to lead to complications. In order to enable IP videoconferencing it is recommended that each local authority deploys either an H.323-aware firewall, or a proxy server alongside the existing firewall. The Videoconferencing document covers firewalls and proxies in relation to H.323 traffic.
The Network Security document discusses firewalls and their implications for security in more detail.
Remote Management
In many cases edge equipment will be managed, either routinely or in emergencies from a remote location using ‘in band’ connectivity, supported by the SNMP protocol. In the event of access link failure this function would be lost along with the ability to properly diagnose the fault that has occurred.
It is therefore, strongly recommended that provision is made for some sort of ‘out of band’ management. This can be achieved by using an analogue modem; ISDN dial up or an ADSL connection.
Interface to the National Interconnect
In summary the Interconnect Service provides:
- An IP-level interconnection between each RBC Network and every other RBC Network subscribing to the Service.
- Connectivity to all organisations connected to JANET.
The Service does not provide transit from a RBC Network to the wider Internet.
The border router nominated by the operator of the RBC Network peers with a SuperJANET backbone router using the BGP4 protocol.
The operator of each RBC Network is required to ensure that the preferred route for traffic between their network and all other RBC Networks and JANET sites is via the Interconnect service.
The technical specifications and recommendations for interfacing with the National Interconnect are explained in the following document:
Provision of Network Services
Schools, of course, require not just IP connectivity but also a number of basic network services, such as Domain Name Service (DNS), e-mail, web services and forms of external access to the network. The latter may be, for example, for students to access their e-mail or work from home, or for service suppliers to make connections to administer their services.
Domain Name System (DNS)
DNS enables the mapping of names to IP addresses and vice versa, and underpins almost all other network services. Following network connectivity, it is perhaps the most important part of an IP network. There are two main areas to consider; providing service to schools to enable the lookup of external information (a "resolver"), and providing information about how to reach services on the RBC/LA network ("name service").
Internet Domain Names
A complete Internet domain name consists of two parts - the name of the piece of equipment (for example, a PC) and the domain name in which it is named. Such names are referred to as "fully qualified domain names" in DNS terminology.
For example, a PC named server1 within an organisation using the domain name something.co.uk would have a fully qualified domain name of server1.something.co.uk.
Many organisations also choose to name services in the DNS the web site of something.co.uk would most likely be reachable at www.something.co.uk.
Any one piece of equipment may have multiple names - the DNS allows for the configuration of aliases. pc23.something.co.uk might well be operating a Web server that was reached as www.something.co.uk at the same time as an E-mail service reached as mail.something.co.uk.
There is a standard domain naming scheme for schools in the UK. Each school has a standard name built from the name of the school, the LA in which is situated, and the ending of "sch.uk". For more details, see:
Standard school domain names:
DNS Structure
The DNS architecture is based on a hierarchy of nameservers, arranged in a tree structure. A request for a name to address (or vice versa) mapping traverses this tree until it reaches a server that holds the required information. Servers towards the leaves of the tree hold more and more specific information; those towards the root hold more general information, sufficient to refer queries onwards to nameservers that hold more detailed data for the required mapping.
The data configured on a nameserver about a domain is known as a "zone" file. In nameserver terminology a zone is a set of domains under one administrative management. An end site will typically require at least two zones to be operated - one to map names to IP addresses (the "forward" zone) and one to map IP addresses to names (the "reverse" zone).
As DNS service is crucial to the straightforward use of the Internet, the standards permit both a primary (or "master") nameserver for a domain, and one or more secondary (or "slave") servers. This helps to make the DNS resilient to failure, subject to careful consideration of each nameserver.
At start up, a secondary server retrieves a copy of the zone files it is serving from the primary servers for each zone. It then periodically checks with the master for updates (more recent nameserving software allows the primary to notify secondary servers after changes are made).
Operation of forward and reverse domains provides information to external networks; to enable mappings for external networks from internal hosts, a name lookup (or "resolver") service should be provided. Most (if not all) nameserver software is capable of acting as both a nameserver and a resolver.
When using private IP address space, DNS services become slightly more complex to configure. Hosts will use their private address space to communicate with other internal hosts - applications and users will be unaware that NAT is in use when communicating with networks external to the RBC/LA. (This in itself causes problems for some applications - see earlier NAT discussion).
The nameserver and resolver configurations in such cases need to provide different information depending upon the source of the query. Lookups from internal hosts will need to return the correct internal information; those from external hosts will have to return information based on the public addresses. This is often referred to as operating "split view" DNS.
In many cases, nameservice, resolving and split view can be configured on the same server, although many choose to operate two sets of servers internally. One to serve and resolve internal queries (returning information based on private IP addresses), and the other to serve the data from the zone files based on public IP addresses to external networks.
RFC1034 and RFC1035 define the base standards for DNS; several other RFCs provide updates.
DNS standards:
Nameserver Provision and Operation
When provisioning nameservers to provide information about local host addresses for external hosts, thought should be given to their location. It would appear that providing a secondary server (or servers) should improve resiliency; however if the secondary server is connected to the same LAN, with the same path to the Internet as the primary server, any failure other than that of the primary itself will also disconnect the secondary from the network. It is therefore sensible to provide at least one off-site secondary server, preferably outside the local network.
Availability of DNS data when the access link to a site or network is down greatly assists applications such as e-mail. In the case of e-mail, a link failure may disconnect the primary DNS server and local e-mail server, but remote e-mail servers can still lookup e-mail delivery information in the DNS, from the secondary servers.
In some cases, this will prevent e-mail being returned to the sender, as no mail routing information can be found. In others, there may be backup e-mail servers that can accept the message, or it may be that the e-mail service for the site is located off-site anyway, so the message can be delivered directly.
Many service providers provide secondary DNS service, or indeed, contract to operate a well engineered primary and secondary DNS service on behalf of their customers.
Zone File Maintenance
Close interworking between the schools and their RBC/LA is required to maintain correct name to address mappings. A mechanism must be in place where either the RBC/LA network provides standard name and address mappings for each school, or for the school to administer its own naming and addressing, and feed this information back to the RBC/LA so that appropriate DNS entries can be configured.
Once the information has been gathered, its configuration into the nameserver software itself will vary with different software products. Typically a Unix based nameserver will be based around plain text files which are directly edited, either manually or via locally customised interfaces. Other platforms will use graphical user interfaces and other methods.
Some operating systems choose to notify nameservers of their IP address and hostname upon boot, or acquisition of an IP address. This is handled using a DNS feature known as dynamic update, and was primarily intended to allow hosts whose IP address regularly changed to maintain a valid entry in the DNS. (For example, users of most Internet dial-up services receive a different IP address each time they connect).
It is unclear how useful this facility is in the schools' networking environment; however the feature is mentioned here for completeness. The service is not essential, but should be implemented by local agreement when it is required.
Dynamic DNS updates:
4.1.5 Resolver Service
Providing resolver service to end sites has other considerations. Where resolvers are provided off-site, any link failure will also break the DNS resolution. As the link is down, this may not initially seem to be a problem - what good is DNS resolution if the wide area network is unreachable?
However, there may be reasons why a site still needs DNS resolution for internal purposes, such as an on-site e-mail server or interactions between local proprietary networking protocols and the DNS.
In many such cases, some form of resolver service (and also local DNS zone service) is needed on site. This resolver may simply be configured to resolve local information only, and forward all other queries off-site, but in the case of link failure this will mean that local information is always available. Under normal operation, this will prevent unnecessary DNS requests being passed to the wide area, helping more efficient operation of both the IP and other networking.
(Note the distinction between resolution on site, where the local server is configured to interact directly with the Internet, and the forwarding of queries, where the local server hands off a query to a server on its upstream network. In the first case, the local server will require external connectivity via NAT, in the latter it does not).
4.1.6 Private and Public Address Space Interactions
In the situation where hosts are reachable at a different address internally than externally (for example, where private address space is being used with NAT), DNS servers are required to return different information for internally sourced queries than those sourced externally. As discussed earlier, this is often referred to as split-view DNS.
The most straightforward solution for this is to run two separate nameserver setups; one configured to serve internal data and resolve queries from internal hosts. A separate nameserver configuration handles external queries. Each nameserver setup by default then returns appropriate information depending upon the source of the query.
Some nameserver software can be configured to conditionally return different information, depending on the source address of the query.
E-Mail
Schools require e-mail services, which must interoperate with Internet RFC2822 standard based e-mail systems. The service may operate other proprietary standards in addition, but the ability for schools to communicate with each other and external networks using Internet standard e-mail is vital. E-mail is one of the fundamental applications required by schools, and it is essential that a robust and resilient service is provided.
This section provides an overview of e-mail services in so far as they relate to network design; the main considerations being the location of, access to, and responsibility for such services. E-mail security issues are also addressed in the Network Security document.
It is recommended that email services are hosted by Local Authorities or RBCs rather than schools. As stated in the Network Security document, school management must recognise that maintaining anti-virus measures requires considerable resource and determination.
However, if schools choose to operate their own e-mail systems, co-ordination with the RBC/LA is essential to ensure that these systems can interoperate smoothly with the RBC/LA supplied e-mail service. The school's mail system should also support interoperation with other systems using RFC2822 e-mail, even if a proprietary messaging system is used within the school.
Direct external access to a school's own e-mail servers is not recommended. E-mail should be accessible via a Web mail interface; if required, using the Internet standard Post Office Protocol version 3 (POP3) or Internet Message Access Protocol version 4 (IMAP4). See section 4.4 on external access for further information.
As with other content delivery to schools, a filtering system must be available, to help ensure that inappropriate messages do not reach end user mailboxes. Where a school operates its own mail server, the school must also maintain such a filtering system if it does not make use of the RBC/LA provided filtering mechanism.
It is emphasised that if a school chooses to host their own email service then they are liable for the support, maintenance and security of that service.
References:
RFC2822 Internet e-mail:
Post Office Protocol version 3:
Internet Message Access Protocol version 4:
Web Services
Web browsing is the prime application requirement for the majority of organisations connected to the Internet. This section provides an overview of Web services in so far as they relate to network design; the main considerations being the location of, access to, and responsibility for such services.
Schools Web browsing requires that inappropriate content can be filtered out, although the filtering configuration should be flexible enough to allow it to be tailored for different groups. For example, some content deemed inappropriate in normal circumstances may be directly relevant to advanced studies in later years of secondary education. The Network Security document covers content filtering issues in further detail.
Many schools have a requirement to operate their own Web sites, for business and teaching purposes. Industry best practice typically employs a dedicated server hosting location, located on a dedicated local network at a service provider's data centre. Space is made available to customers on managed servers - the service provider undertakes the system administration, including operating system upgrades and patches, and the operation of the web serving software.
Users are assigned individual administrative logins to the servers, allowing them to directly manipulate the files and scripts making up the Web site, without the significant responsibility of maintaining the server itself.
The Web server software should support version 1.1 of the HTTP protocol, as specified in RFC2616. One particular feature of version 1.1 that is not present in the 1.0 standard is virtual hosting. This allows many web sites to be operated on the same server, yet be accessible by individual standard school domain names.
In some cases schools may wish to operate their own web server on-site, which is undesirable. In addition to the issues associated with direct external access into the RBC/LA and school's networks (see section 4.4), all traffic to and from the web site will be traversing both the schools access link, and the RBC/LA backbone. If the site is located at a central hosting facility this extra traffic load is relieved.
Schools may wish to consider operating two web sites; one which is located at the school and is accessible within the RBC/LA network only and another on a managed server. When the same data is required to be visible from both servers, replication/synchronisation techniques could be used so that the data is only uploaded and edited on one server. For example, a school may nominate the on-site server as the master, and make changes and additions on this server only. With appropriate file or web site synchronisation software, the server at the hosting centre would automatically update itself from the master.
It is most unlikely that there are circumstances in which it makes sense to place web servers on-site at a school. A well provisioned web hosting location should be able to offer the same degree of flexibility in operation and maintenance of the web site, and significant benefits in terms of reducing traffic load on slower links.
HTTP 1.1:
External Access
As already noted, schools may request that direct external access to a server on site be enabled; alternatively, it may be absolutely necessary to enable external access to hosts on a school's internal network to allow suppliers administrative access to software services that they provide.
Direct, unauthenticated access to machines on the school's network puts not only that school's network at a higher risk of security incidents, but may very well put the level of security for the entire RBC/LA network at risk. Schools should be clear of the level of responsibility that they are taking on if they create a route for possible unauthorised intrusions in this manner.
Better solutions for external access will almost certainly be available, using authenticated proxy servers or virtual private network (VPN) solutions.
In the case of e-mail, for example, messages to and from the school should be configured to relay through (and be filtered by) the LBC/RA e-mail system. As already discussed, the serving of web sites can be provided by a central RBC/LA run server hosting facility to which the schools have their own direct access from the network at their school.
Other access might be provided to students and teachers through proxy web interfaces to the school's facilities - for example, products are available to permit access to filestores via a Web interface. Management access for service suppliers could be provided via a VPN solution, allowing support staff direct access to their servers inside the school - but only via a more secure, authenticated link.
Enabling any form of external access will always increase the risk of incidents; carefully thought out methods of providing the access should be able to keep this risk as low as possible.
This section is not intended to be an exhaustive review of the issues associated with external network access. Security issues are covered in more depth in the Network Security document.
Location of Network Services
The location of network services within the network topology is an important issue. As these services require a level of unsolicited, inbound external interaction, best industry practice dictates that they be logically (and physically if possible) separate from the network infrastructure providing the basic connectivity.
The majority of service providers locate their DNS servers, e-mail servers, web servers and other network service devices on separate IP networks - not only from the customer network, but also from each other. This allows security policies to be tailored for each service, and in the event of an incident on one of these services networks, the others are better protected than if they were to share the same IP subnet and/or physical equipment.
It is critical that DNS service is resilient, and that at least one secondary server is provided, preferably outside the RBC/LEA network, but at least in a separate physical location. In a network with only a single core location, this server could be accommodated at an access/aggregation PoP, or perhaps even on-site at a school.
Ideally, all services should be provided in a resilient fashion, perhaps by engineering a network topology with two nodes hosting duplicate sets of network services.
Disaster Recovery
It is important to consider what happens when there is a major failure on the network. This may be anything from the failure of a vital piece of equipment or major node, to the theft of routers or servers, or the commercial failure of a telecommunications supplier. This section suggests some possible disaster scenarios, but by no means covers them all. Even if it is not feasible (for example, for cost reasons) to have a disaster recovery plan in place, it is vital to have the plan itself.
Designing a network with duplicated core nodes is a good step towards disaster recovery. However, the nodes must be in sufficiently geographically distinct places so as to reduce the risk of common power or telecommunications grid failures affecting them. The loss of one node will still affect all sites that are only connected to it. If there is a major disaster that takes days, weeks or even months to fix, arrangements will be needed to restore service, perhaps by temporarily re-homing links to the other node(s).
Equipment failures are generally covered by maintenance contracts; it is important to ensure than an appropriate response time is agreed. Failure of equipment causing half the network to be disconnected clearly cannot be covered by a next business day response time. Equally, it is vital to check that the response time quoted by a supplier is not just the time within which they will acknowledge and start working on a maintenance request.
Where it has not been possible to engineer resilience into a network, it may make sense to hold some degree of spares on-site. For example, anything from holding a spare interface card to a duplicate router configuration on stand-by where a major portion of the network relies on a single router. At very remote end sites, it may be advisable to keep a pre-configured, duplicate router that can be quickly swapped in by local staff.
When spares are held, it is vital that they are checked regularly, to ensure as far as possible that they are not found to be faulty when needed.
Purchasing a network solution from a single supplier almost always results in cost savings. However, this means that the network will rely almost exclusively on common infrastructure. If there is a disastrous failure on that single provider's network, all of the RBC/LA network will be lost until it is rectified. Purchasing links from a variety of suppliers reduces the risk of losing an entire network (although suppliers do share cable routes, so this is not a foolproof guarantee).
Worse, if the single supplier suffers commercial failure, the network will have to be quickly reprovisioned. Some advance planning for this (hopefully unlikely) event will undoubtedly be of benefit.
The theft of running equipment seems unlikely, yet it has been known to occur.
Support Services
Technical Support
Schools must be able to report network problems and request assistance with other networking problems. The RBC/LA should operate a support centre with a single point of contact for all schools on their network This support centre should be capable of communicating with both technical and non-technical school staff - smaller primary schools are not likely to have highly technical staff on site, yet will still require fast resolution of network faults and other difficulties.
The support centre should provide fault reporting and resolving services, liaising with service providers, including the provider of the National Interconnect, and other suppliers of the RBC/LA network to resolve problems with the network (e.g. link or equipment failures), and issues or problems with services provided by the RBC/LA such as e-mail or web. The support centre should also liaise closely with its counterparts in other RBCs.
A flexible approach to providing effective communications of major network events, such as an outage, must be provided. Local agreements can require the use of messages on well-known web sites, e-mail, faxes and text messages to keep nominated schools' network managers and technicians advised of these events. Effective communication of events with a broad user impact helps keep the user community informed, and as a result can reduce the call-in load on support staff. This enables more effort to be directed at problem resolution.
From time to time it may also be necessary for schools to nominate contacts at their suppliers to deal directly with the RBC/LA support centre. This will most likely be to resolve network related problems with supplier's servers, or access to servers on the school site.
The support centre should offer advice and assistance for both wide area network issues, and also for issues related to the school's internal network where at all possible. Administrative issues such as those related to DNS data updates must always be actioned, with care to ensure that the requirement is well specified; third parties involved should also be well briefed. Once the change has been actioned, it must be tested and the result checked to be correct.
The responsibilities with respect to the technical support, reporting, and repair of faults on the Interconnect are set out in the Interconnect Agreements:
Network Monitoring
The RBC/LA support centre should maintain a network monitoring system (NMS), allowing staff to proactively respond to network outages during working hours as they occur. The system should be capable of quickly bringing link outages, equipment failures and other major network problems to the attention of support centre staff.
The NMS, or an alternative dedicated system, should also measure and record traffic and utilisation levels on links in the network for both reporting and capacity planning.
The NMS should be capable of identifying links that are approaching congestion in real time. Congestion levels may be simply a product of demand, which requires the adding of extra capacity, or may be an indication of some form of problem. An unusually high amount of traffic on a school's link may be an indication of a widespread virus infection on the school's PCs, for example.
Increasing error rates on a link indicate a growing problem that will often result in complete loss of the link in the short term. This information is usually available to telecommunications suppliers, but is seldom acted upon unless reported by the customer. The support centre's NMS should be capable of providing notification of such deteriorating links, to allow staff to report the problem to the supplier before the links fails.
Where possible (and requested), the support centre should be capable of remotely monitoring the local networks at schools, including the view of wide area performance from the perspective of a user at the school. This will particularly aid the analysis and diagnosis of problems involving smaller schools where dedicated network staff are not available full-time on site.
More detailed traffic analysis tools are desirable, particularly to identify the different services and protocols being used on a school's link. Some RBC/LAs have already found that congestion on links in their networks has been caused by heavy use of applications such as Internet gaming and radio. By removing access to these services, either completely or on a time basis (e.g. not in school hours) network performance for users has been greatly improved.
Information Dissemination and Staff Development
High-quality technical support at both school and RBC/LA levels is essential to the success of broadband in schools. The construction of the RBC/LA networks is a complex undertaking and all those involved are on steep learning curves. Every opportunity should therefore be taken to size opportunities for staff development.
A major factor in the uptake of network standards will be the dissemination of information to all involved. RBCs/LAs will need to provide a Web site with a full range of information including:
- National standards documents
- Service level agreements between schools and the RBC
- Guidance notes
It may also be important to use conferences, meetings and newsletters to draw attention to this material at both management and technical levels.
The RBC/LA may be required to provide training for end users of the network services it provides, but not where a school is choosing to operate its own service independently of the RBC/LA offering.
For example, a school may reasonably expect that the RBC/LA would provide training on using its Web mail or Web hosting service, or be provided with information on how to interact with the support centre. However, a school operating its own e-mail server should not expect training from the RBC/LA on anything other than how their system would interoperate with the RBC/LA e-mail relay system.
Advanced and Emerging Technologies
Some more advanced technologies are currently emerging, however it is not expected that these will become requirements for the majority of schools' networking in the near future. This section notes some of these technologies for reference.
Where local conditions require these or any other additional services, coordination between the schools, RBC/LA and suppliers is essential. This should help to ensure that the required services are developed, and are maintained in line with industry standards as they themselves develop.
IPv6
The initial driver for IPv6 was the perception that IPv4 address space would soon be exhausted; this observation was made in the early 1990s. In the intervening time, much stricter controls on assignment of IPv4 address space, and methods such as IPv4 private address space have slowed this rate of exhaustion considerably.
IPv6 is not yet widely adopted, and techniques such as private IP address space and NAT satisfy many organisations' requirements. However, using IPv6 addressing is sometimes seen as an attractive alternative to the rigorous application process for large amounts of public IPv4 addresses.
For the moment, though, the vast majority of network resources are only available using IPv4. Mechanisms to enable IPv6 access to IPv4 service, and vice versa, are not yet widely deployed and in some cases may not be able to provide a transparent service.
For further information on how the Higher and Further Education communities are trialling IPv6, see: .
IP Multicast
IP multicast is a bandwidth conserving technology that can reduce traffic by transporting single streams of information across the network backbone to regional and local distribution points, where the data is replicated for simultaneous delivery to multiple users. Some applications that can take advantage of multicast include videoconferencing, video serving and news distribution.
Again, this is seen as an advanced technology, which is not yet required for use on schools' networks, but for further information on how JANET uses multicast see:
The JANET Multicast technical guide, which provides information for sites wishing to use multicast on JANET is available from:
IP Quality of Service (QoS)
An IP network has traditionally offered a "best-efforts" service, where all IP packets were treated in the same manner, regardless of application. When congestion occurs on a network link, any one packet is as likely to be dropped as any other.
With the increasing use of multimedia applications such as videoconferencing and voice over IP (VoIP), this best-efforts behaviour is undesirable. While a web browsing session would tolerate packet loss (albeit at a detrimental performance to the user), packet loss on voice and video can make application useless. Being able to handle such multimedia packets with a higher priority than others is seen as a useful tool.
Where schools are making heavy use of videoconferencing, VoIP or other delay sensitive interactive multimedia services, IP QoS may become a requirement, so that traffic for these services can be prioritised. This is particularly true for networks that are running close to congestion. However, ultimately, IP QoS is no substitute for the provisioning of sufficient end to end capacity in a network to support the services that schools require.
IP QoS is still somewhat in its infancy, however, and perfectly acceptable voice and video connections can be made over a network provisioned with suitable capacity.
The Videoconferencing document provides an overview of IP QoS and classes of service.
For further information on JANET's QoS implementation, see:
7 References
DfES Standards Fund Guidance
ICT in Schools Standards Fund Grant 2004-05
Guidance for Schools and LEAs
DfES Policy on Connectivity
ICT in Schools Standards Fund Grant 2003-04
NGfL Grant 601a: Information for LEAs and Schools
UK Government’s e-Government Interoperability Framework (e-GIF)
.
IETF RFC standards
ITU standards
Private IP address ranges
Standard school domain names:
DNS standards:
Internet e-mail RFC2822
Post Office Protocol version 3
Internet Message Access Protocol version 4
HTTP 1.1
SNMP
JANET IPv6
JANET IP Multicast, Multicast technical guide
JANET Quality of Service
E2BN Products & Practice
UK Broadband Stakeholder Group
Broadband connectivity choices
JISC/UKERNA Satellite Trial Results
JANET National User Group Networking Glossary
National Interconnect Technical Specifications
National Interconnect Agreements
Regional broadband Consortia (RBC)
Network Security
DfES ICT in Schools Network Services Project
UKERNA, March 2004
Videoconferencing
DfES ICT in Schools Network Services Project
UKERNA, March 2004
Appendix A: Network Topology Discussion
Some bodies may choose to contract the operation of their network to a third party supplier, where the RBC/LA delegates responsibility for the design, engineering and operation of the network to the third party. This can be thought of as a "cloud" network, where the schools are provided with a link to the network, but do not necessarily have any visibility or interest in the design of the network. (Other than that which provides the required level of service, of course)?
Where the RBC/LA chooses to build the network itself, the construction of the backbone is certainly of interest; this appendix discusses some possible topologies/architectures. The technologies discussed are by no means an exhaustive list, but reflect those found to be already in common use on RBC or LA networks.
A.1 Star Networks
In a pure star topology, the backbone consists of a single location (the "hub"), to which all customer links are directly connected. This topology has the advantage of simplicity, but if many long distance (expensive) links is involved it may not be cost-effective.
Star topologies can also raise question over resilience. Whilst the failure of any one link should only cause loss of connectivity to a single site, failures at the hub may cause serious problems for many, if not all, sites. For example, power failure to the equipment at the hub could bring down the entire network.
Providing further interlinked hubs improves the physical resiliency of the network, as failures at individual hubs affects only those sites connected there. However, failure of inter-hub links could cause network services (such as DNS or e-mail) located at one hub to be unavailable to sites connected elsewhere.
This topology now starts to reflect the typical Internet network architecture. Each hub aggregates customer access links, and interconnects them with the others. This interconnectivity needs to be engineered in some way; the vast majority of service providers (including academic networks) do this using IP routing.
Interconnecting aggregation points in a star network can provide a degree of resilience, as shown below. Alternative routes from the two outlying core nodes to the centre node are provided via aggregation hubs. This has implications for the equipment located at the aggregation points, as it now has to be capable of handing the extra traffic load in case of failure.
Equipment configuration at the aggregation hubs will be more complex. However, as the overall topology is now a dual-ring solution (see below for ring topology discussion) this provides a great degree of resiliency. Even failure of both links between the core hubs will not isolate any part of the network (although performance may degrade if the network links are not provisioned with sufficient capacity to cope with failure modes).
However, it is still possible to operate a logical star topology where there is more than one hub, using a technology such as MPLS to carry VLANs from each site to a central location. This causes inefficient use of network capacity - traffic between two sites on the same hub may cross the backbone twice (to the centre and back out again), where in an IP routed network traffic may simply flow from one port to another on the same router.
This effect could be particularly undesirable if two sites on the same hub were to hold a high bit rate point-to-point H.323 video conference, say at 768kbps. In a logical star network this would almost overload all 2Mbps links between the hub and the central location. Worse, H.323 video can peak at double the configured bit rate, so the single videoconference would overload the entire backbone network between the hub and central point.
On a routed network, this point-to-point video traffic would not leave the local hub, maybe not even a single router if both sites were connected to the same equipment.
A.2 Ring Networks
The final topology example discussed in the last section can be improved upon by the addition of a single extra link, to interconnect the hubs on the left and right in the illustration. This connects the three hubs in a ring, so that any failure of one of the three backbone links does not necessarily cause loss of connectivity. If direct connectivity fails, the affected hubs still have the potential to pass traffic between each other, via the third hub.
The rerouting of traffic is the default behaviour for a dynamically routed IP network, and can also be engineered for other technologies such as MPLS. Traffic congestion may well occur, of course, as two remaining links are carrying extra traffic; again, this effect will be magnified in the case of a logical star network.
A.3 Resilience
Using a ring topology for the backbone certainly helps to prevent loss of connectivity to any set of sites; however it is not sufficient simply to implement a ring architecture without consideration to the underlying physical infrastructure.
Supplier diversity or diverse delivery of circuits is an important factor. If all circuits are obtained from the same supplier, and delivered over the same path, to the same equipment rack, a common failure to the supplier's network, the physical path to the local delivery point or failure of the supplier's on site equipment will take down all circuits at that location.
In the last topology discussed above, this could mean that both backbone links from the hub would go down and the hub would be isolated. If all links at that hub were delivered in the same way, it is arguable that this makes no difference, as all customer links will go down too. However, in this case, what is the point of the cost of providing two backbone links and router interfaces when only marginal resiliency is gained?
If one or more customer links are delivered in some way independently of the backbone supplier, then more careful specification of the backbone links could have retained wide area connectivity for those customers.
Where attention has been paid to resiliency in the network design, similar careful attention should be paid to the procurement of the actual network links.
Similar considerations can also be applied to sites, where it may be desirable to have more than one connection to the wide area network. In such cases, it is usually more straightforward to operate an access router (or more) at the local site end, allowing central control over the interconnectivity.
Formulating a routing plan is critical to the success of resilient configurations. The plan should detail where particular traffic is expected to go under normal operations, and consider traffic paths under failure conditions. This helps to adequately provision network links at the planning stage, and also to troubleshoot when problems occur on the deployed network.
Where multiple IP routed paths are available to a destination, particularly if there are links to multiple external networks, an IP routing plan is essential. For example, if a network has a connection to a service provider and JANET, how should IP traffic be routed?
External routing protocols such as BGP allow network reachability information to be exchanged, but require careful configuration of BGP metrics to attempt to influence the flow of traffic. Unlike internal routing protocols, BGP does not intrinsically have any knowledge of link capacities or utilisation, and does not have a particularly intelligent path selection algorithm.
In the case of two links, the plan is not particularly difficult - the connection to JANET is likely to be used to exchange traffic with JANET networks only, and the remainder of the traffic will use the service provider connection. The connection to JANET in this case is known as a "private peering" - traffic is only exchanged between customers of the two networks.
However, where multiple service provider links are available for full Internet access, how should traffic be exchanged? Which service provider link should be used for outbound traffic for which networks, and (more problematically) which service provider link should take traffic from the Internet to the local network?
This sort of arrangement is known as "multi-homing", and is one of the hardest IP networking issues to solve. As such, it is beyond the scope of this document, and it should simply be noted that extreme care is needed in such situations, and close co-operation between all networks involved is required.
