Advanced Network Services

Home ITS Services Network Services Advanced Network Services

Corporation for Educational Network Initiatives in California (CENIC)

Education in California is at the cusp of pulling the plug on SONET transport and the first CalREN-2 network. In 1997 a joint grant proposal from CSU, UC, Stanford, CalTech, USC and JPL was submitted to the National Science Foundation. Together these member institutions are the charter associates of the Corporation for Educational Network Initiatives in California (CENIC), a nonprofit corporation. CENIC asked for money to build a state education network. Our proposal for $3.5M was funded and was the start of the CalREN-2 network we are about to retire at the ripe old age of 5+ years.

The retiring network is built on metropolitan OC-48 SONET rings from SBC Pacific Bell and OC-12 (622 Mb/s) circuits from Qwest Communications. SONET (Synchronous Optical NETwork) adds a layer of complexity and cost that is not required in a pure IP network. The replacement for SONET is ONI, CENIC's Optical Network Initiative.

Planning for the replacement network started two years ago. At that time that the economy was declining and the telecom industry in particular was imploding. Wild over-expansion by telecommunications companies left billions of dollars in fiber optic cables throughout the world without subscribers. CENIC initially hoped to lease packet-over-fiber circuits from a carrier that would agree that some of our money was better than no money at all.

In the intervening two years the ONI plan has changed. Ownership of glass (dark fiber) implies responsibility for light amplification every 100 Km. CENIC was initially reluctant to accept the challenge of managing equipment in remote locations like Soledad. Trading off against the risks of facilities ownership are the risks of having contracts abrogated in bankruptcy should some of our business partners run out of money. We almost had deals with a company that withdrew their partnership offer at the 11th hour. Failed deals for services each drove us closer to the ownership direction. CENIC is now the owner of 1000+ route miles of fiber. We are purchasing Cisco 15808 fiber multiplexors to light the fiber. Our direct ownership of the glass strands will give us the ability to run at increasing high backbone speeds in the future; 1 Gb/s now, 10 Gb/s later, 40 Gb/s over the horizon. And those link speeds can be repeated at up to 40 different colors of light.

In the old network, routing nodes are in metropolitan campus machine rooms. Our experience with the reliability of campus-run facilities has led to the decision to pull back to hardened telco-grade facilities for most nodes. Campuses are connected by spur circuits. Notable exceptions are UC Santa Barbara and CSU San Luis Obispo where the network design required optical amplification and suitable campus facilities were available.

The new network has an overall three part architecture.

CalREN-XD is our eXperimental and Development network. Where it is funded, it will permit network researchers to have direct access to communications facilities in ways that might be disruptive to production traffic for their bleeding edge research. University of California Institutes for Science and Innovation will do their work at this level of the network.

CalREN-HPR is CENIC's High Performance Research network. It will be the preferred path to Internet2's Abilene network and other research nets. IP version 6 will see first light in California on this net. We will also use it for high rate video multicasting. Campus connection speeds to HPR are starting at 1 Gb/s and are expected to move to 10 Gb/s as soon as router interfaces are available.

CalREN-DC is the Digital California network. Private ISP subscribers will reach the University network over DC. And DC will be the conduit to the commercial Internet outside the research community. CENIC connects California K-14 educational institutions to this network. For K-12, the point of action for network connectivity are the 58 County Offices of Education.

What this means to UCSC

• From today's perspective, all this is happening pretty fast. We have been making spaces ready including cleaning paths in underground ducts. SBC PacBell will be pulling fiber on April 28. They will install our equipment on April 30. We expect to see test bits in late May and to be able to move client traffic as soon as final exams are over.
• No fiber optic cables were ever installed in Santa Cruz County by distressed communications companies. Hence there was nothing for us to purchase. We received a bid for new construction from campus to Los Gatos. We did not believe that fiber on poles on Hwy 17 was worth $5M. Our connections to Sunnyvale and Palo Alto will be SBC managed wavelength services. SBC calls the new service Enhanced Gigaman. Managed services means that SBC dedicates fiber strands for the contract duration. They also furnish and maintain the the optical equipment on the campus and at intermediate sites where optical amplification is required.
• Frontdoor is our present entrance router. It's outside world interfaces are ATM. There is no ATM in the new network architecture. It is being replaced.
• We will have two equal equivalent connections coming to the campus. We are installing them in different buildings. A power shutdown in the Communications Building will not mean that the whole network will go dark -- if other buildings have power. Our new entrance is in a building with the generator capacity to run backbone gear during a PG&E outage.
• Commercial Internet traffic will enter and leave the campus on paths logically separate from other traffic. This would permit us to squeeze on the pipe to limit our bandwidth use while minimizing the effect on research network use. We have no reason to believe that the money currently allocated to purchase these services would require squeezing any time soon. We need to be prepared.
• UCSC's second connection at the alternate location will be delayed until August. We have a legacy 45 Mb/s backup circuit. This is due for retirement as well, as soon as the second connection is up.
• XD is a place holder in the overall network architecture that is not fully funded. In the short term, there will be no XD network activity at UCSC.
• More than once, we've been asked How do I access Internet-2? The answer to this is not new. The network uses the best path available for all traffic. That just happens. No client action is either required or possible. This would have to change when the XD network comes to UCSC, but only for the directly affected researchers.

Who's been doing the work

At UCSC, the team that's been working with me to getting ready for UCSC's part in ONI includes especially ITS staffers Mark Boolootian, Rich Chew, Neil Goldstein, Russ Johnson, Frank Koch, Paul Marundee, Matt McKenna, Michael O'Connell, Tad Reynales and David Stihler. We get cooperation from many parts of the University support system that makes things happen -- notably Physical Planning and Construction, Physical Plant and University Purchasing.

K-12 net use in Santa Cruz and Monterey counties

UCSC's networking group provides engineering services to the public schools under contract with CTAP -- the California Technology Assistance Project. We participate in region V which covers Santa Cruz, Santa Clara, Monterey and San Benito Counties.

Prior to December, Santa Cruz Schools were connected to San Francisco State and both county offices used Frame Relay to get their internet service. The counties still use Frame Relay for distribution of internet service to client schools. The Telco circuits connecting to the County Offices are point to point T-1s. The current service arrangement for Santa Cruz, Monterey and San Benito schools is as clients of the CSU Monterey Bay campus. The K-12 router shown in the diagram is hosted in the CSUMB telco and network room.

More information on the County schools network is available from them directly: http://www.santacruz.k12.ca.us/

The combined use of two counties can be seen by looking at the traffic on the router's Ethernet interface. In and out are relative to the router. Serial-out (i.e. T-1 out) is data moving towards schools. Likewise, Ethernet-In is also data moving towards schools. While schools do host some internet content (web sites), they are predominantly data consumers rather than producers.

Some two year old historical data

Credits: The data presented in these plots was collected from network routers using the SNMP network monitoring tools produced by Carnegie Mellon University.

Fiber cable optical launch info

NTS uses (or will use) at least five fiber standards. Sadly, fiber standards are not uniquely identified by connector types.

10 and 100 Mb/s use diode light emitters; 1 Gb/s uses class I laser emitters

** A special mode conditioning cable is required to achieve more than 200 meters with multimode fiber.

Emerging standards for 10 G/s LAN fiber interconnect.

** The multimode fiber contemplated here is "laser enhanced" 50 micron core diameter. 65 meters may be a soft limit depending on the detailed bandwidth specification of the fiber. LX4 encodes the data over four separate wavelengths.

dBm is a measure of power, like watts.

0 dBm = 0.001 Watt
10 dBm = 0.010 Watt
-10 dBm = 0.0001 Watt

Loss budget = Min_Transmit_Power - Rcvr_Sensitivity

If you attach a power meter with a 62.5 micron jumper to one of the mm sources above, you expect to see a power level between Min and Max.

Rcvr sensitivity means with at least this much power delivered to the receiver the design error rate can be achieved. 1:1,000,000,000

At 10 and 100 Mb/s, the distance limit for full duplex systems is due to loss of light. At 1 Gb/s over multimode fiber, the limit is due to fiber bandwidth.

If too much power is delivered to the receiver it is possible to overload it. With long range ZX optics, if the fiber link provides less than 8 dB of loss, additional attenuation should be added.

It is technically permissible to use LX for short range fiber applications because the fiber path is not required to have attenuation to prevent overload.

Rules of thumb

• At Gig speeds, most of our light losses come from the connectors and only a very small amount from the fiber.
• The faster we go, the lower the loss budget we get to work with. The more important it is to treat connectors with great respect and minimize patch cables. Using ZX to make up for connector losses is a very expensive proposition.
• Over its useful life, an ordinary light bulb loses about half its brightness before it burns out. The same is true for fiber optic light sources. The loss budget numbers allow for expected lower output over the useful life of the fiber transceiver. The aging process "settles down" after a few thousand hours. Fiber light sources don't have a "burn out" mechanism that is similar to light bulb filaments.

Back to top

Version of Dec 15, 2002