Building a scalable next-genera2on cheap-ass na2onwide network.

Transkript

1 Building a scalable next-genera2on cheap-ass na2onwide network. Rakennus skaala*ava seuraavan sukupolven halpa-ass valtakunnallinen. hugge@nordu.net TREX 2016

2 (In two slides) Joint project by all five Nordic NRENs, running since 1989 AS2603, from California to Helsinki. Self-operated 100G DWDM in northern europe, leased or swapped 100G and 10G con2nentally. 19 Current worldwide pops, expanding to 21 this year. 2Tbit/s of current edge-capacity. Ingress peak at about 600G Ac2vely evolving addi2onal services outside of network services. Shares the same engineering and opera2onal department as SUNET One of the prime R&E networks in the world Currently searching for two System Engineers and one Media engineer ;)

3

4

5 Current Topology 3 systems North, West and South 2 networks per system Red and Green Redundancy on layer 3 Fully Redundant Network Red Green separahon [>10m] Desire for High Capacity Growth Support for Special ApplicaHons For example FiberChannel. Alien waves Support both; Router ConnecHvity Dedicated High BW Pt-Pt ConnecHvity Cost SensiHve GE and 10GE But also 100G Ready

6 Current Topology drawbacks Fixed point to point links, as no ROADMs Only one pop in each city Loose the pop and everything behind gets disconnected. Crappy DCN Long last mile Ex Stockholm Umeå Link failure get nohced by users via BGP Single homed users get longer outages. Dual link error can loose a 1/3 of the network. CombinaHon of Corestream and CN4200 Corestream can t take CWDM/DWDM XFP:s

7 Scale: this is approx km fiber path. New Topology 3 fiber paths to all regions Redundancy on layer 3 Fully Redundant Network SeparaHon between fiber paths [>10m] Desire for High Capacity Growth Support for Special ApplicaHons Transfer of frequency of light Support both; Router ConnecHvity Dedicated High BW Pt-Pt ConnecHvity Cost SensiHve 10GE and 100GE But also 1TE Ready Leased capacity to Visby

8 Fiberplant Core fibers mostly Aerial in high voltage power lines New and old fibers, G.652B some G.652D OTDR and attenuation data delivered as they measured Kmz data About 8000km Access fiber from city networks Two separate paths to university No KMZ data

9 A (new) way of building networks IP over DWDM (IPoDWDM). Eliminates unnecessary and expensive components => less components that fails Gridless Reroute traffic before fail Much lower CAPEX but also lower OPEX The worlds first nahonwide network using this technology

10 Juniper 100-Gigabit DWDM OTN for MX FEC Soj-decision FEC (SD-FEC) Acacia, Min OSNR 14.0 db High-gain FEC (Staircase) (HG-FEC) CorHna, Min OSNR 16.0 db G.709 FEC (GFEC), Min OSNR 24 db CFP2-ACO 100 channels on C-band ITU 50GHz spacing Output power -8 through 0 dbm Input power -18 through 0 dbm ChromaHc dispersion tolerance: +/ 40,000 ps/nm PMD tolerance: 25 ps (mean) PolarizaHon tracking: 50 krad/s

11

12 Equipment DWDM ADVA FSP3000R7 Routers Amps and Roadm Gridless Colorless Juniper MX 100G DWDM Coherent 100GE 1GE

13 Nordunet DWDM Nordunet DWDM Nordunet DWDM Nordunet DWDM Köpenhamn Kalix Narvik Gävle Sundsvall Umeå Luleå Åkroken Kiruna Hjälta Borlänge Östersund Stockholm Uppsala Västerås Norrköping Karlstad Kalmar Växjö Karlskrona Kristianstad Malmö TUG Falun Jönköping Linköping Borås Skövde Trollhättan Göteborg Lund Kungsbacka Varla FRE Halmstad Eskilstuna Alvesta Ulricehamn ROADM University Router n x 100G Core 100G Access POP Router Line fiber 10G PTP SVA SBO KIR5 KIR3 KLX1 LLA1 AJR1 SEA1 UME8 OSD FLN4 BOR10 UPA LBA FSN2 KKA8 TRH NSNIQ BOS ULR5 JKG SKE109 ETA10 LIN3 NKG188 AVA3 VXJ KMR88 KKA8 KSD1 MCEN1 LND88 HSD1 VLA HKR1 HKR2 UU1 UU2 MAH2 MAH1 BTH2 BTH1 LNU4 LNU3 LNU2 LNU1 HJ2 HJ1 LIU1 LIU2 HB2 HB1 Onsala1 HH1 HH2 LU1 LU2 CTH2 CTH1 GU2 GU1 HIS2 HIS1 KAU1 KAU2 ORU1 ORU2 MDH1 MDH2 DU1 DU2 MIUN1 MIUN2 HIG1 HIG2 UMU2 UMU1 LTU2 LTU1 IRF1 IRF2 KTH2 KTH1 SU2 SU1 KI2 KI1 SH2 SH1 HV1 HV2 MIUN4 MIUN3 DU3 DU4 MDH4 MDH Nordunet DWDM Borås Skövde Trollhättan Göteborg Kungsbacka Varla Alves Ulriceh TRH NSNIQ BOS SKE VLA HB2 HB1 Onsala1 HH1 HH2 LU2 CTH2 CTH1 GU2 GU1 HIS2 HIS1 HV1 HV Fiber plant

14 IRF2 IRF Narvik KIR5 1 KIR3 Kiruna Åkroken 3 Luleå LTU1 LTU OpHcal ROADM POP Router University Router Line fiber n x 100G Core 100G Access 10G PTP Trollhättan HV1 HV2 Göteborg Kungsbacka Varla 334 GU1 GU CTH1 CTH Onsala KAU1 KAU2 TRH 48 NSNIQ VLA Karlstad Nordunet DWDM KKA8 Borås Nordunet DWDM BOS HB1 HB2 HH2 HH1 MIUN3 16 MIUN4 391 DU1 DU Skövde HIS1 HIS Östersund DU3 DU BOR10 15 ORU2 ORU SKE109 Alvesta Hjälta OSD FLN4 LBA ULR5 Ulricehamn LU2 LU1 10 Falun 36 AJR1 SEA1 Umeå Gävle Uppsala Borlänge MDH2 MDH FSN2 Västerås AVA3 ETA10 MDH4 MDH3 UME8 SVA SBO UPA Eskilstuna HJ HJ2 JKG Jönköping VXJ LNU1 LNU2 LLA1 Växjö Kalix Sundsvall Stockholm FRE LIU2 LIU LIN3 MIUN2 42 KLX1 UMU2 UMU1 315 MIUN1 HIG2 HIG UU2 UU Linköping Kalmar NKG188 TUG Norrköping 357 LNU3 LNU4 KTH1 KTH2 SU1 KI1 SU2 KI2 SH1 SH2 ROADM site in each University city Four pure ROADM sites No traffic are drop InterconnecHon with Nordunet Nordunet bypass Norway ESS connechvity to Copenhagen South of Sweden only has two paths ConnecHng Malmö Stockholm over Nordunet, 100G transponder or Coherent Sunet Aerial, Nordunet in the ground 25 Halmstad HSD1 50 Nordunet DWDM 26 Lund LND88 MAH2 MAH1 44 HKR2 HKR1 Karlskrona KMR88 31 KKA BTH1 BTH Köpenhamn Nordunet DWDM 49 MCEN1 Malmö 32 KSD1 Kristianstad 33

15 IRF2 IRF1 106 ROADM POP Router University Router Line fiber n x 100G Core 100G Access 10G PTP Trollhättan HV1 HV2 GU1 GU2 CTH1 CTH2 Göteborg Onsala1 Kungsbacka Varla 342 KAU1 KAU2 119 Halmstad Köpenhamn Karlstad Borås Narvik 106 Skövde Åkroken Östersund Alvesta Lund Malmö Hjälta BOR ORU2 ORU1 Ulricehamn TRH NSNIQ VLA 123 KKA8 BOS HB1 HB2 HH2 HH1 HSD1 MIUN3 MIUN4 402 DU1 DU DU3 DU HIS1 HIS2 401 KIR5 SKE109 LU2 MCEN1 OSD FLN4 LBA ULR5 LU1 135 LND88 Falun Kiruna Luleå Umeå Gävle Uppsala Borlänge 125 KIR3 117 Västerås 118 ETA AVA3 MAH2 MAH1 AJR1 100 SEA1 MDH2 MDH FSN2 Eskilstuna 407 HJ1 407 MDH4 MDH3 127 HJ2 134 JKG UME8 SVA SBO UPA Jönköping VXJ LNU1 LNU2 LTU1 LTU2 KSD1 Växjö 130 HKR2 HKR1 LLA1 Kalix Sundsvall Stockholm 128 Karlskrona Kristianstad FRE LIU2 LIU1 LIN3 MIUN KLX1 UMU2 UMU1 UU2 UU Linköping Kalmar KMR88 KKA8 MIUN1 HIG2 HIG NKG188 TUG Norrköping LNU3 LNU4 BTH1 BTH2 KTH1 KTH2 SU1 KI1 SU2 KI2 SH1 SH2 Design, ROADM single Core POP Router University Router Line fiber Connected n x 100G via CoreER4/LR4 Östersund Coherent 100G Access MIUN3 MIUN4 interface to the 402 next city/ G PTP OSD pop DU1 DU In Stockholm, Göteborg and Luleå 316 DU3 DU4 Two routers at the PoP LR4 on both links to the University Trollhättan HV1 GU1 HV2 GU2 CTH1 CTH2 342 Göteborg KAU1 KAU2 119 Karlstad Borås Skövde Hjälta BOR ORU2 ORU TRH NSNIQ KKA8 FLN4 LBA HB1 HB2 HIS1 BOS HIS2 SKE109 ULR5 Ulricehamn Falun AJR1 110 SEA1 Umeå Gävle Uppsala Borlänge MDH2 MDH FSN2 117 Västerås 118 ETA10 Eskilstuna 407 HJ1 407 MDH4 MDH3 127 HJ2 JKG UME8 SVA SBO UPA Jönköping VXJ LIU Vä

16 Stamnätsnod Sites Degree A Local loop 1-8dB Högskolenod Typ A 3 gridless coherent portar expantion Option 10x10GE Muxponder Degree C Svartfiber till Stam LR4 MX960 Shelf 1 Shelf 2 Degree U1 Svartfiber till Univ LR4 Degree U2 Högskolenod Typ B 3 gridless coherent portar expantion Option 10x10GE Muxponder Degree B Degree D Svartfiber till Univ Svartfiber till Stam 100G Cohernet Förstärkarnod Linje fiber En gridless coherent wave Coherent mellan stam MX960 Coherent mellan stam MX960 och Univ MX480 LR4 mellan stam MX960 och lokal Univ MX480 4 degree Core site (Stamnätsnod) Juniper MX960 4 Coherent 100GE interfaces 1 LR4 100GE interface 2 OpHcal Shelves University site (Högskolenod) 2 MX480 1 LR4 100GE interface 1 Coherent 100GE interface OpHonally 10x10GE Muxponder including OpHcal Shelf MulHple 10GE interconnect interfaces Högskola MX480 MX480

17 Local loop 1-8dB University Högskolenod Typ A 3 gridless coherent portar expantion Option 10x10GE Muxponder Högskola Svartfiber till Stam LR4 MX480 Högskolenod Typ B 3 gridless coherent portar expantion Option 10x10GE Muxponder Svartfiber till Stam 100G Cohernet MX480 Type A Type B One LR4 with ophon for 3-4 coherent waves Preferable passive equipment OpHon can require PoP rebuild. One Coherent 100G Juniper interface Need to be able to provide 3-4 extra coherent waves without and extra equipment No fixed filters Preferable passive combiners/spliters

18 Routers University PoPs Malmö Göteborg Luleå Stockholm 53 University, MX PoP, MX960 5 PoP, MX PoP, MX2020

19

20

21 South has only two paths All nodes

22 Add path from Nordunet All nodes

23 New creature

24 Hybrid Raman EDFA Amplifiers Raman at all places where span-loss >17DB Optimize for high OSNR Low Noise EDFA Preparation for the future, 400G/1T Gain 9-15dB, depends on fiber and connectors

25 OTDR RAMAN SECTION

26 OTDR RAMAN SECTION SHORT DISTANCE

27

28 1383NM WATERPEAK

29 Site

30 Backup Power

31 Site Two racks per site Dual -48V DC batery bank feed Generator backup Free cooling 3G/4G Out of band

32 Site

33 Racks Custom made Slide bearings Tip safe

34 Rack Cable chains

35 Load test Juniper mx960 fully loaded kg

36 Console server + SW

37 ImplementaHon DWDM nodes come prebuilt from manufacturer with internal fiber connected. Shelf interconnects are done on site, and fibers are scoped Routers and DWDM nodes coms un-configured Permanent 3G/4G connechon to all Roadm nodes. ConfiguraHon is done via RS232 and Ethernet by Sunet Junos image is copied to flashdisk on OOB router and SCP to router. 3G copy of image takes ~1,5h 1.5GB

38

39

40

41 Monitoring show interfaces transport pm optics current et-0/0/0 Physical interface: et-0/0/0, SNMP ifindex :30-current Suspect Flag:True Reason:Not Applicable PM CURRENT MIN MAX AVG THRESHOLD TCA-ENABLED TCA-RAISED (MIN) (MAX) (MIN) (MAX) (MIN) (MAX) Lane chromatic dispersion(ps/nm) NA NA NA NA Lane differential group delay(ps) NA NA NA NA Lane Q2 factor(0.1db) NA NA NA NA SNR(0.1dB) NA NA NA NA Carrier frequency offset(mhz) No No No No Tx output power(0.01dbm) No No No No Rx input total power(0.01dbm) No No No No Module temperature(celsius) No No No No hugge@fre-r1-re0> show interfaces transport pm otn current et-0/0/0physical interface: et-0/0/0, SNMP ifindex 554 FEC Suspect Flag:False Reason:Not Applicable PM COUNT THRESHOLD TCA-ENABLED TCA-RAISED FEC-CorrectedErr NA NA FEC-UncorrectedWords 0 0 NA NA BER Suspect Flag:False Reason:Not Applicable PM MIN MAX AVG THRESHOLD TCA-ENABLED TCA-RAISED BER 9.8e-6 1.2e-5 1.1e-5 1.0e-2 No No hugge@fre-r1-re0> show interfaces diagnostics optics et-0/0/0 grep dbm Tx power : mw / 1.00 dbm Rx power (total) : mw / dbm Rx power (signal) : mw / dbm

42 Management Waves provisioning with GMPLS inside ADVA NE. Provision from NMS will be supported in next release. ADVA NMS supports control of Juniper PTX interfaces, work in progress to support MX DWDM interfaces Netconf Yang support in % of the configurahon inside the routers is controlled centrally with NSO4 from Cisco (NCS from Tail-F before acquishon). YANGmodels and Netconf is uhlized.

43 Next generation network SunetC July 2015 Procurement transmissionequipment 16 sept All offers recieved Nov Order ophcs and routers. Jan Delivery of ophcs and routers. Feb Building the core. Summer 2016 MigraHon of customers. Oct-nov 2016 Everything operahonal + teardown of old network. 31 Dec 2016 TDC will do a full poweroff off our racks Questions?

44 QUESTIONS? (or find me on any IRC network) htps://

45 QUESTIONS? (or find me on any IRC network) htps://