Building a scalable next-genera2on cheap-ass na2onwide network.

Building a scalable next-genera2on cheap-ass na2onwide network. Rakennus skaala*ava seuraavan sukupolven halpa-ass valtakunnallinen. hugge@nordu.net TREX 2016

(In two slides) Joint project by all five Nordic NRENs, running since 1989 AS2603, from California to Helsinki. Self-operated 100G DWDM in northern europe, leased or swapped 100G and 10G con2nentally. 19 Current worldwide pops, expanding to 21 this year. 2Tbit/s of current edge-capacity. Ingress peak at about 600G Ac2vely evolving addi2onal services outside of network services. Shares the same engineering and opera2onal department as SUNET One of the prime R&E networks in the world Currently searching for two System Engineers and one Media engineer ;)

Current Topology 3 systems North, West and South 2 networks per system Red and Green Redundancy on layer 3 Fully Redundant Network Red Green separahon [>10m] Desire for High Capacity Growth Support for Special ApplicaHons For example FiberChannel. Alien waves Support both; Router ConnecHvity Dedicated High BW Pt-Pt ConnecHvity Cost SensiHve GE and 10GE But also 100G Ready

Current Topology drawbacks Fixed point to point links, as no ROADMs Only one pop in each city Loose the pop and everything behind gets disconnected. Crappy DCN Long last mile Ex Stockholm Umeå Link failure get nohced by users via BGP Single homed users get longer outages. Dual link error can loose a 1/3 of the network. CombinaHon of Corestream and CN4200 Corestream can t take CWDM/DWDM XFP:s

Scale: this is approx. 1.000 km fiber path. New Topology 3 fiber paths to all regions Redundancy on layer 3 Fully Redundant Network SeparaHon between fiber paths [>10m] Desire for High Capacity Growth Support for Special ApplicaHons Transfer of frequency of light Support both; Router ConnecHvity Dedicated High BW Pt-Pt ConnecHvity Cost SensiHve 10GE and 100GE But also 1TE Ready Leased capacity to Visby

Fiberplant Core fibers mostly Aerial in high voltage power lines New and old fibers, G.652B some G.652D OTDR and attenuation data delivered as they measured Kmz data About 8000km Access fiber from city networks Two separate paths to university No KMZ data

A (new) way of building networks IP over DWDM (IPoDWDM). Eliminates unnecessary and expensive components => less components that fails Gridless Reroute traffic before fail Much lower CAPEX but also lower OPEX The worlds first nahonwide network using this technology

Juniper 100-Gigabit DWDM OTN for MX FEC Soj-decision FEC (SD-FEC) Acacia, Min OSNR 14.0 db High-gain FEC (Staircase) (HG-FEC) CorHna, Min OSNR 16.0 db G.709 FEC (GFEC), Min OSNR 24 db CFP2-ACO 100 channels on C-band ITU 50GHz spacing Output power -8 through 0 dbm Input power -18 through 0 dbm ChromaHc dispersion tolerance: +/ 40,000 ps/nm PMD tolerance: 25 ps (mean) PolarizaHon tracking: 50 krad/s

Equipment DWDM ADVA FSP3000R7 Routers Amps and Roadm Gridless Colorless Juniper MX 100G DWDM Coherent 100GE 1GE

2015-07-07 1 2 3 4 5 6 8 9 11 12 14 15 17 19 23 24 25 26 27 28 29 30 31 32 34 35 36 37 38 39 40 41 42 43 44 45 20 22 13 10 7 18 33 16 21 46 47 48 49 50 Nordunet DWDM Nordunet DWDM Nordunet DWDM Nordunet DWDM Köpenhamn Kalix Narvik Gävle Sundsvall Umeå Luleå Åkroken Kiruna Hjälta Borlänge Östersund Stockholm Uppsala Västerås Norrköping Karlstad Kalmar Växjö Karlskrona Kristianstad Malmö TUG Falun Jönköping Linköping Borås Skövde Trollhättan Göteborg Lund Kungsbacka Varla FRE Halmstad Eskilstuna Alvesta Ulricehamn ROADM University Router n x 100G Core 100G Access POP Router Line fiber 10G PTP SVA SBO KIR5 KIR3 KLX1 LLA1 AJR1 SEA1 UME8 OSD FLN4 BOR10 UPA LBA FSN2 KKA8 TRH NSNIQ BOS ULR5 JKG SKE109 ETA10 LIN3 NKG188 AVA3 VXJ KMR88 KKA8 KSD1 MCEN1 LND88 HSD1 VLA HKR1 HKR2 UU1 UU2 MAH2 MAH1 BTH2 BTH1 LNU4 LNU3 LNU2 LNU1 HJ2 HJ1 LIU1 LIU2 HB2 HB1 Onsala1 HH1 HH2 LU1 LU2 CTH2 CTH1 GU2 GU1 HIS2 HIS1 KAU1 KAU2 ORU1 ORU2 MDH1 MDH2 DU1 DU2 MIUN1 MIUN2 HIG1 HIG2 UMU2 UMU1 LTU2 LTU1 IRF1 IRF2 KTH2 KTH1 SU2 SU1 KI2 KI1 SH2 SH1 HV1 HV2 MIUN4 MIUN3 DU3 DU4 MDH4 MDH3 316 347 331 333 345 342 343 344 341 336 334 391 318 301 303 306 304 307 309 315 313 310 312 319 321 330 328 325 327 351 353 350 348 354 356 359 357 362 360 365 363 366 368 371 369 374 372 340 376 337 339 389 390 392 391 392 19 23 24 20 18 21 48 Nordunet DWDM Borås Skövde Trollhättan Göteborg Kungsbacka Varla Alves Ulriceh TRH NSNIQ BOS SKE VLA HB2 HB1 Onsala1 HH1 HH2 LU2 CTH2 CTH1 GU2 GU1 HIS2 HIS1 HV1 HV2 347 345 342 343 344 341 336 334 371 340 376 337 339 Fiber plant

IRF2 IRF1 301 303 Narvik KIR5 1 KIR3 Kiruna 2 2015-07-07 Åkroken 3 Luleå LTU1 LTU2 304 306 OpHcal ROADM POP Router University Router Line fiber n x 100G Core 100G Access 10G PTP Trollhättan HV1 HV2 Göteborg Kungsbacka Varla 334 GU1 GU2 344 341 CTH1 CTH2 336 342 343 Onsala1 345 347 17 18 24 KAU1 KAU2 TRH 48 NSNIQ VLA 331 333 23 45 Karlstad Nordunet DWDM KKA8 Borås 340 376 Nordunet DWDM BOS HB1 HB2 HH2 HH1 MIUN3 16 MIUN4 391 DU1 DU2 21 47 Skövde HIS1 HIS2 337 389 Östersund 392 318 316 DU3 DU4 390 339 13 14 BOR10 15 ORU2 ORU1 19 20 327 325 SKE109 Alvesta 27 29 28 5 9 34 35 22 7 Hjälta OSD FLN4 LBA ULR5 Ulricehamn LU2 LU1 10 Falun 36 AJR1 SEA1 Umeå Gävle 37 43 6 8 11 12 Uppsala Borlänge MDH2 MDH1 330 328 391 392 FSN2 Västerås AVA3 ETA10 MDH4 MDH3 UME8 SVA SBO UPA Eskilstuna HJ1 348 350 HJ2 JKG 41 39 Jönköping 354 356 VXJ LNU1 LNU2 LLA1 Växjö 30 4 46 Kalix Sundsvall Stockholm FRE LIU2 LIU1 353 309 313 312 351 321 LIN3 MIUN2 42 KLX1 UMU2 UMU1 315 MIUN1 HIG2 HIG1 310 307 UU2 UU1 319 40 38 Linköping Kalmar NKG188 TUG Norrköping 357 LNU3 LNU4 KTH1 KTH2 SU1 KI1 SU2 KI2 SH1 SH2 ROADM site in each University city Four pure ROADM sites No traffic are drop InterconnecHon with Nordunet Nordunet bypass Norway ESS connechvity to Copenhagen South of Sweden only has two paths ConnecHng Malmö Stockholm over Nordunet, 100G transponder or Coherent Sunet Aerial, Nordunet in the ground 25 Halmstad 374 372 HSD1 50 Nordunet DWDM 26 Lund 371 369 LND88 MAH2 MAH1 44 HKR2 HKR1 Karlskrona KMR88 31 KKA8 360 359 BTH1 BTH2 362 368 366 365 363 Köpenhamn Nordunet DWDM 49 MCEN1 Malmö 32 KSD1 Kristianstad 33

IRF2 IRF1 106 ROADM POP Router University Router Line fiber n x 100G Core 100G Access 10G PTP Trollhättan HV1 HV2 GU1 GU2 CTH1 CTH2 Göteborg Onsala1 Kungsbacka Varla 342 KAU1 KAU2 119 Halmstad Köpenhamn Karlstad Borås Narvik 106 Skövde Åkroken Östersund Alvesta Lund Malmö Hjälta BOR10 111 ORU2 ORU1 Ulricehamn 103 114 115 116 120 121 122 124 TRH NSNIQ VLA 123 KKA8 BOS HB1 HB2 HH2 HH1 HSD1 MIUN3 MIUN4 402 DU1 DU2 405 404 316 DU3 DU4 404 136 403 405 HIS1 HIS2 401 KIR5 SKE109 LU2 MCEN1 OSD FLN4 LBA ULR5 LU1 135 LND88 Falun 102 110 Kiruna Luleå Umeå Gävle Uppsala Borlänge 125 KIR3 117 Västerås 118 ETA10 131 AVA3 MAH2 MAH1 AJR1 100 SEA1 MDH2 MDH1 406 406 FSN2 Eskilstuna 407 HJ1 407 MDH4 MDH3 127 HJ2 134 JKG UME8 SVA SBO UPA 138 126 105 107 108 109 113 Jönköping VXJ LNU1 LNU2 LTU1 LTU2 KSD1 Växjö 130 HKR2 HKR1 LLA1 Kalix Sundsvall Stockholm 128 Karlskrona Kristianstad 104 403 101 FRE LIU2 LIU1 LIN3 MIUN2 402 137 KLX1 UMU2 UMU1 UU2 UU1 140 112 Linköping Kalmar KMR88 KKA8 MIUN1 HIG2 HIG1 132 133 129 NKG188 TUG Norrköping LNU3 LNU4 BTH1 BTH2 KTH1 KTH2 SU1 KI1 SU2 KI2 SH1 SH2 Design, ROADM single Core POP Router University Router Line fiber Connected n x 100G via CoreER4/LR4 Östersund Coherent 100G Access MIUN3 MIUN4 interface to the 402 next city/ 403 10G PTP OSD pop DU1 DU2 405 404 In Stockholm, Göteborg and Luleå 316 DU3 DU4 Two routers at the PoP 404 405 LR4 on both links to the University Trollhättan HV1 GU1 HV2 GU2 CTH1 CTH2 342 Göteborg KAU1 KAU2 119 Karlstad Borås Skövde Hjälta BOR10 111 ORU2 ORU1 114 115 116 120 121 122 TRH NSNIQ KKA8 FLN4 LBA 123 125 HB1 HB2 HIS1 BOS HIS2 SKE109 ULR5 Ulricehamn Falun AJR1 110 SEA1 Umeå Gävle Uppsala Borlänge MDH2 MDH1 406 406 FSN2 117 Västerås 118 ETA10 Eskilstuna 407 HJ1 407 MDH4 MDH3 127 HJ2 JKG UME8 SVA SBO UPA 126 10 10 10 10 11 Jönköping VXJ LIU Vä

Stamnätsnod Sites Degree A Local loop 1-8dB Högskolenod Typ A 3 gridless coherent portar expantion Option 10x10GE Muxponder Degree C Svartfiber till Stam LR4 MX960 Shelf 1 Shelf 2 Degree U1 Svartfiber till Univ LR4 Degree U2 Högskolenod Typ B 3 gridless coherent portar expantion Option 10x10GE Muxponder Degree B Degree D Svartfiber till Univ Svartfiber till Stam 100G Cohernet Förstärkarnod Linje fiber En gridless coherent wave Coherent mellan stam MX960 Coherent mellan stam MX960 och Univ MX480 LR4 mellan stam MX960 och lokal Univ MX480 4 degree Core site (Stamnätsnod) Juniper MX960 4 Coherent 100GE interfaces 1 LR4 100GE interface 2 OpHcal Shelves University site (Högskolenod) 2 MX480 1 LR4 100GE interface 1 Coherent 100GE interface OpHonally 10x10GE Muxponder including OpHcal Shelf MulHple 10GE interconnect interfaces Högskola MX480 MX480

Local loop 1-8dB University Högskolenod Typ A 3 gridless coherent portar expantion Option 10x10GE Muxponder Högskola Svartfiber till Stam LR4 MX480 Högskolenod Typ B 3 gridless coherent portar expantion Option 10x10GE Muxponder Svartfiber till Stam 100G Cohernet MX480 Type A Type B One LR4 with ophon for 3-4 coherent waves Preferable passive equipment OpHon can require PoP rebuild. One Coherent 100G Juniper interface Need to be able to provide 3-4 extra coherent waves without and extra equipment No fixed filters Preferable passive combiners/spliters

Routers University PoPs Malmö Göteborg Luleå Stockholm 53 University, MX480 24 PoP, MX960 5 PoP, MX2010 2 PoP, MX2020

South has only two paths All nodes

Add path from Nordunet All nodes

New creature

Hybrid Raman EDFA Amplifiers Raman at all places where span-loss >17DB Optimize for high OSNR Low Noise EDFA Preparation for the future, 400G/1T Gain 9-15dB, depends on fiber and connectors

OTDR RAMAN SECTION

OTDR RAMAN SECTION SHORT DISTANCE

1383NM WATERPEAK

Backup Power

Site Two racks per site Dual -48V DC batery bank feed Generator backup Free cooling 3G/4G Out of band

Racks Custom made Slide bearings Tip safe

Rack Cable chains

Load test Juniper mx960 fully loaded 158.76 kg

Console server + SW

ImplementaHon DWDM nodes come prebuilt from manufacturer with internal fiber connected. Shelf interconnects are done on site, and fibers are scoped Routers and DWDM nodes coms un-configured Permanent 3G/4G connechon to all Roadm nodes. ConfiguraHon is done via RS232 and Ethernet by Sunet Junos image is copied to flashdisk on OOB router and SCP to router. 3G copy of image takes ~1,5h 1.5GB

Monitoring hugge@fre-r1-re0> show interfaces transport pm optics current et-0/0/0 Physical interface: et-0/0/0, SNMP ifindex 554 12:30-current Suspect Flag:True Reason:Not Applicable PM CURRENT MIN MAX AVG THRESHOLD TCA-ENABLED TCA-RAISED (MIN) (MAX) (MIN) (MAX) (MIN) (MAX) Lane chromatic dispersion(ps/nm) 2795 0 0 0 0 0 NA NA NA NA Lane differential group delay(ps) 7 3 7 3 0 0 NA NA NA NA Lane Q2 factor(0.1db) 125 125 125 125 0 0 NA NA NA NA SNR(0.1dB) 132 131 134 131 0 0 NA NA NA NA Carrier frequency offset(mhz) 62 49 64 54-3600 3600 No No No No Tx output power(0.01dbm) 99 93 107 97-1100 300 No No No No Rx input total power(0.01dbm) -1051-1413 -1048-1062 -3000 300 No No No No Module temperature(celsius) 31 31 31 31-5 75 No No No No hugge@fre-r1-re0> show interfaces transport pm otn current et-0/0/0physical interface: et-0/0/0, SNMP ifindex 554 FEC Suspect Flag:False Reason:Not Applicable PM COUNT THRESHOLD TCA-ENABLED TCA-RAISED FEC-CorrectedErr 816882837 0 NA NA FEC-UncorrectedWords 0 0 NA NA BER Suspect Flag:False Reason:Not Applicable PM MIN MAX AVG THRESHOLD TCA-ENABLED TCA-RAISED BER 9.8e-6 1.2e-5 1.1e-5 1.0e-2 No No hugge@fre-r1-re0> show interfaces diagnostics optics et-0/0/0 grep dbm Tx power : 1.259 mw / 1.00 dbm Rx power (total) : 0.089 mw / -10.51 dbm Rx power (signal) : 0.018 mw / -17.45 dbm

Management Waves provisioning with GMPLS inside ADVA NE. Provision from NMS will be supported in next release. ADVA NMS supports control of Juniper PTX interfaces, work in progress to support MX DWDM interfaces Netconf Yang support in 2017 85% of the configurahon inside the routers is controlled centrally with NSO4 from Cisco (NCS from Tail-F before acquishon). YANGmodels and Netconf is uhlized.

Next generation network SunetC July 2015 Procurement transmissionequipment 16 sept. 2015 All offers recieved Nov. 2015 Order ophcs and routers. Jan. 2016 Delivery of ophcs and routers. Feb. 2016 Building the core. Summer 2016 MigraHon of customers. Oct-nov 2016 Everything operahonal + teardown of old network. 31 Dec 2016 TDC will do a full poweroff off our racks Questions?

QUESTIONS? hugge@nordu.net (or find me on any IRC network) htps://www.sunet.se/bloggdel/teknikbloggen/