Greenfielding a scalable next- generation (cheap) nationwide network. hugge@nordu.net NETNOD Autumn 2016
(In one slide) Joint project by all five Nordic NRENs, running since 1989 AS2603, from California to Helsinki. Self- operated 100G DWDM in Europe, leased or swapped 100G and 10G continentally. 19 Current worldwide pops, expanding to 21 this year. 3Tbit/s of current edge- capacity. Actively evolving additional services outside of network services. Shares the same engineering and operational department as SUNET One of the prime R&E networks in the world Currently searching for one Systems Engineer ;)
Current Topology 3 systems North, West and South 2 networks per system Red and Green Redundancy on layer 3 Fully Redundant Network Red Green separation [>10m] Support for Special Applications For example FiberChannel. Alien waves Support both; Router Connectivity Dedicated High BW Pt- Pt Connectivity Cost Sensitive GE and 10GE But also 100G Ready
Current Topology drawbacks Fixed point to point links, as no ROADMs Only one pop in each city Loose the pop and everything behind gets disconnected. Crappy DCN Long last mile Ex Stockholm Umeå Link failure get noticed by users via BGP Single homed users get longer outages. Dual link error can loose a 1/3 of the network. Combination of Corestream and CN4200 Corestream can t take CWDM/DWDM XFP:s
Scale: this is approx. 1.000 km fiber path. New Topology 3 fiber paths to all regions Redundancy on layer 3 Fully Redundant Network Separation between fiber paths [>10m] Desire for High Capacity Growth Support for Special Applications Transfer of frequency of light Support both; Router Connectivity Dedicated High BW Pt- Pt Connectivity Cost Sensitive 10GE and 100GE But also 1TE Ready Leased capacity to Visby
Fiberplant Core fibers mostly Aerial in high voltage power lines New and old fibers, G.652B some G.652D OTDR and attenuation data delivered as they measured Kmz data About 8000km Access fiber from city networks Two separate paths to university No KMZ data
Fiber Tender Only one patch the first 20km and >=1.5db attenuation >=- 42dB reflection.25db/km Measurement of water peak OTDR Measurement Short Pulse Long Pulse Require all connectors to pass IEC 61300-3- 35 Splice >= 0.2dB Connector >= 0.4dB (Proof of this with pictures of all connectors) Equal Spacing between sites Connector- Splice plan / fiber Stretch Attenuation/Stretch shall include all connectors and splices
A (new) way of building networks IP over DWDM (IPoDWDM). Eliminates unnecessary and expensive components => less components that fails Reroute traffic before fail Much lower CAPEX but also lower OPEX Gridless The worlds first nationwide network using this technology
Juniper 100- Gigabit DWDM OTN for MX FEC Soft- decision FEC (SD- FEC) Acacia, Min OSNR 14.0 db High- gain FEC (Staircase / HG- FEC / Cortina), Min OSNR 16.0 db G.709 FEC (GFEC), Min OSNR 24 db CFP2- ACO 100 channels on C- band ITU 50GHz spacing Output power - 8 through 0 dbm Input power - 28 through 0 dbm Chromatic dispersion tolerance: +/ 40,000 ps/nm PMD tolerance: 25 ps (mean) Polarization tracking: 50 krad/s
2015-07-07 1 2 3 4 5 6 8 9 11 12 14 15 17 19 23 24 25 26 27 28 29 30 31 32 34 35 36 37 38 39 40 41 42 43 44 45 20 22 13 10 7 18 33 16 21 46 47 48 49 50 Nordunet DWDM Nordunet DWDM Nordunet DWDM Nordunet DWDM Köpenhamn Kalix Narvik Gävle Sundsvall Umeå Luleå Åkroken Kiruna Hjälta Borlänge Östersund Stockholm Uppsala Västerås Norrköping Karlstad Kalmar Växjö Karlskrona Kristianstad Malmö TUG Falun Jönköping Linköping Borås Skövde Trollhättan Göteborg Lund Kungsbacka Varla FRE Halmstad Eskilstuna Alvesta Ulricehamn ROADM University Router n x 100G Core 100G Access POP Router Line fiber 10G PTP SVA SBO KIR5 KIR3 KLX1 LLA1 AJR1 SEA1 UME8 OSD FLN4 BOR10 UPA LBA FSN2 KKA8 TRH NSNIQ BOS ULR5 JKG SKE109 ETA10 LIN3 NKG188 AVA3 VXJ KMR88 KKA8 KSD1 MCEN1 LND88 HSD1 VLA HKR1 HKR2 UU1 UU2 MAH2 MAH1 BTH2 BTH1 LNU4 LNU3 LNU2 LNU1 HJ2 HJ1 LIU1 LIU2 HB2 HB1 Onsala1 HH1 HH2 LU1 LU2 CTH2 CTH1 GU2 GU1 HIS2 HIS1 KAU1 KAU2 ORU1 ORU2 MDH1 MDH2 DU1 DU2 MIUN1 MIUN2 HIG1 HIG2 UMU2 UMU1 LTU2 LTU1 IRF1 IRF2 KTH2 KTH1 SU2 SU1 KI2 KI1 SH2 SH1 HV1 HV2 MIUN4 MIUN3 DU3 DU4 MDH4 MDH3 316 347 331 333 345 342 343 344 341 336 334 391 318 301 303 306 304 307 309 315 313 310 312 319 321 330 328 325 327 351 353 350 348 354 356 359 357 362 360 365 363 366 368 371 369 374 372 340 376 337 339 389 390 392 391 392 19 23 24 20 18 21 48 Nordunet DWDM Borås Skövde Trollhättan Göteborg Kungsbacka Varla Alves Ulriceh TRH NSNIQ BOS SKE VLA HB2 HB1 Onsala1 HH1 HH2 LU2 CTH2 CTH1 GU2 GU1 HIS2 HIS1 HV1 HV2 347 345 342 343 344 341 336 334 371 340 376 337 339 Fiber plant
IRF2 IRF1 301 303 Narvik KIR5 1 KIR3 Kiruna 2 2015-07-07 Åkroken 3 Luleå LTU1 LTU2 304 306 Optical ROADM POP Router University Router Line fiber n x 100G Core 100G Access 10G PTP Trollhättan HV1 HV2 Göteborg Kungsbacka Varla 334 GU1 GU2 344 341 CTH1 CTH2 336 342 343 Onsala1 345 347 17 18 24 KAU1 KAU2 TRH 48 NSNIQ VLA 331 333 23 45 Karlstad Nordunet DWDM KKA8 Borås 340 376 Nordunet DWDM BOS HB1 HB2 HH2 HH1 MIUN3 16 MIUN4 391 DU1 DU2 21 47 Skövde HIS1 HIS2 337 389 Östersund 392 318 316 DU3 DU4 390 339 13 14 BOR10 15 ORU2 ORU1 19 20 327 325 SKE109 Alvesta 27 29 28 5 9 34 35 22 7 Hjälta OSD FLN4 LBA ULR5 Ulricehamn LU2 LU1 10 Falun 36 AJR1 SEA1 Umeå Gävle 37 43 6 8 11 12 Uppsala Borlänge MDH2 MDH1 330 328 391 392 FSN2 Västerås AVA3 ETA10 MDH4 MDH3 UME8 SVA SBO UPA Eskilstuna HJ1 348 350 HJ2 JKG 41 39 Jönköping 354 356 VXJ LNU1 LNU2 LLA1 Växjö 30 4 46 Kalix Sundsvall Stockholm FRE LIU2 LIU1 353 309 313 312 351 321 LIN3 MIUN2 42 KLX1 UMU2 UMU1 315 MIUN1 HIG2 HIG1 310 307 UU2 UU1 319 40 38 Linköping Kalmar NKG188 TUG Norrköping 357 LNU3 LNU4 KTH1 KTH2 SU1 KI1 SU2 KI2 SH1 SH2 ROADM site in each University city Four pure ROADM sites No traffic are drop Interconnection with Nordunet Nordunetbypass Norway ESS connectivity to Copenhagen South of Sweden only has two paths Connecting Malmö Stockholm over Nordunet, 100G alien coherent Sunet Aerial, Nordunet in the ground 25 Halmstad 374 372 HSD1 50 Nordunet DWDM 26 Lund 371 369 LND88 MAH2 MAH1 44 HKR2 HKR1 Karlskrona KMR88 31 KKA8 360 359 BTH1 BTH2 362 368 366 365 363 Köpenhamn Nordunet DWDM 49 MCEN1 Malmö 32 KSD1 Kristianstad 33
Stamnätsnod Sites Degree A Local loop 1-8dB Högskolenod Typ A 3 gridless coherent portar expantion Option 10x10GE Muxponder Degree C Svartfiber till Stam LR4 MX960 Shelf 1 Shelf 2 Degree U1 Svartfiber till Univ LR4 Degree U2 Högskolenod Typ B 3 gridless coherent portar expantion Option 10x10GE Muxponder Degree B Degree D Svartfiber till Univ Svartfiber till Stam 100G Cohernet Förstärkarnod Linje fiber En gridless coherent wave Coherent mellan stam MX960 Coherent mellan stam MX960 och Univ MX480 LR4 mellan stam MX960 och lokal Univ MX480 4 degree Core site (Stamnätsnod) Juniper MX960 4 Coherent 100GE interfaces 1 LR4 100GE interface 2 Optical Shelves University site (Högskolenod) 2 MX480 1 LR4 100GE interface 1 Coherent 100GE interface Optionally 10x10GE Muxponder including Optical Shelf Multiple 10GE interconnect interfaces Högskola MX480 MX480
Routers 15 MX80 53 University, MX480 29 PoP, MX960 5 PoP, MX2010 2 PoP, MX2020 Optical 35 ROADM FSP 3000 80 ILA FSP 3000 University PoPs Malmö Göteborg Luleå Stockholm
All nodes
Hybrid Raman EDFA Amplifiers Raman at all places with more than 17db span loss Optimize for high OSNR Low Noise EDFA Preparation for the future, 400G/1T Gain 9-15dB, depends on fiber and connectors
OTDR RAMAN SECTION
OTDR RAMAN SECTION SHORT DISTANCE
1383NM WATERPEAK
Site
Backup Power
Site Two racks per site Dual - 48V DC battery bank feed Generator backup Free cooling 3G/4G Out of band
Racks Custom made Slide bearings Tip safe
Rack Cable chains
Load test Juniper mx960 fully loaded 158.76 kg
Console server + SW
Implementation DWDM nodes comes prebuilt from manufacture with internal fiber connected. Shelf interconnects are done on site, and fibers are scoped Routers and DWDM nodes coms un- configured Permanent 3G/4G connection to all Roadm nodes. Configuration is done via RS232 and Ethernet by Sunet Junos image is copied to flashdisk on OOB router and SCP to router over 4G
Monitoring hugge@fre-r1-re0> show interfaces transport pm optics current et-0/0/0 Physical interface: et-0/0/0, SNMP ifindex 554 12:30-current Suspect Flag:True Reason:Not Applicable PM CURRENT MIN MAX AVG THRESHOLD TCA-ENABLED TCA-RAISED (MIN) (MAX) (MIN) (MAX) (MIN) (MAX) Lane chromatic dispersion(ps/nm) 2795 0 0 0 0 0 NA NA NA NA Lane differential group delay(ps) 7 3 7 3 0 0 NA NA NA NA Lane Q2 factor(0.1db) 125 125 125 125 0 0 NA NA NA NA SNR(0.1dB) 132 131 134 131 0 0 NA NA NA NA Carrier frequency offset(mhz) 62 49 64 54-3600 3600 No No No No Tx output power(0.01dbm) 99 93 107 97-1100 300 No No No No Rx input total power(0.01dbm) -1051-1413 -1048-1062 -3000 300 No No No No Module temperature(celsius) 31 31 31 31-5 75 No No No No hugge@fre-r1-re0> show interfaces transport pm otn current et-0/0/0physical interface: et-0/0/0, SNMP ifindex 554 FEC Suspect Flag:False Reason:Not Applicable PM COUNT THRESHOLD TCA-ENABLED TCA-RAISED FEC-CorrectedErr 816882837 0 NA NA FEC-UncorrectedWords 0 0 NA NA BER Suspect Flag:False Reason:Not Applicable PM MIN MAX AVG THRESHOLD TCA-ENABLED TCA-RAISED BER 9.8e-6 1.2e-5 1.1e-5 1.0e-2 No No hugge@fre-r1-re0> show interfaces diagnostics optics et-0/0/0 grep dbm Tx power : 1.259 mw / 1.00 dbm Rx power (total) : 0.089 mw / -10.51 dbm Rx power (signal) : 0.018 mw / -17.45 dbm
Management Waves provisioning with GMPLS inside ADVA NE.. ADVA NMS supports control of Juniper PTX interfaces, work in progress to support MX DWDM interfaces Netconf Yang support in 2017 95% of the configuration inside the routers is controlled centrally with NSO from Cisco (NCS from Tail- F before acquistion). YANG- models and Netconf is the name of the game.
QUESTIONS? hugge@nordu.net (or find me on any IRC network) https://www.sunet.se/bloggdel/teknikbloggen/