Greenfielding a scalable next- generation (cheap) nationwide network. NETNOD Autumn 2016

Transkript

1 Greenfielding a scalable next- generation (cheap) nationwide network. hugge@nordu.net NETNOD Autumn 2016

2 (In one slide) Joint project by all five Nordic NRENs, running since 1989 AS2603, from California to Helsinki. Self- operated 100G DWDM in Europe, leased or swapped 100G and 10G continentally. 19 Current worldwide pops, expanding to 21 this year. 3Tbit/s of current edge- capacity. Actively evolving additional services outside of network services. Shares the same engineering and operational department as SUNET One of the prime R&E networks in the world Currently searching for one Systems Engineer ;)

3

4 Current Topology 3 systems North, West and South 2 networks per system Red and Green Redundancy on layer 3 Fully Redundant Network Red Green separation [>10m] Support for Special Applications For example FiberChannel. Alien waves Support both; Router Connectivity Dedicated High BW Pt- Pt Connectivity Cost Sensitive GE and 10GE But also 100G Ready

5 Current Topology drawbacks Fixed point to point links, as no ROADMs Only one pop in each city Loose the pop and everything behind gets disconnected. Crappy DCN Long last mile Ex Stockholm Umeå Link failure get noticed by users via BGP Single homed users get longer outages. Dual link error can loose a 1/3 of the network. Combination of Corestream and CN4200 Corestream can t take CWDM/DWDM XFP:s

6 Scale: this is approx km fiber path. New Topology 3 fiber paths to all regions Redundancy on layer 3 Fully Redundant Network Separation between fiber paths [>10m] Desire for High Capacity Growth Support for Special Applications Transfer of frequency of light Support both; Router Connectivity Dedicated High BW Pt- Pt Connectivity Cost Sensitive 10GE and 100GE But also 1TE Ready Leased capacity to Visby

7 Fiberplant Core fibers mostly Aerial in high voltage power lines New and old fibers, G.652B some G.652D OTDR and attenuation data delivered as they measured Kmz data About 8000km Access fiber from city networks Two separate paths to university No KMZ data

8 Fiber Tender Only one patch the first 20km and >=1.5db attenuation >=- 42dB reflection.25db/km Measurement of water peak OTDR Measurement Short Pulse Long Pulse Require all connectors to pass IEC Splice >= 0.2dB Connector >= 0.4dB (Proof of this with pictures of all connectors) Equal Spacing between sites Connector- Splice plan / fiber Stretch Attenuation/Stretch shall include all connectors and splices

9 A (new) way of building networks IP over DWDM (IPoDWDM). Eliminates unnecessary and expensive components => less components that fails Reroute traffic before fail Much lower CAPEX but also lower OPEX Gridless The worlds first nationwide network using this technology

10 Juniper 100- Gigabit DWDM OTN for MX FEC Soft- decision FEC (SD- FEC) Acacia, Min OSNR 14.0 db High- gain FEC (Staircase / HG- FEC / Cortina), Min OSNR 16.0 db G.709 FEC (GFEC), Min OSNR 24 db CFP2- ACO 100 channels on C- band ITU 50GHz spacing Output power - 8 through 0 dbm Input power - 28 through 0 dbm Chromatic dispersion tolerance: +/ 40,000 ps/nm PMD tolerance: 25 ps (mean) Polarization tracking: 50 krad/s

11

12 Nordunet DWDM Nordunet DWDM Nordunet DWDM Nordunet DWDM Köpenhamn Kalix Narvik Gävle Sundsvall Umeå Luleå Åkroken Kiruna Hjälta Borlänge Östersund Stockholm Uppsala Västerås Norrköping Karlstad Kalmar Växjö Karlskrona Kristianstad Malmö TUG Falun Jönköping Linköping Borås Skövde Trollhättan Göteborg Lund Kungsbacka Varla FRE Halmstad Eskilstuna Alvesta Ulricehamn ROADM University Router n x 100G Core 100G Access POP Router Line fiber 10G PTP SVA SBO KIR5 KIR3 KLX1 LLA1 AJR1 SEA1 UME8 OSD FLN4 BOR10 UPA LBA FSN2 KKA8 TRH NSNIQ BOS ULR5 JKG SKE109 ETA10 LIN3 NKG188 AVA3 VXJ KMR88 KKA8 KSD1 MCEN1 LND88 HSD1 VLA HKR1 HKR2 UU1 UU2 MAH2 MAH1 BTH2 BTH1 LNU4 LNU3 LNU2 LNU1 HJ2 HJ1 LIU1 LIU2 HB2 HB1 Onsala1 HH1 HH2 LU1 LU2 CTH2 CTH1 GU2 GU1 HIS2 HIS1 KAU1 KAU2 ORU1 ORU2 MDH1 MDH2 DU1 DU2 MIUN1 MIUN2 HIG1 HIG2 UMU2 UMU1 LTU2 LTU1 IRF1 IRF2 KTH2 KTH1 SU2 SU1 KI2 KI1 SH2 SH1 HV1 HV2 MIUN4 MIUN3 DU3 DU4 MDH4 MDH Nordunet DWDM Borås Skövde Trollhättan Göteborg Kungsbacka Varla Alves Ulriceh TRH NSNIQ BOS SKE VLA HB2 HB1 Onsala1 HH1 HH2 LU2 CTH2 CTH1 GU2 GU1 HIS2 HIS1 HV1 HV Fiber plant

13 IRF2 IRF Narvik KIR5 1 KIR3 Kiruna Åkroken 3 Luleå LTU1 LTU Optical ROADM POP Router University Router Line fiber n x 100G Core 100G Access 10G PTP Trollhättan HV1 HV2 Göteborg Kungsbacka Varla 334 GU1 GU CTH1 CTH Onsala KAU1 KAU2 TRH 48 NSNIQ VLA Karlstad Nordunet DWDM KKA8 Borås Nordunet DWDM BOS HB1 HB2 HH2 HH1 MIUN3 16 MIUN4 391 DU1 DU Skövde HIS1 HIS Östersund DU3 DU BOR10 15 ORU2 ORU SKE109 Alvesta Hjälta OSD FLN4 LBA ULR5 Ulricehamn LU2 LU1 10 Falun 36 AJR1 SEA1 Umeå Gävle Uppsala Borlänge MDH2 MDH FSN2 Västerås AVA3 ETA10 MDH4 MDH3 UME8 SVA SBO UPA Eskilstuna HJ HJ2 JKG Jönköping VXJ LNU1 LNU2 LLA1 Växjö Kalix Sundsvall Stockholm FRE LIU2 LIU LIN3 MIUN2 42 KLX1 UMU2 UMU1 315 MIUN1 HIG2 HIG UU2 UU Linköping Kalmar NKG188 TUG Norrköping 357 LNU3 LNU4 KTH1 KTH2 SU1 KI1 SU2 KI2 SH1 SH2 ROADM site in each University city Four pure ROADM sites No traffic are drop Interconnection with Nordunet Nordunetbypass Norway ESS connectivity to Copenhagen South of Sweden only has two paths Connecting Malmö Stockholm over Nordunet, 100G alien coherent Sunet Aerial, Nordunet in the ground 25 Halmstad HSD1 50 Nordunet DWDM 26 Lund LND88 MAH2 MAH1 44 HKR2 HKR1 Karlskrona KMR88 31 KKA BTH1 BTH Köpenhamn Nordunet DWDM 49 MCEN1 Malmö 32 KSD1 Kristianstad 33

14 Stamnätsnod Sites Degree A Local loop 1-8dB Högskolenod Typ A 3 gridless coherent portar expantion Option 10x10GE Muxponder Degree C Svartfiber till Stam LR4 MX960 Shelf 1 Shelf 2 Degree U1 Svartfiber till Univ LR4 Degree U2 Högskolenod Typ B 3 gridless coherent portar expantion Option 10x10GE Muxponder Degree B Degree D Svartfiber till Univ Svartfiber till Stam 100G Cohernet Förstärkarnod Linje fiber En gridless coherent wave Coherent mellan stam MX960 Coherent mellan stam MX960 och Univ MX480 LR4 mellan stam MX960 och lokal Univ MX480 4 degree Core site (Stamnätsnod) Juniper MX960 4 Coherent 100GE interfaces 1 LR4 100GE interface 2 Optical Shelves University site (Högskolenod) 2 MX480 1 LR4 100GE interface 1 Coherent 100GE interface Optionally 10x10GE Muxponder including Optical Shelf Multiple 10GE interconnect interfaces Högskola MX480 MX480

15 Routers 15 MX80 53 University, MX PoP, MX960 5 PoP, MX PoP, MX2020 Optical 35 ROADM FSP ILA FSP 3000 University PoPs Malmö Göteborg Luleå Stockholm

16

17

18 All nodes

19 Hybrid Raman EDFA Amplifiers Raman at all places with more than 17db span loss Optimize for high OSNR Low Noise EDFA Preparation for the future, 400G/1T Gain 9-15dB, depends on fiber and connectors

20 OTDR RAMAN SECTION

21 OTDR RAMAN SECTION SHORT DISTANCE

22

23 1383NM WATERPEAK

24 Site

25 Backup Power

26 Site Two racks per site Dual - 48V DC battery bank feed Generator backup Free cooling 3G/4G Out of band

27 Racks Custom made Slide bearings Tip safe

28 Rack Cable chains

29 Load test Juniper mx960 fully loaded kg

30 Console server + SW

31 Implementation DWDM nodes comes prebuilt from manufacture with internal fiber connected. Shelf interconnects are done on site, and fibers are scoped Routers and DWDM nodes coms un- configured Permanent 3G/4G connection to all Roadm nodes. Configuration is done via RS232 and Ethernet by Sunet Junos image is copied to flashdisk on OOB router and SCP to router over 4G

32

33

34

35 Monitoring show interfaces transport pm optics current et-0/0/0 Physical interface: et-0/0/0, SNMP ifindex :30-current Suspect Flag:True Reason:Not Applicable PM CURRENT MIN MAX AVG THRESHOLD TCA-ENABLED TCA-RAISED (MIN) (MAX) (MIN) (MAX) (MIN) (MAX) Lane chromatic dispersion(ps/nm) NA NA NA NA Lane differential group delay(ps) NA NA NA NA Lane Q2 factor(0.1db) NA NA NA NA SNR(0.1dB) NA NA NA NA Carrier frequency offset(mhz) No No No No Tx output power(0.01dbm) No No No No Rx input total power(0.01dbm) No No No No Module temperature(celsius) No No No No hugge@fre-r1-re0> show interfaces transport pm otn current et-0/0/0physical interface: et-0/0/0, SNMP ifindex 554 FEC Suspect Flag:False Reason:Not Applicable PM COUNT THRESHOLD TCA-ENABLED TCA-RAISED FEC-CorrectedErr NA NA FEC-UncorrectedWords 0 0 NA NA BER Suspect Flag:False Reason:Not Applicable PM MIN MAX AVG THRESHOLD TCA-ENABLED TCA-RAISED BER 9.8e-6 1.2e-5 1.1e-5 1.0e-2 No No hugge@fre-r1-re0> show interfaces diagnostics optics et-0/0/0 grep dbm Tx power : mw / 1.00 dbm Rx power (total) : mw / dbm Rx power (signal) : mw / dbm

36 Management Waves provisioning with GMPLS inside ADVA NE.. ADVA NMS supports control of Juniper PTX interfaces, work in progress to support MX DWDM interfaces Netconf Yang support in % of the configuration inside the routers is controlled centrally with NSO from Cisco (NCS from Tail- F before acquistion). YANG- models and Netconf is the name of the game.

37 QUESTIONS? (or find me on any IRC network)