Terabit Ethernet

Terabit Ethernet or TbE is Ethernet with speeds above 100 Gbit/s. 400 Gigabit Ethernet (400G, 400GbE) and 200 Gigabit Ethernet (200G, 200GbE)[1] standards developed by the IEEE P802.3bs Task Force using broadly similar technology to 100 Gigabit Ethernet[2][3] were approved on December 6, 2017.[4][5] In 2016, several networking equipment suppliers were already offering proprietary solutions for 200G and 400G.[5]

The Ethernet Alliance's 2020 technology roadmap expects speeds of 800 Gbit/s and 1.6 Tbit/s become IEEE standard after 2020, possibly between 2023 and 2025.[6][7] Doubling to 800 GbE is expected to occur after 112 Gbit/s SerDes become available. The Optical Internetworking Forum (OIF) has already announced five new projects at 112 Gbit/s which would also make 4th generation (single-lane) 100 GbE links possible.[8]

History

Facebook and Google, among other companies, have expressed a need for TbE.[9] While a speed of 400 Gbit/s is achievable with existing technology, 1 Tbit/s (1000 Gbit/s) would require different technology.[2][10] Accordingly, at the IEEE Industry Connections Higher Speed Ethernet Consensus group meeting in September 2012, 400 GbE was chosen as the next generation goal.[2] Additional 200GbE objectives were added in January 2016.

The University of California, Santa Barbara (UCSB) attracted help from Agilent Technologies, Google, Intel, Rockwell Collins, and Verizon Communications to help with research into next generation Ethernet.[11]

As of early 2016, chassis/modular based core router platforms from Cisco, Juniper and other major manufacturers support 400 Gbit/s full duplex data rates per slot. One, two and four port 100GbE and one port 400GbE line cards are presently available. As of early 2019, 200GbE line cards became available after 802.3cd standard ratification.[12][13]

200G Ethernet uses PAM4 signaling which allows 2 bits to be transmitted per clock cycle, but at a higher implementation cost.[14]

Standards development

The IEEE formed the "IEEE 802.3 Industry Connections Ethernet Bandwidth Assessment Ad Hoc", to investigate the business needs for short and long term bandwidth requirements.[15][16][17]

IEEE 802.3's "400 Gb/s Ethernet Study Group" started working on the 400 Gbit/s generation standard in March 2013.[18] Results from the study group were published and approved on March 27, 2014. Subsequently, the IEEE 802.3bs Task Force[19] started working to provide physical layer specifications for several link distances.[20]

The IEEE 802.3bs standard was approved on December 6, 2017[4] and is available online.[21]

The IEEE 802.3cd standard was approved on December 5, 2018.

The IEEE 802.3cn standard was approved on December 20, 2019.

The IEEE 802.3cm standard was approved on January 30, 2020.

IEEE project objectives

Like all speeds since 10 Gigabit Ethernet, the standards support only full-duplex operation. Other objectives include:[20]

1. Support MAC data rates of 400 Gbit/s and 200 Gbit/s[1]
2. Preserve the Ethernet frame format utilizing the Ethernet MAC
3. Preserve minimum and maximum frame size of current Ethernet standard
4. Support a bit error ratio (BER) of 10⁻¹³, which is an improvement over the 10⁻¹² BER that was specified for 10GbE, 40GbE, and 100GbE.
5. Support for OTN (transport of Ethernet across optical transport networks), and optional support for Energy-Efficient Ethernet (EEE).

802.3bs project

Define physical layer specifications supporting:[20]

• 400 Gbit/s Ethernet
  • at least 100 m over multi-mode fiber (400GBASE-SR16) using sixteen parallel strands of fiber each at 25 Gbit/s[22][23]
  • at least 500 m over single-mode fiber (400GBASE-DR4) using four parallel strands of fiber each at 100 Gbit/s[24][25]
  • at least 2 km over single-mode fiber (400GBASE-FR8) using eight parallel wavelengths (CWDM) each at 50 Gbit/s[24][26][27]
  • at least 10 km over single-mode fiber (400GBASE-LR8) using eight parallel wavelengths (CWDM) each at 50 Gbit/s[24][27][28]
  • eight and sixteen lane chip-to-chip/chip-to-module electrical interfaces (400GAUI-8 and 400GAUI-16)
• 200 Gbit/s Ethernet
  • at least 500 m over single-mode fiber (200GBASE-DR4) using four parallel strands of fiber each at 50 Gbit/s[29][30]
  • at least 2 km over single-mode fiber (200GBASE-FR4) using four parallel wavelengths (CWDM) each at 50 Gbit/s[1][30]
  • at least 10 km over single-mode fiber (200GBASE-LR4) using four parallel wavelengths (CWDM) each at 50 Gbit/s[1][30]
  • four and eight lane chip-to-chip/chip-to-module electrical interfaces (200GAUI-4 and 200GAUI-8)

802.3cd project

• Define four-lane 200 Gbit/s PHYs for operation over:
  • copper twin-axial cables with lengths up to at least 3 m (200GBASE-CR4).
  • printed circuit board backplane with a total channel insertion loss of ≤ 30 dB at 13.28125 GHz (200GBASE-KR4).
• Define 200 Gbit/s PHYs for operation over MMF with lengths up to at least 100 m (200GBASE-SR4).

802.3ck project

• 200 Gbit/s Ethernet
  • Define a two-lane 200 Gbit/s Attachment Unit interface (AUI) for chip-to-module applications, compatible with PMDs based on 100 Gbit/s per lane optical signaling (200GAUI-2 C2M)
  • Define a two-lane 200 Gbit/s Attachment Unit Interface (AUI) for chip-to-chip applications (200GAUI-2 C2C)
  • Define a two-lane 200 Gbit/s PHY for operation over electrical backplanes an insertion loss ≤ 28 dB at 26.56 GHz (200GBASE-KR2)
  • Define a two-lane 200 Gbit/s PHY for operation over twin axial copper cables with lengths up to at least 2 m (200GBASE-CR2)
• 400 Gbit/s Ethernet
  • Define a four-lane 400 Gbit/s Attachment Unit interface (AUI) for chip-to-module applications, compatible with PMDs based on 100 Gbit/s per lane optical signaling (400GAUI-4 C2M)
  • Define a four-lane 400 Gbit/s Attachment Unit Interface (AUI) for chip-to-chip applications (400GAUI-4 C2C)
  • Define a four-lane 400 Gbit/s PHY for operation over electrical backplanes an insertion loss ≤ 28 dB at 26.56 GHz (400GBASE-KR4)
  • Define a four-lane 400 Gbit/s PHY for operation over twin axial copper cables with lengths up to at least 2 m (400GBASE-CR4)

802.3cm project

• 400 Gbit/s Ethernet
  • Define a physical layer specification supporting 400 Gbit/s operation over 8 pairs of MMF with lengths up to at least 100 m (400GBASE-SR8)
  • Define a physical layer specification supporting 400 Gbit/s operation over 4 pairs of MMF with lengths up to at least 100 m (400GBASE-SR4.2)

802.3cn project

• 200 Gbit/s Ethernet
  • Provide a physical layer specification supporting 200 Gbit/s operation over four wavelengths capable of at least 40 km of SMF (200GBASE-ER4) [31]
• 400 Gbit/s Ethernet
  • Provide a physical layer specification supporting 400 Gbit/s operation over eight wavelengths capable of at least 40 km of SMF (400GBASE-ER8)[31]

802.3cu project

• Define a four-wavelength 400 Gbit/s PHY for operation over SMF with lengths up to at least 2 km (400GBASE-FR4)
• Define a four-wavelength 400 Gbit/s PHY for operation over SMF with lengths up to at least 6 km (400GBASE-LR4-6) [32]

802.3cw project

• Provide a physical layer specification supporting 400 Gbit/s operation on a single wavelength capable of at least 80 km over a DWDM system (400GBASE-ZR) [33]

802.3db project

• Define a physical layer specification that supports 200 Gbit/s operation over 2 pairs of MMF with lengths up to at least 50 m
• Define a physical layer specification that supports 400 Gbit/s operation over 4 pairs of MMF with lengths up to at least 50 m [34]

200G port types

Name	Clause	Medium	Media count	Lanes	Gigabaud per lane	Reach
200GBASE-CR4	136 (802.3cd)	twinaxial copper cable	04		26.5625 (PAM4)	00.003 m
200GBASE-KR4	137 (802.3cd)	electrical backplane			26.5625 (PAM4)
200GBASE-SR4	138 (802.3cd)	multimode fibre 850 nm laser			26.5625 (PAM4)	OM3: 070 m OM4: 100 m
200GBASE-DR4	121 (802.3bs)	single-mode fibre WDM 1304.5−1317.5 nm			26.5625 (PAM4)	00.500 m
200GBASE-FR4	122 (802.3bs)	single-mode fibre WDM 1271−1331 nm	01	04	26.5625 (PAM4)	02,000 m
200GBASE-LR4	122 (802.3bs)	single-mode fibre WDM 1295−1309 nm[35]			26.5625 (PAM4)	10,000 m
200GBASE-ER4	122 (802.3cn)				26.5625 (PAM4)	40,000 m
200GAUI-8	120B/C (802.3bs)	Chip-to-module/ Chip-to-chip interface	08		26.5625 (NRZ)	000.25 m
200GAUI-4	120D/E (802.3bs)		04		26.5625 (PAM4)

400G port types

Name	Clause	Medium	Media count	Lanes	Gigabaud per lane	Reach
400GBASE-SR16	123 (802.3bs)	multimode fibre 850 nm laser	16		26.5625 (NRZ)	OM3: 070 m OM4: 100 m
400GBASE-SR8	138 (802.3cm)		08		26.5625 (PAM4)	OM5: 100 m
400GBASE-SR4.2	150 (802.3cm)		04	02	26.5625 (PAM4)	uses 4 pairs of MMF OM3: 070 m OM4: 100 m OM5: 150 m
400GBASE-DR4	124 (802.3bs)	single-mode fibre 1304.5−1317.5 nm	04		53.125 (PAM4)	00.500 m
400GBASE-FR8	122 (802.3bs)	single-mode fibre WDM 1273−1309 nm[36]	01	08	26.5625 (PAM4)	02,000 m
400GBASE-LR8	122 (802.3bs)				26.5625 (PAM4)	10,000 m
400GBASE-ER8	122 (802.3cn)				26.5625 (PAM4)	40,000 m
400GBASE-FR4	151 (802.3cu)	single-mode fibre WDM 1271−1331 nm	01	04	53.125 (PAM4)	02,000 m
400GBASE-LR4-6	151 (802.3cu)				53.125 (PAM4)	06,000 m
400GBASE-ZR	155/156 (802.3cw)	single-mode fibre	01	02	59.84375 (DP-16QAM)	80,000 m
400GAUI-16	120B/C (802.3bs)	Chip-to-module/ Chip-to-chip interface	16		26.5625 (NRZ)	000.25 m
400GAUI-8	120D/E (802.3bs)		08		26.5625 (PAM4)

See also

• Ethernet Alliance
• Interconnect bottleneck
• Optical fiber cable
• Optical communication
• Parallel optical interface

References

1. "IEEE 802.3 NGOATH SG Adopted Changes to 802.3bs Project Objectives" (PDF).
2. "Network boffins say Terabit Ethernet is TOO FAST: Sticking to 400Gb for now".
3. On-board optics: beyond pluggables
4. "[STDS-802-3-400G] IEEE P802.3bs Approved!". IEEE 802.3bs Task Force. Retrieved 2017-12-14.
5. "High-Speed Transmission Update: 200G/400G". 2016-07-18.
6. "The 2020 Ethernet Roadmap". Ethernet Alliance.
7. Jain, P. C. (2016). "Recent trends in next generation terabit Ethernet and gigabit wireless local area network". 2016 International Conference on Signal Processing and Communication (ICSC). IEEE. pp. 106–110. doi:10.1109/ICSPCom.2016.7980557. ISBN 978-1-5090-2684-5. S2CID 25506683.
8. "OIF Launches CEI-112G Project for 100G Serial Electrical Links". Businesswire. 30 Aug 2016.
9. Feldman, Michael (February 3, 2010). "Facebook Dreams of Terabit Ethernet". HPCwire. Tabor Communications, Inc.
10. Matsumoto, Craig (March 5, 2010). "Dare We Aim for Terabit Ethernet?". Light Reading. UBM TechWeb.
11. Craig Matsumoto (October 26, 2010). "The Terabit Ethernet Chase Begins". Light Reading. Retrieved December 15, 2011.
12. "Cisco 4 x 100 Gbit/s interface".
13. "Alcatel-Lucent boosts operator capacity to deliver big data and video over existing networks with launch of 400G IP router interface".
14. Smith, Ryan. "Micron Spills on GDDR6X: PAM4 Signaling For Higher Rates, Coming to NVIDIA's RTX 3090". www.anandtech.com.
15. Stephen Lawson (May 9, 2011). "IEEE Seeks Data on Ethernet Bandwidth Needs". PC World. Retrieved May 23, 2013.
16. "IEEE Industry Connections Ethernet Bandwidth Assessment" (PDF). IEEE 802.3 Ethernet Working Group. July 19, 2012. Retrieved 2015-03-01.
17. Max Burkhalter Brafton (May 12, 2011). "Terabit Ethernet could be on its way". Perle. Retrieved December 15, 2011.
18. "400 Gb/s Ethernet Study Group". Group web site. IEEE 802.3. Retrieved May 23, 2013.
19. IEEE 802.3bs Task Force
20. "Objectives" (PDF). IEEE 802.3bs Task Force. Mar 2014. Retrieved 2015-03-01.
21. "P802.3bs - IEEE Standard for Ethernet Amendment: Media Access Control Parameters, Physical Layers and Management Parameters for 200 Gb/s and 400 Gb/s Operation". IEEE 802.3bs Task Force. Retrieved 2017-12-14.
22. 100 m MMF draft proposal
23. "400GBase-SR16 draft specifications" (PDF).
24. IEEE 802.3 Ethernet Working Group Liaison letter to ITU-T Questions 6/15 and 11/15
25. 400G-PSM4: A Proposal for the 500 m Objective using 100 Gbit/s per Lane Signaling
26. 400Gb/s 8x50G PAM4 WDM 2km SMF PMD Baseline Specifications
27. Baseline Proposal for 8 x 50G NRZ for 400GbE 2 km and 10 km PMD
28. "400 GbE technical draft specifications" (PDF).
29. IEEE 802.3 NGOATH SG Adopted Changes to 802.3bs Project Objectives Updated by IEEE 802.3 NGOATH Study Group, Mar 16, 2016, IEEE 802 Mar 2016 Plenary, Macau, China.
30. IEEE 802.3bs 200/400 Gb/s Ethernet (Standards Informant)
31. http://www.ieee802.org/3/cn/proj_doc/3cn_Objectives_181113.pdf
32. https://www.ieee802.org/3/cu/Objectives_Approved_Sept_2019.pdf
33. http://www.ieee802.org/3/cw/proj_doc/3cw_Objectives_190911.pdf
34. http://www.ieee802.org/3/db/P802d3db_Objectives_Approved_May_2020.pdf
35. 1295.5595/1300.0540/1304.5799/1309.1374 nm (229.0/229.8/230.6/231.4 THz)
36. 1273.5449/1277.8877/1282.2603/1286.6629 + 1295.5595/1300.0540/1304.5799/1309.1374 nm (229.0/229.8/230.6/231.4 + 233.0/233.8/234.6/235.4 THz)

Further reading

• Chris Jablonski. "Researchers to develop 1 Terabit Ethernet by 2015". ZD Net. Retrieved October 9, 2011.
• Iljitsch van Beijnum (August 2011). "Speed matters: how Ethernet went from 3 Mbps to 100 Gbps... and beyond". Ars Technica. Retrieved October 9, 2011.
• Rick Merritt (May 9, 2011). "IEEE Looks beyond 100G Ethernet". The Cutting Edge. Retrieved October 9, 2011.
• Stephen Lawson (February 2, 2010). "Facebook Sees Need for Terabit Ethernet". PC World. Retrieved December 15, 2011.
• IEEE Reports
  • "100 gigabit Ethernet and beyond". IEEE Optical Communications : Design, Technologies, and Applications. March 2010. doi:10.1109/MCOM.2010.5434372. ISSN 0163-6804.
  • Elby, Stuart (July 2011). "The drive towards Terabit Ethernet". 2011 IEEE Photonics Society Summer Topical Meeting Series. pp. 104–105. doi:10.1109/PHOSST.2011.6000067. ISBN 978-1-4244-5730-4. S2CID 9077455.
  • Detwiler, Thomas; Stark, Andrew; Basch, Bert; Ralph, Stephen E. (July 2011). "DQPSK for Terabit Ethernet in the 1310 nm band". 2011 IEEE Photonics Society Summer Topical Meeting Series. pp. 143–144. doi:10.1109/PHOSST.2011.6000087. ISBN 978-1-4244-5730-4. S2CID 44199212.

External links

• West, John (April 3, 2009). "Terabit Ethernet on the way". insideHPC.
• Mellor, Chris (February 15, 2009). "Terabit Ethernet possibilities". The Register.
• Wang, Brian (April 24, 2008). "Terabit Ethernet around 2015".
• Duffy, Jim (April 20, 2009). "100 Gigabit Ethernet: Bridge to Terabit Ethernet". Network World. Archived from the original on May 14, 2010.
• Fleishman, Glenn (February 13, 2009). "Terabit Ethernet becomes a photonic possibility". Ars Technica. Condé Nast.