Most network switches come pre-loaded with the manufactures Operating System. Cisco, Juniper and HP ship always their hardware with OS bundled together. Over the last couple of years, more and more white box switching and routing gear found its way into the enterprise sector. With software-defined-networks becoming more and more popular, white box switches become more relevant and many modern datacenters like the idea of a Linux based OS for their switches. It makes the administrative process easier, since Linux/Unix skills are widely common in this day and age.
Cumulus Networks makes use of white box switches and develops their own OS, called Cumulus OS. Cumulus OS is Debian based and provides you the common Linux commands plus some additional OS specific commands. Currently Cumulus OS supports only a couple of switches ranging from 1GbE to 10GbE and all the way to 40GbE. Cumulus Networks does not sell the actual switches, instead it only sells their software which is available as a yearly subscription.
I got my hands on a Quanta T3048-LY2R. The T3048-LY2R has 48x 10 GbE SFP+ and 4x 40 GbE QSFP+ ports. Each 40 GbE port can be configured for 40 GbE or 4x 10GbE ports which would offer up to 64x 10GbE SFP+ ports. Further details can be found here. The white box switch together with a Cumulus OS license cost you roughly $6000. Price-wise this switch cannot be beaten by any of the big brands out there.
The T3048-LY2R gets shipped with ONIE installed. Open Network Install Environment combines a bootloader with a modern Linux kernel and BusyBox. ONIE provides a platform which allows you to install any network OS on a bare metal switch. Once you power on the switch, you’ll automatically be dropped into the ONIE:/ # shell.
I used a USB stick to install Cumulus OS on my T3048-LY2R:
- Download the latest Cumulus OS to your local machine
- Rename the Cumulus OS image to onie-installer and place it on a usb drive which has been formatted with VFAT.
- Plug-in the USB stick.TheONIE auto-discovery should automatically detect the USB stick and find the image.
- Ifauto-discoveryhas been disabled, you can re-enable it by running
onie# /etc/init.d/discover.sh start
- Ifauto-discoveryhas been disabled, you can re-enable it by running
- Once the auto install starts, you should see the installation running through several steps
ONIE:/ # tiaONIE: Executing installer: file://dev/sda1/onie-installer Verifying image checksum ...l /bin/sh: tial: not found ONIE:/ # OK. Preparing image archive ... OK. Dumping image info... Control File Contents ===================== Description: Cumulus Linux OS-Release: 2.5.1-5227ded-201503051128-build Architecture: powerpc Date: Thu, 05 Mar 2015 11:55:31 -0800 Installer-Version: 1.2 Platforms: accton_as4600_54t, accton_as6701_32x, accton_5652, accton_as5610_52x, dni_6448, dni_7448, dni_c7448n, cel_kennisis, cel_redstone, cel_smallstone, cumulus_p2020, quanta_lb9, quanta_ly2, quanta_ly2r, quanta_ly6_p2020 Homepage: http://www.cumulusnetworks.com/ Data Archive Contents ===================== -rw-r--r-- build/Development 96092160 2015-03-05 19:55:30 sysroot.squash.xz -rw-r--r-- build/Development 44 2015-03-05 19:55:31 sysroot.squash.xz.sha1 -rw-r--r-- build/Development 1824 2015-03-05 19:55:31 uboot_env.tar.xz -rw-r--r-- build/Development 44 2015-03-05 19:55:31 uboot_env.tar.xz.sha1 -rw-r--r-- build/Development 4186672 2015-03-05 19:55:30 uImage-powerpc.itb -rw-r--r-- build/Development 44 2015-03-05 19:55:31 uImage-powerpc.itb.sha1 Partitioning /dev/mmcblk0... done. Formatting /dev/mmcblk0p1... done. Formatting /dev/mmcblk0p3... done. Validating sha1 for uImage-powerpc.itb... done. Validating sha1 for sysroot.squash.xz... done. Installing OS-Release 2.5.1-5227ded-201503051128-build into image slot 1 ... Copying sysroot into /dev/mmcblk0p5... done. Verifying sysroot copy... OK. Verifying sysroot copy... OK. Erasing 128 Kibyte @ 800000 - 100% complete. Validating sha1 for uboot_env.tar.xz... done. Updating u-boot environment variables... done. Updating u-boot environment variables. Success: System provisioning complete. Rebooting... umount: can't umount /tmp.iYvd9w: Invalid argument umount: can't remount rootfs read-only The system is going down NOW! Sent SIGTERM to all processes Sent SIGKILL toRestarting system. U-Boot 2010.12 (Oct 15 2013 - 13:54:14) CPU0: P2020, Version: 2.1, (0x80e20021) Core: E500, Version: 5.1, (0x80211051) Clock Configuration: CPU0:1200 MHz, CPU1:1200 MHz, CCB:600 MHz, DDR:400 MHz (800 MT/s data rate) (Asynchronous), LBC:37.500 MHz L1: D-cache 32 kB enabled I-cache 32 kB enabled Board: Quanta LY2R P2020, Rev A I2C: ready DRAM: Detected UDIMM TS256MSK64V3N Detected UDIMM(s) DDR: 2 GiB (DDR3, 64-bit, CL=6, ECC off) SD/MMC : 4-bit Mode FLASH: 128 MiB L2: 512 KB enabled MMC: FSL_ESDHC: 0 LY2R U-Boot Product Name : LY2R Model Name : QUANTA LY2R Serial Number : SN Part Number : 1LY2BZZ000X Label Revision Number : 1 Hardware Version : 1.0 Platform Version : 0xf2f1 Release Date : 2014/4/12 MAC Address : c4:54:44:61:ea:b4 Set ethaddr MAC address = c4:54:44:61:ea:b4 PCIe1: Root Complex of Slot 1, x2, regs @ 0xffe0a000 01:00.0 - 14e4:b846 - Network controller PCIe1: Bus 00 - 01 In: serial Out: serial Err: serial Net: eTSEC1: PHY is Broadcom BCM5461S (2060c1) eTSEC1 Hit any key to stop autoboot: 0 ** local boot ** Booting primary image...
- After the primary imagehas been loaded successfully, youshould be dropped into the Cumulus OS login prompt.
- The default username is cumulus and the password is CumulusLinux!
- You’re all set. Enjoy your new switch with the Cumulus OS!