MC-LAG Configuration

30 min

\introduction\ mc lag (multi chassis link aggregation group) is a mechanism for achieving link aggregation across devices it involves aggregating links between one device and two other devices, combining the benefits of regular link aggregation while providing device level redundancy mc lag introduces a form of horizontal virtualization, where two physical devices are virtualized as a single logical device this virtual "single device" is used to perform "one to one" link aggregation with the connected upstream or downstream devices the following information is synchronized between mc lag peers system information ensures synchronization of mac addresses for mc lag member ports, enabling the "system id" field in lacp (link aggregation control protocol) messages sent to the server to be the same, achieving cross device link aggregation mc lag member port configuration information records details like names of local and remote mc lag member ports for consistency checks mc lag member port status information keeps track of the status of local and remote mc lag member ports to ensure isolation between the peer link and mc lag member ports in fault free scenarios, and to release isolation between the peer link and the same named port on the remote side in case of a failure in the remote mc lag member port arp (address resolution protocol) information arp entries related to mc lag member ports are synchronized between mc lag peers fdb (forwarding database) information fdb entries associated with mc lag member ports are synchronized between mc lag peers \explanation of principles\ as shown in the diagram, on two separate switches, a cross device link aggregation group is established and connected to ordinary link aggregation ports on the user side once the mc lag is established, entries can be synchronized between devices the direct link between switch1 and switch2 serves as the peer link interface, used for protocol messages and forwarding traffic during failures the links connecting the switches to users, acting as member interfaces of the mc lag, handle incoming traffic, distribute loads, and provide backup protection for the links in the context of mc lag, sonic employs a lightweight inter chassis communication protocol (iccp) on the control plane, conducting only a limited amount of consistency checks and information synchronization while ensuring the functionality iccp, defined in rfc7275, serves as the standard protocol for inter chassis communication in the mc lag implementation, the iccp protocol establishes connections between mc lag peer devices using tcp port 8888 this streamlined iccp protocol primarily focuses on configuring consistency checks and synchronizing arp and mac table entries between the two mc lag peer devices, the local ip and peer ip addresses are used as the source and destination addresses for establishing the tcp connection to form the iccp neighbor relationship once the iccp connection is successfully established, the system sends heartbeat messages to the peer every 1 second if no heartbeat messages are received for 15 consecutive intervals, the connection is deemed timed out, leading to the termination of the iccp connection \mc lag loop prevention mechanism\ mc lag itself comes with a loop prevention mechanism, as shown in the diagram when a device receives a broadcast packet from the mc lag side, this broadcast packet is transmitted through the peer link link to the opposite device due to the flow isolation between the peer link link and the mc lag member interfaces, any traffic coming in from the peer link port will not be forwarded out through the mc lag member interfaces this effectively prevents the formation of loops \mc lag fault handling\ as illustrated in the diagram, in the event of a link failure on the mc lag side, the device will proactively update the interface information for the corresponding terminal's mac table entries and arp table entries to reflect the peer link interface consequently, downstream traffic will be forwarded through the peer link interface to the other device, ensuring seamless fault handling for the user once the fault is rectified, and the mc lag interface is back up, traffic will resume its normal forwarding behavior \mc lag configuration\ \basic configuration\ operation command description enter the system configuration view configure terminal enter the link aggregation configuration view and create a link aggregation group interface link aggregation lag id only supported single domain the range of 1 to 4095 create a mc lag domain mclag domain domain id delete a mc lag domain no mclag domain domain id configure peerlink interface peer link {ethernet|link aggregation} name configure the peer ip address for the mc lag control link peer address a b c d set the local ip address for the mc lag control link local address a b c d add a member lag port to the mc lag group member lag lag id remove a member lag port from the mc lag group no member lag lag id enable the active standby mode of mc lag member ports member lag lag id active backup add mc lag member ports and configure them in active standby mode the device will automatically determine the active and standby interfaces on the device side based on the traffic conditions of the active and standby ports of the server disable the active standby mode of mc lag member ports no member lag lag id active backup enter the configuration view for a vlan interface interface vlan vlan id modify the mac address of a vlan interface mac address hh\ hh\ hh\ hh\ hh \ hh to make two mc lag switches appear as the same logical gateway to the outside, it is necessary to manually configure the mac addresses of the service gateways on both ends to be consistent this avoids arp conflicts and abnormal traffic forwarding caused by downstream devices learning different mac addresses in actual deployment, the mode of mc lag member ports must correspond to the binding mode of the server's network card when the server adopts the active standby mode (bond mode 1), the active standby mode of the mc lag member ports must be enabled when the server adopts the load balancing mode (such as bond mode 0, 2), the member lag ports should be configured as static aggregation (default) when the server adopts dynamic link aggregation (bond mode 4), the member lag ports need to be configured as dynamic lacp aggregation \dad link configuration (optional)\ the dad link (dual active detection link) is a layer 3 interworking link used for sending dual active detection messages between mc lag peers when a peer link failure is detected, the interface status of the standby device negotiated by mc lag, except for the management and peer link interfaces, is set to error down this ensures that only the active device forwards traffic and prevents network conflicts it is recommended to first complete the mc lag configuration and use the `show mclag state` command to confirm that 【the mclag’s keepalive is ok】 before configuring the dad link, to prevent the device port from being mistakenly error down operation command description enter the system configuration view configure terminal enter the mc lag domain mclagdomain domain id enter the created mc lag domain configure the source ip address of dad heartbeat packets dad local address a b c d it is recommended to use the management port ip address configure the destination ip address for dad heartbeat packets dad peer address a b c d it is necessary to ensure that the configured local address and peer address are reachable at the third layer configure the error down delay time dad detection delay time time value range 0 255 (default 60), unit second when the peer link fails but dad detection is normal, error down will be triggered on the standby device after the configured delay time to prevent session flapping configure the error down recovery delay time for mc lag member lags dad recovery delay mlag time time value range 0 255 (default 60), unit second when the peer link fault is recovered, the error down state of the mc lag member lag will be restored after the configured delay time to prevent session flapping configure the error down recovery delay time for mc lag non member lag dad recovery delay non mlag time time value range 0 255 (default 0), unit second when the peer link fault is recovered, the error down state of the lag that is not an mc lag member will be restored after the configured delay time to prevent session flapping specify the vrf to which the dad message belongs dad vrf vrf name when using the management network port as the source for sending dad packets and the management network port is designated as mgmt vrf, this command needs to be used to specify the source of dad as mgmt vrf \display and maintenance\ operation command description check the relevant information about mc lag show mclag state check the error down status of the interface show interface errdown check the interface that was forced to shut down due to dad disconnection enable the m lag configuration consistency check function and specify the check mode consistency check action {idle|default|graceful} idle only check for consistency without taking any action default check for consistency and shut down the member interfaces that are inconsistent graceful check for consistency, shut down the member ports of devices with the standby role, and keep the active devices in the up state the default state is idle check the results of member consistency checks show mclag consistency check result pass configuration is consistent failed configuration is inconsistent \configuration example\ \network requirements\ a server is dual homed to an ethernet network through m lag the customer requires high service reliability link aggregation between the server and devices only achieves link level reliability, and a fault on a device may cause service interruption m lag can be configured when devices work properly, links load balance traffic and a fault of any device does not affect services high service reliability is therefore ensured \procedure\ create an aggregation port and add the physical interfaces connected to server and switch c to the aggregation port group \# switch a sonic(config)# interface link aggregation 1 sonic(config)# interface ethernet 1 sonic(config if 1)# link aggregation group 1 \# switch b sonic(config)# interface link aggregation 1 sonic(config)# interface ethernet 1 sonic(config if 1)# link aggregation group 1 create an aggregation port and add the interconnection port between switch a and switch b to the aggregation group as the peerlink interface \# switch a sonic(config)# interface link aggregation 2 sonic(config)# interface ethernet 2 sonic(config if 2)# link aggregation group 2 sonic(config)# interface ethernet 3 sonic(config if 3)# link aggregation group 2 \# switch b sonic(config)# interface link aggregation 2 sonic(config)# interface ethernet 2 sonic(config if 2)# link aggregation group 2 sonic(config)# interface ethernet 3 sonic(config if 3)# link aggregation group 2 create vlan100 to add all aggregation ports to vlan \# switch a sonic(config)# vlan 100 sonic(config)# interface link aggregation 1 sonic(config lagif 1)# switchport access vlan 100 sonic(config)# interface link aggregation 2 sonic(config lagif 2)# switchport access vlan 100 sonic(config)# interface vlan 100 sonic(config vlanif 100)# ip address 101 1 1 1/24 sonic(config vlanif 100)# mac address 00 00 00 11 22 10 \# switch b sonic(config)# vlan 100 sonic(config)# interface link aggregation 1 sonic(config lagif 1)# switchport access vlan 100 sonic(config)# interface link aggregation 2 sonic(config lagif 2)# switchport access vlan 100 sonic(config)# interface vlan 100 sonic(config vlanif 100)# ip address 101 1 1 1/24 sonic(config vlanif 100)# mac address 00 00 00 11 22 10 create vlan200 and configure the ip address to add the peerlink interface to the vlan \# switch a sonic(config)# vlan 200 sonic(config)# interface link aggregation 2 sonic(config lagif 2)# switchport trunk vlan 200 sonic(config lagif 2)# switchport trunk vlan 100 sonic(config)# interface vlan 200 sonic(config vlanif 200)# ip address 10 1 1 24/24 \# switch b sonic(config)# vlan 200 sonic(config)# interface link aggregation 2 sonic(config lagif 3)# switchport trunk vlan 200 sonic(config lagif 2)# switchport trunk vlan 100 sonic(config)# interface vlan 200 sonic(config vlanif 200)# ip address 10 1 1 48/24 create an mc lag and specify the member interface, peerlink interface and peerlink port ip address \# switch a sonic(config)# mclag domain 1 sonic(mclag domain)# local address 10 1 1 24 sonic(mclag domain)# peer address 10 1 1 48 sonic(mclag domain)# peer link lag 2 sonic(mclag domain)# member lag 1 \# switch b sonic(config)# mclag domain 1 sonic(mclag domain)# local address 10 1 1 48 sonic(mclag domain)# peer address 10 1 1 24 sonic(mclag domain)# peer link lag 2 sonic(mclag domain)# member lag 1 \verify configuration\ check mc lag status sonic# show mclag state the mclag's keepalive is ok mclag info sync is completed domain id 1 mclag session channel primary channel vrf name default consistency check action idle local ip 10 1 1 48 peer ip 10 1 1 24 peer link interface portchannel0003 keepalive time 1 sesssion timeout 15 peer link mac 00 11 24 1a 1b 61 admin role none role standby mclag interface portchannel0001 loglevel debug the server can ping the gateway and check the mac address of the device \# use the server ping gateway address testuser\@78 $ping 101 1 1 1 ping 101 1 1 1 (101 1 1 1) 56(84) bytes of data 64 bytes from 101 1 1 1 icmp seq=1 ttl=64 time=3 38 ms 64 bytes from 101 1 1 1 icmp seq=2 ttl=64 time=1 33 ms 64 bytes from 101 1 1 1 icmp seq=3 ttl=64 time=1 32 ms 64 bytes from 101 1 1 1 icmp seq=4 ttl=64 time=1 29 ms ^c \ 101 1 1 1 ping statistics 4 packets transmitted, 4 received, 0% packet loss, time 3005ms rtt min/avg/max/mdev = 1 291/1 835/3 389/0 897 ms # display the mac address on switch b device sonic# show mac address no vlan macaddress port type \ 1 101 00 51 82 11 0f 78 portchannel0001 dynamic cut off the link between switchb and switchc, use the server ping gateway again, and still be able to ping, and check the mac address of the device at this time \# use the server ping gateway address testuser\@78 $ping 101 1 1 1 ping 101 1 1 1 (101 1 1 1) 56(84) bytes of data 64 bytes from 101 1 1 1 icmp seq=1 ttl=64 time=3 38 ms 64 bytes from 101 1 1 1 icmp seq=2 ttl=64 time=1 33 ms 64 bytes from 101 1 1 1 icmp seq=3 ttl=64 time=1 32 ms 64 bytes from 101 1 1 1 icmp seq=4 ttl=64 time=1 29 ms ^c \ 101 1 1 1 ping statistics 4 packets transmitted, 4 received, 0% packet loss, time 3005ms rtt min/avg/max/mdev = 1 291/1 835/3 389/0 897 ms \# display the mac address on your switch b device sonic# show mac address no vlan macaddress port type \ 1 101 00 51 82 11 0f 78 portchannel0002 dynamic \mc lag+stp configuration example\ m lag itself has a loop avoidance mechanism under normal circumstances, no loops will be generated in an m lag network in a multi level m lag network, loops may occur in the network when there are errors in network setup, when initializing m lag configurations, or when devices restart with empty configurations stp can be deployed to avoid loops \networking requirements \ the switch adopts cross device link aggregation technology to ensure high reliability of services at the device level during normal operation, the links perform load sharing, and when the stp configuration is enabled, the links can successfully break loops, ensuring that data packet forwarding is not affected \configuration steps \ 1\ create an aggregation port and add the physical interfaces connected to the switch to the aggregation port group respectively \#switch a sonic(config)# interface link aggregation 1 sonic(config)# interface ethernet 20 sonic(config if 20)# link aggregation group 1 sonic(config if 20)# exit sonic(config)# interface ethernet 35 sonic(config if 35)# link aggregation group 1 sonic(config if 35)# exit \#switch b sonic(config)# interface link aggregation 1 sonic(config)# interface link aggregation 2 sonic(config)# interface link aggregation 3 sonic(config)# interface ethernet 14 sonic(config if 14)# link aggregation group 1 sonic(config if 14)# exit sonic(config)# interface ethernet 11 sonic(config if 11)# link aggregation group 2 sonic(config if 11)# exit sonic(config)# interface ethernet 12 sonic(config if 12)# link aggregation group 3 sonic(config if 12)# exit \#switch c sonic(config)# interface link aggregation 1 sonic(config)# interface link aggregation 2 sonic(config)# interface link aggregation 3 sonic(config)# interface ethernet 35 sonic(config if 35)# link aggregation group 1 sonic(config if 35)# exit sonic(config)# interface ethernet 21 sonic(config if 21)# link aggregation group 2 sonic(config if 21)# exit sonic(config)# interface ethernet 49 sonic(config if 49)# link aggregation group 3 sonic(config if 49)# exit \#switch d sonic(config)# interface link aggregation 3 sonic(config)# interface ethernet 12 sonic(config if 12)# link aggregation group 3 sonic(config if 12)# exit sonic(config)# interface ethernet 49 sonic(config if 49)# link aggregation group 3 sonic(config if 49)# exit 2\ create a vlan interface and add physical interfaces and aggregated interface groups to the vlan \#switch a sonic(config)# vlan 10 sonic(config)# interface link aggregation 1 sonic(config lagif 1)# switchport trunk vlan 10 sonic(config lagif 1)# exit sonic(config)# interface ethernet 37 sonic(config if 37)# switchport trunk vlan 10 sonic(config if 37)# exit sonic(config)# interface ethernet 10 sonic(config if 10)# switchport trunk vlan 10 sonic(config if 10)# exit \#switch b sonic(config)# vlan 10 sonic(config)# vlan 400 sonic(config)# interface link aggregation 1 sonic(config lagif 1)# switchport trunk vlan 10 sonic(config lagif 1)# exit sonic(config)# interface link aggregation 2 sonic(config lagif 2)# switchport trunk vlan 400 sonic(config lagif 2)# switchport trunk vlan 10 sonic(config lagif 2)# stp bpdu filter enable sonic(config lagif 2)# exit sonic(config)# interface link aggregation 3 sonic(config lagif 3)# switchport trunk vlan 10 sonic(config lagif 3)# exit \#switch c sonic(config)# vlan 10 sonic(config)# vlan 400 sonic(config)# interface link aggregation 1 sonic(config lagif 1)# switchport trunk vlan 10 sonic(config lagif 1)# exit sonic(config)# interface link aggregation 2 sonic(config lagif 2)# switchport trunk vlan 400 sonic(config lagif 2)# switchport trunk vlan 10 sonic(config lagif 2)# stp bpdu filter enable sonic(config lagif 2)# exit sonic(config)# interface link aggregation 3 sonic(config lagif 3)# switchport trunk vlan 10 sonic(config lagif 3)# exit \#switch d sonic(config)# vlan 10 sonic(config)# interface link aggregation 3 sonic(config lagif 3)# switchport trunk vlan 10 sonic(config lagif 3)# exit sonic(config)# interface ethernet 38 sonic(config if 38)# switchport trunk vlan 10 sonic(config if 38)# exit sonic(config)# interface ethernet 3 sonic(config if 3)# switchport trunk vlan 10 sonic(config if 3)# exit 3\ create mc lag and configure bridge mac \#switch b sonic(config)# bridge mac 00 00 04 12 05 01 sonic(config)# interface vlan 400 sonic(config vlanif 400)# ip address 50 1 1 2/24 sonic(config vlanif 400)# exit sonic(config)# mclag domain 1 sonic(mclag domain)# peer address 50 1 1 1 sonic(mclag domain)# local address 50 1 1 2 sonic(mclag domain)# peer link link aggregation 2 sonic(mclag domain)# member lag 1 sonic(mclag domain)# member lag 3 sonic(mclag domain)# commit sonic(mclag domain)# exit \#switch c sonic(config)# bridge mac 00 00 04 12 05 01 sonic(config)# interface vlan 400 sonic(config vlanif 400)# ip address 50 1 1 1/24 sonic(config vlanif 400)# exit sonic(config)# mclag domain 1 sonic(mclag domain)# peer address 50 1 1 2 sonic(mclag domain)# local address 50 1 1 1 sonic(mclag domain)# peer link link aggregation 2 sonic(mclag domain)# member lag 1 sonic(mclag domain)# member lag 3 sonic(mclag domain)# commit sonic(mclag domain)# exit 4\ enable mstp on the switch and check the stp loop breaking status of the entire device \#switch a sonic(config)# stp enable mstp sonic(config)# stp instance 1 sonic(config stp 1)# exit sonic(config)# stp bind vlan 10 1 sonic(config)# stp name test \#switch b sonic(config)# stp enable mstp sonic(config)# stp instance 1 sonic(config stp 1)# exit sonic(config)# stp bind vlan 10 1 sonic(config)# stp name test \#switch c sonic(config)# stp enable mstp sonic(config)# stp instance 1 sonic(config stp 1)# exit sonic(config)# stp bind vlan 10 1 sonic(config)# stp name test \#switch d sonic(config)# stp enable mstp sonic(config)# stp instance 1 sonic(config stp 1)# exit sonic(config)# stp bind vlan 10 1 sonic(config)# stp name test \verification \ 1\ check the mc lag status sonic# show mclag state the mclag's keepalive is ok mclag info sync is completed domain id 1 mclag session channel primary channel vrf name default consistency check action idle local ip 50 1 1 1 peer ip 50 1 1 2 peer link interface portchannel0002 keepalive time 1 sesssion timeout 15 peer link mac 00 11 24 1a 1b 61 admin role none role standby mclag interface portchannel0001, portchannel0003 loglevel debug 2\ check the stp status on switch d sonic# show stp status role flags root root, desg designated, altn alternate, back backup, mstr master, disa disabled state flags disc discard/blocking/listening, lear learning, forw forwarding spanning tree mode mstp mst instance port port role stp state \ 1 ethernet3 desg forw 1 portchannel0003 root forw 1 ethernet38 altn disc