An aggregation is similar to an IPMP link created for failover, but has other advantages:
- An aggregation is treated as one device for configuration and management, and uses only one IP address.
- In-bound as well as outbound load spreading is provided, so extra bandwidth is available.
- Automatic failover is provided, to the remaining working links in the aggregation.
- Policies can be applied to the traffic flows.
- Parallel aggregations between servers can be created, without going through a switch.
Early versions of Solaris 10 do not have this facility – you would need to download the Sun Trunking software to get it.
### Solaris 10 configuration:
To show the available network interfaces:
[root @ soltest]~ # dladm show-link LINK CLASS MTU STATE OVER eri0 phys 1500 up -- hme0 phys 1500 up -- qfe0 phys 1500 down --
There are conditions to consider:
- If you need to run an aggregation through a network switch, that switch must support aggregation technology.
- An interface that has been plumbed cannot become a member of an aggregation.
- Currently, interfaces must be of the GLDv3 type: xge, e1000g, and bge.
- NOTE: Later releases of Solaris 10 will support aggregations on what are now considered legacy devices such as eri. qfe and hme.
- All interfaces in the aggregation must run at the same speed and in full duplex mode.
To set up an aggregation using dladm, the basics steps are:
[root @ soltest]~ # dladm create -aggr -d bge0 -d bge1 1
Create an aggregation number 1 from interfaces bge0 and bge1.
[root @ soltest]~ # ifconfig aggr1 plumb 192.168.200.98 up
Brings the aggregation interface up.
[root @ soltest]~ # dladm show-aggr
key: 1 (0x0001) policy: L4 address: 0:3:ba:7:84:5e (auto)
device address speed duplex link state
bge0 0:3:ba:7:84:5e 1000 Mbps full up attached
bge1 0:3:ba:7:84:5e 1000 Mbps full up attached
To add another interface to the aggregation:
[root @ soltest]~ # dladm add-aggr -d bge2 -d bge3 1
[root @ soltest]~ # dladm show-aggr 1
key: 1 (0x0001) policy: L4 address:0:3:ba:7:84:5e (auto)
device address speed duplex link state
bge0 0:14:4f:1:c8:b0 1000 Mbps full up attached
bge1 0:14:4f:1:c8:b3 1000 Mbps full up attached
bge2 0:14:4f:1:c8:b1 1000 Mbps full up attached
bge3 0:14:4f:1:c8:b2 1000 Mbps full up attached
To remove a device from the aggregation:
[root @ soltest]~ # dladm remove-aggr -d bge0 1
[root @ soltest]~ # dladm show-aggr 1
key: 1 (0x0001) policy: L4 address: 0:3:ba:7:84:5e (auto)
device address speed duplex link state
bge1 0:14:4f:1:c8:b3 1000 Mbps full up attached
bge2 0:14:4f:1:c8:b1 1000 Mbps full up attached
bge3 0:14:4f:1:c8:b2 1000 Mbps full up attached
It is possible to set policies for aggregations, such as whether to use LACP protocol or not.
Here is an example which sets LACP mode to active, and the timer interval to short.
[root @ soltest]~ # dladm modify-aggr -l active -t short 1
The default outbound load spreading is based on ports, but this can also be changed - see the dladm man page for full details.
Aggregations can be assigned to zones using IP instances (Update 4 onwards) - see zonecfg for details.
dladm can assign an aggregation temporarily to a zone, until the next reboot:
[root @ soltest]~ # dladm set-linkprop -t -p zone=apache aggr1
The -t option must be used.
Use zonecfg to assign aggr1 permanently to the apache zone.
dladm also provides functions to display statistics, and not just for aggregations:-
[root @ soltest]~ # dladm show-dev -s eri0
ipackets rbytes ierrors opackets obytes oerrors
eri0 806693 122733634 0 808629 336565292 0
[root @ soltest]~ # dladm show-aggr -s
key: 1 ipackets rbytes opackets obytes %ipkts %opkts
Total 158030 31173663 12 504
qfe1 132878 26014941 12 504 84.1 100.0
qfe2 8818 1818890 0 0 5.6 0.0
qfe3 8174 1671697 0 0 5.2 0.0
qfe0 8160 1668135 0 0 5.2 0.0
### Solaris 11 configuration:
There are two types of link aggregation supported in Solaris 11:
- Trunk (802.3ad) aggregations, which provide load spreading at MAC/IP/PORT level, but need 802.3ad network switch support.
- DLMP (Data Link MultiPathing) where the network links can be connected to separate switches. DLMP load spreading is at link level, and is intended primarily as the underlying link for VNICs. (Virtual Network Interfaces) These are used extensively with virtual machines calless zones.
To show the available network interfaces:
[root @ soltest]~ # dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE net1 Ethernet unknown 0 unknown e1000g1 net2 Ethernet unknown 0 unknown e1000g2 net0 Ethernet up 1000 full e1000g0 net3 Ethernet unknown 0 unknown e1000g3 [root @ soltest]~ # dladm show-link LINK CLASS MTU STATE OVER net1 phys 1500 unknown -- net2 phys 1500 unknown -- net0 phys 1500 up -- net3 phys 1500 unknown --
Firstly, an 802.3ad compliant aggregation
There are conditions to consider:
If you need to run a trunk aggregation through a switch, that switch must support 802.3ad aggregation technology.
Interfaces must be of the GLDv3 type: xge, e1000g, and bge.
It is unlikely that other types would be supported in Solaris 11 in any case.
All interfaces in the aggregation must run at the same speed and in full duplex mode.
No IP instances must initially exist on the component interfaces.
To set up an aggregation using dladm, the basics steps are:
[root @ soltest]~ # dladm create-aggr -L active -l net1 -l net2 1
Create an aggregation number 1 (aggr1) from interfaces net1 and net2.
-L active enables active LACP.
[root @ soltest]~ # ipadm create-ip aggr1
[root @ soltest]~ # ipadm create-addr -T static -a 130.100.1.5/16 aggr1/v4
Brings the aggregation interface aggr1 up with address 130.100.1.5
[root @ soltest]~ # dladm show-aggr
LINK MODE POLICY ADDRPOLICY LACPACTIVITY LACPTIMER
aggr1 trunk L4 auto off short
Use dlstat to display link statistics.
To add another interface to the aggregation:
[root @ soltest]~ # dladm add-aggr -l net3 aggr1
[root @ soltest]~ # dladm show-aggr -x
LINK PORT SPEED DUPLEX STATE ADDRESS PORTSTATE
aggr1 -- 1000Mb full up 0:14:4f:97:7d:f --
net1 1000Mb full up 0:14:4f:97:7d:f attached
net2 1000Mb full up 0:14:4f:97:7d:10 attached
net3 1000Mb full up 0:14:4f:97:7d:11 attached
[root @ soltest]~ # dlstat show-aggr aggr1
LINK PORT IPKTS RBYTES OPKTS OBYTES
aggr1 -- 5.34M 435.78M 1.48M 190.05M
aggr1 net1 2.02M 159.87M 570.29K 73.00M
aggr1 net2 2.13M 181.61M 570.09K 72.97M
aggr1 net3 1.19M 94.31M 344.38K 44.08M
To remove a device from the aggregation:
[root @ soltest]~ # dladm remove-aggr -l net0 1
To remove aggr1:
[root @ soltest]~ # dladm delete-aggr 1
or..
[root @ soltest]~ # dladm delete-aggr aggr1
Secondly, a DLMP aggregation
To create a DLMP-mode aggregation:
[root @ soltest]~ # dladm create-aggr -m dlmp -l net1 -l net2 -l net3 aggr1
-m dlmp denotes that this is a DLMP-mode aggregation.
The links that form part of the aggregation can be connected to different switches which do not need 802.3ad support, thus eliminating the switch as a point of failure, and the need for any switch configuration.
As VNICs are created they are associated with an aggregation port, and if the port fails, they will be switched to another (working) port.
If the number of VNICs exceeds the number of aggregation ports, then the ports are shared between them.
VNICs to be connected to the aggr1 interface are created as follows:
[root @ soltest]~ # dladm create-vnic -l aggr1 vnic0