Target OS: Solaris 11 and later. Some of these commands may work on earlier versions too.
-1-
Interrupt Statistics : intrstat utility
intrstat utility can be used to monitor interrupt activity generated by various hardware devices along with the CPU that serviced the interrupt and the CPU time spent in servicing those interrupts on a system. On a busy system, intrstat reported stats may help figure out which devices are busy at work, and keeping the system busy with interrupts.
.. [idle system] showing the interrupt activity on first two vCPUs ..
# intrstat -c 0-1 5
device | cpu0 %tim cpu1 %tim
-------------+------------------------------
cnex#0 | 0 0.0 0 0.0
ehci#0 | 0 0.0 0 0.0
hermon#0 | 0 0.0 0 0.0
hermon#1 | 0 0.0 0 0.0
hermon#2 | 0 0.0 0 0.0
hermon#3 | 0 0.0 0 0.0
igb#0 | 0 0.0 0 0.0
ixgbe#0 | 0 0.0 0 0.0
mpt_sas#0 | 18 0.0 0 0.0
vldc#0 | 0 0.0 0 0.0
device | cpu0 %tim cpu1 %tim
-------------+------------------------------
cnex#0 | 0 0.0 0 0.0
ehci#0 | 0 0.0 0 0.0
hermon#0 | 0 0.0 0 0.0
hermon#1 | 0 0.0 0 0.0
hermon#2 | 0 0.0 0 0.0
hermon#3 | 0 0.0 0 0.0
igb#0 | 0 0.0 0 0.0
ixgbe#0 | 0 0.0 0 0.0
mpt_sas#0 | 53 0.2 0 0.0
vldc#0 | 0 0.0 0 0.0
^C
Check the outputs of the following as well.
# echo ::interrupts | mdb -k
# echo ::interrupts -d | mdb -k
-2-
Physical Location of Disk : croinfo & diskinfo commands
Both croinfo and diskinfo commands provide information about the chassis, receptacle, and occupant relative to all disks or a specific disk. Note that croinfo and diskinfo utilities share the same executable binary and function in a identical manner. The main difference being the defaults used by each of the utilities.
# croinfo D:devchassis-path t:occupant-type c:occupant-compdev ------------------------------ --------------- --------------------- /dev/chassis//SYS/MB/HDD0/disk disk c0t5000CCA0125411FCd0 /dev/chassis//SYS/MB/HDD1/disk disk c0t5000CCA0125341F0d0 /dev/chassis//SYS/MB/HDD2 - - /dev/chassis//SYS/MB/HDD3 - - /dev/chassis//SYS/MB/HDD4/disk disk c0t5000CCA012541218d0 /dev/chassis//SYS/MB/HDD5/disk disk c0t5000CCA01248F0B8d0 /dev/chassis//SYS/MB/HDD6/disk disk c0t500151795956778Ed0 /dev/chassis//SYS/MB/HDD7/disk disk c0t5001517959567690d0 # diskinfo -oDcpd D:devchassis-path c:occupant-compdev p:occupant-paths d:occupant-devices ------------------------------ --------------------- ----------------------------------------------------------------------------- ----------------------------------------- /dev/chassis//SYS/MB/HDD0/disk c0t5000CCA0125411FCd0 /devices/pci@400/pci@1/pci@0/pci@0/LSI,sas@0/iport@1/disk@w5000cca0125411fd,0 /devices/scsi_vhci/disk@g5000cca0125411fc /dev/chassis//SYS/MB/HDD1/disk c0t5000CCA0125341F0d0 /devices/pci@400/pci@1/pci@0/pci@0/LSI,sas@0/iport@2/disk@w5000cca0125341f1,0 /devices/scsi_vhci/disk@g5000cca0125341f0 /dev/chassis//SYS/MB/HDD2 - - - /dev/chassis//SYS/MB/HDD3 - - - /dev/chassis//SYS/MB/HDD4/disk c0t5000CCA012541218d0 /devices/pci@700/pci@1/pci@0/pci@0/LSI,sas@0/iport@1/disk@w5000cca012541219,0 /devices/scsi_vhci/disk@g5000cca012541218 /dev/chassis//SYS/MB/HDD5/disk c0t5000CCA01248F0B8d0 /devices/pci@700/pci@1/pci@0/pci@0/LSI,sas@0/iport@2/disk@w5000cca01248f0b9,0 /devices/scsi_vhci/disk@g5000cca01248f0b8 /dev/chassis//SYS/MB/HDD6/disk c0t500151795956778Ed0 /devices/pci@700/pci@1/pci@0/pci@0/LSI,sas@0/iport@4/disk@w500151795956778e,0 /devices/scsi_vhci/disk@g500151795956778e /dev/chassis//SYS/MB/HDD7/disk c0t5001517959567690d0 /devices/pci@700/pci@1/pci@0/pci@0/LSI,sas@0/iport@8/disk@w5001517959567690,0 /devices/scsi_vhci/disk@g5001517959567690
-3-
Monitoring Network Traffic Statistics : dlstat command
dlstat command reports network traffic statistics for all datalinks or a specific datalink on a system.
# dlstat -i 5 net0
LINK IPKTS RBYTES OPKTS OBYTES
net0 163.12M 39.93G 206.14M 43.63G
net0 312 196.59K 146 370.80K
net0 198 172.18K 121 121.98K
net0 168 91.23K 93 195.57K
^C
For the complete list of options along with examples, please consult the Solaris Documentation.
-4-
Fault Management : fmstat utility
Solaris Fault Manager gathers and diagnoses problems detected by the system software, and initiates self-healing activities such as disabling faulty components. fmstat utility can be used to check the statistics associated with the Fault Manager.
fmadm config lists out all active fault management modules that are currently participating in fault management. -m option can be used to report the diagnostic statistics related to a specific fault management module. fmstat without any option report stats from all fault management modules.
# fmstat 5
module ev_recv ev_acpt wait svc_t %w %b open solve memsz bufsz
cpumem-retire 0 0 1.0 8922.5 96 0 0 0 12b 0
disk-diagnosis 1342 0 1.1 8526.0 96 0 0 0 0 0
disk-transport 0 0 1.0 8600.3 96 1 0 0 56b 0
...
...
zfs-diagnosis 139 75 1.0 8864.5 96 0 4 12 672b 608b
zfs-retire 608 0 0.0 15.2 0 0 0 0 4b 0
...
...
# fmstat -m cpumem-retire 5
NAME VALUE DESCRIPTION
auto_flts 0 auto-close faults received
bad_flts 0 invalid fault events received
cacheline_fails 0 cacheline faults unresolveable
cacheline_flts 0 cacheline faults resolved
cacheline_nonent 0 non-existent retires
cacheline_repairs 0 cacheline faults repaired
cacheline_supp 0 cacheline offlines suppressed
...
...
-5-
InfiniBand devices : List & Show Information about each device
ibv_devices lists out all available IB devices whereas ibv_devinfo shows information about all devices or a specific IB device.
# ibv_devices
device node GUID
------ ----------------
mlx4_0 0021280001cee63a
mlx4_1 0021280001cee492
mlx4_2 0021280001cee4aa
mlx4_3 0021280001cee4ea
# ibv_devinfo -d mlx4_0
hca_id: mlx4_0
transport: InfiniBand (0)
fw_ver: 2.7.8130
node_guid: 0021:2800:01ce:e63a
sys_image_guid: 0021:2800:01ce:e63d
vendor_id: 0x02c9
vendor_part_id: 26428
hw_ver: 0xB0
board_id: SUN0160000002
phys_port_cnt: 2
port: 1
state: PORT_ACTIVE (4)
max_mtu: 2048 (4)
active_mtu: 2048 (4)
sm_lid: 56
port_lid: 95
port_lmc: 0x00
link_layer: IB
port: 2
state: PORT_ACTIVE (4)
max_mtu: 2048 (4)
active_mtu: 2048 (4)
sm_lid: 56
port_lid: 96
port_lmc: 0x00
link_layer: IBOther commands and utilities such as ibstatus, fwflash or cfgadm can also be used to retrieve similar information.
-6-
PCIe Hot-Plugging : hotplug command
When the hotplug service is enabled on a Solaris system, hotplug command to bring hot pluggable devices online or offline without physically adding or removing the device from the system.
The following command lists out the all physical [hotplug] connectors along with the current status.
# hotplug list -c Connection State Description ________________________________________________________________________________ IOU2-EMS2 ENABLED PCIe-Native IOU2-PCIE6 ENABLED PCIe-Native IOU2-PCIE7 EMPTY PCIe-Native IOU2-PCIE4 EMPTY PCIe-Native IOU2-PCIE1 EMPTY PCIe-Native
For detailed instructions to hotplug a device, check the Solaris documentation out.