Fun with IPMI
IPMI is a standard protocol which allows out of band access to hardware features like reading sensors and error logs, turning the power on or off or accessing the serial console. This is made possible by the Baseboard Management Controller (BMC), an independent CPU which is accessed from the operating system or by sharing an Ethernet port with the OS.
Most modern servers offer an IPMI 1.5 or 2.0 interface and usually loading the ipmi_si driver is all that is needed to verify if it is supported, but some may need additional parameters and/or a kernel upgrade. dmidecode is also useful to determine if the system has a BMC.
After loading the ipmi_si and ipmi_devintf drivers the BMC will be accessible from Linux, e.g. using the ipmitool command. The ipmitool package will also install the ipmievd daemon, which logs events like overheating a failed fan (beware: it uses local.* syslog facility which on Debian systems is not configured by default).
It is also useful to install the ipmi_poweroff driver, which will generate an ACPI power button event when a clean shutdown is requested by the BMC (you need to install acpid too).
The details of accessing the BMC over an Ethernet connection varies depending on the server model and manufacturer, e.g. IBM xSeries servers are shipped with the same default IP address, username and password which means that anybody on the same L2 network (no gateway address is configured) can shut down your system:
ipmitool -I lan -H 10.1.1.97 -A PASSWORD -U USERID -P PASSW0RD shell
The most convenient way to experiment with IPMI is to run ipmitool shell locally and trying the available commands. These are some trivial examples:
# show the parameters for Ethernet access
lan print
# reset the system
chassis power reset
# cleanly shut down the OS and then power off the system
chassis power soft
# at the next boot, boot from the network
chassis bootdev pxe
# turn on the locator LED
chassis identify 255
# print the system event log
sel list
# read a specific sensor
# (with enough servers you could create a 3D thermal map of the room...)
sensor get "Ambient Temp"