HowTo: applying kernel updates to ganeti node

Print Friendly, PDF & Email

To apply a standard kernel update to a same version kernel (e.g. 2.6.18-6 security update where the kernel version doesn’t change).

  1. Check running instances on master node:
  2. $ ssh gnt-cluster0
    $ sudo gnt-instance list
    Instance                 OS          Primary_node            Status     Memory
    id-zprod.ilrt.bris.ac.uk centos-5    leda.ilrt.bris.ac.uk    running      1024
    ldap.ilrt.bris.ac.uk     debian-etch himalia.ilrt.bris.ac.uk running        64
    www1.ilrt.bris.ac.uk     debian-etch himalia.ilrt.bris.ac.uk ADMIN_down      -

  3. Failover running instances from node which is going to reboot (himalia in this example).
  4. $ sudo gnt-instance failover ldap.ilrt.bris.ac.uk
    Failover will happen to image ldap.ilrt.bris.ac.uk. This requires a
    shutdown of the instance. Continue?
    y/[n]/?: y
    * checking disk consistency between source and target
    * shutting down instance on source node
    * deactivating the instance's disks on source node
    * activating the instance's disks on target node
    * starting the instance on the target node

  5. Check instances started correctly on target node.
  6. Failover the master node if necessary. First determine secondary node in this cluster.
  7. $ sudo gnt-cluster getmaster
    himalia.ilrt.bris.ac.uk
    $ sudo gnt-node list
    Node                    DTotal  DFree MTotal MNode MFree Pinst Sinst
    himalia.ilrt.bris.ac.uk 276444 152028   4095   257  3793     1     2
    leda.ilrt.bris.ac.uk    276444 207324   4095   257  2705     2     1

    Free node in this cluster is leda.

    $ ssh leda
    $ sudo gnt-cluster masterfailover

    Test the new master from another host.

    $ ssh gnt-cluster0

  8. If all instances migrated successfully it is safe to reboot the node.

IMPORTANT: if the kernel update involves a new release upgrade, e.g. 2.6.18-6 -> 2.6.18-7 then you must first update the drbd kernel modules and deploy them via bcfg2. This is a dangerous process and must be performed carefully!

  1. Check the contents of https://svn.ilrt.bris.ac.uk/repos/cfgman/trunk/scripts/update-kernel.sh for files in bcfg2 that need to be updated with the current kernel version. (This might change so the files are automatically generated with the correct kernel version in the future.)
  2. Then update the kernel packages on the xen hypervisor host that the deployer sits on.
  3. $ ssh callisto
    $ sudo bcfg2 -qvn
    $ sudo bcfg2 -kqv
    $ sudo aptitude update
    $ sudo aptitude upgrade

  4. Then update the deployer
  5. $ ssh deployer
    $ sudo bcfg2 -qvn
    $ sudo bcfg2 -kqv
    $ sudo aptitude update
    $ sudo aptitude upgrade

  6. See what services a reboot will affect back on callisto
  7. $ sudo xm list

  8. Then reboot callisto
  9. $ sudo reboot

  10. Update the local debian repo packages for the DRBD modules.
  11. $ sudo /usr/local/projects/pkg-repo/bin/update-drbd.sh

  12. Then update the modules on the ganeti cluster node via bcfg2.
  13. $ sudo bcfg2 -qvn
    $ sudo bcfg2 -qv
    $ sudo aptitude update
    $ sudo aptitude upgrade

  14. Perform the steps outlined above before rebooting the node.
  15. Then you must update all ganeti cluster nodes repeating steps 6-7 on each node.
  16. When all is done remove the old kernel packages.