resize linux root disk in Oracle VM

I just needed to grow my root fs on an Oracle Linux guest VM running in OVM 3.2 and I thought I’d share the process with you. A lot of the steps are taken from a similar blog on doing a similar thing on vmware.

To start with, I have a linux vm which started as a clone of one of the Linux VM templates that Oracle provides. It comes with a 20G disk which is partitioned into two pieces. One for /boot and the other one is lvm physical volume that holds the rest of the system including the root partition. Continue reading

firewall port for ZFS Appliance with Oracle VM Storageconnect

I just stumbled across this and could not find it anywhere else on the net. I set up a ZFS Appliance with Oracle VM and their storageconnect plugin according to the documentation pdf (which are pretty easy step-by-step instructions) but in this case the OVM Server and the ZFS Appliance were not in the same network and access is denied by default in the firewall between those nets. So trying to register the appliance as a FC Storage led to this error that just tells me that the connection timed out. Continue reading

Linux Huge Pages in Oracle VM 3

My last post was about properly determining the number of hugepages needed with the newest version of the unbreakable enterprise kernel UEK R3 in Oracle Linux. Aside from the little issue with the proper kernel version detection in the script, the process is quite simple and well documented on the web:

  • turn off automatic memory management AMM by setting the parameter MEMORY_TARGET to 0
  • determine the number of hugepages needed by running the hugepages_settings.sh script
  • add an entry to /etc/sysctl.conf
  • reload the config with sysctl -p or reboot the server
  • edit /etc/security/limits.conf
  • bounce the instance(s)

Unfortunately, this was not it for me. You can check the effect of the setting like this:

[root@linuxbox ~]# grep Huge /proc/meminfo
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

This should have showed more than 0 hugepages configured. There is however an error logged by dmesg.

[root@linuxbox ~]# dmesg | tail -1
Guest pages are not properly aligned to use hugepages

Alright, the problem is that while Oracle VM may be engineered to run the Oracle database best, someone actually forgot to add support for huge pages in Para-Virtualized (PVM) guests. This should work for HVM guests (I have not tried it) but the general recommendation is to use PVM for Linux guests. The workaround for OVM 3.2.4 and later is described in metalink note 1529373.1. You have to add the parameter ‘allowsuperpage’ to the grub kernel boot options in /etc/grub.conf like this (added parameter is bold):

#boot=/dev/sda
default=0
timeout=5
title Oracle VM Server (2.6.39-300.32.5.el5uek)
        root (hd0,0)
        kernel /xen.gz dom0_mem=1160M allowsuperpage
        module /vmlinuz-2.6.39-300.32.5.el5uek ro root=UUID=d15aad1f-668b-4358-a54a-d7163a15198d
        module /initrd-2.6.39-300.32.5.el5uek.img

You need to do this on all nodes that you may want to run or live-migrate a VM with huge pages to and reboot the servers. I had to perform an update to 3.2.7 anyway so a reboot was neccessary anyway.

Next step, you need to add the parameter “superpages=1” at the end of the vm.cfg file of this vm and stop/start (restart is not enough) the vm.

But you are not supposed to edit the vm.cfg manually and an edit of the VM in the manager (like changing number of vcpus, memory or boot options) will remove that line again. Which will then lead to this VM not actually allocating any hugepages after the next boot. The note says they are working on it and I really hope they will fix this rather sooner than later.

And in case you are wondering what all this fuss is about and why I do not simply want to live with good old standard shared memory, let Mark Bobak convince you that if you are not using Huge Pages, you are doing it wrong.

Oracle VM 3 storage choices – part 1: intro, types and support

During my most recent installation of an Oracle VM 3.2 server pool, I was pondering again all the different options to manage storage with OVM, wrote them down together with pros and cons (to help me decide which one to implement) and now I am sharing these notes.

First of all, you will need to understand the different kinds of storage that OVM can use for all its data. We will start with the easier ones and work our way up. Continue reading

taking hot backups with Oracle VM

Propably my major complaint about oracle vm is it’s lack of a hot backup solution for virtual machines. There is simply no documented or supported way to back up running virtual machines – crazy. Of course you can install good old backup agents inside of your VMs but that is one of the things I want to avoid when deploying virtualization. Or, you could suspend or halt a machine, copy the virtual disk off to somewhere else and be happy. But I feel like we should not have to stop our services for backup. This is 2012 afterall! Continue reading

Oracle VM and bond0 aka the management interface

While working with Oracle VM one thing we stumbled across was that it always created a bond interface for management purposes. While it is perfectly fine to have a bond interface for failover reasons – the interface created is per default in Active Backup mode and cannot be modified according to the documentation. Now in Active Backup mode one interface is basically wasted (because only one interface is used at a time). OVM does offer two other modes though: Dynamic Link Aggregation and Adaptive Load Balancing.

Since we got LACP capable network infrastructure we wanted to use Dynamic Link Aggregation instead of Active Backup but OVM doesn’t offer a way to change the mode of the bond0 interface. That said, this is only true for the web interface. When SSH into the OVM server itself (the actual server where you want to change the bond, not the manager server) you can change the mode of the bond. This is the way we chose.

When doing this you should make sure that there are no virtual machines running on that box at the moment, because it might lead to a reboot (if it doesn’t you want to reboot anyways). Now when connected via SSH you simply execute the following command:

/etc/xen/scripts/ovs-bond-mode bondm=bond0 bondmode=4

This will change the config files for that interface and reconfigure the interface. In 2 out of 3 servers this lead to an instant reboot of the machine. On the one that didn’t reboot the interface came back up right away and was now in Dynamic Link Aggregation mode. The web interface however still showed Active Backup mode but after a reboot it showed the correct mode (that is why you want to reboot anyways). The servers which rebooted instantly came up with the interface in the right mode and the web interface showed the right mode too.

From now on when provisioning new OVM servers this is the very first thing we execute on every server.

UPDATE 2013-06-17 Bjoern Rost: The script does not exist anymore in Version 3.2 of OVM but changing the bond mode is now possible through the OVM manager GUI. It still leads to a reboot everytime I tried (and did not warn you) but other than that it works and looks much more like a supported action than the old way of calling the script directly.

Migrating a solaris server into Oracle VM using p2v

For my latest experiment with oracle VM I took a look at the p2v capabilty.
p2v stands for “physical to virtual” and does exactly, what the name suggests: converting a physical server to a virtual machine. In the following text I will describe some of the obstacles I faced when trying to convert a solaris server as well as my solutions.
Maybe this might help if somebody got stuck at the same issues.

The first few steps are fairly simple. All you need to start is the oracle VM Server boot medium (which can be downloaded here), a working Oracle VM environment and, of course, a physical server to convert.
And, last but not least, this howto.

Continue reading

Creating a virtual RAC using oracle VM templates

Our latest toy at portrix systems is an oracle VM testing environment, consisting of three sun servers, a 7120 Storage an a lot of enthusiasm to explore this exciting technology as deeply as possible.

A few days ago I challenged myself to install a virtual two node cluster into this environment.
Luckily I did not have to perform the installation from scratch, because oracle offers some preconfigured templates to download and simply import into the OVM environment.
I was most interested in “Oracle Real Application Cluster (RAC) 11g R2” which you will find here.

Unfortunately the first steps of the documentation, though very detailed, did not work in every aspect. It seems to refer to an older release of Oracle VM Manager and some important parts of the descriptions worked a bit differently 3.0.3.
If somebody else maybe faced the same problem, the following description might help.
Continue reading