revisiting Solaris 11 Automatic Installer without DHCP

A while ago I blogged about my experience with the new automatic installer in Solaris 11 and my special setup in which I refused to use a DHCP service because DHCP is simply something we do not use in our datacenter. That particular blog post has been one of my most popular (by visits) so this must be something that other admins experience aswell.

One of the issues I came across was that while you can easily specify IP, gateway and installation server on the commandline without DHCP, the install will break at a point where it needs to install some packages from ‘pkg.oracle.com’ but could not resolve that name due to a lack of DNS servers being set up. I described a rather tedious workaround in that post but came up with something a little better today. Continue reading

Cisco AnyConnect Static IPv6

We are using Cisco AnyConnect to provide VPN for our users (which we authenticate via RADIUS). Recently we needed to provide some users with static IP addresses.

For IPv4 this was easy since the ASA we are using supports RADIUS attribute 8 (Framed-IP-Address). For IPv6 there is an equivalent RADIUS attribute 168 (Framed-IPv6-Address) defined in RFC 6911. Unfortunately the ASA doesn’t support that attribute.

But thanks to the Cisco support we found out that the ASA (starting with version 9.0(1)) does support RFC 3162 (Cisco Bug ID CSCtr65342). What RFC 3162 provides are RADIUS attribute 96 (Framed-Interface-Id) and 97 (Framed-IPv6-Prefix). With these two you can easily provide static IPv6 addresses to your AnyConnect users.

To assign a user the address 2001:db8::42/64 you just set the following attributes in your RADIUS server:

Framed-IPv6-Prefix = 2001:db8:0:0::/64
Framed-Interface-Id = 0:0:0:42

For setting a static IPv6 address for local users one can use vpn-framed-ipv6-address which is documented in the Release Notes and the Command Reference.

firewall port for ZFS Appliance with Oracle VM Storageconnect

I just stumbled across this and could not find it anywhere else on the net. I set up a ZFS Appliance with Oracle VM and their storageconnect plugin according to the documentation pdf (which are pretty easy step-by-step instructions) but in this case the OVM Server and the ZFS Appliance were not in the same network and access is denied by default in the firewall between those nets. So trying to register the appliance as a FC Storage led to this error that just tells me that the connection timed out. Continue reading

upgrading OPatch through rpm

Every DBA has been there. You “just” want to apply a PSU, one-off patch or mostly anything in Oracle and quickly browse through the release notes or Readme when you encounter this:

You must use the OPatch utility version 12.1.0.1.1 or later to apply this patch. Oracle recommends that you use the latest released OPatch version for 12.1, which is available for download from My Oracle Support patch 6880880 by selecting the 12.1.0.1.0 release.

Almost every patch requires you to patch OPatch first. Which really is not a big deal: download patch 6880880, move the old $ORACLE_HOME/OPatch out of the way (or back up) and unzip the new one in it’s place. A very boring task, especially of you maintain a number of Homes on a box or multiple database servers. I am tired of performing these tasks and came up with this: Continue reading

OTN interview about #RACAttack at OpenWorld 2013

We had so much fun with RAC Attack during OpenWorld in San Francisco that people soon started to ask questions. Luckily, none of those were about why we were wearing bandanas, carrying an inflatable kangaroo and took jumping pictures with all the participants all day. For most other questions, see this interview between OTN’s Laura Ramsey and me.

RMOUG 2014 – Let’s go to Colorado

If there is is one thing that can be said about Colorado, it’s that it is huge. Really, really huge. In fact it is an mindblowing, ridiculous and completely exaggerated version of general hugeness. I beleive, you can leave Denver and walk for days straight ahead and you won’t meet a single soul. Some Elks maybe or sometimes a lost penguin.
There is another thing about Colorado: it is cold. Absolutely freezing cold. In fact, if there is anything, that exceeds Colorados hugeness, it might be its coldness. And yes, after your brain freezes for about a week, “coldness” suddenly becomes a valid word and you don’t bother anymore about some grammatical abomination like “hugeness”.

I have come to know Colorado as a harsh place, full of vast nothingness covered in snow (which, when I think about it, might be the reason for the cold). But, and thats a fact, Colorado is also one one of the most beautiful places I have ever been. When you stand at the bottom of the Rocky Mountains and just take a deep breath while looking around, you suddenly begin to feel very small compared to the vast amount of everything, nature throws in your face. And that is just an awesome feeling. Continue reading

Linux Huge Pages in Oracle VM 3

My last post was about properly determining the number of hugepages needed with the newest version of the unbreakable enterprise kernel UEK R3 in Oracle Linux. Aside from the little issue with the proper kernel version detection in the script, the process is quite simple and well documented on the web:

  • turn off automatic memory management AMM by setting the parameter MEMORY_TARGET to 0
  • determine the number of hugepages needed by running the hugepages_settings.sh script
  • add an entry to /etc/sysctl.conf
  • reload the config with sysctl -p or reboot the server
  • edit /etc/security/limits.conf
  • bounce the instance(s)

Unfortunately, this was not it for me. You can check the effect of the setting like this:

[root@linuxbox ~]# grep Huge /proc/meminfo
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

This should have showed more than 0 hugepages configured. There is however an error logged by dmesg.

[root@linuxbox ~]# dmesg | tail -1
Guest pages are not properly aligned to use hugepages

Alright, the problem is that while Oracle VM may be engineered to run the Oracle database best, someone actually forgot to add support for huge pages in Para-Virtualized (PVM) guests. This should work for HVM guests (I have not tried it) but the general recommendation is to use PVM for Linux guests. The workaround for OVM 3.2.4 and later is described in metalink note 1529373.1. You have to add the parameter ‘allowsuperpage’ to the grub kernel boot options in /etc/grub.conf like this (added parameter is bold):

#boot=/dev/sda
default=0
timeout=5
title Oracle VM Server (2.6.39-300.32.5.el5uek)
        root (hd0,0)
        kernel /xen.gz dom0_mem=1160M allowsuperpage
        module /vmlinuz-2.6.39-300.32.5.el5uek ro root=UUID=d15aad1f-668b-4358-a54a-d7163a15198d
        module /initrd-2.6.39-300.32.5.el5uek.img

You need to do this on all nodes that you may want to run or live-migrate a VM with huge pages to and reboot the servers. I had to perform an update to 3.2.7 anyway so a reboot was neccessary anyway.

Next step, you need to add the parameter “superpages=1″ at the end of the vm.cfg file of this vm and stop/start (restart is not enough) the vm.

But you are not supposed to edit the vm.cfg manually and an edit of the VM in the manager (like changing number of vcpus, memory or boot options) will remove that line again. Which will then lead to this VM not actually allocating any hugepages after the next boot. The note says they are working on it and I really hope they will fix this rather sooner than later.

And in case you are wondering what all this fuss is about and why I do not simply want to live with good old standard shared memory, let Mark Bobak convince you that if you are not using Huge Pages, you are doing it wrong.

setting up Huge Pages with UEK R3 (kernel 3.8)

I came across a little hiccup when configuring huge pages on my oracle linux 6.5 playground machine with the latest unbreakable linux kernel. Part of the documentation and also this blog post by Tim Hall have a nifty script that outputs the setting one should tweak to set the nr_hugepages parameter correctly. Unfortunately, this script fails on the most recent version of the kernel shipped with Oracle Linux:

[root@linuxbox ~]# /usr/local/bin/hugepages_setting.sh 
Unrecognized kernel version 3.8. Exiting.

Fortunately, the fix is really easy. There is a wrapper script that will fake the output of uname back to 2.6 and the method of setting the parameter with vm.nr_hugepages has stayed the same.

[root@linuxbox ~]$ yum install uname26
[root@linuxbox ~]$ uname26 /usr/local/bin/hugepages_setting.sh 
Recommended setting: vm.nr_hugepages = 516

But even better than faking your way around this would be to modify the script and to accept 3.8 as a valid kernel version like this.

#!/bin/bash
#
# hugepages_settings.sh
#
# Linux bash script to compute values for the
# recommended HugePages/HugeTLB configuration
#
# Note: This script does calculation for all shared memory
# segments available when the script is run, no matter it
# is an Oracle RDBMS shared memory segment or not.
# Check for the kernel version
KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`
# Find out the HugePage size
HPG_SZ=`grep Hugepagesize /proc/meminfo | awk {'print $2'}`
# Start from 1 pages to be on the safe side and guarantee 1 free HugePage
NUM_PG=1
# Cumulative number of pages required to handle the running shared memory segments
for SEG_BYTES in `ipcs -m | awk {'print $5'} | grep "[0-9][0-9]*"`
do
   MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
   if [ $MIN_PG -gt 0 ]; then
      NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
   fi
done
# Finish with results
case $KERN in
   '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
          echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
   '2.6' | '3.8' ) echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
      *) echo "Unrecognized kernel version $KERN. Exiting." ;;
esac

Btw: my test returned only 1 when I first ran it. After some advice by Frits Hoogland on twitter and a little study of this blog post by Tanel Poder it became quite obvious. I forgot to unset the MEMORY_TARGET parameter in my spfile and thus the shared memory was allocated in chunks visible in /dev/shm but not through ‘ipcs -m’. I do not trust AMM anyway, set MEMORY_MANAGEMENT to 0 and went on.

Rad am Ring 2013

Last September the portrix crew participated in the Rad am Ring bike challenge after being invited by our data centre provider, the Equinix Inc. The goal was to drive as many laps as possible in 24 hours around the german formular one GP race track, the Nuerburgring and it´s former historical pendant, the most famous Nordschleife. Each lap would let us give a donation to the Projekt Schmetterling e.V. foundation which helps people suffering from cancer to cope with their burden.

There were 16 miles of roller-coaster bike action switching between steep hills to climb and prone slopes often droping into fierce hairpin turns which had to be mastered to finish one lap of the track.

Fortunately Equinix did a tremendous job on supporting their guests as they managed everything from the hotel to catering and even putting up a massage couch for the whole upcoming 24 hours staged in a formular one racing box. It was just great as the race would become a tough challenge against nature itself.

Weather changed quickly from slightly cloudy to atrocious. After two laps of racing it started to rain heavily escorted by fog as the sun set behind the Eifel Mountains. Eventually a nearby thunderstorm would force the responsibles to interrupt the race for several hours. There is a video filmed by rheinahr.tv which gives you an idea about the experience we´ve had there including a group picture of us at 06:50. Enjoy!

Summarized Team portrix had raced a whole of 15 laps donating 390€ to Projekt Schmetterling e.V. Riding in the mountains even if odds are against you due to nasty weather conditions was a great experience so we are looking forward to compete again this year related to the commitment of doing more laps to do even more good in 2014.

Best Regards
Ole

top 3 posts and review of 2013

I did a similar post last year and in trying to start a bit of a tradition looked at the same statistics again this year. And the result was quite a surprise. While I thought that 2013 was the year of the 12c database it actually was still the year of the infrastructure. At least here at this blog.

stanley calendar 20132013 has been an awesome ride for me and did not leave much more to wish for. I was able to attend and speak at a number of conferences and user group meetings around the world: IOUG Collaborate in Denver, OUGN vårseminar on a boat from Oslo to Kiel, OUG Scotland in Edinburgh, Oracle OpenWorld in San Francisco, the OTN APAC tour in Auckland and INSYNC in Perth, DOAG in Nuremberg and UKOUG in Manchester. I was awarded ACE Directorship and was elected vice president of the RAC special interest group. How much better could it get?
Continue reading