Quick look inside linux ‘top’ for APEX

Imagine… you’re a SysAdmin…
Imagine… you’re “terminal schooled”…
And you are running Oracle Application Express

You want to be able to support your APEX-team by helping them diagnose performance-related issues, but all you have is good, old ‘top‘ to get you some insight… Continue reading

debugging puppet and facter with strace

I worked on a slightly annoying problem yesterday and thought I’d share the process with you. On one of our linux boxes puppet stopped working with an error that was not very helpful:

[root@portrix]# puppet agent --test
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve local facts: private method `split' called for nil:NilClass
Error: Failed to apply catalog: Could not retrieve local facts: private method `split' called for nil:NilClass

All this told me was that something was wrong with facter during the puppet run. I was able to reproduce the error with just facter but even the debug output was not much more helpful:
Continue reading

quick tip: build debian package from sqldeveloper rpm

Oracle has made the installation of sqldeveloper easy on distributions that use rpm by providing a nice package. But we are using ubuntu for our development workstations at the office and thus need a debian .deb package. I do not want to use the ‘alien’ workaround and we already build or rebuild some of our packages for our own repository anyway. This can easily be built with the help of fpm like this after downloading the Linux rpm from OTN:

fpm --no-auto-depends -d 'oracle-jdk-7' -s rpm -t deb -m "portrix Systems GmbH " /var/tmp/sqldeveloper-

Upload this to your local repository and you can then install this with apt-get or simply show the details like this.

portrix@vm-04:~$ sudo apt-cache show sqldeveloper
Package: sqldeveloper
Architecture: all
Maintainer: portrix Systems GmbH 
Installed-Size: 305622
Depends: oracle-jdk-7
Provides: sqldeveloper
Filename: pool/main/sqldeveloper_4.
Size: 246647402
MD5sum: 1a7127bde4521e0a5bc7160ee3f891a0
SHA1: 55f12afe5075bcbd9223f61fe8f183fd0570c295
SHA256: aeb07cede84298c7c8a9bf8caf0c42ccf560840a857415be81bb4189c8939ba2
Section: extra
Priority: Priority
Homepage: http://example.com/no-uri-given
Description: Oracle SQL Developer is a new, free graphical tool that enhances productivity
             and simplifies database development tasks. With SQL Developer, you can browse database objects, 
             run SQL statements and SQL scripts, and edit and debug PL/SQL statements. 
             You can also run any number of provided reports, as well as create and save your own.
License: Oracle

Update October 2015:
I tried the same procedure for sqldeveloper 4.1.1 but this failed on installation with an error complaining about the Provides meta-information:

root@PS-112:~# dpkg -i /tmp/sqldeveloper_4.
dpkg: error processing archive /tmp/sqldeveloper_4. (--install):
 parsing file '/var/lib/dpkg/tmp.ci/control' near line 9 package 'sqldeveloper':
 `Provides' field, invalid package name `/home/twall/jna/build/native-linux-x86-64/libjnidispatch.so': must start with an alphanumeric character
Errors were encountered while processing:

So I went through some extra steps to modify the package (unpack, edit, repack) and remove the two references starting with /home/twall like this:

root@PS-112:~# dpkg-deb -R /tmp/sqldeveloper_4. /tmp/rebuild-sqldev
root@PS-112:~# vim /tmp/rebuild-sqldev/DEBIAN/control 
# edit the Provides: line to show only "Provides: sqldeveloper"
root@PS-112:~# dpkg-deb -b /tmp/rebuild-sqldev /tmp/sqldeveloper_4.

resize linux root disk in Oracle VM

I just needed to grow my root fs on an Oracle Linux guest VM running in OVM 3.2 and I thought I’d share the process with you. A lot of the steps are taken from a similar blog on doing a similar thing on vmware.

To start with, I have a linux vm which started as a clone of one of the Linux VM templates that Oracle provides. It comes with a 20G disk which is partitioned into two pieces. One for /boot and the other one is lvm physical volume that holds the rest of the system including the root partition. Continue reading

Linux Huge Pages in Oracle VM 3

My last post was about properly determining the number of hugepages needed with the newest version of the unbreakable enterprise kernel UEK R3 in Oracle Linux. Aside from the little issue with the proper kernel version detection in the script, the process is quite simple and well documented on the web:

  • turn off automatic memory management AMM by setting the parameter MEMORY_TARGET to 0
  • determine the number of hugepages needed by running the hugepages_settings.sh script
  • add an entry to /etc/sysctl.conf
  • reload the config with sysctl -p or reboot the server
  • edit /etc/security/limits.conf
  • bounce the instance(s)

Unfortunately, this was not it for me. You can check the effect of the setting like this:

[root@linuxbox ~]# grep Huge /proc/meminfo
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

This should have showed more than 0 hugepages configured. There is however an error logged by dmesg.

[root@linuxbox ~]# dmesg | tail -1
Guest pages are not properly aligned to use hugepages

Alright, the problem is that while Oracle VM may be engineered to run the Oracle database best, someone actually forgot to add support for huge pages in Para-Virtualized (PVM) guests. This should work for HVM guests (I have not tried it) but the general recommendation is to use PVM for Linux guests. The workaround for OVM 3.2.4 and later is described in metalink note 1529373.1. You have to add the parameter ‘allowsuperpage’ to the grub kernel boot options in /etc/grub.conf like this (added parameter is bold):

title Oracle VM Server (2.6.39-300.32.5.el5uek)
        root (hd0,0)
        kernel /xen.gz dom0_mem=1160M allowsuperpage
        module /vmlinuz-2.6.39-300.32.5.el5uek ro root=UUID=d15aad1f-668b-4358-a54a-d7163a15198d
        module /initrd-2.6.39-300.32.5.el5uek.img

You need to do this on all nodes that you may want to run or live-migrate a VM with huge pages to and reboot the servers. I had to perform an update to 3.2.7 anyway so a reboot was neccessary anyway.

Next step, you need to add the parameter “superpages=1” at the end of the vm.cfg file of this vm and stop/start (restart is not enough) the vm.

But you are not supposed to edit the vm.cfg manually and an edit of the VM in the manager (like changing number of vcpus, memory or boot options) will remove that line again. Which will then lead to this VM not actually allocating any hugepages after the next boot. The note says they are working on it and I really hope they will fix this rather sooner than later.

And in case you are wondering what all this fuss is about and why I do not simply want to live with good old standard shared memory, let Mark Bobak convince you that if you are not using Huge Pages, you are doing it wrong.

setting up Huge Pages with UEK R3 (kernel 3.8)

I came across a little hiccup when configuring huge pages on my oracle linux 6.5 playground machine with the latest unbreakable linux kernel. Part of the documentation and also this blog post by Tim Hall have a nifty script that outputs the setting one should tweak to set the nr_hugepages parameter correctly. Unfortunately, this script fails on the most recent version of the kernel shipped with Oracle Linux:

[root@linuxbox ~]# /usr/local/bin/hugepages_setting.sh 
Unrecognized kernel version 3.8. Exiting.

Fortunately, the fix is really easy. There is a wrapper script that will fake the output of uname back to 2.6 and the method of setting the parameter with vm.nr_hugepages has stayed the same.

[root@linuxbox ~]$ yum install uname26
[root@linuxbox ~]$ uname26 /usr/local/bin/hugepages_setting.sh 
Recommended setting: vm.nr_hugepages = 516

But even better than faking your way around this would be to modify the script and to accept 3.8 as a valid kernel version like this.

# hugepages_settings.sh
# Linux bash script to compute values for the
# recommended HugePages/HugeTLB configuration
# Note: This script does calculation for all shared memory
# segments available when the script is run, no matter it
# is an Oracle RDBMS shared memory segment or not.
# Check for the kernel version
KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`
# Find out the HugePage size
HPG_SZ=`grep Hugepagesize /proc/meminfo | awk {'print $2'}`
# Start from 1 pages to be on the safe side and guarantee 1 free HugePage
# Cumulative number of pages required to handle the running shared memory segments
for SEG_BYTES in `ipcs -m | awk {'print $5'} | grep "[0-9][0-9]*"`
   MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
   if [ $MIN_PG -gt 0 ]; then
      NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
# Finish with results
case $KERN in
   '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
          echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
   '2.6' | '3.8' ) echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
      *) echo "Unrecognized kernel version $KERN. Exiting." ;;

Btw: my test returned only 1 when I first ran it. After some advice by Frits Hoogland on twitter and a little study of this blog post by Tanel Poder it became quite obvious. I forgot to unset the MEMORY_TARGET parameter in my spfile and thus the shared memory was allocated in chunks visible in /dev/shm but not through ‘ipcs -m’. I do not trust AMM anyway, set MEMORY_MANAGEMENT to 0 and went on.

troubleshooting a CPU eating java process

Just yesterday we had a situation where a java process (a JBOSS app server) started to raise the CPU load on a machine significantly. Load average increased and top showed that the process consumed a lot of CPU. The planned “workaround”? Blame something (garbage collection is always an easy victim), restart and hope for the best. After all, there is not much to diagnose here if there is nothing in the logs, right? wrong! There are a few things that we can do with a running java vm to diagnose these issues from a solaris or linux commandline and find the root cause. Continue reading

Upgrading to Oracle Linux 6.5 and UEK3

OL 6.5 was released a few days ago to ULN and public yum, and of course I was going to upgrade some machines immediately. But first: Have I mentioned before that I really love Oracle Linux for providing ISOs and patches for free? One of our customers insists on using Red Hat instead of OL. No problem so far but a few days ago we needed to clone that VM for some testing and staging. Only problem: to do this we needed another support subscription from RH which the customer is paying for but more importantly it delayed the project since we now had to wait for procurement. This is why I appreciate the license model of OL: you can use it for free and have the option of purchasing support for the machines that are most important to you. Also, licensing is per (hardware) server, not VM. Read more about OL licensing directly from the horses belgian’s mouth. Continue reading