WebSphere MQ with Solaris Zones

Solaris Zones are Sun’s approach to virtualization which was introduced in Solaris 10. You can have several ‘zones’ running on a single server, and they all look like separate machines. I got to play with them towards the end of last year, but it’s been a quiet week this week so I thought I’d raise it and see if it would spark any discussion.

There are loads of ways that you can configure Solaris Zones, but essentially the idea is that you can have multiple lightweight Zones by having them all share the core system files. With only one copy of the bulk of the core OS, you can have fairly small zones – each of them accessing the core files across a read-only mount.

In this post, I discuss some of the problems that you may face when using WebSphere MQ with Solaris Zones and the background behind them. I also outline a number of practical approaches to deal with these problems.

Installation is really the bit that really causes problems when using WebSphere MQ in a Solaris Zones environment. Once installed, WMQ should work as normal. The problems that we have are mainly due to attempts to write to /usr, which is mounted as a read-only directory in a default zone configuration. These attempts are:

soft-links

soft-links in /usr/bin to WMQ executables in /opt/mqm/bin
soft-links in /usr/include to include files in /opt/mqm/inc
soft-links in /usr/lib to WMQ libraries in /opt/mqm/lib

These are provided to avoid users needing to explicitly add WMQ paths to their environment when using WMQ, compiling WMQ applications, etc. However, these are not essential to WMQ if you add WMQ paths to your environment explicitly. Executables provided with WMQ make no reliances on these soft-links, and have hard-coded references to the actual locations in /opt/mqm anyway.

As such, although this part of an installation would fail, it really shouldn’t matter, and if you can get the rest of the installation to complete, then you can carry on without it (remembering to modify the environment to allow WMQ executables to be run directly from /opt/mqm/bin).

message catalog

message catalog in /usr/lib/locale

Unlike the soft-links, these actually are required. However, these can be stored in an alternative location, such as within a local zone. The environment variable NLSPATH can be used to specify the location of the message catalogue file.

man pages

man pages in /usr/share/man

These are obviously not critical to WMQ functionality, so the easiest approach is just to not include them in a zones installation.

request scripts

The other problem with installs surround the use of ‘request scripts‘. These are scripts used to request user input, providing the menu that allows selection of which parts to install. As stated in the man pages for zoneadm and pkgadd “Any package that contains a request script is added to the global zone only.“.

The WMQ installer uses a request script, and so this restricts the installation options. (Interestingly, GSKit – the package used to provide WMQ’s SSL support – does not use a request script. So care does need to be taken to ensure that these are installed in the same places.)

Configurations

So with all of these in mind, last year I had a go at creating as many different types of Zones configuration I could think of, and tried installing WebSphere MQ v6 into each. Some were easier than others, and a couple had problems that I couldn’t find a practical workaround for. One configuration in particular needed some hacking of the installer to get to work, and as such really isn’t recommended (although I’ve included it here out of interest, and for completeness’ sake).

I’ve included the outcome of this work below – for each configuration I’ve listed the major problems I encountered, and the workarounds that I employed.

Installing into all Zones at once

Overview

On systems with multiple non-global zones, Solaris allows an install into all non-global ‘whole root’ zones at once.

Work required

Problem 1: This is not possible with WebSphere MQ. As discussed above, the Solaris installer, pkgadd, is not able to install into all zones for packages which include a request script.
Workaround: Install into each non-global zone manually – it just takes longer.
When installing in the global zone, you should use ‘pkgadd -G‘ so that installing into all zones is not attempted.

Installing WMQ into the Global Zone

Overview

This is the closest approach to the existing non-Zones installation.

Work required

Problem 1: Inconsistent behavior when installing into Global Zone. By default, when installing into the global zone, software is also installed into all non-global ‘whole root’ zones. This is not possible with WebSphere MQ (because of the use of request scripts discussed above). As a result, WebSphere MQ is installed in the global zone only, while GSKit (the package used to provide WMQ’s SSL support and which does not use request scripts) is installed into global zone and all non-global ‘whole root’ zones.
Workaround: To ensure consistent behavior for both WebSphere MQ and GSkit (that is, to install into global zone only), use ‘pkgadd -G‘.

Installing WMQ into the Global Zone and using in non-global ‘Sparse Root’ zones

Overview

The principle here is to have one instance of your executables (/opt/mqm) which is visible to all non-global sparse root zones. Each non-global zone would have it’s own /var/mqm and so it’s own queue managers. With each zone having it’s own IP address etc. this essentially gives you the appearance of multiple machines with WMQ, but all sharing the same installed executables.

Work required

Problem 1: Installing into the Global Zone does not create the necessary /var/mqm structure in non-global zones.
Workaround: The /var/mqm directory structure can be copied from the global zone to a non-global zone. Note that the security permissions will need to be carefully checked to ensure that they are copied correctly. Queue manager data files (i.e. contents of /var/mqm/qmgrs, with the exception of /var/mqm/qmgrs/@SYSTEM) should not be copied – having multiple copies of the same queue manager running in different zones should not be attempted, as it would almost certainly cause problems.
Problem 2: The installer ensures that an ‘mqm’ user and ‘mqm’ group are created. As the installer is not being run in the non-global zones, these user and group ids are missing.
Workaround: An mqm user and group should be manually created in each non-global zone, and these used in security permission settings for /var/mqm/ files and directories in the zone.

Installing WMQ into a non-global ‘Whole Root’ zone

Overview

This is the most straight-forward non-global zone option. It allows WMQ to be used from within the zone, but puts nothing into the global zone that would cause it to visible outside of the zone in which it is installed. Installs in different zones would not share anything.
It is unappealing in some ways because it requires such a large zone – the default (having only a subset of the root) is to have much smaller zones.

Work required

Problem 1: The WebSphere MQ postinstall script attempts to verify if kernel parameters are set correctly. Commands used to do this are not available within non-global zones, and fail with error messages displayed at install-time

## Executing postinstall script.
Insufficient privileges to load a module
Insufficient privileges to load a module
Insufficient privileges to load a module

Workaround: These error messages can be ignored. This occurs post-install, and does not effect the install itself. Kernel parameters should be verified manually using guidance available in the WMQ v6 product README.

Installing WMQ into a non-global ‘Sparse Root’ zone

Overview

Note 1: On a system where WebSphere MQ is already installed in the global zone (when the non-global zone is created).
Note 2: Assuming a default zone configuration – therefore one without it’s own copy of /usr.
When running the installer in the non-global zone, the files that pkgadd is normally unable to create (files in /usr) are already present due to the installation in the global zone. As a result, pkgadd just verifies that the properties are okay and continues, allowing the installation to complete successfully.

Work required

Problem 1: If WebSphere MQ is uninstalled from the global zone, the non-global zone installation will stop working (Attempts to run WMQ commands fail, displaying error messages reporting that WMQ is unable to open message catalog.)
Workaround: None.
Problem 2: As files in /usr are shared between the different installations, installing different versions of WebSphere MQ in the global and non-global zones will fail, as the files will not be consistent.
Workaround: None.
Problem 3: Attempts to uninstall from non-global zone fail to complete, as access is denied when it tries to remove files from /usr. The non-global zone installation remains in a ‘partially installed’ state.
Workaround: Retry uninstall to remove entry from package list. Files in /usr will remain untouched.

Installing WMQ into a custom zone configuration

Overview

Installing into a custom zone configuration – based on a Sparse Root approach but including a local copy of /usr.
This is configured in zonecfg using: remove inherit-pkg-dir dir=/usr

Work required

## Executing postinstall script.
Insufficient privileges to load a module
Insufficient privileges to load a module
Insufficient privileges to load a module

Installing WMQ into a non-global ‘Sparse Root’ zone

Overview

Note 1: On a system where WMQ is not already installed in the global zone when the non-global zone is created
Note 2: Assuming a default zone configuration – therefore one without it’s own copy of /usr.
This is a complicated approach. It has the benefit of being the most efficient in regards to the filesize of the zone – as the zone includes only a minimum subset of the root.

Work required

Problem 1: Multiple errors are displayed reporting access denied when we try to write to read-only directories in /usr. The installation fails (status shown as ‘partially installed’).
Problem 2: Attempts to run WMQ commands fail, displaying error messages reporting that WMQ is unable to open message catalog.
Workaround: This requires modification to the installation scripts:

Unpack a local copy of the installation files to filespace available to the local zone

cp -rf mq_installer_directory/ /zones/comet/FileSystems/RootDir/root/tmp/.

If you do not have ‘mqm’, ‘gsk7bas’ and ‘gsk7bas64′ directories, and only have a .img file, then unpack it using pkgtrans

bash-3.00# pkgtrans /tmp/mq_installer_directory/installer.img /tmp/mq_installer_directory/.
bash-3.00# ls
PreReqs       READMEs       gsk7bas       gsk7bas64    
lap           licenses      mqlicense.sh  mqm           
readadd.txt   silent

Create an mqm user and group within the non-global zone
```
groupadd mqm
useradd -g mqm mqm
```
Move the message catalog files to model the intended new location (e.g. /myzoneusr/lib/locale)
1. Move mqm/root/usr to mqm/root/myzoneusr
```
cd /tmp/mq_installer_directory
mv mqm/root/usr/ mqm/root/myzoneusr
```
Modify mqm/pkgmap
1. Alter lines which contain a path beginning /usr/lib/locale to a location within the zone (e.g. /myzoneusr/lib/locale)
```
vi /tmp/mq_installer_directory/mqm/pkgmap
:%s/\/usr\/lib\/locale/\/myzoneusr\/lib\/locale/g
```
2. Remove all remaining lines which contain a path beginning /usr (soft-links and man pages)
```
vi /tmp/mq_installer_directory/mqm/pkgmap
:g/\/usr/d
```
Modify mqm/install/postinstall
1. Alter lines which contain a path beginning /usr/lib/locale to a location within the zone (e.g. /myzoneusr/lib/locale)
```
vi /tmp/mq_installer_directory/mqm/install/postinstall
:%s/\/usr\/lib\/locale/\/myzoneusr\/lib\/locale/g
```
Modify mqm/install/preremove
1. Alter lines which contain a path beginning /usr/lib/locale to a location within the zone (e.g. /myzoneusr/lib/locale)
```
vi /tmp/mq_installer_directory/mqm/install/preremove
:%s/\/usr\/lib\/locale/\/myzoneusr\/lib\/locale/g
```
Modify gsk7bas/pkgmap
1. Remove all remaining lines which contain a path beginning /usr (soft-links and man pages)
```
vi /tmp/mq_installer_directory/gsk7bas/pkgmap
:g/\/usr/d
```
Modify gsk7bas64/pkgmap
1. Remove all remaining lines which contain a path beginning /usr (soft-links and man pages)
```
vi /tmp/mq_installer_directory/gsk7bas64/pkgmap
:g/\/usr/d
```

Run the mqlicense script

/tmp/mq_installer_directory/mqlicense.sh

Make an attempt to install mqm (only mqm – not GSkit) from within the non-global zone using pkgadd
```
pkgadd -d /tmp/mq_installer_directory/.
```
1. This attempt will fail because changes to mqm/install/postinstall and mqm/install/preremove will mean that the filesize and checksum value for these files stored in mqm/pkgmap are no longer correct. This will cause pkgadd to report these files as corrupt.
```
pkgadd: ERROR: packaging file </tmp/mq_installer_directory/mqm/install/postinstall> is corrupt
    file size <10797> expected <11175> actual
    file cksum <8181> expected <50643> actual
pkgadd: ERROR: packaging file </tmp/mq_installer_directory/mqm/install/preremove> is corrupt
    file size <2659> expected <2731> actual
    file cksum <20944> expected <29032> actual
```
2. Use the pkgadd error message as a guide to correct the filesize and checksum values in mqm/pkgmap. The pkgadd error message reports ‘expected X, found Y’, and this can be used to search for and correct the values adjacent to ‘postinstall’ and ‘preremove’.
```
vi /tmp/mq_installer_directory/mqm/pkgmap
:%s/postinstall 10797 8181/postinstall 11175 50643/g
:%s/preremove 2659 20944/preremove 2731 29032/g
```
Install WMQ from within the zone using pkgadd using the corrected files
```
pkgadd -d /tmp/mq_installer_directory/.
```
1. Ensure that Manual Pages are not included in the install packages list – as these have been removed from the package map
Modify environment variables for users intended to be used with WebSphere MQ to include WMQ paths
```
export PATH=/opt/mqm/bin:/opt/mqm/samp/bin:/opt/ibm/gsk7/bin/:$PATH
```

Modify NLSPATH to point at custom location for message catalog

export LANG=C
export NLSPATH=/myzoneusr/lib/locale/%L/LC_MESSAGES/%N

So there are lots of ways to try and install WebSphere MQ with Solaris Zones!

I don’t think any of us would claim that what we have now is an ideal situation, and obviously to have a more transparent installation story in the future would be much better. But at the moment, with a little care it is certainly possible to run queue managers in Zones in various different configurations to suit a variety of needs.

Are you using any of these approaches at the moment? Or have you found another way to do it? Comments (even anonymous ones) would be very welcome, as it would be useful for us to know how people are using Zones and what more we could do to make this easier.