WebSphere MQ with Solaris Zones

March 2, 2007 in install, webspheremq | by Dale Lane

Solaris Zones are Sun’s approach to virtualization which was introduced in Solaris 10. You can have several ‘zones’ running on a single server, and they all look like separate machines. I got to play with them towards the end of last year, but it’s been a quiet week this week so I thought I’d raise it and see if it would spark any discussion.

There are loads of ways that you can configure Solaris Zones, but essentially the idea is that you can have multiple lightweight Zones by having them all share the core system files. With only one copy of the bulk of the core OS, you can have fairly small zones – each of them accessing the core files across a read-only mount.

In this post, I discuss some of the problems that you may face when using WebSphere MQ with Solaris Zones and the background behind them. I also outline a number of practical approaches to deal with these problems.

Installation is really the bit that really causes problems when using WebSphere MQ in a Solaris Zones environment. Once installed, WMQ should work as normal. The problems that we have are mainly due to attempts to write to /usr, which is mounted as a read-only directory in a default zone configuration. These attempts are:

soft-links

soft-links in /usr/bin to WMQ executables in /opt/mqm/bin
soft-links in /usr/include to include files in /opt/mqm/inc
soft-links in /usr/lib to WMQ libraries in /opt/mqm/lib

These are provided to avoid users needing to explicitly add WMQ paths to their environment when using WMQ, compiling WMQ applications, etc. However, these are not essential to WMQ if you add WMQ paths to your environment explicitly. Executables provided with WMQ make no reliances on these soft-links, and have hard-coded references to the actual locations in /opt/mqm anyway.

As such, although this part of an installation would fail, it really shouldn’t matter, and if you can get the rest of the installation to complete, then you can carry on without it (remembering to modify the environment to allow WMQ executables to be run directly from /opt/mqm/bin).

message catalog

message catalog in /usr/lib/locale

Unlike the soft-links, these actually are required. However, these can be stored in an alternative location, such as within a local zone. The environment variable NLSPATH can be used to specify the location of the message catalogue file.

man pages

man pages in /usr/share/man

These are obviously not critical to WMQ functionality, so the easiest approach is just to not include them in a zones installation.

request scripts

The other problem with installs surround the use of ‘request scripts‘. These are scripts used to request user input, providing the menu that allows selection of which parts to install. As stated in the man pages for zoneadm and pkgadd “Any package that contains a request script is added to the global zone only.“.

The WMQ installer uses a request script, and so this restricts the installation options. (Interestingly, GSKit – the package used to provide WMQ’s SSL support – does not use a request script. So care does need to be taken to ensure that these are installed in the same places.)

Configurations

So with all of these in mind, last year I had a go at creating as many different types of Zones configuration I could think of, and tried installing WebSphere MQ v6 into each. Some were easier than others, and a couple had problems that I couldn’t find a practical workaround for. One configuration in particular needed some hacking of the installer to get to work, and as such really isn’t recommended (although I’ve included it here out of interest, and for completeness’ sake).

I’ve included the outcome of this work below – for each configuration I’ve listed the major problems I encountered, and the workarounds that I employed.

Installing into all Zones at once

Overview

On systems with multiple non-global zones, Solaris allows an install into all non-global ‘whole root’ zones at once.

Work required

Problem 1: This is not possible with WebSphere MQ. As discussed above, the Solaris installer, pkgadd, is not able to install into all zones for packages which include a request script.
Workaround: Install into each non-global zone manually – it just takes longer.
When installing in the global zone, you should use ‘pkgadd -G‘ so that installing into all zones is not attempted.

Installing WMQ into the Global Zone

Overview

This is the closest approach to the existing non-Zones installation.

Work required

Problem 1: Inconsistent behavior when installing into Global Zone. By default, when installing into the global zone, software is also installed into all non-global ‘whole root’ zones. This is not possible with WebSphere MQ (because of the use of request scripts discussed above). As a result, WebSphere MQ is installed in the global zone only, while GSKit (the package used to provide WMQ’s SSL support and which does not use request scripts) is installed into global zone and all non-global ‘whole root’ zones.
Workaround: To ensure consistent behavior for both WebSphere MQ and GSkit (that is, to install into global zone only), use ‘pkgadd -G‘.

Installing WMQ into the Global Zone and using in non-global ‘Sparse Root’ zones

Overview

The principle here is to have one instance of your executables (/opt/mqm) which is visible to all non-global sparse root zones. Each non-global zone would have it’s own /var/mqm and so it’s own queue managers. With each zone having it’s own IP address etc. this essentially gives you the appearance of multiple machines with WMQ, but all sharing the same installed executables.

Work required

Problem 1: Installing into the Global Zone does not create the necessary /var/mqm structure in non-global zones.
Workaround: The /var/mqm directory structure can be copied from the global zone to a non-global zone. Note that the security permissions will need to be carefully checked to ensure that they are copied correctly. Queue manager data files (i.e. contents of /var/mqm/qmgrs, with the exception of /var/mqm/qmgrs/@SYSTEM) should not be copied – having multiple copies of the same queue manager running in different zones should not be attempted, as it would almost certainly cause problems.
Problem 2: The installer ensures that an ‘mqm’ user and ‘mqm’ group are created. As the installer is not being run in the non-global zones, these user and group ids are missing.
Workaround: An mqm user and group should be manually created in each non-global zone, and these used in security permission settings for /var/mqm/ files and directories in the zone.

Installing WMQ into a non-global ‘Whole Root’ zone

Overview

This is the most straight-forward non-global zone option. It allows WMQ to be used from within the zone, but puts nothing into the global zone that would cause it to visible outside of the zone in which it is installed. Installs in different zones would not share anything.
It is unappealing in some ways because it requires such a large zone – the default (having only a subset of the root) is to have much smaller zones.

Work required

Problem 1: The WebSphere MQ postinstall script attempts to verify if kernel parameters are set correctly. Commands used to do this are not available within non-global zones, and fail with error messages displayed at install-time

## Executing postinstall script.
Insufficient privileges to load a module
Insufficient privileges to load a module
Insufficient privileges to load a module

Workaround: These error messages can be ignored. This occurs post-install, and does not effect the install itself. Kernel parameters should be verified manually using guidance available in the WMQ v6 product README.

Installing WMQ into a non-global ‘Sparse Root’ zone

Overview

Note 1: On a system where WebSphere MQ is already installed in the global zone (when the non-global zone is created).
Note 2: Assuming a default zone configuration – therefore one without it’s own copy of /usr.
When running the installer in the non-global zone, the files that pkgadd is normally unable to create (files in /usr) are already present due to the installation in the global zone. As a result, pkgadd just verifies that the properties are okay and continues, allowing the installation to complete successfully.

Work required

Problem 1: If WebSphere MQ is uninstalled from the global zone, the non-global zone installation will stop working (Attempts to run WMQ commands fail, displaying error messages reporting that WMQ is unable to open message catalog.)
Workaround: None.
Problem 2: As files in /usr are shared between the different installations, installing different versions of WebSphere MQ in the global and non-global zones will fail, as the files will not be consistent.
Workaround: None.
Problem 3: Attempts to uninstall from non-global zone fail to complete, as access is denied when it tries to remove files from /usr. The non-global zone installation remains in a ‘partially installed’ state.
Workaround: Retry uninstall to remove entry from package list. Files in /usr will remain untouched.

Installing WMQ into a custom zone configuration

Overview

Installing into a custom zone configuration – based on a Sparse Root approach but including a local copy of /usr.
This is configured in zonecfg using: remove inherit-pkg-dir dir=/usr

Work required

## Executing postinstall script.
Insufficient privileges to load a module
Insufficient privileges to load a module
Insufficient privileges to load a module

Installing WMQ into a non-global ‘Sparse Root’ zone

Overview

Note 1: On a system where WMQ is not already installed in the global zone when the non-global zone is created
Note 2: Assuming a default zone configuration – therefore one without it’s own copy of /usr.
This is a complicated approach. It has the benefit of being the most efficient in regards to the filesize of the zone – as the zone includes only a minimum subset of the root.

Work required

Problem 1: Multiple errors are displayed reporting access denied when we try to write to read-only directories in /usr. The installation fails (status shown as ‘partially installed’).
Problem 2: Attempts to run WMQ commands fail, displaying error messages reporting that WMQ is unable to open message catalog.
Workaround: This requires modification to the installation scripts:

Unpack a local copy of the installation files to filespace available to the local zone

cp -rf mq_installer_directory/ /zones/comet/FileSystems/RootDir/root/tmp/.

If you do not have ‘mqm’, ‘gsk7bas’ and ‘gsk7bas64′ directories, and only have a .img file, then unpack it using pkgtrans

bash-3.00# pkgtrans /tmp/mq_installer_directory/installer.img /tmp/mq_installer_directory/.
bash-3.00# ls
PreReqs       READMEs       gsk7bas       gsk7bas64
lap           licenses      mqlicense.sh  mqm
readadd.txt   silent

Create an mqm user and group within the non-global zone
```
groupadd mqm
useradd -g mqm mqm
```
Move the message catalog files to model the intended new location (e.g. /myzoneusr/lib/locale)
1. Move mqm/root/usr to mqm/root/myzoneusr
```
cd /tmp/mq_installer_directory
mv mqm/root/usr/ mqm/root/myzoneusr
```
Modify mqm/pkgmap
1. Alter lines which contain a path beginning /usr/lib/locale to a location within the zone (e.g. /myzoneusr/lib/locale)
```
vi /tmp/mq_installer_directory/mqm/pkgmap
:%s/\/usr\/lib\/locale/\/myzoneusr\/lib\/locale/g
```
2. Remove all remaining lines which contain a path beginning /usr (soft-links and man pages)
```
vi /tmp/mq_installer_directory/mqm/pkgmap
:g/\/usr/d
```
Modify mqm/install/postinstall
1. Alter lines which contain a path beginning /usr/lib/locale to a location within the zone (e.g. /myzoneusr/lib/locale)
```
vi /tmp/mq_installer_directory/mqm/install/postinstall
:%s/\/usr\/lib\/locale/\/myzoneusr\/lib\/locale/g
```
Modify mqm/install/preremove
1. Alter lines which contain a path beginning /usr/lib/locale to a location within the zone (e.g. /myzoneusr/lib/locale)
```
vi /tmp/mq_installer_directory/mqm/install/preremove
:%s/\/usr\/lib\/locale/\/myzoneusr\/lib\/locale/g
```
Modify gsk7bas/pkgmap
1. Remove all remaining lines which contain a path beginning /usr (soft-links and man pages)
```
vi /tmp/mq_installer_directory/gsk7bas/pkgmap
:g/\/usr/d
```
Modify gsk7bas64/pkgmap
1. Remove all remaining lines which contain a path beginning /usr (soft-links and man pages)
```
vi /tmp/mq_installer_directory/gsk7bas64/pkgmap
:g/\/usr/d
```

Run the mqlicense script

/tmp/mq_installer_directory/mqlicense.sh

Make an attempt to install mqm (only mqm – not GSkit) from within the non-global zone using pkgadd
```
pkgadd -d /tmp/mq_installer_directory/.
```
1. This attempt will fail because changes to mqm/install/postinstall and mqm/install/preremove will mean that the filesize and checksum value for these files stored in mqm/pkgmap are no longer correct. This will cause pkgadd to report these files as corrupt.
```
pkgadd: ERROR: packaging file </tmp/mq_installer_directory/mqm/install/postinstall> is corrupt
    file size <10797> expected <11175> actual
    file cksum <8181> expected <50643> actual
pkgadd: ERROR: packaging file </tmp/mq_installer_directory/mqm/install/preremove> is corrupt
    file size <2659> expected <2731> actual
    file cksum <20944> expected <29032> actual
```
2. Use the pkgadd error message as a guide to correct the filesize and checksum values in mqm/pkgmap. The pkgadd error message reports ‘expected X, found Y’, and this can be used to search for and correct the values adjacent to ‘postinstall’ and ‘preremove’.
```
vi /tmp/mq_installer_directory/mqm/pkgmap
:%s/postinstall 10797 8181/postinstall 11175 50643/g
:%s/preremove 2659 20944/preremove 2731 29032/g
```
Install WMQ from within the zone using pkgadd using the corrected files
```
pkgadd -d /tmp/mq_installer_directory/.
```
1. Ensure that Manual Pages are not included in the install packages list – as these have been removed from the package map
Modify environment variables for users intended to be used with WebSphere MQ to include WMQ paths
```
export PATH=/opt/mqm/bin:/opt/mqm/samp/bin:/opt/ibm/gsk7/bin/:$PATH
```

Modify NLSPATH to point at custom location for message catalog

export LANG=C
export NLSPATH=/myzoneusr/lib/locale/%L/LC_MESSAGES/%N

So there are lots of ways to try and install WebSphere MQ with Solaris Zones!

I don’t think any of us would claim that what we have now is an ideal situation, and obviously to have a more transparent installation story in the future would be much better. But at the moment, with a little care it is certainly possible to run queue managers in Zones in various different configurations to suit a variety of needs.

Are you using any of these approaches at the moment? Or have you found another way to do it? Comments (even anonymous ones) would be very welcome, as it would be useful for us to know how people are using Zones and what more we could do to make this easier.

10 comments

Comments feed for this article

March 2, 2007 at 2:07 am

Dale Lane

The official support statement for the use of WebSphere MQ with Solaris Zones can be found on ibm.com.

Some of the outcome of my investigation from last year (that I discuss in this post) was used as the basis for this support statement, so you may recognise some of the discussion there.

March 3, 2007 at 12:18 pm

protocol7 » Blog Archive » links for 2007-03-03

[...] WebSphere MQ with Solaris Zones (tags: wmq ibm websphere solaris zones by:dale_lane) [...]

March 3, 2007 at 2:49 pm

Anonymous

Applying service would require all zones to have all qmgrs shutdown, if they are sharing /opt/mqm. And there’s probably a bunch of other issues around applying service – can you just apply it once to the core zone? What if something in the service update happened to change the contents of /var/mqm (which would need to be applied to all zones).

March 4, 2007 at 2:37 pm

Dale Lane

You’ve raised an interesting point – it’s all well and good to talk about the implications of Zones on installing WebSphere MQ, but the initial install is only part of the story. Applying maintenance does need consideration.

To address your first point, you’re quite right – sharing a single set of WMQ executables is a double-edged sword. It has the benefit that you’ve only got to look after one installation, and only need to apply maintenance in one place. However, it does leave you vulnerable to losing access to all queue managers in all Zones at once in the event of errors or scheduled maintenance. There is likely to be lessons that can be learnt here from other HA approaches to WMQ. This is not something that I’ve got much experience with so others may be able to provide more insight here.

As for problems with changes to /var/mqm not being applied to all Zones, that is an interesting question. I don’t think that we make changes to /var/mqm in maintenance. At first glance, the only thing that I can think of that could cause problems would be changes to internal structures – leading to inconsistencies such as in queue manager shared memory directories. But if queue managers are stopped before applying maintenance, then I think this should be fine. And utilities such as amqiclen can help if one gets into trouble.

I’ll have to look into this some more. In the meantime, it would be interesting to hear if anyone has any experience in using WMQ in Zones and has come across these (or any other) issues.

March 5, 2007 at 9:14 am

Russell Finn

The approach taken with Refresh Packs (CSDs) is that we don’t get the upgrade process to change any data under /var/mqm. Instead, we update the executables and if they detect that some data needs to be upgraded then it is done the first time the executable is used.

However it isn’t impossible that we would want to update files under /var/mqm as part of an upgrade. For example, if new code pages became available, we might want to update the file /var/mqm/conv/table/ccsid.tbl I don’t know if we’ve actually ever updated this file under a CSD though, but potentially we might want to one day.

May 8, 2007 at 2:07 am

Anonymous

Is there a best practice that we can recommend out of all this?

May 8, 2007 at 10:02 am

Dale Lane

I guess it depends on what you mean by “best practice”.

“Which is the most effective zones configuration?” seems like a tough question to answer, because it depends on what you want to achieve with your zones. If you see each different approach to setting up zones as being a way to implement a solution to a different problem, then the best zones config is the one that helps you solve your problem.

I’m not sure that there should be a “best zones configuration for WebSphere MQ”. In my opinion, WMQ should not overly direct your decisions on how to set up zones – rather you should identify which approach to Zones is the solution to your particular scenario, and then get WMQ to fit in with that.

Okay, so this is perhaps an unrealistic ideal at the moment – as my post shows that some zone configs are a little more complex to get WMQ working in than others.

If by “best practice” you mean which is the “easiest zones configuration for WebSphere MQ”, then I suppose a global zone or a non-global ‘Whole Root’ zone are the easiest.

Does this sound like too much of a cop-out? Is there any better guidance that we could give?

May 9, 2007 at 7:44 am

Andy Piper

No, I think you’re spot on there, Dale – it’s also my opinion that WMQ should not dictate a zone topology.

On a broader note, I don’t think there is a “best practice”, for most things, ever. There are good practices that fit situations. There are practices and configurations that are known to have worked under certain situations. “Best practice” implies that there is only one, allegedly “best”, way to do something. In many cases this is not true – there are several configuration options, and one may fit better than others.

September 25, 2007 at 7:31 pm

George Efsen

I have been asked to participate in a proof of concept where we will be using Solaris zones. I will be responsible to install and configure WMQ to run in this architecture. The policy at this point is that we will NOT be installing applications in the Global zone (including WMQ). We will be going with Sparse zones and will look at what customization will be necessary to make all the applications work as part of the POC.

Is there anymore information as to the success of installs in the Solaris zones that is available since March? I found the position statement for support on the IBM website and the brief statement in regards to Solaris zones in the WMQ v6 Info Center.

The POC will be starting in the next week or so, I need to hit the ground running. Modifying the install scripts seems to be what will fit our initial requirements, but was not included in the support position statement, not to mention a lot of work going forward. I can see that CSD’s and follow on releases would not be pleasant.

January 7, 2008 at 10:53 am

Rik Baeten

I’ve had the following issue when installing WMQ into a non-global ‘Sparse Root’ zone without anything installed in the global zone.

I’m running an application with a separate app user that connects to mq.

When I launch the WMQ binaries that are setuid mqm:mqm as the app user, I get the following error:
================
AMQ6090: WebSphere MQ was unable to display an error message for message id
hexadecimal %6, with inserts %1, %2, %3, %4 and %5.
AMQ7047
================

Using truss I’ve found that the the NLSPATH environment variable is not used.

On the web I’ve found “The GNU C library ignores NLSPATH for setuid/setgid programs” for security reasons:
http://www.linux.com/base/ldp/howto/Secure-Programs-HOWTO/locale.html

I suspect the Solaris C library does the same thing.

You must be logged in to post a comment.

WebSphere MQ with Solaris Zones

Configurations

Installing into all Zones at once

Overview

Work required

Installing WMQ into the Global Zone

Overview

Work required

Installing WMQ into the Global Zone and using in non-global ‘Sparse Root’ zones

Overview

Work required

Installing WMQ into a non-global ‘Whole Root’ zone

Overview

Work required

Installing WMQ into a non-global ‘Sparse Root’ zone

Overview

Work required

Installing WMQ into a custom zone configuration

Overview

Work required

Installing WMQ into a non-global ‘Sparse Root’ zone

Overview

Work required

a

Archives

Links

Meta

10 comments