I saw Dale’s Diagnosing Triggering Problems post, and thought I’d add a checklist I put together.

The idea was to give some help diagnosting common triggering issues when setting up a new triggered queue for an app-team, or diagnosing a queue-depth alert at 2am on a callout.

The Conditions for a Trigger Event section in the APG is the definitive reference, but this checklist is written more from a diagnosis point of view.

The document has a UNIX bias I’m afraid…

The Checklist

This checklist applies when messages are building up on a queue, and you would normally expect an application to be triggered by WMQ to process these messages.

There are a number of checks to perform (you are suggested go through each of these in order).
A possible action is provided for each check, if an issue is highlighted.

After performing the action, you may need to nudge WMQ to re-check the criteria for a trigger event to occur, and trigger the application if the issue is resolved.
To do this, alter the queue to disable and then re-enable triggering:

  • ALTER QL(TRIGGERED_QUEUE) NOTRIGGER
  • ALTER QL(TRIGGERED_QUEUE) TRIGGER

1. Triggering must be enabled, and of the correct type:

Check:

  • Display the TRIGGER TRIGTYPE and TRIGDPTH attributes of the queue to determine the type of triggering that is enabled:
    DIS QL(TRIGGERED_QUEUE) TRIGGER TRIGTYPE TRIGDPTH
    Sample output:

    AMQ8409: Display Queue details. 
       QUEUE(TRIGGERED_QUEUE)                  TYPE(QLOCAL) 
       TRIGGER                                 TRIGDPTH(1) 
       TRIGTYPE(FIRST)

Action:

  • If NOTRIGGER is seen in the output, then triggering is disabled. Enable it as follows:
    ALTER QL(TRIGGERED_QUEUE) TRIGGER
    In most cases you can expect the type of triggering (TRIGTYPE) to be FIRST.
    A summary of the three types is as follows:

    • EVERY
      Trigger events occur every time a message is delivered to the queue. This occurs even if applications already have the queue open to retrieve messages.
    • FIRST
      Trigger event occur when the depth of the queue transitions from zero to one.
    • DEPTH
      Trigger events occur when the depth of the queue transitions from
      (TRIGDPTH-1) to TRIGDPTH.

      Triggering is automatically disabled on the queue, and must be re-enabled by the application (using MQSET).

2. No application can already have the queue open to get messages:

This only applies to FIRST or DEPTH triggering.
For EVERY triggering, a trigger event occurs every time a message arrives, regardless of how many applications are already actively retrieving messages from the queue.

Check:

  • Check the IPPROCS attribute in the QSTATUS for the initiation queue:
    DIS QS(TRIGGERED_QUEUE) IPPROCS
    Sample Output:

    AMQ8450: Display queue status details. 
       QUEUE(TRIGGERED_QUEUE) 
       TYPE(QUEUE)                             CURDEPTH(0) 
       IPPROCS(0)
  • You can find which application has the queue open (IPPROCS>0) as follows:
    DIS QS(TRIGGERED_QUEUE) TYPE(HANDLE) APPLTAG PID
    Sample Output:

    AMQ8450: Display queue status details. 
       QUEUE(TRIGGERED_QUEUE) 
       TYPE(HANDLE) 
       APPLTAG(/home/myuser/bin/mycmd) PID(1234)

Action:

  • For FIRST or DEPTH triggering, if the IPPROCS number is any number greater than zero, then an application has the queue open to retrieve messages and triggering will not occur.
  • If the application is not the one expected to run against this queue, then it must end before the intended application can be triggered.
    The name and PID of the application is shown in the QSTATUS output. If the name contains ‘amqcrsta’ or ‘amqrmppa’, then a remote client application has the queue open (this could be an administrator on another machine).

    If an ‘amqrmppa’ process is terminated (kill -9 PID), this will shut down any other channels running within that process.

  • If the application connected to the queue is the triggered application for this queue, then the application team should be involved to investigate why the application is failing to process messages from the queue.

3. The initiation queue must be valid:

Check:

  • Display the INITQ attribute of the triggered queue:
    DIS QL(TRIGGERED_QUEUE) INITQ
    Sample Output:

    AMQ8409: Display Queue details. 
      QUEUE(TRIGGERED_QUEUE)                    TYPE(QLOCAL) 
      INITQ(SYSTEM.DEFAULT.INITIATION.QUEUE)
  • Display the initiation queue to check it exists, is not an XMITQ, and is not inhibited for put or get:
    DIS QL(SYSTEM.DEFAULT.INITIATION.QUEUE) USAGE PUT GET
    Sample Output:

    AMQ8409: Display Queue details. 
       QUEUE(SYSTEM.DEFAULT.INITIATION.QUEUE) 
       TYPE(QLOCAL)                      GET(ENABLED) 
       PUT(ENABLED)                      USAGE(NORMAL)

    Remember to use single ticks for queues with lower case names:
    DIS QL('inititaion.queue') USAGE PUT GET

Action:

  • If the INITQ attribute is blank, or incorrect, then change it to the name of the initiation queue against which the trigger monitor will run.
  • If the initiation queue is inhibited for put, then enable it as follows:
    ALTER QL(SYSTEM.DEFAULT.INITIATION.QUEUE) PUT(ENABLED)

4. A trigger monitor must be active against the initiation queue:

Check:

  • Check the IPPROCS attribute in the QSTATUS for the initiation queue:
    DIS QS(SYSTEM.DEFAULT.INITIATION.QUEUE) IPPROCS
    Sample Output:

    AMQ8450: Display queue status details. 
       QUEUE(SYSTEM.DEFAULT.INITIATION.QUEUE) 
       TYPE(QUEUE)                             CURDEPTH(0) 
       IPPROCS(1)

    IPPROCS is the number of applications that have the queue open for input, or browse. This should be exactly 1.

  • Check the app which has the queue open is the Trigger Monitor:
    DIS QS(SYSTEM.DEFAULT.INITIATION.QUEUE) TYPE(HANDLE) APPLTAG
    Sample Output:

    AMQ8450: Display queue status details. 
       QUEUE(SYSTEM.DEFAULT.INITIATION.QUEUE) 
       TYPE(HANDLE) 
       APPLTAG(/opt/mqm/bin/runmqtrm)

Action:

  • The trigger monitor can be started in the background as follows on UNIX:
    nohup runmqtrm -m QMGR_NAME -q SYSTEM.DEFAULT.INITIATION.QUEUE >/dev/null 2>&1 &

5. The process definition must be valid

Check:

  • Display the PROCESS attribute of the triggered queue:
    DIS QL(TRIGGERED_QUEUE) PROCESS
    Sample Output:

    AMQ8409: Display Queue details. 
       QUEUE(TRIGGERED_QUEUE)                  TYPE(QLOCAL) 
       PROCESS(TRIGGERED_PROCESS)
  • Ensure the process object exists, and validate the command which will be run by the trigger monitor when the trigger event occurs:
    DIS PROCESS(TRIGGERED_PROCESS) APPLICID
    Sample Output:

    AMQ8407: Display Process details. 
       PROCESS(TRIGGERED_PROCESS)     APPLICID(/home/myuser/bin/mycmd)

Action:

  • If the PROCESS attribute is blank, or incorrect, then change it to the name of a valid process object.
  • If the APPLICID attribute of the process object is incorrect, then alter it to correctly match the command which should be run.

    Remember that on UNIX machines commands are case-sensitive, so must be entered in single ticks when defining/altering a process object in MQSC:
    DEF PROCESS(TRIGGERED_PROCESS) APPLICID('/home/myusr/bin/mycmd')

6. The trigger monitor must be able to start the application

Check:

  • If the application fails to start an application, it places the trigger message on the dead-letter queue, with a feedback code in the ‘Reason’ field.

    The Application Programming Reference manual describes the possible MQFB_* feedback codes

  • You can use the WebSphere MQ Explorer (or MO71) to view the message:
    • Connect to the queue manager using the WebSphere MQ Explorer.
    • Browse the contents of the dead-letter queue for the queue manager.
      The WMQ Explorer understands the format of the DLH header that is added to a message when it is placed on the DLQ.
  • Look at the ‘Reason’ field in the dead-letter header

    If it is displayed in numeric form, the following link to the Constants manual should help to translate it to a MQFB_* code:

Action:

  • Investigate and resolve the problem starting the application.

7. The application must connect to the queue after it is started

Check:

From a WMQ administrator’s point of view, you can check whether the application is being started by the trigger monitor. Further investigation of the application’s operation would be performed by inspecting the application’s own logging/diagnostic info.

If you need to absolutely confirm that an application is being started by a trigger monitor, then the following process can be performed to restart the trigger monitor in the foreground (so you can see the output on your screen):

  1. The best way to stop a trigger monitor is to inhibit the initiation queue for get:
    ALTER QL(SYSTEM.DEFAULT.INITIATION.QUEUE) GET(DISABLED)
  2. You can then immediately un-inhibit get on the queue:
    ALTER QL(SYSTEM.DEFAULT.INITIATION.QUEUE) GET(ENABLED)
  3. You can now run the trigger monitor as a foreground process, so you can monitor the output:
    runmqtrm -m QMGR_NAME -q SYSTEM.DEFAULT.INITIATION.QUEUE
  4. You will now need to open a new terminal (telnet/ssh session)
  5. 5. Disable and re-enable triggering on the queue to see the output
    from the trigger monitor when a trigger message is processed:
    ALTER QL(TRIGGERED_QUEUE) NOTRIGGER
    ALTER QL(TRIGGERED_QUEUE) TRIGGER
Advertisements