The post has seemed very popular, so I thought I might be helpful to follow it up with some thoughts on diagnosing problems when using data conversion.
The document Saket pointed at is pretty much the definitive reference document, but in this post I will try and add some general observations, and describe a simple technique for diagnosing data conversion problems if things go wrong.
The most common cause of problems relating to data conversion is that messages do not have the expected CCSID values. To start with, let’s recap the two main ways to set message CCSIDs:
Applications setting the CCSID
When putting a message, the “putting” application can set the CCSID field in the message descriptor.
The “getting” application can set the same CCSID field in the message descriptor when it gets the message.
Note that WebSphere MQ returns the actual CCSID of the message when MQGET returns. Often, this is the CCSID in the MQGET request. However, in the result of a conversion error, MQGET returns an unconverted message, with the CCSID field of the original message in the message descriptor.
A common error in WMQ application programs is to not set a requested CCSID in the message descriptor before every MQGET, resulting in the unexpected use of a different CCSID overwritten by a previous failed MQGET.
Using queue manager defaults
When putting a message, the “putting” application can set the CCSID field in the message descriptor to
MQCCSI_Q_MGR. This means the queue manager’s character set should be used – “the queue manager changes this value in the message descriptor to the true character-set identifier of the queue manager“.
The “getting” application can set the CCSID field in the message descriptor to
MQCCSI_Q_MGR. MQGET will return the converted message together with the CCSID used.
This approach is simple, and useful when you are transferring messages in one language and one CCSID.
But what about transferring messages using multiple languages and CCSIDS?
The Data Conversion document referred to above describes some approaches, and is a good place to start.
In general, a good plan is to convert data into Unicode as soon as possible – allowing you to do the bulk of your work in a “language neutral” way.
In situations where this is not an option, a good way to process mixed language CCSIDs is to use separate queue managers, or at least separate queues, per language. In this way, you can predetermine the CCSID to use.
An alternative approach is to use BROWSE to get each message first, with the expectation that it will fail due to a conversion error. This means that the message will not be removed from the queue, allowing a second MQGET to be safely carried out with the correct CCSID value – obtained from the message descriptor returned by the failed browse.
Similar to an earlier post of mine on handling truncated messages, care needs to be taken to ensure that the second MQGET gets the same message that is got by the initial BROWSE using MQGMO_BROWSE_MSG_UNDER_CURSOR, and MQGMO_LOCK to prevent another application from removing the message inbetween MQI calls.
Debugging data conversion problems
Any such approach is inevitably complex, and customers wanting to mix languages in a WMQ system need to carefully review their overall system design.
In the event that this results in unexplained data conversion errors, the following is one way to debug the problem.
- Stop the sending channel.
- Modify the “putting” application to dump the CCSID in the message descriptor together with some of the message – ideally in hexadecimal.
- Run the “putting” application so that it sends a message which causes a conversion problem. (The stopped channel means that the message will end up on the transmission queue.) Store the dumped CCSID and message contents from the application.
- Browse the transmission queue (with data conversion disabled) so that you can view the contents. Store the CCSID and the message contents there.
- Compare the dump from the application with the contents of the transmission queue.
- Disable the “getting” application, then start the sending channel so that the message is taken from the transmission queue.
- Browse the destination queue (with data conversion disabled) so that you can view the contents. Store the CCSID and message contents there.
- Compare the contents of the destination queue with the previous output
- If the CCSID has changed, this will be because the channel has data conversion enabled. This is a common source of problems – it always converts to the receiving queue mangers CCSID which is not always the intended behaviour. It is often more straightforward to perform conversion by the MQGET issued by the “getting” application – rather than altering messages before they reach the destination queue.
- Modify the “getting” application to dump the CCSID it sets in the message descriptor before the MQGET call, the CCSID in the message descriptor returned by the MQGET call, and some of the message returned – ideally in hexadecimal.
- Compare the output with previous output.
Following these steps is often enough to identify the point at which the problem occurs.
This can either uncover a problem in your configuration, or provide useful information if you need to seek further assistance from WMQ Service.