reached." and on MOM this is seen: "com.wily.isengard.message.MessageUndeliverableEx"" /> reached." and on MOM this is seen: "com.wily.isengard.message.MessageUndeliverableEx"" />

This message is frequently seen in Collectors: "Timed out adding to outgoing message queue. Limit of reached." and on MOM this is seen: "com.wily.isengard.message.MessageUndeliverableEx"

Document ID:  TEC1606598
Last Modified Date:  06/13/2017
{{active ? 'Hide' : 'Show'}} Technical Document Details

Products

  • CA Application Performance Management

Releases

  • CA Application Performance Management:Release:10.0
  • CA Application Performance Management:Release:10.1
  • CA Application Performance Management:Release:10.2
  • CA Application Performance Management:Release:9.5
  • CA Application Performance Management:Release:9.5.1
  • CA Application Performance Management:Release:9.5.2
  • CA Application Performance Management:Release:9.5.3
  • CA Application Performance Management:Release:9.5.6
  • CA Application Performance Management:Release:9.6
  • CA Application Performance Management:Release:9.6.1
  • CA Application Performance Management:Release:9.7
  • CA Application Performance Management:Release:CA APM 9.5
  • CA Application Performance Management:Release:CA APM 9.6
  • CA Application Performance Management:Release:CA APM 9.7
  • CA Application Performance Management:Release:10.3
  • CA Application Performance Management:Release:10.5
  • CA Application Performance Management:Release:10.5.1
  • CA Application Performance Management:Release:10.5.2

Components

  • INTROSCOPE:APMISP
Introduction:

  Frequently this message is seen in Collectors, where XXXX generally shows a number anywhere from 3000 and up.

  "Timed out adding to outgoing message queue. Limit of XXXX reached"

  Which is followed by this message on the MOM:

  com.wily.isengard.message.MessageUndeliverableException: Outgoing mailbox is closed. Message cannot be sent

  Other symptoms can include:

    -Agents disconnecting and moving to other Collectors

    -MOM disconnecting Collectors

    -Workstation and/or Webview running slow

  

  The above generally points to a performance issue.  But we will concentrate more on the outgoing message queue.

Background:

The Enterprise Manager uses a "message queue" to communicate back and forth between EM's.  So when you encounter a "Timed out adding to outgoing message queue. Limit of XXXX reached" message, this means that the message queue for the EM's communication protocol has filled up and no new messages will be added.  This causes the Collector to cease sending out messages and all queries from the MOM to that Collector will error out.  Outgoing delivery threads are fixed in number and shared by all the clients. These threads deliver messages from the outgoing message queue to the client, and only when there are no outgoing messages left in current queue move on to the next queue.

  Some mail threads may be executing a large number of queries marking them busy on the connected Collectors.  In this situation, the mail threads must wait for locks that are required to execute the requests.  Since this is also blocked for a response until it times out, all of the mail threads will slow down as a result.  

  To alleviate this situation, there are 3 properties that work together.

    transport.outgoingMessageQueueSize

    This property is a hidden property inside the IntroscopeEnterpriseManger.properties file.  This sets the size of the messaging queue.  This property is based on the number of messages, not the size of the message content.  If you increase this property value, you should also increase the heap size as the higher the number, the more RAM it uses.  Basically the higher the value you use, the more resources it takes up.

 

    transport.override.isengard.high.concurrency.pool.min.size

    This property determines the minimum number of available outgoing delivery threads.  We recommend to set the minimum and maximum the same.  By doing this, we eliminate the adding/removing of pools when needed.  These important adjustments are required when, sending messages, usually between collectors and MOM, becomes a bottleneck for performance.  However, as in the outgoing setting above, the higher the value, the more resources are taken up.

    transport.override.isengard.high.concurrency.pool.max.size

    This property determines the maximum number of available outgoing delivery threads.  By increasing the maximum pool size, this allows more worker threads to send outgoing messages concurrently.  These important adjustments are required when, sending messages, usually between collectors and MOM, becomes a bottle neck for performance.  However, as in the outgoing setting above, the higher the value, the more resources taken up.

The transport.override.isengard.high.concurrency.pool.max.size property in the IntroscopeEnterpriseManager properties file determines the maximum number of available outgoing delivery threads. 

 

 In a healthy cluster, the delivery threads run smoothly and it doesn't take long for a filled outgoing message queue to have space freed up for more messages. Delivery threads take messages off the outgoing message queues. If there are fewer delivery threads than active client connections, outgoing messages must wait in the queue, and the queue can reach its capacity. When there is no free space in the outgoing message queue, the Enterprise Manager cannot send requested data back to the client. This can result in serious Enterprise Manager performance problems. Increasing the queue size and thread pool size provides more capacity and can resolve this situation.  These important adjustments are required when, sending messages, usually between collectors and MOM, becomes a bottleneck for performance, but care must be taken when making the adjustments.

Environment:
All APM versions on all supported operating systems.
Instructions:

The following are the defaults for 9.x and 10.x versions of the Enterprise Manager, if not specified in the IntroscopeEnterpriseManager.properties file or commented out:

    transport.outgoingMessageQueueSize=3000

    transport.override.isengard.high.concurrency.pool.min.size=5

    transport.override.isengard.high.concurrency.pool.max.size=5

 

  These settings are considered low for the current and most recent versions and/or if your cluster has a very heavy load.

  The recommended settings are below.

    transport.outgoingMessageQueueSize=8000

    transport.override.isengard.high.concurrency.pool.min.size=14

    transport.override.isengard.high.concurrency.pool.max.size=14

 

  The settings above can be increased but keep in mind the higher you increase them to, the more resources you will take up and can wind up choking the EM.  Applying any changes above requires a restart of the EM.

 

  NOTE: These settings should be applied to MOM and all Collectors.

  If, after applying the above settings, you do not see any improvement, then there is more you can check on for your performance issue.  See links below.

  Note: Please keep in mind that you may encounter different recommended values.  This is because, over time with Introscope, and as clusters grow, the recommended values need to be increased.

Additional Information:

Please help us improve!

Will this information enable you to resolve your issue?

Please tell us what we can do better.

{{feedbackText.length ? feedbackText.length : '0'}}/255

{{status}}

Not what you were looking for?

Search Again >

Product Information

Support by Product >

Communities

Join a Community >

Chat with CA

Just give us some brief information and we'll connect you to the right CA ExpertCA sales representative.

Our hours of availability are 8AM - 5PM CST.

All Fields Required

connecting

We're matching your request.

Unfortunately, we can't connect you to an agent. If you are not automatically redirected please click here.

  • {{message.agentProfile.name}} will be helping you today.

    View Profile


  • Transfered to {{message.agentProfile.name}}

    {{message.agentProfile.name}} joined the conversation

    {{message.agentProfile.name}} left the conversation

  • Your chat with {{$storage.chatSession.messages[$index - 1].agentProfile.name}} has ended.
    Thank you for your interest in CA.


    Rate Your Chat Experience.

    {{chat.statusMsg}}

agent is typing