WebLogic Overload Protection and OFM

[NEW in 2011: For more details please take a look at my book Middleware and Cloud Computing.]

Even when running your application in the cloud with the most careful capacity planning in place, autoscaling enabled, carefully tuned, well-written and load tested applications without any design flaws you better prepare yourself for instantaneous growth.

There is number of core WebLogic settings as well as various settings for the individual subcomponents of WebLogic such as JMS or JDBC that enable you to limit the effect of excessive load. I recommend to use the following enumeration of topics as a checklist for your own WebLogic settings. All of these settings apply to non-cloud environments as well.

The basic settings make sense for other Oracle Fusion Middleware products  as well which are running on top of WebLogic. Examples for these products are Oracle BPM, Service Bus, Service Registry and so on.

Some of these settings are documented at Oracle as well, but most are scattered throughout the PDFs.

Certainly there is much more to overload protection in the cloud such as dealing with offensive traffic from attackers, system architecture issues such as distributed JMS in the cloud or a service oriented application design that allows to dynamically disable non-critical parts of your application if  Armageddon is close to peak.

This article is a shortened excerpt from my upcoming cloud computing book.

Enable WebLogic Administration Port

Enabling the administration port is not the same as setting a port number for the admin server. Enabling the administration does the following: It reserves a thread and a separate port number for all administration communication within a  WebLogic server domain, enables SSL and disables non-SSL administration communication. Using the administration port feature increases the likelihood that admin server communication will be functional under high load.

Workmanager Capacity Constraint

WebLogic uses work managers with a variable and self-tuning number of worker threads. There is a default work manager but you can define your own work manager and assign a particular application or even a part of it such as a JSP to your custom work manager. When configuring a custom work manger you can add restrictions such as the minimum or maximum number of threads, a fair-share usage policy or a capacity constraint. The capacity constraint defines the maximum number of requests that can be queued or are executing at any given point in time.

Incoming requests that exceed the number of execute threads will be queued.

Incoming requests over the capacity constraint are rejected and result in a “503- Service Unavailable” response code for web applications. This capacity constraint can be shared across multiple work managers.

Maximum Request Queue Length

You can define a maximum queue length shared across all work managers with the setting “Shared Capacity For Work Managers” field in the WebLogic admin console. The default value of this field is 65536. This setting does not apply for the administration port, so you will not risk the access to the admin server even if the maximum number of requests queued is reached.

Maximum Thread Setting

Although the work managers use a self-tuning thread pool it is still possible to limit the upper bound of the pool. Note, that in general I reckon that the self-tuning work manger is doing fine and I do not recommend to set a maximum number of threads. However if your load test reveals that an excessive amount of threads will make your system slow or unstable, you could try to run your load test with a maximum constraint for the thread pool.

There is no way to set the maximum thread count directly from the WebLogic admin GUI, but you can provide an additional startup argument in your server start script:

-Dweblogic.threadpool.MaxPoolSize=500

You can achieve the same by editing the WebLogic config.xml in the config subdirectory of your WebLogic domain. Add the &ltself-tuning-thread-pool-size-max> element with the maximum number of threads to the <server> element. As always, make a backup copy and stop the admin server first before editing the config.xml because a running admin server will overwrite your changes.

Maximum Heap Setting and Panic Setting

Define the maximum heap size for the JVM with the –Xmx parameter. The maximum heap should never exceed the available physical memory in your machine since pageing for virtual memory will slow the system down extremely.

Define which percentage of free heap triggers an out of memory situation in the WebLogic admin console under Configuration / Overload. The “Panic Action” setting defines what action will be taken if an out of memory situation occurs. The default setting is “Ignore, take no action”, but you can change it to “Exit the server process” and let the nodemanager restart your server.

Restrict the number of HTTP sessions

For a WebLogic web application you can limit the maximum number of HTTP sessions created by setting the max-in-memory-sessions tag within the session-descriptor of the weblogic.xml file. Otherwise creating more and more sessions due to user requests can eventually cause an out of memory. When this number is exceeded, an weblogic.servlet.SessionCreationException is thrown for further attempts This setting applies to both replicated and non-replicated in-memory sessions.

Define JMS quota

Limit the number of pending JMS messages on a particular destination (queue or durable topic) by specifying a quota. Use a quota resource that defines byte and messages maximums and assign the quota to the destination.

There is also quota for destinations that do not explicitly set a value, these destinations share the quota of the JMS server

Specifying a Blocking Send Policy on JMS Servers

Specify for blocking sends whether all send requests for a particular destination are queued until space is available (FIFO setting). Then, no send request is permitted to complete when there is another send request is waiting for space.

With the preemptive setting a blocking send can preempt other blocking send operations if there is sufficient space available.

JMS Message Buffer Size

The Message Buffer Size option specifies how much of the heap memory JMS will consume to store message bodies before they are paged out to disk. There is a default for this setting of one-third of the maximum heap size for the JVM, or a maximum of 512 megabytes.

Writing JMS messages to disk will slow down JMS but prevent an out of memory. You trade in performance for stability.

Maximum Number of JDBC Database Connections

Set the maximum number of connections to the value determined by load testing the application (maximum number determined during load test plus some headroom). Set the initial size of the connection pool to the number of used connections.

Note, that the WebLogic JDBC pinned to thread feature is particularly dangerous in overload situations. With pinned to thread enabled for a connection pool the connections are not returned to the pool but remain attached to the execute thread. The pinned to thread feature will save connection wait time if there is a high competition for database connections for a busy connection pool but the number of database connections increase beyond the maximum number of connections set for the connection pool.