WebLogic 12c Proxy Plugin for Web Servers is missing (kinda solved)

According to the documentation for the brand new WebLogic 12c the WLS mod_wl proxy plugin for web servers should be located under the following location:

$WL_SERVER/server/plugin

and this is where the trouble starts:

  • This directory does not exists for WebLogic 12c.
  • You can download the plugins seperately from here. Yet there is no version for WebLogic 12c either.
  • You must not simply use an lower plugin version (e.g. the one that comes with WebLogic 11g) because lower versions of the plugin e.g. 10.3.5 are not guaranteed to work (only higher versions). Now, this is between you me: yes they do work typically.  Yet I remember at least one situation when I spent over a day debugging an issue because an older plugin version was used. I am pretty sure you can imagine serveral way to spend your time better.
  • The behavior for the Apache mod_wl 10.x plugins is known to be rather strange after a a managed server drops out of the dynamic server list (clustered or not).

 

Here are some possible solutions:

I briefly checked the 10.3.5 plugin for Apache 2.2 with a WebLogic 12c cluster and it seems to work okay so far. Actually even the glitch with the failing load balancing after a server drops out of a cluster has improved. Probably that’s the best solution currently, but remember it’s at your own risk.

Follow up with my OTN thread and some valuable input from René and Sandeep.

For a more critical production environment I recommend to open a support ticket with Oracle. If you do so, please post an update here and share your findings.

 

So how are you using WebLogic 12 in production? If you do, how is your clustering set up?

 

UPDATE  / Jan 2013:

Thanks to input from my friend Rene I am aware now that Oracle officially recommends to use the latest 10.3.x plugin even for WebLogic 12c. Still it is causing a lot of confusion and the WLS 12.1.1 documentation is wrongly stating that the plugin is part of the WebLogic distribution.

 

Select “Generally Available System Configurations”:

http://www.oracle.com/technetwork/middleware/ias/downloads/wls-1211x-certmatrix-1395408.xls

 

 

Win a Copy of Middleware and Cloud Computing Book

If you are interested in Middleware and Cloud Computing subscribe to my my newsletter and win one of the first two copies of my brand new cloud computing book. The drawing will be Dec. 15th 2010.

There are only 4 mailings per year and you can unsubscribe any time from the newsletter.

Oracle Middleware and Cloud Computing Book

WebLogic Whole-Server Migration on EC2 Cloud?

WebLogic whole-server migration is a compelling solution for the availability of servers and pinned resources such as JMS  on those servers. It simply moves the IP address to the new candidate machine before starting the managed server there. From the perspective of the managed server there is no change at all, since even the IP address is the same. The crucial point however is the availability of floating IP addresses that can be automatically reassigned by the wlsifconfig.sh script.

On EC2 there are only two IP addresses per instance, a public and a private one. You cannot get additional IP addresses, so whole-server migration isn’t of much use.

In fact, you could get an elastic IP address, which is a reassignable, static IP address assigned to an EC instance. Still an elastic IP is not useful for server-migration because assignment of the address requires the EC2 command-line and cannot be done directly from the cluster script using ifconfig.

This text is a digest of my upcoming book about cloud computing with Oracle.

RackspaceCloud Support for MultiCast?

A WebLogic server cluster used to use IP multicast (MC) for cluster heartbeats and global JNDI updates. Today there is another option with Unicast. Still large cluster deployments benefit from multicast. Often using MC is problematic, there is a whole list of possible problems with Windows, routers and firewalls swallow the MC packets (by design, unless you configure them otherwise) and on Unix it could be disabled for the NIC. Apart from WLS clustering it is used by products such as TIBCO EMS for failover.

I had an excellent discussion with the support team of RackspaceCloud about using IP multicast. “There is no obvious reason why it shouldn’t work“, they told me, but at the end there was some doubt left. Since I know AWS doesn’t support MC on EC2 I decided to give it a try.

How to reproduce

  1. Launch two instance in RSC. Should take some 3 minutes. 1 went for a 1GB Win2008 because I wanted to check for poetential Win problems. Note: Currently you cannot create an image from a Windows server, that’s why I started two images from the very beginning, otherwise it makes more sense to prepare one and then clone it.
  2. I downloaded Mozilla FireFox because InternetExplorer drives my crazy with all its security fuss when doing a simple proof of concept.
  3. Download and install WLS10.3.2
  4. I turned off the Win firewall just in case. I don’t wanted the firewall to block the MC packets.
  5. I shared my WebLogic installation directory and copied it over. Yes, you can do that in the RS cloud and at least for a POC it saves a lot of time.
  6. Open a cmd prompt cd to server\bin and run the setWLSEnv to set the environment:

    C:\Oracle\Middleware\wlserver_10.3\server\bin>setWLSEnv.cmd

  7. Repeat step 2 to 6 on the other instance, then run the MC test utility on both sides using a different name with -N, e.g use -N Tom on the other side:
    C:\Oracle\Middleware\wlserver_10.3\server\bin>java utils.MulticastTest -n
    Frank -A 237.0.0.1 -P 8001
    ***** WARNING ***** WARNING ***** WARNING *****
    Do NOT use the same multicast address as a running WLS cluster.
    Starting test.  Hit any key to abort
    Using multicast address 237.0.0.1:8001
    Will send messages under the name Frank every 2 seconds
    Will print warning every 600 seconds if no messages are received
     I (Frank) sent message num 1
    Received message 2 from Frank
     I (Frank) sent message num 2
    Received message 3 from Frank
     I (Frank) sent message num 3
    Received message 4 from Frank
     I (Frank) sent message num 4
  8. If MC was working correctly you’d expect to see your own messages as well as the messages from the remote side, but we just seeing the local ones.

Conclusion: there is no support for Multicast between two instances in the RackspaceCloud. The RSC support team officially confirmed that later as well :” Unfortunately, multicasting is disabled on our hypervisors to preserve network sanity. We apologize for the inconvenience.

So what?

Note: Oracle states MC is not working for AWS either and the preferred solution is Unicast (which is acceptable IMHO).

WebLogic 10, 11g and WebLogic 12c and Apache mod_wl Plugin Problems

Unfortunately there is a problem with the Apache plugin for WLS10.3.1. It is rather annoying because it spoils the WebLogic cluster experience as such and even worse: it spoils my distributed cluster demo. I reproduced it working with different teams using WLS 10.3.1 / Windows 32 bit / Apache with mod_wl_20 and also mod_wl_22. And typically I encourage all my customers to create a support case concerning this.

Let me explain what happens. Typically the symptoms are the following: In a distributed cluster setup with three managed server m1, m2, and m3 you configure the plugin for load balancing which initially seems to work fine. You can monitor your requests being distributed in a round-robin way, e.g. m2, m3, m1, m2, m3, m1, m2, m3, … Then, if you kill one managed server, let’s say m2, the pattern changes to m1, m3, m1, m1, m1, m1, m1,…. So the load balancing breaks.

UPDATE:

There is a bug now mentioned in the release notes of WLS 10.3.2 which is pointing towards the same direction but the description is rather vague.

UPDATE2:

Last week I could verify the same behaviour on two clusters with WLS 10.3.3 under Windows. I’d recommend you to try it with your Windows installation. It is reproducible even on one machine running three managed servers.

UPDATE3 WebLogic 10.3.5 and WebLogic 12c:

I thought it would be interesting to verify if the error shows up for WebLogic 12c as well – in contrast to the documentation there is currently no proxy plugin with the WebLogic 12c (12.1.1) distribution. So I used the 10.3.5 version for Linux 32 bit and guess what? The behavior is different! It is not perfect (as it used to be in the good old WLS9 days) but Oracle does have a sense of humor apparently. When running 3 managed servers I observe m1,m1,m1,m2,m2,m2,m3,m3,m3,m1,m1,m1,m2… Interesting, hmm? So it load balances but kind of addresses on server n times. Now here is the good surprise: After killing e.g. m2, I see the following pattern: m1,m1,m1,m3,m3,m3,m1,m1,m1,m3…  So unlike with the previous 10.x versions there is still load balancing but the plugin keeps assigning a particular server more than once.

UPDATE^2 WebLogic 10.3.5 Apache 2.2 plugin used for WebLogic 12c: I have to correct myself. When trying WLS 10.3.5 plugin for Apache 2.2 on CentOS6 again in 2 different setups, it seems to work fine. I could’t observe the strange lb pattern anymore. So maybe Oracle has some sense of humor but they don’t express it in the way the plugin load balances your request. Better like this ;)