WebLogic 12.1.2 Per Domain Nodemanger Command Line Arguments

Stepping out of any WebLogic admin training you will know that before WebLogic 12.1.2 we could start the node manager (NM) using its listen address and its port number as command-line arguments overwriting the settings in the nodemanager.properties file.

This is still possible, but doesn’t work OOTB for the per domain NM which was introduced in 12.1.2.

The reason is, that the start script in

DOMAIN_HOME/bin/startNodeManager.sh

is calling

WL_HOME/server/bin/startNodemanager.sh

The script located in WL_HOME is still okay, but the one in DOMAIN_HOME doesn’t forward the command-line parameters. So if you just want to be able to run it as before, then replace

${WL_HOME}/server/bin/startNodeManager.sh

with

${WL_HOME}/server/bin/startNodeManager.sh ${1} ${2}

in the startNodeManager.sh in DOMAIN_HOME/bin.

Have a great day!

ps. Are you observing this 12.1.2 NM bug? Let me know if you don’t.

WebLogic SNMP UDP Port – Misleading Error Logs

Have you lately tried to use SNMP with WebLogic? I did, because I upgraded an earlier Windows-based domain to WebLogic 12.1.2 running on CentOS Linux.

SNMP Configuration

WebLogic is showing some strange and undesired behaviour here… Firstly, to get started with SNMP basically all you need to do is to enable the SNMP agent.

WebLogic SNMP Agent

 

This is good enough when running under Windows. Then you might specify a trap destination such as localhost 4000 and run the WebLogic command-line tool to listen to the destination and verify if your setup is working.

WebLogic Trap Destination

 

The command line to run the trap listener looks like this

$ java weblogic.diagnostics.snmp.cmdline.Manager SnmpTrapMonitor -p 4000
Listening on port:4000

Now, of course you (well, in this case me) forgot to modify SNMP UDP Listen Port and the Master AgentX Port. For sure you know that you cannot bind to ports less than 1024 in UNIX.

WebLogic Bind Exception

What happens

  • Interesting enough, there is an error message in the Weblogic startup – it is just pointing you to the wrong direction since it complains about port 171 not being available. Now, you might wonder, port 171 is configured nowhere!
  • Apparently WebLogic is trying 161 and the following 10 port numbers, eg. to avoid conflicts when several servers are running on a single machine (nothing to complain about, this behavior is documented, just click on “help”).
  • If all ports are unavailable (or not allowed for non-root users in UNIX), WebLogic does not complain about the configured port not available, but about the last one it tried. Not such a good solution…
  • Looking at the Master AgentX port, the situation is different again. Of course the restriction for binding to ports less than 1024 still applies, but there is no auto-increment on WebLogic. Unfortunately, the error message in the log file does not even mention the port number.

To conclude, when specifying a SNMP port number that you cannot bind to, you might see an error message containing a different port number than the one you configured for SNMP UDP Listen Port.
For the Master AgentX Port you will see a rather generic message, not containing the problematic port number at all.

Both is good to know and leaves some space for improvement.

<Sep 21, 2013 8:53:54 AM CEST> <Error> <SNMP> <BEA-320951> <Failed to initalize the SNMP AgentX master subagent. One possible cause is the failure to bind the Master agentX port. If this is the case, check to see if the specified AgentX port is available, or ensure that the process has permissions to bind to the desired port.
weblogic.diagnostics.snmp.agent.SNMPAgentToolkitException: java.net.BindException: Permission denied
 at weblogic.diagnostics.snmp.agent.monfox.MonfoxToolkit.initializeMasterAgentX(MonfoxToolkit.java:186)

 


What happened with the Web Service Test Client in WebLogic 12.1.2 ?

Trying to run the Web Service Test Client as you used to do it until WebLogic version 12.1.1 fails in WebLogic 12.1.2. It took me a while to find out why.

My 12.1.2 domain was running in development mode, I deployed a web service but could not access the web service test client. The admin console did not display the link under Deployment / Testing and also the typical URL /wls_utc didn’t work.

Web Service Test Client / Console in WebLogic 12.1.2

Now here comes the news.

New URL

First of all, the link to access the test client has changed in a way which is hard to spot. No word about this in the release notes. To access the web service test client use the following new URL:

http://localhost:7001/ws_utc

(until 12.1.1 it was wls_utc)

tc1

Testing from Console is BROKEN

Of course then you should be able to access the test client using the admin console via Deployments / Testing. Surprisingly this didn’t work for a really simple web service I used for testing. For exactly the same web service, only deployed in 12.1.1 domain, the console displays the testing link.

So not sure if it was only my example, which couldn’t be any simpler, but the testing link was not displayed.

testcl2

UPDATE: I was trying this with more different web services. I did not work with any of my examples. I looks like a bug to me. I recommend to ask your friendly Oracle consultant if it is a feature.

Using Web Service Test Client from an URL

Although the bug described above, the good news is that you can still use the

http://ADMIN_SERVER:PORT/ws_utc/?wsdlURL=XYZ

for testing, eg.

http://localhost:7001/ws_utc/?wsdlUrl=http://localhost:7001/Timing/DelayService?WSDL

More Features

Those of you who know the possibilities of the SOA suite test client (part of EM) or the OSB test client might have missed some more features in the WLS web service test client.  The test client coming with 12.1.2 is overhauled (or new?) and comes with a number of additional possibilities like settings for authentication, WS-Adressing, WS-Atomic TX etc. (see the tabs in the first screenshot above).

Let me know if it is working for you.

WebLogic 12.1.2 Node Manager Bug

While working with WebLogic 12.1.2 I discovered the following bug. Not sure though if the same would happen with 10.3.X.

Try the following:

- Create a domain with one managed server

- Configure a machine, then start the per domain node manager. Using the admin console start the managed server. All working lovely so far.

- It’s lunch time. Stop node manager. Then stop the managed server using the admin console. This shouldn’t be a big deal.

- After lunch: start admin server and node manager. Node manager reports managed server is running (which is not correct!).

- Try to start managed server and it will report an error message that it is in incompatible state. :(

bug

 

Check the video below to reproduce it (just a quick recording, not a nice one):

 

1.) Interesting enough the server.PID file is removed when the managed server is shut down. However when restarting the node manager the managed server it is detected as running.

2.) With the node manager stopped, when forcing the managed server to shut down via admin console, ms1.state file shows FORCE_SHUTTING_DOWN, whereas when forcing the managed server to shut down (same button) with node manager running, the state is SHUTDOWN.

 

As you could guess, I wanted to spend my morning with sth else.
Comments welcome! Lunchtime for me now.

 

AWS outage destroys EBS-based AMIs in Europe region

I always recommend to create your own EBS-based AMIs (e.g. for running complex software such as Oracle Fusion Middleware). This hold true for the classic AMIs as well as for the converted Oracle VM templates. Never rely on the existence of AMIs provided by Oracle because:

- Oracle can change or update (or remove) them any time.

- They often don’t exist for certain AWS regions, they are S3-based or only exist based on 32-bit OEL instead of 64-bit.

- Also, the AMIs provided often don’t exist for a specific version of Oracle products.

So always create your own copy! Yet here is something to consider:

AWS broke an EBS-based AMI of mine by deleting arbitrary block in the image. This is particularly annoying since there is no easy way to create an offline copy an EBS-based AMI. You could rsync the running image to local computer but there is absolutely no support to get this done in a user-friendly way from the AWS console.

The good: They informed me in time (being in Sydney if something happens in the EU regions gives you an advantage) and sent an apology. They also replaced the deleted blocks with empty blocks.

The bad: It cost me several days to create this AMI which was an OEL EBS-based, full-blown installation of Oracle SOA Suite 11.1.1.5 (I still have to check if it will be usable after a file system check).

For a more detailed explanation of what happened take a look at Amazon’s summary of the events. It summarizes to an error in the EBS software that was overlayed with a power outage in Dublin.

Hello Amazon: Why don’t you provide an easy way to have an offline backup of EBS-based AMIs for disaster recovery?