wls12c — munz & more

Review: WebLogic 12c Distinctive Recipes Workshops in AUS (Brisbane, Adelaide, Perth)

June 7, 2013 by frank 1 Comment

Reviews and Attendee’s Opinions

My WebLogic 12c training courses in AUS were a great success so I’d like to share the reviews with you. My special thanks go to AUSOUG president John Bushell for making this possible and Burke Scheld for organizing the events.

what Did you like about the workshop?

I got some really nice feedback after the presentation, here is all the answers.
Tricks and Tips
4/22/2013 3:31
Info about the non-core tools and tricks, very handy to get to hear about little odd things like that.
4/22/2013 3:24
Useful hints and tips, and a good overview of WebLogic 12c
4/22/2013 3:18
Useful open source software and administration tips
4/20/2013 2:13
It covered some left of field topics and points – some of the things I was hoping for.
4/19/2013 1:46 PMView
Good tips on tuning. Good opensource tool suggestions.
4/19/2013 11:44
Grand
4/18/2013 5:35
Tips and tricks on weblogic server management.
4/18/2013 4:10
Live demos, interesting content
4/18/2013 3:04
Opportunity to see things that are new to me in a class room environment
4/18/2013 2:55
Good independently sourced information about Weblogic features and capabilities. The idea of using virtual images helps to set a test environment quickly. The expertise of the presenter makes him capable of providing information that is beyond what you can find on the official Oracle docs.
4/18/2013 2:47
Good presentation, good explanations, aimed at the right level for me.
4/18/2013 2:16
We can discuss the good, as well as criticise the bad, features of Oracle software.
4/18/2013 2:02
Frank provided information about some useful tools that I wasn’t aware of. The overview of the new features in 12 was good. The configuration tips were handy. The cost of the workshop was very reasonable for the information gained from it.
4/18/2013 1:54
To get an overview of technologies and to discuss those.
4/18/2013 1:51
Well presented with references to resources for continuing help
4/18/2013 1:49
Nice pens from the presenter. And the idea of giving away a free book.
4/11/2013 2:59
Interesting insite into some open source tools that enhance the administration features of WebLogic
4/11/2013 2:43
Frank obviously knows his stuff, and backs up recommendations with excellent real world examples.
4/10/2013 1:26
Very well presented. Topics were relevant and current

Comments about presenter:

Good presenter, and well prepared. Always interesting
4/22/2013 3:18
Experienced and keen to answer questions and offer advice
4/20/2013 2:13
Good relaxed presentation with time to digest the material – thankyou well done
4/19/2013 1:46 PMView
Frank was a great presenter and his style was very professional. English was very good and understandable.
4/19/2013 11:44
Knowlegable, responsive, good communicator
4/19/2013 1:21
Frank had good experience and knowledge of weblogic server.
4/18/2013 4:10
Frank Munz was knowledgeable, concise and to the point
4/18/2013 3:04
Friendly, knowledgeable
4/18/2013 2:55
Well prepared, you can tell that he has a long experience with the platform
4/18/2013 2:47
Very good, knowledgeable, easily understood
4/18/2013 2:16
Frank is an excellent presenter and very knowledgeable about almost every aspects of Weblogic software.
4/18/2013 2:02
I liked Frank’s relaxed presentation style and the depth of his knowledge of the product.
4/18/2013 1:54
Excellent guy who knows what he’s talking about, broad range of experience and good examples.
4/18/2013 1:51
excellent presentation style
4/18/2013 1:49
I like how he is not biased towards a brand of technology but instead rationalises upon technology choice depending on immediate/long term goals.
4/11/2013 2:59
Great presentation – really informative and easy to follow.
4/22/2013 3:24
Good presenter, though he gets a little carried away reiterating key points
4/11/2013 2:43
Frank is organised and knowledgeable and presents with a comfortable, easy to listen to style.
4/10/2013 1:26
Very nicely presented
4/10/2013 12:34

More?

Learn about my internal WebLogic 12c training offerings.

Filed Under: oracle, WebLogic 12c Book Tagged With: weblogic, wls12c, workshop

Oracle Coherence with WebLogic 12c

April 2, 2013 by frank 1 Comment

Collecting material for the next edition of my WebLogic 12c book I began writing a primer about Oracle Coherence from a WebLogic perspective. Coherence is part of the WebLogic Suite and you can start using it right away.

Many WebLogic professionals still consider Coherence as something complicated and challenging to get started with, so I made a number of hands-on Coherence primer screencasts available – also more blog postings with best practices will follow.

Make sure to subscribe to WebLogic book YouTube channel: http://www.youtube.com/weblogicbook

Basic Introduction

Coherence is a coherent data grid that provides a number of different cache implementations, which are well described here. The following figure illustrates how data A. B, C is stored into different nodes with the same put(key, value) method.

Oracle Coherence Basics, Multicast Test and Firewall Issues

Oracle Coherence Command Line, Replication and Availability

Development with NetBeans Introduction

Oracle Coherence NetBeans API and NamedCache

Coherence ActiveCache with WebLogic

Coherence ActiveCache can be used to offload session data from WebLogic into Coherence cache nodes. These cache nodes can be managed with WebLogic Nodemanager. A detailed explanation can be found here.

Oracle Coherence ActiveCache for WebLogic 12c configuration

Oracle Coherence ActiveCache Demo for WebLogic 12c

Filed Under: oracle Tagged With: coherence, video on youtube, wls12c

Oracle Service Bus (OSB) for the Busy IT Profesional

March 26, 2013 by frank 1 Comment

Audience

Oracle Service Bus is one of these software products that haven’t changed much in their core since many years. These days I tend to see more and more OSB projects and one reason certainly is that license wise OSB is included with Oracle SOA Suite.

Compared to SOA Suite, OSB is conceptually different though. Even with a good understanding of SOA Suite (which I will cover in one of the following posts), there will be a slightly steep, but rather short learning curve for OSB.

The following recipes tries to motivate and explain the usage of OSB without the usual marketing hype. Make sure to read the previous posting about getting started with SOA in general and Oracle Fusion Middleware.

Technology

OSB is virtualizing service calls from clients to the service implementation. By using so called proxy and business services (both are components within OSB) incoming and outgoing requests can be arbitrarily connected and protocols can be converted. Proxy services contain the processing logic (validation, enrichment, transformation etc.).

Download

If you are trying to get started with OSB, here is the link to download Oracle Service Bus Overview and Getting Started (PDF).

Enjoy!

PS. Actually this recipe is taken out of my book Oracle WebLogic Server 12c – Distinctive Recipes.

Filed Under: oracle, WebLogic 12c Book Tagged With: best practice, oracle service bus, weblogic, wls12c

Simple Custom JMX MBeans with WebLogic 12c and Spring

July 25, 2012 by frank Leave a Comment

Using the Spring framework bundled with WebLogic 12 the programming of JMX MBeans becomes as easy as writing a simple POJO within a web application. After coding the MBean as a POJO all that is left to do is creating the Spring configuration file and editing the web.xml deployment decriptor:

- Firstly you have to add the Spring <listener> to the web.xml deployment descriptor with the web.xml to get Spring started.

- Secondly create the applicationContext.xml for Spring in the WEB-INF subdirectory. Declare the Spring bean and let Spring export it to the platform MBean server.

- Thirdly deploy the web application and test it with JConsole, WLST or j4psh.

For your convinience I already built an example. You can download the deployable SimpleJMX.war file or the complete NetBeans project.

You can see the application in action, including the involved files and the deployment in the following screencast:

How to access custom MBeans from WLST is shown in the second part:

The third part is showing how to retrieve MBean attributes from a JSP:

Filed Under: oracle, WebLogic 12c Book Tagged With: admin, jmx, monitoring, video on youtube, weblogic, wls12c

Amazon / Oracle Cloud Workshops: Mission Completed

July 20, 2012 by frank Leave a Comment

I am just back from a successful, long and intensive trip to Australia. If you are waiting for a reply to an email there is a good chance you will get it soon now.

I delivered an Amazon and Oracle public cloud workshop series on behalf of AUSOUG, based on my cloud computing book: 6 cities, 6 workshops of 4 h each, almost 200 attendees, hands-on, live development in the cloud (WebLogic with 61 GB heap deployed live on 3 continents, cloud storage, auto scaling, transformation of a classical Java EE app in the cloud), 16 flights (including a 1 week trip to the outback, flying 2,5 h north of Perth and driving 2200 km for shooting the cover image for my upcoming WebLogic 12c: Advanced Recipes book).

Check out the flattering reviews for the event.

thanks to everybody who attended!

Filed Under: amazon, CloudComputing, oracle Tagged With: aws, book, cloud computing, ec2, weblogic, wls12c, workshop

EJB 3.1: Stateless Session Bean Deployed as .war, Dependency Injection, Asynchronous Methods

June 4, 2012 by frank 5 Comments

Development of an EJB 3.1 stateless session bean has become incredibly simple. You add an annotation, there is no XML required at all, no need to implement an interface anymore, and you can easily implement asynchronous methods that decouple your client calls from the method execution. Then, at the end you deploy the EJB as a simple .war file – no need to warp it in an .ear file.

The following screencasts explain you how to do it. You can also download the whole NetBeans project or just a deployable .war file exactly the sources created in the screencasts.

Simple Stateless Session Bean with asynchronous method, developed with NetBeans 7.1 and deployed as .war file:

Asynchronous Methods with Stateless Session Bean:

Inject Stateless Session Bean into Servlet:

Here is a tough question at the end:

Imagine you have to add a WebLogic specific settings for your stateless session bean. There is no annotation for that. These settings have to be put in the deployment descriptor which typically used to be the weblogic-ejb-jar.xml file within an .ear file. But what if you have already build a .war file which is perfectly possible now (and neither an .ear nor an ejb.jar file).

Where do you put these settings then? Post your answer here.

Filed Under: oracle, WebLogic 12c Book Tagged With: ejb3.1, netbeans, simple example, video on youtube, wls12c

Web Service Development with NetBeans and Testing with WebLogic Admin Console

May 23, 2012 by frank Leave a Comment

Developing of SOAP based web services for WebLogic 12c (or any other application server) with NetBeans is a breeze. You can do it without using any templates or IDE wizards, no XML is required, no modification of deployment descriptors.

The following screencast shows how to develop the most simple web service pobbislbe, how to test it entirely with WebLogic admin console, and how modifying the web service will work without redeployment.

Filed Under: oracle, WebLogic 12c Book Tagged With: development, netbeans, web service, weblogic, wls12c

Another Myth Debunked: More than 200 Continuous Redeployments with WebLogic

May 9, 2012 by frank 4 Comments

The other day I overheard the myth that you simply cannot redeploy several times with WebLogic because inevitably you would end up having an out of memory error. Since I am currently writing on an WebLogic 12c book I decided to verify that.

Certainly this depends a lot on your heap settings and what your application is doing, eg. it is easier to get it wrong if you are writing your own classloaders etc. I certainly admit that these problems, if they occur, are particularly hard to sort out.

Hands-on Test

More important you should detect them early. So go and test it! It’s a simple 10 line WLST script. I have done it for you to verifiy if there is a general problem with redeployment in WebLogic.

I redeployed the MedRec application over 200 times. MedRec comes with the WebLogic 12c installation (only if you decide to install the samples). It consists of over 20 EJBs, a web part and a few web services.

The sample domain can be found in the WL_HOME/samples/domains/medrec directory.

Download

A little WLST script that you can download does the trick (it’s a quick hack, you can certainly beautify and add error handling to it!). Download it (it comes zipped) and unzip it to the domain directory. It even contains comments about how to run it.

The script deploys the MedRec application (so undeploy it first!), waits for 30 seconds, undeploys, waits again and deploys again. At the same time it prints the number of loaded classes, heap and non-heap runtime values.

Results

I was running this experiment for 200 redeployments. Apart from the output of the program, I recorded some data plots with jconsole.

The non-heap usage after 200 redeployments is shown on the following screenshot.

Everything was fine. I guess I could also do 1000 redeployments. I just stopped it because I got bored. I believe it would go on forever and I needed to shut down my computer.

Also here is the heap usage:

So test your own application deployment! It’s an easy test. IMHO it should be run for every important application. Add it to your tests when load testing the application.

Also I recommend to create some load for the app while it’s deployed. You may want shorten the interval between the deployments to a few seconds to get more deployments done faster.

Let me know if it breaks in your case

Also stay informed about other exciting topics covered in the my WebLogic 12c book. I just created a Youtube channel with many WebLogic 12c screencasts (more to come every week).

Filed Under: oracle Tagged With: admin, book, deploymnet, weblogic, wls12c

WebLogic Scripting Tool on Capsaicin: j4psh JMX Shell with Syntax Highlighting, Wildcards and Tab-Completion

April 24, 2012 by frank Leave a Comment

[This posting will be a part of my upcoming WebLogic 12c book]

Kudos Oracle! Over a year ago I mentioned in my Oracle Middleware and Cloud Computing book that developing WebLogic Scripting Tool (WLST) scripts with vi or Notepad++ is so 80′ies. It took you only a couple of months to respond with an Eclipse facet. Now everyone can use the bundled Eclipse with OEPE for WLST scripting . Thanks for that – even if you had it on your to-do list anyway

WLST itself is a fantastic tool to interactively explore WebLogic MBean world with all their attributes, operations, and relationships (it’s described here - did I mention that this post is not for the beginners?). WLST is using its own proprietary syntax to navigate to MBeans though.

Are you willing to look beyond your own backyard? Ready to move to real MBean names? Then let’s spice it up.

j4psh JMX Shell

j4psh is a JMX shell which is using standard MBean names, provides syntax highlighting together with code completion and even wildcard patterns! In j4psh you will be using similar command as in WLST. Here is an incomplete list of the most useful commands:

connect, to connect to an Java EE server with the jolokia.war deployed or using the Java agent
pwd, to find out the current MBean name or domain
cd, to navigate to an MBean or domain
ls, to list MBean details (works with * and can be restricted as “ls -a” for attributes and “ls -o” for operations).
cat, to display attribute values

Unlike WLST, where every command is basically a Python method, the j4psh syntax is as easy as it gets.

For the bigger picture how j4psh can be downloaded, installed and how nicely it fits into the Jolokia open source project see my previous posting.

Screencasts

In my opinion j4psh is easiest to understand when seen in action, so here are two examples of j4psh sessions.

1.) Basic usage of j4psh with WebLogic. The JMX “HelloWord”: enabling verbose garbage collection without server restart.

2.) Advanced Usage of j4psh with WebLogic: How to find the right MBean for a JDBC connection? Retrieve JMX attributes from JDBC connection pool, then use the MBean Name for the same RESTful request.

Give it a try! It’s free anyway. And let me know what you think!
And by the way, after writing about open source Jolokia, I am now determined to grow a real Jolokia plant myself…

Update: Many thanks to the Jolokia team for the plant!!

Filed Under: oracle Tagged With: jolokia, monitoring, RESTful, weblogic, wls12c

RESTful WebLogic Monitoring of Servers, Applications, JDBC and JMS with Jolokia

April 19, 2012 by frank 5 Comments

[This posting will be a part of my upcoming WebLogic 12c book]

This article is part II of III. Make sure you read part I explaining the basics of monitoring WebLogic with WLST, the WebLogic 12c REST-ful management API and the open source framework Jolokia.

There will be a part III in a few of days, explaining how to use j4psh with WebLogic. j4psh is a JMX shell that resembles WLST in interactive mode but includes features such as syntax highlighting and tab-completion of commands and MBean names. j4psh is highly useful to interactively find out the correct MBean names used for the requests below.

WebLogic with Jolokia

Did you ever wonder how to retrieve the configuration values for a WebLogic managed server, runtime data of a JDBC connection pool or the number of messages in a particular JMS queue with a fast and simple HTTP GET?

The examples show typical use cases for monitoring a WebLogic domain with the following configuration:

- Admin server running at localhost:7001
- Managed server with name surf1 running at localhost:7003
- JDBC datasource with the name emergencyDB and associated connection pool (target set to surf1)
- JMS configuration with a JMS server surfJMS running on managed server surf1, a module surfJMSModule containing a queue name jms/ShippingRequestQueue (target of JMS server is surf1, JMS module subdeployment set to surfJMS)
- A deployment of a web service with the name SubmitOrder and target AdminServer and surf1.

An installation of Jolokia with target set to all server of the domain is recommended to follow the examples in this part.

General: List Configuration Details of Managed Server with Name surf1

http://localhost:7001/jolokia/read/com.bea:Name=surf1,Type=Server?ignoreErrors=true

Note, that without the ?ignoreErrors=true parameter the request would fail with an “NoAccessRuntimeException” because you cannot access the SSL keystore passphrases without proper authentication. Authentication is easily possible, but beyond of the scope of this article.

{ "request" : { "mbean" : "com.bea:Name=surf1,Type=Server",
      "type" : "read"
    },
  "status" : 200,
  "timestamp" : 1334778534,
  "value" : { "AcceptBacklog" : 300,
      "AddWorkManagerThreadsByCpuCount" : false,
      "AdminReconnectIntervalSeconds" : 10,
      "AdministrationPort" : 9002,
      "AdministrationPortEnabled" : false,
      "AdministrationProtocol" : "t3s",
      "AutoKillIfFailed" : false,
      "AutoMigrationEnabled" : false,
      "AutoRestart" : true,
      "COM" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=COM" },
      "COMEnabled" : false,
      "CandidateMachines" : [  ],
      "ClasspathServletDisabled" : false,
      "ClientCertProxyEnabled" : false,
      "Cluster" : { "objectName" : "com.bea:Name=surfCluster,Type=Cluster" },
      "ClusterRuntime" : null,
      "ClusterWeight" : 100,
      "CoherenceClusterSystemResource" : null,
      "CompleteCOMMessageTimeout" : -1,
      "CompleteHTTPMessageTimeout" : -1,
      "CompleteIIOPMessageTimeout" : -1,
      "CompleteMessageTimeout" : 60,
      "CompleteT3MessageTimeout" : -1,
      "ConnectTimeout" : 0,
      "ConsensusProcessIdentifier" : -1,
      "ConsoleInputEnabled" : false,
      "CustomIdentityKeyStoreFileName" : null,
      "CustomIdentityKeyStorePassPhrase" : "ERROR: Access to sensitive attribute in clear text is not allowed due to the setting of ClearTextCredentialAccessEnabled attribute in SecurityConfigurationMBean. Attr: CustomIdentityKeyStorePassPhrase, MBean name: com.bea:Name=surf1,Type=Server (class weblogic.management.NoAccessRuntimeException)",
      "CustomIdentityKeyStorePassPhraseEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: CustomIdentityKeyStorePassPhraseEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "CustomIdentityKeyStoreType" : null,
      "CustomTrustKeyStoreFileName" : null,
      "CustomTrustKeyStorePassPhrase" : "ERROR: Access to sensitive attribute in clear text is not allowed due to the setting of ClearTextCredentialAccessEnabled attribute in SecurityConfigurationMBean. Attr: CustomTrustKeyStorePassPhrase, MBean name: com.bea:Name=surf1,Type=Server (class weblogic.management.NoAccessRuntimeException)",
      "CustomTrustKeyStorePassPhraseEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: CustomTrustKeyStorePassPhraseEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "CustomTrustKeyStoreType" : null,
      "DGCIdlePeriodsUntilTimeout" : 5,
      "DataSource" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=DataSource" },
      "DefaultFileStore" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=DefaultFileStore" },
      "DefaultIIOPPassword" : "ERROR: Access to sensitive attribute in clear text is not allowed due to the setting of ClearTextCredentialAccessEnabled attribute in SecurityConfigurationMBean. Attr: DefaultIIOPPassword, MBean name: com.bea:Name=surf1,Type=Server (class weblogic.management.NoAccessRuntimeException)",
      "DefaultIIOPPasswordEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: DefaultIIOPPasswordEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "DefaultIIOPUser" : null,
      "DefaultInternalServletsDisabled" : false,
      "DefaultProtocol" : "t3",
      "DefaultSecureProtocol" : "t3s",
      "DefaultTGIOPPassword" : "ERROR: Access to sensitive attribute in clear text is not allowed due to the setting of ClearTextCredentialAccessEnabled attribute in SecurityConfigurationMBean. Attr: DefaultTGIOPPassword, MBean name: com.bea:Name=surf1,Type=Server (class weblogic.management.NoAccessRuntimeException)",
      "DefaultTGIOPPasswordEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: DefaultTGIOPPasswordEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "DefaultTGIOPUser" : "guest",
      "ExecuteQueues" : [  ],
      "ExpectedToRun" : true,
      "ExternalDNSName" : null,
      "ExtraEjbcOptions" : null,
      "ExtraRmicOptions" : null,
      "FederationServices" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=FederationServices" },
      "GatheredWritesEnabled" : false,
      "GracefulShutdownTimeout" : 0,
      "HealthCheckIntervalSeconds" : 180,
      "HealthCheckStartDelaySeconds" : 120,
      "HealthCheckTimeoutSeconds" : 60,
      "HostsMigratableServices" : true,
      "HttpTraceSupportEnabled" : false,
      "HttpdEnabled" : true,
      "IIOP" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=IIOP" },
      "IIOPEnabled" : true,
      "IIOPTxMechanism" : "ots",
      "IdleConnectionTimeout" : 65,
      "IdleIIOPConnectionTimeout" : -1,
      "IdlePeriodsUntilTimeout" : 4,
      "IgnoreSessionsDuringShutdown" : false,
      "InstrumentStackTraceEnabled" : true,
      "InterfaceAddress" : null,
      "JDBCLLRTableName" : null,
      "JDBCLoggingEnabled" : false,
      "JDBCLoginTimeoutSeconds" : 0,
      "JMSDefaultConnectionFactoriesEnabled" : true,
      "JMSThreadPoolSize" : 15,
      "JNDITransportableObjectFactoryList" : [  ],
      "JTAMigratableTarget" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=JTAMigratableTarget" },
      "JavaCompiler" : "javac",
      "JavaCompilerPostClassPath" : null,
      "JavaCompilerPreClassPath" : null,
      "JavaStandardTrustKeyStorePassPhrase" : "ERROR: Access to sensitive attribute in clear text is not allowed due to the setting of ClearTextCredentialAccessEnabled attribute in SecurityConfigurationMBean. Attr: JavaStandardTrustKeyStorePassPhrase, MBean name: com.bea:Name=surf1,Type=Server (class weblogic.management.NoAccessRuntimeException)",
      "JavaStandardTrustKeyStorePassPhraseEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: JavaStandardTrustKeyStorePassPhraseEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "KernelDebug" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=ServerDebug" },
      "KeyStores" : "DemoIdentityAndDemoTrust",
      "ListenAddress" : "",
      "ListenDelaySecs" : 0,
      "ListenPort" : 7003,
      "ListenPortEnabled" : true,
      "ListenThreadStartDelaySecs" : 60,
      "ListenersBindEarly" : false,
      "Log" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=Log" },
      "LogRemoteExceptionsEnabled" : false,
      "LoginTimeoutMillis" : 5000,
      "LowMemoryGCThreshold" : 5,
      "LowMemoryGranularityLevel" : 5,
      "LowMemorySampleSize" : 10,
      "LowMemoryTimeInterval" : 3600,
      "MSIFileReplicationEnabled" : false,
      "Machine" : null,
      "ManagedServerIndependenceEnabled" : true,
      "MaxCOMMessageSize" : -1,
      "MaxHTTPMessageSize" : -1,
      "MaxIIOPMessageSize" : -1,
      "MaxMessageSize" : 10000000,
      "MaxOpenSockCount" : -1,
      "MaxT3MessageSize" : -1,
      "MessageIdPrefixEnabled" : false,
      "MessagingBridgeThreadPoolSize" : 5,
      "MuxerClass" : null,
      "NMSocketCreateTimeoutInMillis" : 180000,
      "Name" : "surf1",
      "NativeIOEnabled" : true,
      "NetworkAccessPoints" : [  ],
      "Notes" : null,
      "OutboundEnabled" : false,
      "OutboundPrivateKeyEnabled" : false,
      "OverloadProtection" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=OverloadProtection" },
      "Parent" : { "objectName" : "com.bea:Name=surfandconsulting,Type=Domain" },
      "PeriodLength" : 60000,
      "PreferredSecondaryGroup" : null,
      "ReliableDeliveryPolicy" : null,
      "ReplicationGroup" : null,
      "ReplicationPorts" : null,
      "RestartDelaySeconds" : 0,
      "RestartIntervalSeconds" : 3600,
      "RestartMax" : 2,
      "ReverseDNSAllowed" : false,
      "SSL" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=SSL" },
      "ScatteredReadsEnabled" : false,
      "SelfTuningThreadPoolSizeMax" : 400,
      "SelfTuningThreadPoolSizeMin" : 1,
      "ServerDebug" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=ServerDebug" },
      "ServerDiagnosticConfig" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=WLDFServerDiagnostic" },
      "ServerLifeCycleTimeoutVal" : 30,
      "ServerStart" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=ServerStart" },
      "ServerVersion" : "unknown",
      "SingleSignOnServices" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=SingleSignOnServices" },
      "SocketBufferSizeAsChunkSize" : false,
      "SocketReaders" : -1,
      "StagingDirectoryName" : "/u01/domains/surfandconsulting/servers/surf1/stage",
      "StagingMode" : "stage",
      "StartupMode" : "RUNNING",
      "StartupTimeout" : 0,
      "StuckThreadMaxTime" : 600,
      "StuckThreadTimerInterval" : 60,
      "SystemPasswordEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: SystemPasswordEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "TGIOPEnabled" : true,
      "ThreadPoolPercentSocketReaders" : 33,
      "TimedOutRefIsolationTime" : 0,
      "TransactionLogFilePrefix" : "./",
      "TransactionLogFileWritePolicy" : "Direct-Write",
      "TransactionLogJDBCStore" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=TransactionLogJDBCStore" },
      "TunnelingClientPingSecs" : 45,
      "TunnelingClientTimeoutSecs" : 40,
      "TunnelingEnabled" : false,
      "Type" : "Server",
      "UploadDirectoryName" : "./servers/surf1/upload",
      "Use81StyleExecuteQueues" : false,
      "UseConcurrentQueueForRequestManager" : false,
      "UseFusionForLLR" : false,
      "VerboseEJBDeploymentEnabled" : "false",
      "VirtualMachineName" : "surfandconsulting_surf1",
      "WebServer" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=WebServer" },
      "WebService" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=WebService" },
      "WeblogicPluginEnabled" : false,
      "XMLEntityCache" : null,
      "XMLRegistry" : null
    }
}

Deployment: List all Deployments
http://localhost:7001/jolokia/search/com.bea:*,Type=ApplicationRuntime
{ "request" : { "mbean" : "com.bea:Type=ApplicationRuntime,*",
 "type" : "search"
 },
 "status" : 200,
 "timestamp" : 1334780578,
 "value" : [ "com.bea:Name=mejb,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=consoleapp,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=bea_wls_management_internal2,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=emergencyDB,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=bea_wls_deployment_internal,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=SubmitOrder,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=wls-management-services,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=bea_wls_internal,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=jolokia,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=bea_wls9_async_response,ServerRuntime=AdminServer,Type=ApplicationRuntime"
 ]
 }
Deployment: List Deployments Details for an Application with the Name SubmitOrder
http://localhost:7001/jolokia/read/com.bea:ServerRuntime=*,Name=SubmitOrder,Type=ApplicationRuntime
{ "request" : { "mbean" : "com.bea:Name=SubmitOrder,ServerRuntime=*,Type=ApplicationRuntime",
      "type" : "read"
    },
  "status" : 200,
  "timestamp" : 1334777478,
  "value" : { "com.bea:Name=SubmitOrder,ServerRuntime=AdminServer,Type=ApplicationRuntime" : { "ActiveVersionState" : 2,
          "ApplicationName" : "SubmitOrder",
          "ApplicationVersion" : null,
          "ClassRedefinitionRuntime" : null,
          "CoherenceClusterRuntime" : null,
          "ComponentRuntimes" : [ { "objectName" : "com.bea:ApplicationRuntime=SubmitOrder,Name=AdminServer_/SubmitOrder,ServerRuntime=AdminServer,Type=WebAppComponentRuntime" } ],
          "EAR" : false,
          "HealthState" : { "critical" : false,
              "mBeanName" : null,
              "mBeanType" : null,
              "reasonCode" : [  ],
              "state" : 0,
              "subsystemName" : null
            },
          "KodoPersistenceUnitRuntimes" : [  ],
          "LibraryRuntimes" : null,
          "MaxThreadsConstraintRuntimes" : [  ],
          "MinThreadsConstraintRuntimes" : [  ],
          "Name" : "SubmitOrder",
          "OptionalPackageRuntimes" : [  ],
          "Parent" : { "objectName" : "com.bea:Name=AdminServer,Type=ServerRuntime" },
          "PersistenceUnitRuntimes" : [  ],
          "QueryCacheRuntimes" : [  ],
          "RequestClassRuntimes" : [  ],
          "Type" : "ApplicationRuntime",
          "WorkManagerRuntimes" : [ { "objectName" : "com.bea:ApplicationRuntime=SubmitOrder,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime" } ],
          "WseeRuntimes" : [ { "objectName" : "com.bea:ApplicationRuntime=SubmitOrder,Name=SubmitOrder!EntryWSProdService,ServerRuntime=AdminServer,Type=WseeRuntime" },
              { "objectName" : "com.bea:ApplicationRuntime=SubmitOrder,Name=SubmitOrder!EntryWSService,ServerRuntime=AdminServer,Type=WseeRuntime" }
            ],
          "WseeV2Runtimes" : [  ]
        } }
}

JDBC Data Source: Retrieve Settings for emergencyDB Data Source with Target Managed Server surf1 Running at localhost:7003
http://localhost:7003/jolokia/read/com.bea:Name=emergencyDB,ServerRuntime=surf1,Type=JDBCDataSourceRuntime?ignoreErrors=true
{ "request" : { "mbean" : "com.bea:Name=emergencyDB,ServerRuntime=surf1,Type=JDBCDataSourceRuntime",
      "type" : "read"
    },
  "status" : 200,
  "timestamp" : 1334823502,
  "value" : { "ActiveConnectionsAverageCount" : 0,
      "ActiveConnectionsCurrentCount" : 0,
      "ActiveConnectionsHighCount" : 1,
      "ConnectionDelayTime" : 44,
      "ConnectionsTotalCount" : 1,
      "CurrCapacity" : 1,
      "CurrCapacityHighCount" : 1,
      "DatabaseProductName" : "Apache Derby",
      "DatabaseProductVersion" : "10.6.1.0 - (938214)",
      "DeploymentState" : 2,
      "DriverName" : "Apache Derby Network Client JDBC Driver",
      "DriverVersion" : "10.6.1.0 - (938214)",
      "Enabled" : true,
      "FailedReserveRequestCount" : 0,
      "FailuresToReconnectCount" : 0,
      "HighestNumAvailable" : 1,
      "HighestNumUnavailable" : 1,
      "JDBCDriverRuntime" : { "objectName" : "com.bea:Name=surfandconsulting_surf1_org.apache.derby.jdbc.ClientDriver,ServerRuntime=surf1,Type=JDBCDriverRuntime" },
      "LastTask" : null,
      "LeakedConnectionCount" : 0,
      "ModuleId" : "emergencyDB",
      "Name" : "emergencyDB",
      "NumAvailable" : 1,
      "NumUnavailable" : 0,
      "Parent" : { "objectName" : "com.bea:Name=surf1,Type=ServerRuntime" },
      "PrepStmtCacheAccessCount" : 0,
      "PrepStmtCacheAddCount" : 0,
      "PrepStmtCacheCurrentSize" : 0,
      "PrepStmtCacheDeleteCount" : 0,
      "PrepStmtCacheHitCount" : 0,
      "PrepStmtCacheMissCount" : 0,
      "Properties" : "ERROR: MBean getAttribute failed: weblogic.common.resourcepool.ResourcePermissionsException: User \"\" does not have permission to perform operation \"admin\" on resource \"emergencyDB\" of module \"null\" of application \"null\" of type \"ConnectionPool\" (class javax.management.MBeanException)",
      "ReserveRequestCount" : 1,
      "State" : "Running",
      "Type" : "JDBCDataSourceRuntime",
      "VersionJDBCDriver" : "org.apache.derby.jdbc.ClientDriver",
      "WaitSecondsHighCount" : 0,
      "WaitingForConnectionCurrentCount" : 0,
      "WaitingForConnectionFailureTotal" : 0,
      "WaitingForConnectionHighCount" : 0,
      "WaitingForConnectionSuccessTotal" : 0,
      "WaitingForConnectionTotal" : 0,
      "WorkManagerRuntimes" : null
    }
}

JDBC Connection Pool: Find all JMS Servers with Target set to Managed Server Running at localhost:7003

http://localhost:7003/jolokia/read/com.bea:Name=emergencyDB,ServerRuntime=surf1,Type=JDBCConnectionPoolRuntime?ignoreErrors=true
which returns the JMS server surfJMS:
{ "request" : { "mbean" : "com.bea:Name=emergencyDB,ServerRuntime=surf1,Type=JDBCConnectionPoolRuntime",
      "type" : "read"
    },
  "status" : 200,
  "timestamp" : 1334823105,
  "value" : { "ActiveConnectionsAverageCount" : 0,
      "ActiveConnectionsCurrentCount" : 0,
      "ActiveConnectionsHighCount" : 0,
      "ConnectionDelayTime" : 44,
      "ConnectionLeakProfileCount" : 0,
      "ConnectionsTotalCount" : 1,
      "CurrCapacity" : 1,
      "DeploymentState" : 2,
      "Enabled" : true,
      "FailuresToReconnectCount" : 0,
      "HighestNumAvailable" : 1,
      "HighestNumUnavailable" : 0,
      "LeakedConnectionCount" : 0,
      "MaxCapacity" : 15,
      "ModuleId" : "emergencyDB",
      "Name" : "emergencyDB",
      "NumAvailable" : 1,
      "NumUnavailable" : 0,
      "Parent" : { "objectName" : "com.bea:Name=surf1,Type=ServerRuntime" },
      "PoolState" : true,
      "Properties" : "ERROR: MBean getAttribute failed: weblogic.common.resourcepool.ResourcePermissionsException: User \"\" does not have permission to perform operation \"admin\" on resource \"emergencyDB\" of module \"null\" of application \"null\" of type \"ConnectionPool\" (class javax.management.MBeanException)",
      "State" : "Running",
      "StatementProfileCount" : 0,
      "Type" : "JDBCConnectionPoolRuntime",
      "VersionJDBCDriver" : "org.apache.derby.jdbc.ClientDriver",
      "WaitSecondsHighCount" : 0,
      "WaitingForConnectionCurrentCount" : 0,
      "WaitingForConnectionHighCount" : 0,
      "WorkManagerRuntimes" : null
    }
}

JMS: Find all JMS Servers with Target set to Managed Server Running at localhost:7003
http://localhost:7003/jolokia/search/com.bea:Type=JMSServerRuntime,*
which returns the JMS server surfJMS:
{
   "timestamp":1334776840,
   "status":200,
   "request":{
      "mbean":"com.bea:Type=JMSServerRuntime,*",
      "type":"search"
   },
   "value":[
      "com.bea:Name=surfJMS,ServerRuntime=surf1,Type=JMSServerRuntime"
   ]
}

JMS: Runtime Properties for a Particular Queue
Request the MessagesCurrentCount, MessagesHighCount and MessagesReceivedCount for a queue with the name jms/ShippingRequestQueue (note the “!” is used in the URL to escape the “/” character in the queue name) in a JMS module with the name surfJMSModule (note that JMS module and queue name are typically separated in the MBean Name attribute with a “!” which has to be escaped by another “!”).
http://localhost:7003/jolokia/read/com.bea:JMSServerRuntime=surfJMS,Name=surfJMSModule!!jms!/ShippingRequestQueue,ServerRuntime=surf1,Type=JMSDestinationRuntime/MessagesCurrentCount,MessagesHighCount,MessagesReceivedCount
which returns the following JSON structure.
{
   "request":{
      "attribute":[
         "MessagesCurrentCount",
         "MessagesHighCount",
         "MessagesReceivedCount"
      ],
      "mbean":"com.bea:JMSServerRuntime=surfJMS,Name=surfJMSModule!jms/ShippingRequestQueue,ServerRuntime=surf1,Type=JMSDestinationRuntime",
      "type":"read"
   },
   "status":200,
   "timestamp":1334775025,
   "value":{
      "MessagesCurrentCount":0,
      "MessagesHighCount":1,
      "MessagesReceivedCount":3
   }
}


The output for the queue monitoring shows that there was a total of 3 messages sent, with a maximum of 1 in the queue (because of a deployed receiver retrieving the messages immediately) and currently 0 message in the queue.

Filed Under: oracle Tagged With: availability, best practice, monitoring, weblogic, wls12c

Spicing up WebLogic RESTful Management with Jolokia

April 19, 2012 by frank 1 Comment

[This posting will be part of my upcoming WebLogic 12c book]

Management and monitoring of WebLogic resources is a day-to-day challenge for many administrators. Programming a management solution by writing JMX code in Java is a low-level and time consuming process which can be best avoided using the Jython based WebLogic scripting tool (WLST).

WLST is a higher level, domain specific language (DSL) especially developed to address management issues. A few lines of WLST code typically encapsulate hundreds of lines of Java code using JMX. Although WLST is compact and easy to write it still suffers from the “J” in JMX: Every time you execute a WLST script to monitor some attribute on the application server a JVM has to be started on the client side for WLST. This overhead can become quite substantial if you are monitoring many servers and retrieving attributes at regular intervals. In particular for monitoring systems such as Nagios (or the newer Shinken) the WLST approach is not suitable.

WebLogic 12c RESTful Management Service

One of the distinguishing new features of WebLogic 12c is its RESTful management service which can be enabled by a simple click under DOMAIN / Configuration / General / Advanced / Enable RESTful Management Services followed by a server restart.

Once enabled, you can access the most important WebLogic 12c runtime values using a simple URL syntax from your web browser or a Unix command line tool such as curl. For example you can retrieve runtime data of the administration server with fast HTTP request instead of spawning a JVM process for WLST:

http://localhost:7001/management/tenant-monitoring/servers/AdminServer/

Jolokia

Basics

WebLogic 12c RESTful management is an easy and convinient start, yet there is more possible. The open source framework Jolokia was named after of one of the the hottest chilis on the planet. Jolokia is the kind of chili that most people agree better not to eat (it’s still useful, e.g. to produce military grade tear gas and marine paint).

(Image source: courtesy of R. Huss / Jolokia)

The project is best described by the its developer:

“Jolokia is an agent based approach for remote JMX access. It is an alternative to standard JSR 160 connectors. The communication between client and agent goes over HTTP (either GET or POST), where the request and response payload is represented in JSON. The Jolokia protocol supports the following operations:

- Reading and writing JMX attributes

- Execution of JMX operations

- Searching for MBean Names by pattern

- Listing of MBean Meta-data like supported attributes, operations and notifications”

Installation

The installation process is well described at the Jolokia site. You can simply get the jolokia.war file from the Jolokia download site or the Maven central repository, nothing else is needed for WebLogic.

It’s even possible to use it without any deployment at all. Starting with JDK6 you can run it as an JVM agent, a technique typically used for profilers etc. So your java call starting WebLogic has the following format:

java -javaagent:agent.jar ...

If you like to install the whole Jolokia project (including the JMX shell described later, the Nagios plugin, a Spring Roo addon etc) on a UNIX system with Perl and CPAN already installed it can be as easy as:

[root@ccloud ~]# perl -MCPAN -e shell cpan shell -- CPAN exploration and modules installation (v1.9800)

cpan[1]> install JMX:Jmx4Perl
[... you have to confirm the installation of several dependencies ...]

cpan[2]> exit

After building the Jolokia module it can download the jolokia.war which is then deployed to WebLogic. In case you are playing with an admin server running in development only move it to the DOMAIN_NAME/autodeploy directory otherwise deploy it via the administration console to all servers in the domain:

oracle@ccloud [~]$ jolokia download

* Loading Jolokia meta data from http://www.jolokia.org/jolokia.meta
* Good PGP signature (EF101165)
* Using Jolokia 1.0.3 for Jmx4Perl 1.04
* Downloading war agent version 1.0.3 from repository http://labs.consol.de/maven/repository
* Saved ./jolokia.war
* Good PGP signature (EF101165)

Feature Comparison: Jolokia and WebLogic 12c RESTful Management Service

Now, how does Jolokia compare to what you already have out of the box in WebLogic 12c? Here is a short overview:

	WebLogic 12c Management Service	Jolokia
Installation	Not required: enable from admin console	Simple installation: deploy prebuilt jolokia.war
Availability	Only for WebLogic 12c	Multi vendor, multi version support: WebLogic 9 to 12c Most other application servers Mule agent JVM agent OSGi agent
Approach	Central (available on admin server only)	Can be distributed (target on every server in domain)
Access syntax	Easy, proprietary URL format	Standard MBean name
Return format	HTML, JSON, XML	JSON
Accessible Data	Servers, Clusters, Deployed Apps, DataSources Not supported: JMS	All accessable MBeans and attributes
Security	Requires authentication with admin or monitor role	Standard web based security (can be restricted to certain IPs, roles, HTTPprotocols, etc.)
Documentation	Oracle documentation	Very good online documentation
Access for single attributes only example: HeapMemoryUsage	No	Yes
Access for inner path example HeapMemoryUsage/max	No	Yes
Bulk requests	No	Yes (One HTTPrequest for different MBeans is translated into multiple JMX requests within WebLogic JVM and yet one result is returned)
Additional utilities		JMX shell j4psh Nagios plugin check_jmx4perl

Jolokia Request

The GET URL for a Jolokia read request has the following format:

<base-url>/read/<mbean name>/<attribute name>/<inner path>

To get started just paste the following URL into your browser or use the Unix command line utility curl (assuming you have the admin server running at localhost:7001). It will retrieve the configured ListenPort attributes:

http://localhost:7001/jolokia/read/com.bea:*,Type=Server/ListenPort

which will return the following result for a WebLogic domain with 3 managed servers configured:

{ "request" : { "attribute" : "ListenPort",
      "mbean" : "com.bea:Type=Server,*",
      "type" : "read"
    },
  "status" : 200,
  "timestamp" : 1334776908,
  "value" : { "com.bea:Name=AdminServer,Type=Server" : { "ListenPort" : 7001 },
      "com.bea:Name=surf1,Type=Server" : { "ListenPort" : 7003 },
      "com.bea:Name=surf2,Type=Server" : { "ListenPort" : 7005 },
      "com.bea:Name=surf3,Type=Server" : { "ListenPort" : 7007 }
    }
}

Your response will be displayed in a single line. To get the format above you can past the result into one of the many online JSON formatters (or pipe the curl output to an appropriate utility).
Also I recommend to have a look at some examples about how to use Jolokia from Java, Perl and JavaScript.
Now continue reading part II of this posting with examples showing how to use Jolokia to retrieve monitoring values for servers, deployments, JDBC and JMS.

Filed Under: oracle Tagged With: monitoring, RESTful, weblogic, wls12c

WebLogic Stuck Threads: Creating, Understanding and Dealing with them

April 10, 2012 by frank 23 Comments

Using the time off during the bank holidays over Easter I spent some time coding and looking into more unknown details of WebLogic stuck thread behavior. (Actually I started to write this posting because I was told by my doctor to keep my mouth shut for some days, but that’s another story…).

My personal task was to answer some of the most common questions I’ve encountered while consulting and running WebLogic 12c workshops. As often with my postings, this article is not meant to explain the basic concept of thread pools or workmanagers. I recommend to read the Oracle WebLogic 12c documentation about stuck thread handling first which explains how you can deal with stuck threads by configuring a workmanager.

Also there are some excellent details about stuck threads (including WLST scripting and monitoring) to be found at the Middleware Magic site - a site run by a group of really knowledgeable guys.

Now, typically customers tell me that they “observe some stuck threads”, “sometimes”, but often they are “not sure what caused them” and typically they “don’t know what exact state these thread are in” and in addition nobody seems to know if “the stuck threads ever clear up again without rebooting”. I am a pragmatic guy. I enjoy having little applications or tools to demonstrate and measure how WebLogic is working. Keen to play around with the newest edition of Netbeans (I used to be an Eclipse guy) and EJB 3.1 in WLS12c I built a small application to easily test WebLogic stuck thread settings and countermeasures.

Here are some more details about the StuckThreadForFree application:

The application allows you to create threads which are busy or which are waiting long enough to be detected as “stuck” by WebLogic.
This little application will only work with WLS12c. I intentionally avoided JSF, so a plain JSP page is used to set your parameters. The JSP is calling a simple Servlet which in a for loop is calling an asynchronous business method of an injected stateless session bean. @Asynchronous and no-interface session beans are only available in EJB3.1 so you have to run it on WLS12c. Unlike in previous versions, the EJB is directly packaged into the .war file for deployment.
Every call to the stateless session bean is serialized by the EJB container, so every EJB method is executing in its own thread.
Depending on which method was called on the EJB is either waiting n seconds using Thread.sleep() or calculating some trigonometric function for n seconds. Both methods will cause stuck threads.
There is zero configuration in the deployment descriptors for the EJB! Only context-root for the web part is set (which could be avoided as well).
Building the StuckThreadForFree app with Netbeans was a smooth ride and a real pleasure.
The app is provided as is. You can have it for free, yet there is no guarantee for anything but it shouldn’t cause any problems either. Better don’t run it on your production system.
It’s just a hack. It demonstrates what it should, nothing else.

DOWNLOAD: for your conveninience you can download the StuckThreadForFree.war from here and follow the example yourself (here is the link to the whole Netbeans project). After downloading you can easily deploy it to WebLogic. To follow the example it’s good enough to run it the admin server. Then you can start with the following URL:

http://localhost:7001/StuckThreadForFree

Now, lets use the app to answer some typical questions.

What are hogging threads? When do threads become hogged? After what period of time?

According to the Oracle doc hogging threads “.. will either be declared as stuck after the configured timeout or will return to the pool before that. The self-tuning mechanism will backfill if necessary.”

So how long does it take for them to become hogged? Nobody (including Google) seemed to know. Trust me I did some research and asked plenty of colleagues about this. Here is the answer:

If you run the application with 3 threads / 100 seconds / Thread.sleep() and immediately switch to the WebLogic 12c admin console Admin Server / Monitoring / Threads you will observe the following:

So interestingly hogging threads are detected right away! In my case it took about 2 seconds (I had to hit reload once).

So WebLogic transitions into FAILED state when a certain number of stuck threads are detected, right?

That’s a common misconception! The default configuration of WLS 12c (I also checked for WLS 11 = 10.3.3) is Stuck Thread Count = 0, which means the server “never transitions into FAILED server irrespective of the number of stuck threads”. You will only see the FAILED state only when you set the value to a positive number of threads!

Once the server transitions into FAILED, you can define if WLS should be shut down (and restarted by WLS nodemanager) or suspended.

Remember: WLS will not transition into FAILED state when StuckThreadCount is set to zero. Only the health runtime value is set to Warning (but this will be cleared if the hogging thread conditions clears) as shown below:

What exactly causes a stuck thread? What state does a thread have to be in to be marked as stuck?

In general there is a number of different thread states in Java: NEW, RUNNABLE, BLOCKED, WAITING, TIMED_WAITING, TERMINATED.

But which state has a thread to be in to be marked as stuck later? If you run the StuckThreadForFree application and create a stack trace with WebLogic admin console under Server / ServerName / Monitoring / Threads you can observe that the thread state is ACTIVE/TIMED_WAITING when using the Thread.sleep() method to block it:

"[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" TIMED_WAITING
            	java.lang.Thread.sleep(Native Method)
            	com.munzandmore.stuckthread.LongRunningEJB.threadSleep(LongRunningEJB.java:26)
            	com.munzandmore.stuckthread.LongRunningEJB_x9v26k_NoIntfViewImpl.__WL_invoke(Unknown Source)

when using the calc() method to keep the threads busy they are state ACTIVE/RUNNABLE :

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE
            	com.munzandmore.stuckthread.LongRunningEJB.threadCalc(LongRunningEJB.java:40)
            	com.munzandmore.stuckthread.LongRunningEJB_x9v26k_NoIntfViewImpl.__WL_invoke(Unknown Source)
            	weblogic.ejb.container.internal.SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:31)

So both states can become stuck. Also, I am pretty sure I could also show the BLOCKED state when using a monitor lock for synchronization but due to time restrictions this is not included in the app.

Can a stuck thread still do reasonable work?

Absolutely! Just because a thread is marked as stuck it doesn’t mean it is frozen or unusable. Imagine you wanted to calculate PI, you are creating PDFs, distance maps, mapping the human genome or you have deployed some JCA adapter talking to MQ-Series, SAP or PeopleSoft which is internally using a Thread.sleep() method call. All of this is are reasonable usages likely to occur in the wild.

Do stuck threads ever dissapear? Can they be cleared somehow? Are they stuck forever?

First of all you cannot get rid of a stuck thread by simply “killing it”. You cannot cancel or kill any thread in Java. However, stuck threads automatically will disappear if the condition clears up which caused them to be marked as stuck (e.g. the sleep period is over or the calculation is done).

To prove the point, switch to the WebLogic admin console and under Server / ServerName / Configuration set StuckThreadCount to 3 and StuckThreadTime to 60 seconds then restart the server and run the StuckThreadForFree app to create 3 threads running for 120 seconds using the Thread.sleep() method (the other method will work as well, there is no difference, but keeping 3 threads busy by doing math proves to be a fan test of your machine as well):

In the WebLogic log file you will find three entries logging the stuck thread state after a while:

<05.04.2012 10:55 Uhr MESZ> <Critical> <WebLogicServer> <BEA-000385> <Server health failed. Reason: health of
critical service 'Thread Pool' failed>
<05.04.2012 10:55 Uhr MESZ> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '4' for queue: 'webl
ogic.kernel.Default (self-tuning)' has been busy for "85" seconds working on the request "Workmanager: default
, Version: 1, Scheduled=false, Started=true, Started time: 85443 ms
", which is more than the configured time (StuckThreadMaxTime) of "60" seconds. Stack trace:
 java.lang.Thread.sleep(Native Method)

After waiting about one minute you will observe that WebLogic is transitioning into FAILED state as configured:

Wait another minute, then check the thread states under Server / ServerName / Monitoring / Threads which reveals the following:

So once the condition causing the stuck threads is cleared also the stuck threads will disappear again! Stuck threads are not stuck forever. Phew!

When should I use StuckThreadCount in the admin console or a Workmanager stuck-thread setting then?

Very good question. Use StuckThreadCount from the WebLogic admin console or with a <work-manager-shutdown-trigger> definition moving the application into ADMIN mode if you can react on the FAILED state.

Do not use StuckThreadCount if the threads might be doing something useful and you cannot react on the situation anyway. Obviously transitioning into FAILED state and restarting WLS with the nodemanager is counterproductive if you threads are doing something useful.

More?

The following posting shows how simple tools like ps, top and jcmd can track down the exact line of Java code causing a thread to use a high amount of CPU. Exactly the same StuckThreadForFree application is used as here.

Filed Under: oracle Tagged With: availability, performance, weblogic, wls12c

WebLogic Feature Timeline: What is new in WLS 12, 11, 10, 9?

February 24, 2012 by frank 11 Comments

To whom it may concern, here is a quick overview of the new features as they were included in WebLogic Server versions 9 to 12c. Destilled for my first WebLogic 12c workshop running a fortnight ago and including some links to the WebLogic documentation and some other useful resources.

happy weekend then … !

Download WebLogic Feature Timeline (PDF): What is new in WebLogic 12c? WebLogic 11? WebLogic 10? WebLogic 9?

Update (05-March-2012):I just uploaded a newer version after adding valuable input from Steve and Jan. (Due to other changes, those of you linking to the PDF directly need to update their link this time).

Filed Under: oracle Tagged With: weblogic, wls12c, workshop

WebLogic 12c: Node Manager Best Practices

January 31, 2012 by frank 9 Comments

During the last couple of years (and the last couple of WebLogic versions) I collected a number of best practices regarding WebLogic nodemanager. All of them hold true for WebLogic 12c as well. This posting is not a step-by-step beginners guide and it will not save you from attending some training or studying the Oracle documentation regarding node manager yourself. Anyway, here are some suggestions, check if the apply for your environments:

Node Manager Best Practices

At first, take a decision to start servers with or without NM. Note, that is not absolutely necessary. You can always start your servers with the scripts generated by the config wizzard. I personally know rather big companies building lovely cars who took the decsicion not to use node manager.
Would I use nodemanger myself? For an “average” project: yes! Only after configuring node manager you can use the WebLogic admin console to start and stop managed servers and node manager will restart you failed servers as well. However, if you consider restarting you servers automatically because of out-of-memory problems, better read this article about “surviving generations” to understand how to track down memory leaks and fix them. Anyway, you still want to use node manager.
Make sure you understand that nodemanger will use default values to start your servers unless you specify them yourself in the admin console under server startup parameters.
Make sure you always start your servers with same startup parameters! This is really important. You end up in deep trouble if you don’t. Believe me.
Imagine somebody is starting a managed server using the admin console and the provided values there. Next day somebody else starts a server using the provided scripts (which – at least in real life – will never be identical to the startup values configured in the admin console). Now depending on the way the server was started it will behave differently and show erratic behaviour or not.
Document and communicate the usage of node manager. Write it down in the operations manual. If you ever hire me as a consultant for some performance tuning it helps to know if you are actually using node manager or not.
Don’t forget to enroll new machines for NM usingnmEnroll()
A good way to overcome the potential problem with startup parameters configured in admin console is to use:
```
[file: nodemanger.properties]startScriptEnabled=true
stopScriptEnabled=true
```
Then node manager will use the generated start script and you do not need to configure startup values in the admin server console.
If you are not using SSL for your domain the default option for node manager to use encrypted communication does not make that much sense for you. Disable it. On the admin server site switcht to “plain text” for node manager communication and in the node manager.properties located in WL_HOME/common/nodemanager set
```
secureListener=false
```
If you decide to use SSL for the node manager communication, get correct certificates! The demo certificate will not work in a distributed system. Make sure the hostnames in the certificates are correct. If they are not correct, you may want to consider disabling host name verification on the admin server (which is the client for the node manager).
Remember that node manager is not part of the domain. Still you can check the node manager status and and see the logs directly from the WebLogic admin console.

Some homework for Oracle Here is my personal wishlist for WebLogic Server 13f:

Enable plain text communication for node manager as default. Why should it be SSL?
Set start/stopScriptEnabled as default. This will cause less confusion.

Any settings you would like to share? Something to argue? Let me know.

Follow @frankmunz

Filed Under: oracle Tagged With: admin, best practice, node manager, weblogic, wls12c, workshop

WebLogic 12c Proxy Plugin for Web Servers is missing (kinda solved)

January 15, 2012 by frank 1 Comment

According to the documentation for the brand new WebLogic 12c the WLS mod_wl proxy plugin for web servers should be located under the following location:

$WL_SERVER/server/plugin

and this is where the trouble starts:

This directory does not exists for WebLogic 12c.
You can download the plugins seperately from here. Yet there is no version for WebLogic 12c either.
You must not simply use an lower plugin version (e.g. the one that comes with WebLogic 11g) because lower versions of the plugin e.g. 10.3.5 are not guaranteed to work (only higher versions). Now, this is between you me: yes they do work typically. Yet I remember at least one situation when I spent over a day debugging an issue because an older plugin version was used. I am pretty sure you can imagine serveral way to spend your time better.
The behavior for the Apache mod_wl 10.x plugins is known to be rather strange after a a managed server drops out of the dynamic server list (clustered or not).

Here are some possible solutions:

I briefly checked the 10.3.5 plugin for Apache 2.2 with a WebLogic 12c cluster and it seems to work okay so far. Actually even the glitch with the failing load balancing after a server drops out of a cluster has improved. Probably that’s the best solution currently, but remember it’s at your own risk.

Follow up with my OTN thread and some valuable input from René and Sandeep.

For a more critical production environment I recommend to open a support ticket with Oracle. If you do so, please post an update here and share your findings.

So how are you using WebLogic 12 in production? If you do, how is your clustering set up?

UPDATE / Jan 2013:

Thanks to input from my friend Rene I am aware now that Oracle officially recommends to use the latest 10.3.x plugin even for WebLogic 12c. Still it is causing a lot of confusion and the WLS 12.1.1 documentation is wrongly stating that the plugin is part of the WebLogic distribution.

Select “Generally Available System Configurations”:

http://www.oracle.com/technetwork/middleware/ias/downloads/wls-1211x-certmatrix-1395408.xls

Filed Under: oracle Tagged With: cluster, oracle weblogic, plugin, weblogic, wls12c

Reviews and Attendee’s Opinions

what Did you like about the workshop?

Comments about presenter:

More?

Basic Introduction

Development with NetBeans Introduction

Coherence ActiveCache with WebLogic

Audience

Technology

Download

Hands-on Test

Download

Results

j4psh JMX Shell

Screencasts

WebLogic with Jolokia

General: List Configuration Details of Managed Server with Name surf1

Deployment: List all Deployments

Deployment: List Deployments Details for an Application with the Name SubmitOrder

JDBC Data Source: Retrieve Settings for emergencyDB Data Source with Target Managed Server surf1 Running at localhost:7003

JDBC Connection Pool: Find all JMS Servers with Target set to Managed Server Running at localhost:7003

JMS: Find all JMS Servers with Target set to Managed Server Running at localhost:7003

JMS: Runtime Properties for a Particular Queue

WebLogic 12c RESTful Management Service

Jolokia

Basics

Installation

Feature Comparison: Jolokia and WebLogic 12c RESTful Management Service

Jolokia Request

What are hogging threads? When do threads become hogged? After what period of time?

So WebLogic transitions into FAILED state when a certain number of stuck threads are detected, right?

What exactly causes a stuck thread? What state does a thread have to be in to be marked as stuck?

Can a stuck thread still do reasonable work?

Do stuck threads ever dissapear? Can they be cleared somehow? Are they stuck forever?

When should I use StuckThreadCount in the admin console or a Workmanager stuck-thread setting then?

More?

Node Manager Best Practices

Some homework for Oracle Here is my personal wishlist for WebLogic Server 13f:

UPDATE / Jan 2013:

Tags and Clouds

WebLogic 12c Book

Twitter

Cloud Book