weblogic — munz & more

Oracle Service Bus (OSB) for the Busy IT Profesional

March 26, 2013 by frank Leave a Comment

Audience

Oracle Service Bus is one of these software products that haven’t changed much in their core since many years. These days I tend to see more and more OSB projects and one reason certainly is that license wise OSB is included with Oracle SOA Suite.

Compared to SOA Suite, OSB is conceptually different though. Even with a good understanding of SOA Suite (which I will cover in one of the following posts), there will be a slightly steep, but rather short learning curve for OSB.

The following recipes tries to motivate and explain the usage of OSB without the usual marketing hype. Make sure to read the previous posting about getting started with SOA in general and Oracle Fusion Middleware.

Technology

OSB is virtualizing service calls from clients to the service implementation. By using so called proxy and business services (both are components within OSB) incoming and outgoing requests can be arbitrarily connected and protocols can be converted. Proxy services contain the processing logic (validation, enrichment, transformation etc.).

Download

If you are trying to get started with OSB, here is the link to download Oracle Service Bus Overview and Getting Started (PDF).

Enjoy!

PS. Actually this recipe is taken out of my book Oracle WebLogic Server 12c – Distinctive Recipes.

Filed Under: oracle, WebLogic 12c Book Tagged With: best practice, oracle service bus, weblogic, wls12c

Using Apache Derby Database with WebLogic (the express way)

October 8, 2012 by frank Leave a Comment

this is more a note to myself, because I keep forgetting the details and then I get angry about myself when I am in a hurry

If you quickly require a database with WebLogic 12c (or some other version 10.3.3 or later, because since then WLS comes with Derby) and some content (e.g. to try some monitoring tool or a WLST script or something), do the following:

- make sure you have the WebLogic samples and the sample DB installed.

- create a new domain.

- in the domain’s setDomainEnv.sh under DOMAIN_HOME/bin
change the DERBY_FLAG from false to true.

- start the Admin server which will start the Derby database as well.

- Create a data source using the following values:

 DBName: medrec
 Host: localhost
 Port: 1527
 Username: medrec
 Password: medrec

- Test the connection pool after creating it. It should work right away.

Done!

more details, recipes and discussions …

Filed Under: oracle Tagged With: apache, database, derby, open source, weblogic

DevOps Basics I: Track Down High CPU Thread with ps, top and the new JDK7 jcmd Tool.

September 18, 2012 by frank 2 Comments

I captured the following webcast last night. I was writing about how a clever combination of simple tools like ps, top and jcmd can help to track down the Java code that is causing a thread to consume a high amount CPU time.

The approach is very generic and works for WebLogic, Glassfish or any other Java application. UNIX commands in the example are run on CentOS, so they will work without changes for Oracle Enterprise Linux or RedHat.
Creating the thread dump at the end of the video is done with the jcmd tool from JDK7.

The webcast uses the StuckThreadForFree sample application which is specific for WebLogic 12c and can be downloaded from my stuck thread recipe.

Filed Under: oracle Tagged With: devops, performance, weblogic

Simple Custom JMX MBeans with WebLogic 12c and Spring

July 25, 2012 by frank Leave a Comment

Using the Spring framework bundled with WebLogic 12 the programming of JMX MBeans becomes as easy as writing a simple POJO within a web application. After coding the MBean as a POJO all that is left to do is creating the Spring configuration file and editing the web.xml deployment decriptor:

- Firstly you have to add the Spring <listener> to the web.xml deployment descriptor with the web.xml to get Spring started.

- Secondly create the applicationContext.xml for Spring in the WEB-INF subdirectory. Declare the Spring bean and let Spring export it to the platform MBean server.

- Thirdly deploy the web application and test it with JConsole, WLST or j4psh.

For your convinience I already built an example. You can download the deployable SimpleJMX.war file or the complete NetBeans project.

You can see the application in action, including the involved files and the deployment in the following screencast:

How to access custom MBeans from WLST is shown in the second part:

The third part is showing how to retrieve MBean attributes from a JSP:

Filed Under: oracle, WebLogic 12c Book Tagged With: admin, jmx, monitoring, video on youtube, weblogic, wls12c

Amazon / Oracle Cloud Workshops: Mission Completed

July 20, 2012 by frank Leave a Comment

I am just back from a successful, long and intensive trip to Australia. If you are waiting for a reply to an email there is a good chance you will get it soon now.

I delivered an Amazon and Oracle public cloud workshop series on behalf of AUSOUG, based on my cloud computing book: 6 cities, 6 workshops of 4 h each, almost 200 attendees, hands-on, live development in the cloud (WebLogic with 61 GB heap deployed live on 3 continents, cloud storage, auto scaling, transformation of a classical Java EE app in the cloud), 16 flights (including a 1 week trip to the outback, flying 2,5 h north of Perth and driving 2200 km for shooting the cover image for my upcoming WebLogic 12c: Advanced Recipes book).

Check out the flattering reviews for the event.

thanks to everybody who attended!

Filed Under: amazon, CloudComputing, oracle Tagged With: aws, book, cloud computing, ec2, weblogic, wls12c, workshop

Web Service Development with NetBeans and Testing with WebLogic Admin Console

May 23, 2012 by frank Leave a Comment

Developing of SOAP based web services for WebLogic 12c (or any other application server) with NetBeans is a breeze. You can do it without using any templates or IDE wizards, no XML is required, no modification of deployment descriptors.

The following screencast shows how to develop the most simple web service pobbislbe, how to test it entirely with WebLogic admin console, and how modifying the web service will work without redeployment.

Filed Under: oracle, WebLogic 12c Book Tagged With: development, netbeans, web service, weblogic, wls12c

WebLogic 12c Configuration Management: Change Your Defaults to Track What Happens

May 11, 2012 by frank Leave a Comment

Good morning. I just finished the chapter about configuration management for my new WebLogic book. Here is a very short summary of it.

There is a number of default configuration settings that I recommend to change to better keep track of configuration changes. The screencast showing you hands-on how to do this is still rendering and will be available once you are reading this (you can subscribe to all of them here).

After creating a WebLogic domain, consider the following:

1.) User personalized administration users.

2.) Audit WebLogic configuration changes (including usernames) so you will know who changed what.

3.) Enable the configuration archiving so you will be able to revert to a previous set of configuration files.

None of these settings are enabled per default. All of this you should think about for your environment.

Here is the three-part screencast how to enable it (switch to HD quality and full screen).

Filed Under: oracle, WebLogic 12c Book Tagged With: book, configuration, weblogic

Another Myth Debunked: More than 200 Continuous Redeployments with WebLogic

May 9, 2012 by frank 4 Comments

The other day I overheard the myth that you simply cannot redeploy several times with WebLogic because inevitably you would end up having an out of memory error. Since I am currently writing on an WebLogic 12c book I decided to verify that.

Certainly this depends a lot on your heap settings and what your application is doing, eg. it is easier to get it wrong if you are writing your own classloaders etc. I certainly admit that these problems, if they occur, are particularly hard to sort out.

Hands-on Test

More important you should detect them early. So go and test it! It’s a simple 10 line WLST script. I have done it for you to verifiy if there is a general problem with redeployment in WebLogic.

I redeployed the MedRec application over 200 times. MedRec comes with the WebLogic 12c installation (only if you decide to install the samples). It consists of over 20 EJBs, a web part and a few web services.

The sample domain can be found in the WL_HOME/samples/domains/medrec directory.

Download

A little WLST script that you can download does the trick (it’s a quick hack, you can certainly beautify and add error handling to it!). Download it (it comes zipped) and unzip it to the domain directory. It even contains comments about how to run it.

The script deploys the MedRec application (so undeploy it first!), waits for 30 seconds, undeploys, waits again and deploys again. At the same time it prints the number of loaded classes, heap and non-heap runtime values.

Results

I was running this experiment for 200 redeployments. Apart from the output of the program, I recorded some data plots with jconsole.

The non-heap usage after 200 redeployments is shown on the following screenshot.

Everything was fine. I guess I could also do 1000 redeployments. I just stopped it because I got bored. I believe it would go on forever and I needed to shut down my computer.

Also here is the heap usage:

So test your own application deployment! It’s an easy test. IMHO it should be run for every important application. Add it to your tests when load testing the application.

Also I recommend to create some load for the app while it’s deployed. You may want shorten the interval between the deployments to a few seconds to get more deployments done faster.

Let me know if it breaks in your case

Also stay informed about other exciting topics covered in the my WebLogic 12c book. I just created a Youtube channel with many WebLogic 12c screencasts (more to come every week).

Filed Under: oracle Tagged With: admin, book, deploymnet, weblogic, wls12c

WebLogic Startup Mode: Development or Production? The (hopefully) Complete List of Differences

April 26, 2012 by frank 9 Comments

Let me quickly summarize the all differences between development and production startup mode for WebLogic 12c and previous versions. Information about it is scattered across the Oracle documentation and several blog postings but non of those I checked were complete.

Differences

Startup Mode	Development		Production
Auto deployment (to admin server only)	Yes		No
Automatically created boot.properties for Admin server only under $DOMAIN_HOME/servers/$SERVER_NAME/security	Yes		No
Nodemanger username and password	default admin credentials		randomly generated
SSL demo certificates cause warning in log files	No		Yes
Log file are rotated at server startup	Yes		No
On demand deployment of internal applications as default	Yes		No
Availability of web service test client	Yes		No
Availability of Classloader Analysis Tool	Yes		No
Availability of Fast Swap	Yes		No
Change Center Enabled	No		Yes
Default start parameters when using Sun JDK and the default startWebLogic script	java -client -Xms256m -Xmx512m -XX:CompileThreshold= 8000 -XX:PermSize=128m -XX:MaxPermSize=256m		java -server -Xms256m -Xmx512m -XX:MaxPermSiz=256m
The following settings in the weblogic.xml deployment descriptors default to -1: pageCheckSeconds servlet-reload-check-secs servlet-reload-check resource-reload-check-secs	No		Yes
*Differences in previous* WebLogic versions:**
Up to WebLogic 11: Default JDK used for new domain	Sun	JRockit
Up to WebLogic 8 only: Default number of execute threads	15	25
Up to WebLogic 8 only: maximum number of JDBC connections default setting	15	25

Configuration

There is different ways to configure the WebLogic startup mode:

The startup mode can be configured with the admin console, as shown on the screenshot below (requires a restart). Once it’s set to production mode it cannot be reverted with the admin console.
Supply the -Dweblogic.ProductionModeEnabled=true switch as startup parameter
Set the PRODUCTION_MODE variable in the setDomainEnv script in the directory DOMAIN_NAME/bin.

What you should know

Some issues to be aware of:

The differences between the two settings depend on the WebLogic version. This can be a bit confusing when you are running different environments with different versions. See the table above: For WebLogic 8 the maximum number of connections in a JDBC connection pool defaults to 25 in production mode (but 15 in development mode); for WebLogic 12c it defaults to 15 in both cases.
You cannot revert to development mode from the WebLogic admin console. It’s like the child-proof lock in the back of your car. I understand that is makes sense to protect the children in the back seat your car from themselves but you are are logged with the admin role – still you cannot revert to development mode using the admin console.
Autodeployment is not used that much anymore since it only deploys to the admin server – so it doesn’t make sense for a production domain. What if you are a developer? Then your IDE is deploying to the admin server, that is correct – but your IDE is typically not using autodeployment.
Note, that there will be a randomly set user and password for nodemanager when creating your domain in production mode with config.sh (see comments from Jacco).
You cannot disable the change center in production mode

Something I forgot? I appreciate your feedback.

Follow @frankmunz

[A more detailed discussion with directions will be part of my upcoming WebLogic 12c book]

Filed Under: oracle Tagged With: admin, best practice, weblogic

WebLogic Scripting Tool on Capsaicin: j4psh JMX Shell with Syntax Highlighting, Wildcards and Tab-Completion

April 24, 2012 by frank Leave a Comment

[This posting will be a part of my upcoming WebLogic 12c book]

Kudos Oracle! Over a year ago I mentioned in my Oracle Middleware and Cloud Computing book that developing WebLogic Scripting Tool (WLST) scripts with vi or Notepad++ is so 80′ies. It took you only a couple of months to respond with an Eclipse facet. Now everyone can use the bundled Eclipse with OEPE for WLST scripting . Thanks for that – even if you had it on your to-do list anyway

WLST itself is a fantastic tool to interactively explore WebLogic MBean world with all their attributes, operations, and relationships (it’s described here - did I mention that this post is not for the beginners?). WLST is using its own proprietary syntax to navigate to MBeans though.

Are you willing to look beyond your own backyard? Ready to move to real MBean names? Then let’s spice it up.

j4psh JMX Shell

j4psh is a JMX shell which is using standard MBean names, provides syntax highlighting together with code completion and even wildcard patterns! In j4psh you will be using similar command as in WLST. Here is an incomplete list of the most useful commands:

connect, to connect to an Java EE server with the jolokia.war deployed or using the Java agent
pwd, to find out the current MBean name or domain
cd, to navigate to an MBean or domain
ls, to list MBean details (works with * and can be restricted as “ls -a” for attributes and “ls -o” for operations).
cat, to display attribute values

Unlike WLST, where every command is basically a Python method, the j4psh syntax is as easy as it gets.

For the bigger picture how j4psh can be downloaded, installed and how nicely it fits into the Jolokia open source project see my previous posting.

Screencasts

In my opinion j4psh is easiest to understand when seen in action, so here are two examples of j4psh sessions.

1.) Basic usage of j4psh with WebLogic. The JMX “HelloWord”: enabling verbose garbage collection without server restart.

2.) Advanced Usage of j4psh with WebLogic: How to find the right MBean for a JDBC connection? Retrieve JMX attributes from JDBC connection pool, then use the MBean Name for the same RESTful request.

Give it a try! It’s free anyway. And let me know what you think!
And by the way, after writing about open source Jolokia, I am now determined to grow a real Jolokia plant myself…

Update: Many thanks to the Jolokia team for the plant!!

Filed Under: oracle Tagged With: jolokia, monitoring, RESTful, weblogic, wls12c

RESTful WebLogic Monitoring of Servers, Applications, JDBC and JMS with Jolokia

April 19, 2012 by frank 5 Comments

[This posting will be a part of my upcoming WebLogic 12c book]

This article is part II of III. Make sure you read part I explaining the basics of monitoring WebLogic with WLST, the WebLogic 12c REST-ful management API and the open source framework Jolokia.

There will be a part III in a few of days, explaining how to use j4psh with WebLogic. j4psh is a JMX shell that resembles WLST in interactive mode but includes features such as syntax highlighting and tab-completion of commands and MBean names. j4psh is highly useful to interactively find out the correct MBean names used for the requests below.

WebLogic with Jolokia

Did you ever wonder how to retrieve the configuration values for a WebLogic managed server, runtime data of a JDBC connection pool or the number of messages in a particular JMS queue with a fast and simple HTTP GET?

The examples show typical use cases for monitoring a WebLogic domain with the following configuration:

- Admin server running at localhost:7001
- Managed server with name surf1 running at localhost:7003
- JDBC datasource with the name emergencyDB and associated connection pool (target set to surf1)
- JMS configuration with a JMS server surfJMS running on managed server surf1, a module surfJMSModule containing a queue name jms/ShippingRequestQueue (target of JMS server is surf1, JMS module subdeployment set to surfJMS)
- A deployment of a web service with the name SubmitOrder and target AdminServer and surf1.

An installation of Jolokia with target set to all server of the domain is recommended to follow the examples in this part.

General: List Configuration Details of Managed Server with Name surf1

http://localhost:7001/jolokia/read/com.bea:Name=surf1,Type=Server?ignoreErrors=true

Note, that without the ?ignoreErrors=true parameter the request would fail with an “NoAccessRuntimeException” because you cannot access the SSL keystore passphrases without proper authentication. Authentication is easily possible, but beyond of the scope of this article.

{ "request" : { "mbean" : "com.bea:Name=surf1,Type=Server",
      "type" : "read"
    },
  "status" : 200,
  "timestamp" : 1334778534,
  "value" : { "AcceptBacklog" : 300,
      "AddWorkManagerThreadsByCpuCount" : false,
      "AdminReconnectIntervalSeconds" : 10,
      "AdministrationPort" : 9002,
      "AdministrationPortEnabled" : false,
      "AdministrationProtocol" : "t3s",
      "AutoKillIfFailed" : false,
      "AutoMigrationEnabled" : false,
      "AutoRestart" : true,
      "COM" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=COM" },
      "COMEnabled" : false,
      "CandidateMachines" : [  ],
      "ClasspathServletDisabled" : false,
      "ClientCertProxyEnabled" : false,
      "Cluster" : { "objectName" : "com.bea:Name=surfCluster,Type=Cluster" },
      "ClusterRuntime" : null,
      "ClusterWeight" : 100,
      "CoherenceClusterSystemResource" : null,
      "CompleteCOMMessageTimeout" : -1,
      "CompleteHTTPMessageTimeout" : -1,
      "CompleteIIOPMessageTimeout" : -1,
      "CompleteMessageTimeout" : 60,
      "CompleteT3MessageTimeout" : -1,
      "ConnectTimeout" : 0,
      "ConsensusProcessIdentifier" : -1,
      "ConsoleInputEnabled" : false,
      "CustomIdentityKeyStoreFileName" : null,
      "CustomIdentityKeyStorePassPhrase" : "ERROR: Access to sensitive attribute in clear text is not allowed due to the setting of ClearTextCredentialAccessEnabled attribute in SecurityConfigurationMBean. Attr: CustomIdentityKeyStorePassPhrase, MBean name: com.bea:Name=surf1,Type=Server (class weblogic.management.NoAccessRuntimeException)",
      "CustomIdentityKeyStorePassPhraseEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: CustomIdentityKeyStorePassPhraseEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "CustomIdentityKeyStoreType" : null,
      "CustomTrustKeyStoreFileName" : null,
      "CustomTrustKeyStorePassPhrase" : "ERROR: Access to sensitive attribute in clear text is not allowed due to the setting of ClearTextCredentialAccessEnabled attribute in SecurityConfigurationMBean. Attr: CustomTrustKeyStorePassPhrase, MBean name: com.bea:Name=surf1,Type=Server (class weblogic.management.NoAccessRuntimeException)",
      "CustomTrustKeyStorePassPhraseEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: CustomTrustKeyStorePassPhraseEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "CustomTrustKeyStoreType" : null,
      "DGCIdlePeriodsUntilTimeout" : 5,
      "DataSource" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=DataSource" },
      "DefaultFileStore" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=DefaultFileStore" },
      "DefaultIIOPPassword" : "ERROR: Access to sensitive attribute in clear text is not allowed due to the setting of ClearTextCredentialAccessEnabled attribute in SecurityConfigurationMBean. Attr: DefaultIIOPPassword, MBean name: com.bea:Name=surf1,Type=Server (class weblogic.management.NoAccessRuntimeException)",
      "DefaultIIOPPasswordEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: DefaultIIOPPasswordEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "DefaultIIOPUser" : null,
      "DefaultInternalServletsDisabled" : false,
      "DefaultProtocol" : "t3",
      "DefaultSecureProtocol" : "t3s",
      "DefaultTGIOPPassword" : "ERROR: Access to sensitive attribute in clear text is not allowed due to the setting of ClearTextCredentialAccessEnabled attribute in SecurityConfigurationMBean. Attr: DefaultTGIOPPassword, MBean name: com.bea:Name=surf1,Type=Server (class weblogic.management.NoAccessRuntimeException)",
      "DefaultTGIOPPasswordEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: DefaultTGIOPPasswordEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "DefaultTGIOPUser" : "guest",
      "ExecuteQueues" : [  ],
      "ExpectedToRun" : true,
      "ExternalDNSName" : null,
      "ExtraEjbcOptions" : null,
      "ExtraRmicOptions" : null,
      "FederationServices" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=FederationServices" },
      "GatheredWritesEnabled" : false,
      "GracefulShutdownTimeout" : 0,
      "HealthCheckIntervalSeconds" : 180,
      "HealthCheckStartDelaySeconds" : 120,
      "HealthCheckTimeoutSeconds" : 60,
      "HostsMigratableServices" : true,
      "HttpTraceSupportEnabled" : false,
      "HttpdEnabled" : true,
      "IIOP" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=IIOP" },
      "IIOPEnabled" : true,
      "IIOPTxMechanism" : "ots",
      "IdleConnectionTimeout" : 65,
      "IdleIIOPConnectionTimeout" : -1,
      "IdlePeriodsUntilTimeout" : 4,
      "IgnoreSessionsDuringShutdown" : false,
      "InstrumentStackTraceEnabled" : true,
      "InterfaceAddress" : null,
      "JDBCLLRTableName" : null,
      "JDBCLoggingEnabled" : false,
      "JDBCLoginTimeoutSeconds" : 0,
      "JMSDefaultConnectionFactoriesEnabled" : true,
      "JMSThreadPoolSize" : 15,
      "JNDITransportableObjectFactoryList" : [  ],
      "JTAMigratableTarget" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=JTAMigratableTarget" },
      "JavaCompiler" : "javac",
      "JavaCompilerPostClassPath" : null,
      "JavaCompilerPreClassPath" : null,
      "JavaStandardTrustKeyStorePassPhrase" : "ERROR: Access to sensitive attribute in clear text is not allowed due to the setting of ClearTextCredentialAccessEnabled attribute in SecurityConfigurationMBean. Attr: JavaStandardTrustKeyStorePassPhrase, MBean name: com.bea:Name=surf1,Type=Server (class weblogic.management.NoAccessRuntimeException)",
      "JavaStandardTrustKeyStorePassPhraseEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: JavaStandardTrustKeyStorePassPhraseEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "KernelDebug" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=ServerDebug" },
      "KeyStores" : "DemoIdentityAndDemoTrust",
      "ListenAddress" : "",
      "ListenDelaySecs" : 0,
      "ListenPort" : 7003,
      "ListenPortEnabled" : true,
      "ListenThreadStartDelaySecs" : 60,
      "ListenersBindEarly" : false,
      "Log" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=Log" },
      "LogRemoteExceptionsEnabled" : false,
      "LoginTimeoutMillis" : 5000,
      "LowMemoryGCThreshold" : 5,
      "LowMemoryGranularityLevel" : 5,
      "LowMemorySampleSize" : 10,
      "LowMemoryTimeInterval" : 3600,
      "MSIFileReplicationEnabled" : false,
      "Machine" : null,
      "ManagedServerIndependenceEnabled" : true,
      "MaxCOMMessageSize" : -1,
      "MaxHTTPMessageSize" : -1,
      "MaxIIOPMessageSize" : -1,
      "MaxMessageSize" : 10000000,
      "MaxOpenSockCount" : -1,
      "MaxT3MessageSize" : -1,
      "MessageIdPrefixEnabled" : false,
      "MessagingBridgeThreadPoolSize" : 5,
      "MuxerClass" : null,
      "NMSocketCreateTimeoutInMillis" : 180000,
      "Name" : "surf1",
      "NativeIOEnabled" : true,
      "NetworkAccessPoints" : [  ],
      "Notes" : null,
      "OutboundEnabled" : false,
      "OutboundPrivateKeyEnabled" : false,
      "OverloadProtection" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=OverloadProtection" },
      "Parent" : { "objectName" : "com.bea:Name=surfandconsulting,Type=Domain" },
      "PeriodLength" : 60000,
      "PreferredSecondaryGroup" : null,
      "ReliableDeliveryPolicy" : null,
      "ReplicationGroup" : null,
      "ReplicationPorts" : null,
      "RestartDelaySeconds" : 0,
      "RestartIntervalSeconds" : 3600,
      "RestartMax" : 2,
      "ReverseDNSAllowed" : false,
      "SSL" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=SSL" },
      "ScatteredReadsEnabled" : false,
      "SelfTuningThreadPoolSizeMax" : 400,
      "SelfTuningThreadPoolSizeMin" : 1,
      "ServerDebug" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=ServerDebug" },
      "ServerDiagnosticConfig" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=WLDFServerDiagnostic" },
      "ServerLifeCycleTimeoutVal" : 30,
      "ServerStart" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=ServerStart" },
      "ServerVersion" : "unknown",
      "SingleSignOnServices" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=SingleSignOnServices" },
      "SocketBufferSizeAsChunkSize" : false,
      "SocketReaders" : -1,
      "StagingDirectoryName" : "/u01/domains/surfandconsulting/servers/surf1/stage",
      "StagingMode" : "stage",
      "StartupMode" : "RUNNING",
      "StartupTimeout" : 0,
      "StuckThreadMaxTime" : 600,
      "StuckThreadTimerInterval" : 60,
      "SystemPasswordEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: SystemPasswordEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "TGIOPEnabled" : true,
      "ThreadPoolPercentSocketReaders" : 33,
      "TimedOutRefIsolationTime" : 0,
      "TransactionLogFilePrefix" : "./",
      "TransactionLogFileWritePolicy" : "Direct-Write",
      "TransactionLogJDBCStore" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=TransactionLogJDBCStore" },
      "TunnelingClientPingSecs" : 45,
      "TunnelingClientTimeoutSecs" : 40,
      "TunnelingEnabled" : false,
      "Type" : "Server",
      "UploadDirectoryName" : "./servers/surf1/upload",
      "Use81StyleExecuteQueues" : false,
      "UseConcurrentQueueForRequestManager" : false,
      "UseFusionForLLR" : false,
      "VerboseEJBDeploymentEnabled" : "false",
      "VirtualMachineName" : "surfandconsulting_surf1",
      "WebServer" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=WebServer" },
      "WebService" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=WebService" },
      "WeblogicPluginEnabled" : false,
      "XMLEntityCache" : null,
      "XMLRegistry" : null
    }
}

Deployment: List all Deployments
http://localhost:7001/jolokia/search/com.bea:*,Type=ApplicationRuntime
{ "request" : { "mbean" : "com.bea:Type=ApplicationRuntime,*",
 "type" : "search"
 },
 "status" : 200,
 "timestamp" : 1334780578,
 "value" : [ "com.bea:Name=mejb,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=consoleapp,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=bea_wls_management_internal2,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=emergencyDB,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=bea_wls_deployment_internal,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=SubmitOrder,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=wls-management-services,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=bea_wls_internal,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=jolokia,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=bea_wls9_async_response,ServerRuntime=AdminServer,Type=ApplicationRuntime"
 ]
 }
Deployment: List Deployments Details for an Application with the Name SubmitOrder
http://localhost:7001/jolokia/read/com.bea:ServerRuntime=*,Name=SubmitOrder,Type=ApplicationRuntime
{ "request" : { "mbean" : "com.bea:Name=SubmitOrder,ServerRuntime=*,Type=ApplicationRuntime",
      "type" : "read"
    },
  "status" : 200,
  "timestamp" : 1334777478,
  "value" : { "com.bea:Name=SubmitOrder,ServerRuntime=AdminServer,Type=ApplicationRuntime" : { "ActiveVersionState" : 2,
          "ApplicationName" : "SubmitOrder",
          "ApplicationVersion" : null,
          "ClassRedefinitionRuntime" : null,
          "CoherenceClusterRuntime" : null,
          "ComponentRuntimes" : [ { "objectName" : "com.bea:ApplicationRuntime=SubmitOrder,Name=AdminServer_/SubmitOrder,ServerRuntime=AdminServer,Type=WebAppComponentRuntime" } ],
          "EAR" : false,
          "HealthState" : { "critical" : false,
              "mBeanName" : null,
              "mBeanType" : null,
              "reasonCode" : [  ],
              "state" : 0,
              "subsystemName" : null
            },
          "KodoPersistenceUnitRuntimes" : [  ],
          "LibraryRuntimes" : null,
          "MaxThreadsConstraintRuntimes" : [  ],
          "MinThreadsConstraintRuntimes" : [  ],
          "Name" : "SubmitOrder",
          "OptionalPackageRuntimes" : [  ],
          "Parent" : { "objectName" : "com.bea:Name=AdminServer,Type=ServerRuntime" },
          "PersistenceUnitRuntimes" : [  ],
          "QueryCacheRuntimes" : [  ],
          "RequestClassRuntimes" : [  ],
          "Type" : "ApplicationRuntime",
          "WorkManagerRuntimes" : [ { "objectName" : "com.bea:ApplicationRuntime=SubmitOrder,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime" } ],
          "WseeRuntimes" : [ { "objectName" : "com.bea:ApplicationRuntime=SubmitOrder,Name=SubmitOrder!EntryWSProdService,ServerRuntime=AdminServer,Type=WseeRuntime" },
              { "objectName" : "com.bea:ApplicationRuntime=SubmitOrder,Name=SubmitOrder!EntryWSService,ServerRuntime=AdminServer,Type=WseeRuntime" }
            ],
          "WseeV2Runtimes" : [  ]
        } }
}

JDBC Data Source: Retrieve Settings for emergencyDB Data Source with Target Managed Server surf1 Running at localhost:7003
http://localhost:7003/jolokia/read/com.bea:Name=emergencyDB,ServerRuntime=surf1,Type=JDBCDataSourceRuntime?ignoreErrors=true
{ "request" : { "mbean" : "com.bea:Name=emergencyDB,ServerRuntime=surf1,Type=JDBCDataSourceRuntime",
      "type" : "read"
    },
  "status" : 200,
  "timestamp" : 1334823502,
  "value" : { "ActiveConnectionsAverageCount" : 0,
      "ActiveConnectionsCurrentCount" : 0,
      "ActiveConnectionsHighCount" : 1,
      "ConnectionDelayTime" : 44,
      "ConnectionsTotalCount" : 1,
      "CurrCapacity" : 1,
      "CurrCapacityHighCount" : 1,
      "DatabaseProductName" : "Apache Derby",
      "DatabaseProductVersion" : "10.6.1.0 - (938214)",
      "DeploymentState" : 2,
      "DriverName" : "Apache Derby Network Client JDBC Driver",
      "DriverVersion" : "10.6.1.0 - (938214)",
      "Enabled" : true,
      "FailedReserveRequestCount" : 0,
      "FailuresToReconnectCount" : 0,
      "HighestNumAvailable" : 1,
      "HighestNumUnavailable" : 1,
      "JDBCDriverRuntime" : { "objectName" : "com.bea:Name=surfandconsulting_surf1_org.apache.derby.jdbc.ClientDriver,ServerRuntime=surf1,Type=JDBCDriverRuntime" },
      "LastTask" : null,
      "LeakedConnectionCount" : 0,
      "ModuleId" : "emergencyDB",
      "Name" : "emergencyDB",
      "NumAvailable" : 1,
      "NumUnavailable" : 0,
      "Parent" : { "objectName" : "com.bea:Name=surf1,Type=ServerRuntime" },
      "PrepStmtCacheAccessCount" : 0,
      "PrepStmtCacheAddCount" : 0,
      "PrepStmtCacheCurrentSize" : 0,
      "PrepStmtCacheDeleteCount" : 0,
      "PrepStmtCacheHitCount" : 0,
      "PrepStmtCacheMissCount" : 0,
      "Properties" : "ERROR: MBean getAttribute failed: weblogic.common.resourcepool.ResourcePermissionsException: User \"\" does not have permission to perform operation \"admin\" on resource \"emergencyDB\" of module \"null\" of application \"null\" of type \"ConnectionPool\" (class javax.management.MBeanException)",
      "ReserveRequestCount" : 1,
      "State" : "Running",
      "Type" : "JDBCDataSourceRuntime",
      "VersionJDBCDriver" : "org.apache.derby.jdbc.ClientDriver",
      "WaitSecondsHighCount" : 0,
      "WaitingForConnectionCurrentCount" : 0,
      "WaitingForConnectionFailureTotal" : 0,
      "WaitingForConnectionHighCount" : 0,
      "WaitingForConnectionSuccessTotal" : 0,
      "WaitingForConnectionTotal" : 0,
      "WorkManagerRuntimes" : null
    }
}

JDBC Connection Pool: Find all JMS Servers with Target set to Managed Server Running at localhost:7003

http://localhost:7003/jolokia/read/com.bea:Name=emergencyDB,ServerRuntime=surf1,Type=JDBCConnectionPoolRuntime?ignoreErrors=true
which returns the JMS server surfJMS:
{ "request" : { "mbean" : "com.bea:Name=emergencyDB,ServerRuntime=surf1,Type=JDBCConnectionPoolRuntime",
      "type" : "read"
    },
  "status" : 200,
  "timestamp" : 1334823105,
  "value" : { "ActiveConnectionsAverageCount" : 0,
      "ActiveConnectionsCurrentCount" : 0,
      "ActiveConnectionsHighCount" : 0,
      "ConnectionDelayTime" : 44,
      "ConnectionLeakProfileCount" : 0,
      "ConnectionsTotalCount" : 1,
      "CurrCapacity" : 1,
      "DeploymentState" : 2,
      "Enabled" : true,
      "FailuresToReconnectCount" : 0,
      "HighestNumAvailable" : 1,
      "HighestNumUnavailable" : 0,
      "LeakedConnectionCount" : 0,
      "MaxCapacity" : 15,
      "ModuleId" : "emergencyDB",
      "Name" : "emergencyDB",
      "NumAvailable" : 1,
      "NumUnavailable" : 0,
      "Parent" : { "objectName" : "com.bea:Name=surf1,Type=ServerRuntime" },
      "PoolState" : true,
      "Properties" : "ERROR: MBean getAttribute failed: weblogic.common.resourcepool.ResourcePermissionsException: User \"\" does not have permission to perform operation \"admin\" on resource \"emergencyDB\" of module \"null\" of application \"null\" of type \"ConnectionPool\" (class javax.management.MBeanException)",
      "State" : "Running",
      "StatementProfileCount" : 0,
      "Type" : "JDBCConnectionPoolRuntime",
      "VersionJDBCDriver" : "org.apache.derby.jdbc.ClientDriver",
      "WaitSecondsHighCount" : 0,
      "WaitingForConnectionCurrentCount" : 0,
      "WaitingForConnectionHighCount" : 0,
      "WorkManagerRuntimes" : null
    }
}

JMS: Find all JMS Servers with Target set to Managed Server Running at localhost:7003
http://localhost:7003/jolokia/search/com.bea:Type=JMSServerRuntime,*
which returns the JMS server surfJMS:
{
   "timestamp":1334776840,
   "status":200,
   "request":{
      "mbean":"com.bea:Type=JMSServerRuntime,*",
      "type":"search"
   },
   "value":[
      "com.bea:Name=surfJMS,ServerRuntime=surf1,Type=JMSServerRuntime"
   ]
}

JMS: Runtime Properties for a Particular Queue
Request the MessagesCurrentCount, MessagesHighCount and MessagesReceivedCount for a queue with the name jms/ShippingRequestQueue (note the “!” is used in the URL to escape the “/” character in the queue name) in a JMS module with the name surfJMSModule (note that JMS module and queue name are typically separated in the MBean Name attribute with a “!” which has to be escaped by another “!”).
http://localhost:7003/jolokia/read/com.bea:JMSServerRuntime=surfJMS,Name=surfJMSModule!!jms!/ShippingRequestQueue,ServerRuntime=surf1,Type=JMSDestinationRuntime/MessagesCurrentCount,MessagesHighCount,MessagesReceivedCount
which returns the following JSON structure.
{
   "request":{
      "attribute":[
         "MessagesCurrentCount",
         "MessagesHighCount",
         "MessagesReceivedCount"
      ],
      "mbean":"com.bea:JMSServerRuntime=surfJMS,Name=surfJMSModule!jms/ShippingRequestQueue,ServerRuntime=surf1,Type=JMSDestinationRuntime",
      "type":"read"
   },
   "status":200,
   "timestamp":1334775025,
   "value":{
      "MessagesCurrentCount":0,
      "MessagesHighCount":1,
      "MessagesReceivedCount":3
   }
}


The output for the queue monitoring shows that there was a total of 3 messages sent, with a maximum of 1 in the queue (because of a deployed receiver retrieving the messages immediately) and currently 0 message in the queue.

Filed Under: oracle Tagged With: availability, best practice, monitoring, weblogic, wls12c

Spicing up WebLogic RESTful Management with Jolokia

April 19, 2012 by frank 1 Comment

[This posting will be part of my upcoming WebLogic 12c book]

Management and monitoring of WebLogic resources is a day-to-day challenge for many administrators. Programming a management solution by writing JMX code in Java is a low-level and time consuming process which can be best avoided using the Jython based WebLogic scripting tool (WLST).

WLST is a higher level, domain specific language (DSL) especially developed to address management issues. A few lines of WLST code typically encapsulate hundreds of lines of Java code using JMX. Although WLST is compact and easy to write it still suffers from the “J” in JMX: Every time you execute a WLST script to monitor some attribute on the application server a JVM has to be started on the client side for WLST. This overhead can become quite substantial if you are monitoring many servers and retrieving attributes at regular intervals. In particular for monitoring systems such as Nagios (or the newer Shinken) the WLST approach is not suitable.

WebLogic 12c RESTful Management Service

One of the distinguishing new features of WebLogic 12c is its RESTful management service which can be enabled by a simple click under DOMAIN / Configuration / General / Advanced / Enable RESTful Management Services followed by a server restart.

Once enabled, you can access the most important WebLogic 12c runtime values using a simple URL syntax from your web browser or a Unix command line tool such as curl. For example you can retrieve runtime data of the administration server with fast HTTP request instead of spawning a JVM process for WLST:

http://localhost:7001/management/tenant-monitoring/servers/AdminServer/

Jolokia

Basics

WebLogic 12c RESTful management is an easy and convinient start, yet there is more possible. The open source framework Jolokia was named after of one of the the hottest chilis on the planet. Jolokia is the kind of chili that most people agree better not to eat (it’s still useful, e.g. to produce military grade tear gas and marine paint).

(Image source: courtesy of R. Huss / Jolokia)

The project is best described by the its developer:

“Jolokia is an agent based approach for remote JMX access. It is an alternative to standard JSR 160 connectors. The communication between client and agent goes over HTTP (either GET or POST), where the request and response payload is represented in JSON. The Jolokia protocol supports the following operations:

- Reading and writing JMX attributes

- Execution of JMX operations

- Searching for MBean Names by pattern

- Listing of MBean Meta-data like supported attributes, operations and notifications”

Installation

The installation process is well described at the Jolokia site. You can simply get the jolokia.war file from the Jolokia download site or the Maven central repository, nothing else is needed for WebLogic.

It’s even possible to use it without any deployment at all. Starting with JDK6 you can run it as an JVM agent, a technique typically used for profilers etc. So your java call starting WebLogic has the following format:

java -javaagent:agent.jar ...

If you like to install the whole Jolokia project (including the JMX shell described later, the Nagios plugin, a Spring Roo addon etc) on a UNIX system with Perl and CPAN already installed it can be as easy as:

[root@ccloud ~]# perl -MCPAN -e shell cpan shell -- CPAN exploration and modules installation (v1.9800)

cpan[1]> install JMX:Jmx4Perl
[... you have to confirm the installation of several dependencies ...]

cpan[2]> exit

After building the Jolokia module it can download the jolokia.war which is then deployed to WebLogic. In case you are playing with an admin server running in development only move it to the DOMAIN_NAME/autodeploy directory otherwise deploy it via the administration console to all servers in the domain:

oracle@ccloud [~]$ jolokia download

* Loading Jolokia meta data from http://www.jolokia.org/jolokia.meta
* Good PGP signature (EF101165)
* Using Jolokia 1.0.3 for Jmx4Perl 1.04
* Downloading war agent version 1.0.3 from repository http://labs.consol.de/maven/repository
* Saved ./jolokia.war
* Good PGP signature (EF101165)

Feature Comparison: Jolokia and WebLogic 12c RESTful Management Service

Now, how does Jolokia compare to what you already have out of the box in WebLogic 12c? Here is a short overview:

	WebLogic 12c Management Service	Jolokia
Installation	Not required: enable from admin console	Simple installation: deploy prebuilt jolokia.war
Availability	Only for WebLogic 12c	Multi vendor, multi version support: WebLogic 9 to 12c Most other application servers Mule agent JVM agent OSGi agent
Approach	Central (available on admin server only)	Can be distributed (target on every server in domain)
Access syntax	Easy, proprietary URL format	Standard MBean name
Return format	HTML, JSON, XML	JSON
Accessible Data	Servers, Clusters, Deployed Apps, DataSources Not supported: JMS	All accessable MBeans and attributes
Security	Requires authentication with admin or monitor role	Standard web based security (can be restricted to certain IPs, roles, HTTPprotocols, etc.)
Documentation	Oracle documentation	Very good online documentation
Access for single attributes only example: HeapMemoryUsage	No	Yes
Access for inner path example HeapMemoryUsage/max	No	Yes
Bulk requests	No	Yes (One HTTPrequest for different MBeans is translated into multiple JMX requests within WebLogic JVM and yet one result is returned)
Additional utilities		JMX shell j4psh Nagios plugin check_jmx4perl

Jolokia Request

The GET URL for a Jolokia read request has the following format:

<base-url>/read/<mbean name>/<attribute name>/<inner path>

To get started just paste the following URL into your browser or use the Unix command line utility curl (assuming you have the admin server running at localhost:7001). It will retrieve the configured ListenPort attributes:

http://localhost:7001/jolokia/read/com.bea:*,Type=Server/ListenPort

which will return the following result for a WebLogic domain with 3 managed servers configured:

{ "request" : { "attribute" : "ListenPort",
      "mbean" : "com.bea:Type=Server,*",
      "type" : "read"
    },
  "status" : 200,
  "timestamp" : 1334776908,
  "value" : { "com.bea:Name=AdminServer,Type=Server" : { "ListenPort" : 7001 },
      "com.bea:Name=surf1,Type=Server" : { "ListenPort" : 7003 },
      "com.bea:Name=surf2,Type=Server" : { "ListenPort" : 7005 },
      "com.bea:Name=surf3,Type=Server" : { "ListenPort" : 7007 }
    }
}

Your response will be displayed in a single line. To get the format above you can past the result into one of the many online JSON formatters (or pipe the curl output to an appropriate utility).
Also I recommend to have a look at some examples about how to use Jolokia from Java, Perl and JavaScript.
Now continue reading part II of this posting with examples showing how to use Jolokia to retrieve monitoring values for servers, deployments, JDBC and JMS.

Filed Under: oracle Tagged With: monitoring, RESTful, weblogic, wls12c

WebLogic Stuck Threads: Creating, Understanding and Dealing with them

April 10, 2012 by frank 23 Comments

Using the time off during the bank holidays over Easter I spent some time coding and looking into more unknown details of WebLogic stuck thread behavior. (Actually I started to write this posting because I was told by my doctor to keep my mouth shut for some days, but that’s another story…).

My personal task was to answer some of the most common questions I’ve encountered while consulting and running WebLogic 12c workshops. As often with my postings, this article is not meant to explain the basic concept of thread pools or workmanagers. I recommend to read the Oracle WebLogic 12c documentation about stuck thread handling first which explains how you can deal with stuck threads by configuring a workmanager.

Also there are some excellent details about stuck threads (including WLST scripting and monitoring) to be found at the Middleware Magic site - a site run by a group of really knowledgeable guys.

Now, typically customers tell me that they “observe some stuck threads”, “sometimes”, but often they are “not sure what caused them” and typically they “don’t know what exact state these thread are in” and in addition nobody seems to know if “the stuck threads ever clear up again without rebooting”. I am a pragmatic guy. I enjoy having little applications or tools to demonstrate and measure how WebLogic is working. Keen to play around with the newest edition of Netbeans (I used to be an Eclipse guy) and EJB 3.1 in WLS12c I built a small application to easily test WebLogic stuck thread settings and countermeasures.

Here are some more details about the StuckThreadForFree application:

The application allows you to create threads which are busy or which are waiting long enough to be detected as “stuck” by WebLogic.
This little application will only work with WLS12c. I intentionally avoided JSF, so a plain JSP page is used to set your parameters. The JSP is calling a simple Servlet which in a for loop is calling an asynchronous business method of an injected stateless session bean. @Asynchronous and no-interface session beans are only available in EJB3.1 so you have to run it on WLS12c. Unlike in previous versions, the EJB is directly packaged into the .war file for deployment.
Every call to the stateless session bean is serialized by the EJB container, so every EJB method is executing in its own thread.
Depending on which method was called on the EJB is either waiting n seconds using Thread.sleep() or calculating some trigonometric function for n seconds. Both methods will cause stuck threads.
There is zero configuration in the deployment descriptors for the EJB! Only context-root for the web part is set (which could be avoided as well).
Building the StuckThreadForFree app with Netbeans was a smooth ride and a real pleasure.
The app is provided as is. You can have it for free, yet there is no guarantee for anything but it shouldn’t cause any problems either. Better don’t run it on your production system.
It’s just a hack. It demonstrates what it should, nothing else.

DOWNLOAD: for your conveninience you can download the StuckThreadForFree.war from here and follow the example yourself (here is the link to the whole Netbeans project). After downloading you can easily deploy it to WebLogic. To follow the example it’s good enough to run it the admin server. Then you can start with the following URL:

http://localhost:7001/StuckThreadForFree

Now, lets use the app to answer some typical questions.

What are hogging threads? When do threads become hogged? After what period of time?

According to the Oracle doc hogging threads “.. will either be declared as stuck after the configured timeout or will return to the pool before that. The self-tuning mechanism will backfill if necessary.”

So how long does it take for them to become hogged? Nobody (including Google) seemed to know. Trust me I did some research and asked plenty of colleagues about this. Here is the answer:

If you run the application with 3 threads / 100 seconds / Thread.sleep() and immediately switch to the WebLogic 12c admin console Admin Server / Monitoring / Threads you will observe the following:

So interestingly hogging threads are detected right away! In my case it took about 2 seconds (I had to hit reload once).

So WebLogic transitions into FAILED state when a certain number of stuck threads are detected, right?

That’s a common misconception! The default configuration of WLS 12c (I also checked for WLS 11 = 10.3.3) is Stuck Thread Count = 0, which means the server “never transitions into FAILED server irrespective of the number of stuck threads”. You will only see the FAILED state only when you set the value to a positive number of threads!

Once the server transitions into FAILED, you can define if WLS should be shut down (and restarted by WLS nodemanager) or suspended.

Remember: WLS will not transition into FAILED state when StuckThreadCount is set to zero. Only the health runtime value is set to Warning (but this will be cleared if the hogging thread conditions clears) as shown below:

What exactly causes a stuck thread? What state does a thread have to be in to be marked as stuck?

In general there is a number of different thread states in Java: NEW, RUNNABLE, BLOCKED, WAITING, TIMED_WAITING, TERMINATED.

But which state has a thread to be in to be marked as stuck later? If you run the StuckThreadForFree application and create a stack trace with WebLogic admin console under Server / ServerName / Monitoring / Threads you can observe that the thread state is ACTIVE/TIMED_WAITING when using the Thread.sleep() method to block it:

"[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" TIMED_WAITING
            	java.lang.Thread.sleep(Native Method)
            	com.munzandmore.stuckthread.LongRunningEJB.threadSleep(LongRunningEJB.java:26)
            	com.munzandmore.stuckthread.LongRunningEJB_x9v26k_NoIntfViewImpl.__WL_invoke(Unknown Source)

when using the calc() method to keep the threads busy they are state ACTIVE/RUNNABLE :

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE
            	com.munzandmore.stuckthread.LongRunningEJB.threadCalc(LongRunningEJB.java:40)
            	com.munzandmore.stuckthread.LongRunningEJB_x9v26k_NoIntfViewImpl.__WL_invoke(Unknown Source)
            	weblogic.ejb.container.internal.SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:31)

So both states can become stuck. Also, I am pretty sure I could also show the BLOCKED state when using a monitor lock for synchronization but due to time restrictions this is not included in the app.

Can a stuck thread still do reasonable work?

Absolutely! Just because a thread is marked as stuck it doesn’t mean it is frozen or unusable. Imagine you wanted to calculate PI, you are creating PDFs, distance maps, mapping the human genome or you have deployed some JCA adapter talking to MQ-Series, SAP or PeopleSoft which is internally using a Thread.sleep() method call. All of this is are reasonable usages likely to occur in the wild.

Do stuck threads ever dissapear? Can they be cleared somehow? Are they stuck forever?

First of all you cannot get rid of a stuck thread by simply “killing it”. You cannot cancel or kill any thread in Java. However, stuck threads automatically will disappear if the condition clears up which caused them to be marked as stuck (e.g. the sleep period is over or the calculation is done).

To prove the point, switch to the WebLogic admin console and under Server / ServerName / Configuration set StuckThreadCount to 3 and StuckThreadTime to 60 seconds then restart the server and run the StuckThreadForFree app to create 3 threads running for 120 seconds using the Thread.sleep() method (the other method will work as well, there is no difference, but keeping 3 threads busy by doing math proves to be a fan test of your machine as well):

In the WebLogic log file you will find three entries logging the stuck thread state after a while:

<05.04.2012 10:55 Uhr MESZ> <Critical> <WebLogicServer> <BEA-000385> <Server health failed. Reason: health of
critical service 'Thread Pool' failed>
<05.04.2012 10:55 Uhr MESZ> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '4' for queue: 'webl
ogic.kernel.Default (self-tuning)' has been busy for "85" seconds working on the request "Workmanager: default
, Version: 1, Scheduled=false, Started=true, Started time: 85443 ms
", which is more than the configured time (StuckThreadMaxTime) of "60" seconds. Stack trace:
 java.lang.Thread.sleep(Native Method)

After waiting about one minute you will observe that WebLogic is transitioning into FAILED state as configured:

Wait another minute, then check the thread states under Server / ServerName / Monitoring / Threads which reveals the following:

So once the condition causing the stuck threads is cleared also the stuck threads will disappear again! Stuck threads are not stuck forever. Phew!

When should I use StuckThreadCount in the admin console or a Workmanager stuck-thread setting then?

Very good question. Use StuckThreadCount from the WebLogic admin console or with a <work-manager-shutdown-trigger> definition moving the application into ADMIN mode if you can react on the FAILED state.

Do not use StuckThreadCount if the threads might be doing something useful and you cannot react on the situation anyway. Obviously transitioning into FAILED state and restarting WLS with the nodemanager is counterproductive if you threads are doing something useful.

More?

The following posting shows how simple tools like ps, top and jcmd can track down the exact line of Java code causing a thread to use a high amount of CPU. Exactly the same StuckThreadForFree application is used as here.

Filed Under: oracle Tagged With: availability, performance, weblogic, wls12c

WebLogic Feature Timeline: What is new in WLS 12, 11, 10, 9?

February 24, 2012 by frank 11 Comments

To whom it may concern, here is a quick overview of the new features as they were included in WebLogic Server versions 9 to 12c. Destilled for my first WebLogic 12c workshop running a fortnight ago and including some links to the WebLogic documentation and some other useful resources.

happy weekend then … !

Download WebLogic Feature Timeline (PDF): What is new in WebLogic 12c? WebLogic 11? WebLogic 10? WebLogic 9?

Update (05-March-2012):I just uploaded a newer version after adding valuable input from Steve and Jan. (Due to other changes, those of you linking to the PDF directly need to update their link this time).

Filed Under: oracle Tagged With: weblogic, wls12c, workshop

Audience

Technology

Download

Hands-on Test

Download

Results

Differences

Configuration

What you should know

j4psh JMX Shell

Screencasts

WebLogic with Jolokia

General: List Configuration Details of Managed Server with Name surf1

Deployment: List all Deployments

Deployment: List Deployments Details for an Application with the Name SubmitOrder

JDBC Data Source: Retrieve Settings for emergencyDB Data Source with Target Managed Server surf1 Running at localhost:7003

JDBC Connection Pool: Find all JMS Servers with Target set to Managed Server Running at localhost:7003

JMS: Find all JMS Servers with Target set to Managed Server Running at localhost:7003

JMS: Runtime Properties for a Particular Queue

WebLogic 12c RESTful Management Service

Jolokia

Basics

Installation

Feature Comparison: Jolokia and WebLogic 12c RESTful Management Service

Jolokia Request

What are hogging threads? When do threads become hogged? After what period of time?

So WebLogic transitions into FAILED state when a certain number of stuck threads are detected, right?

What exactly causes a stuck thread? What state does a thread have to be in to be marked as stuck?

Can a stuck thread still do reasonable work?

Do stuck threads ever dissapear? Can they be cleared somehow? Are they stuck forever?

When should I use StuckThreadCount in the admin console or a Workmanager stuck-thread setting then?

More?

Tags and Clouds

WebLogic 12c Book

Twitter

Cloud Book