Frank Munz's Blog: Oracle Fusion Middleware and Cloud Computing

RESTful WebLogic Monitoring of Servers, Applications, JDBC and JMS with Jolokia

April 19, 2012 by frank 5 Comments

[This posting will be a part of my upcoming WebLogic 12c book]

This article is part II of III. Make sure you read part I explaining the basics of monitoring WebLogic with WLST, the WebLogic 12c REST-ful management API and the open source framework Jolokia.

There will be a part III in a few of days, explaining how to use j4psh with WebLogic. j4psh is a JMX shell that resembles WLST in interactive mode but includes features such as syntax highlighting and tab-completion of commands and MBean names. j4psh is highly useful to interactively find out the correct MBean names used for the requests below.

WebLogic with Jolokia

Did you ever wonder how to retrieve the configuration values for a WebLogic managed server, runtime data of a JDBC connection pool or the number of messages in a particular JMS queue with a fast and simple HTTP GET?

The examples show typical use cases for monitoring a WebLogic domain with the following configuration:

- Admin server running at localhost:7001
- Managed server with name surf1 running at localhost:7003
- JDBC datasource with the name emergencyDB and associated connection pool (target set to surf1)
- JMS configuration with a JMS server surfJMS running on managed server surf1, a module surfJMSModule containing a queue name jms/ShippingRequestQueue (target of JMS server is surf1, JMS module subdeployment set to surfJMS)
- A deployment of a web service with the name SubmitOrder and target AdminServer and surf1.

An installation of Jolokia with target set to all server of the domain is recommended to follow the examples in this part.

General: List Configuration Details of Managed Server with Name surf1

http://localhost:7001/jolokia/read/com.bea:Name=surf1,Type=Server?ignoreErrors=true

Note, that without the ?ignoreErrors=true parameter the request would fail with an “NoAccessRuntimeException” because you cannot access the SSL keystore passphrases without proper authentication. Authentication is easily possible, but beyond of the scope of this article.

[JavaScript]
{ "request" : { "mbean" : "com.bea:Name=surf1,Type=Server",
      "type" : "read"
    },
  "status" : 200,
  "timestamp" : 1334778534,
  "value" : { "AcceptBacklog" : 300,
      "AddWorkManagerThreadsByCpuCount" : false,
      "AdminReconnectIntervalSeconds" : 10,
      "AdministrationPort" : 9002,
      "AdministrationPortEnabled" : false,
      "AdministrationProtocol" : "t3s",
      "AutoKillIfFailed" : false,
      "AutoMigrationEnabled" : false,
      "AutoRestart" : true,
      "COM" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=COM" },
      "COMEnabled" : false,
      "CandidateMachines" : [  ],
      "ClasspathServletDisabled" : false,
      "ClientCertProxyEnabled" : false,
      "Cluster" : { "objectName" : "com.bea:Name=surfCluster,Type=Cluster" },
      "ClusterRuntime" : null,
      "ClusterWeight" : 100,
      "CoherenceClusterSystemResource" : null,
      "CompleteCOMMessageTimeout" : -1,
      "CompleteHTTPMessageTimeout" : -1,
      "CompleteIIOPMessageTimeout" : -1,
      "CompleteMessageTimeout" : 60,
      "CompleteT3MessageTimeout" : -1,
      "ConnectTimeout" : 0,
      "ConsensusProcessIdentifier" : -1,
      "ConsoleInputEnabled" : false,
      "CustomIdentityKeyStoreFileName" : null,
      "CustomIdentityKeyStorePassPhrase" : "ERROR: Access to sensitive attribute in clear text is not allowed due to the setting of ClearTextCredentialAccessEnabled attribute in SecurityConfigurationMBean. Attr: CustomIdentityKeyStorePassPhrase, MBean name: com.bea:Name=surf1,Type=Server (class weblogic.management.NoAccessRuntimeException)",
      "CustomIdentityKeyStorePassPhraseEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: CustomIdentityKeyStorePassPhraseEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "CustomIdentityKeyStoreType" : null,
      "CustomTrustKeyStoreFileName" : null,
      "CustomTrustKeyStorePassPhrase" : "ERROR: Access to sensitive attribute in clear text is not allowed due to the setting of ClearTextCredentialAccessEnabled attribute in SecurityConfigurationMBean. Attr: CustomTrustKeyStorePassPhrase, MBean name: com.bea:Name=surf1,Type=Server (class weblogic.management.NoAccessRuntimeException)",
      "CustomTrustKeyStorePassPhraseEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: CustomTrustKeyStorePassPhraseEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "CustomTrustKeyStoreType" : null,
      "DGCIdlePeriodsUntilTimeout" : 5,
      "DataSource" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=DataSource" },
      "DefaultFileStore" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=DefaultFileStore" },
      "DefaultIIOPPassword" : "ERROR: Access to sensitive attribute in clear text is not allowed due to the setting of ClearTextCredentialAccessEnabled attribute in SecurityConfigurationMBean. Attr: DefaultIIOPPassword, MBean name: com.bea:Name=surf1,Type=Server (class weblogic.management.NoAccessRuntimeException)",
      "DefaultIIOPPasswordEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: DefaultIIOPPasswordEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "DefaultIIOPUser" : null,
      "DefaultInternalServletsDisabled" : false,
      "DefaultProtocol" : "t3",
      "DefaultSecureProtocol" : "t3s",
      "DefaultTGIOPPassword" : "ERROR: Access to sensitive attribute in clear text is not allowed due to the setting of ClearTextCredentialAccessEnabled attribute in SecurityConfigurationMBean. Attr: DefaultTGIOPPassword, MBean name: com.bea:Name=surf1,Type=Server (class weblogic.management.NoAccessRuntimeException)",
      "DefaultTGIOPPasswordEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: DefaultTGIOPPasswordEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "DefaultTGIOPUser" : "guest",
      "ExecuteQueues" : [  ],
      "ExpectedToRun" : true,
      "ExternalDNSName" : null,
      "ExtraEjbcOptions" : null,
      "ExtraRmicOptions" : null,
      "FederationServices" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=FederationServices" },
      "GatheredWritesEnabled" : false,
      "GracefulShutdownTimeout" : 0,
      "HealthCheckIntervalSeconds" : 180,
      "HealthCheckStartDelaySeconds" : 120,
      "HealthCheckTimeoutSeconds" : 60,
      "HostsMigratableServices" : true,
      "HttpTraceSupportEnabled" : false,
      "HttpdEnabled" : true,
      "IIOP" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=IIOP" },
      "IIOPEnabled" : true,
      "IIOPTxMechanism" : "ots",
      "IdleConnectionTimeout" : 65,
      "IdleIIOPConnectionTimeout" : -1,
      "IdlePeriodsUntilTimeout" : 4,
      "IgnoreSessionsDuringShutdown" : false,
      "InstrumentStackTraceEnabled" : true,
      "InterfaceAddress" : null,
      "JDBCLLRTableName" : null,
      "JDBCLoggingEnabled" : false,
      "JDBCLoginTimeoutSeconds" : 0,
      "JMSDefaultConnectionFactoriesEnabled" : true,
      "JMSThreadPoolSize" : 15,
      "JNDITransportableObjectFactoryList" : [  ],
      "JTAMigratableTarget" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=JTAMigratableTarget" },
      "JavaCompiler" : "javac",
      "JavaCompilerPostClassPath" : null,
      "JavaCompilerPreClassPath" : null,
      "JavaStandardTrustKeyStorePassPhrase" : "ERROR: Access to sensitive attribute in clear text is not allowed due to the setting of ClearTextCredentialAccessEnabled attribute in SecurityConfigurationMBean. Attr: JavaStandardTrustKeyStorePassPhrase, MBean name: com.bea:Name=surf1,Type=Server (class weblogic.management.NoAccessRuntimeException)",
      "JavaStandardTrustKeyStorePassPhraseEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: JavaStandardTrustKeyStorePassPhraseEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "KernelDebug" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=ServerDebug" },
      "KeyStores" : "DemoIdentityAndDemoTrust",
      "ListenAddress" : "",
      "ListenDelaySecs" : 0,
      "ListenPort" : 7003,
      "ListenPortEnabled" : true,
      "ListenThreadStartDelaySecs" : 60,
      "ListenersBindEarly" : false,
      "Log" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=Log" },
      "LogRemoteExceptionsEnabled" : false,
      "LoginTimeoutMillis" : 5000,
      "LowMemoryGCThreshold" : 5,
      "LowMemoryGranularityLevel" : 5,
      "LowMemorySampleSize" : 10,
      "LowMemoryTimeInterval" : 3600,
      "MSIFileReplicationEnabled" : false,
      "Machine" : null,
      "ManagedServerIndependenceEnabled" : true,
      "MaxCOMMessageSize" : -1,
      "MaxHTTPMessageSize" : -1,
      "MaxIIOPMessageSize" : -1,
      "MaxMessageSize" : 10000000,
      "MaxOpenSockCount" : -1,
      "MaxT3MessageSize" : -1,
      "MessageIdPrefixEnabled" : false,
      "MessagingBridgeThreadPoolSize" : 5,
      "MuxerClass" : null,
      "NMSocketCreateTimeoutInMillis" : 180000,
      "Name" : "surf1",
      "NativeIOEnabled" : true,
      "NetworkAccessPoints" : [  ],
      "Notes" : null,
      "OutboundEnabled" : false,
      "OutboundPrivateKeyEnabled" : false,
      "OverloadProtection" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=OverloadProtection" },
      "Parent" : { "objectName" : "com.bea:Name=surfandconsulting,Type=Domain" },
      "PeriodLength" : 60000,
      "PreferredSecondaryGroup" : null,
      "ReliableDeliveryPolicy" : null,
      "ReplicationGroup" : null,
      "ReplicationPorts" : null,
      "RestartDelaySeconds" : 0,
      "RestartIntervalSeconds" : 3600,
      "RestartMax" : 2,
      "ReverseDNSAllowed" : false,
      "SSL" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=SSL" },
      "ScatteredReadsEnabled" : false,
      "SelfTuningThreadPoolSizeMax" : 400,
      "SelfTuningThreadPoolSizeMin" : 1,
      "ServerDebug" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=ServerDebug" },
      "ServerDiagnosticConfig" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=WLDFServerDiagnostic" },
      "ServerLifeCycleTimeoutVal" : 30,
      "ServerStart" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=ServerStart" },
      "ServerVersion" : "unknown",
      "SingleSignOnServices" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=SingleSignOnServices" },
      "SocketBufferSizeAsChunkSize" : false,
      "SocketReaders" : -1,
      "StagingDirectoryName" : "/u01/domains/surfandconsulting/servers/surf1/stage",
      "StagingMode" : "stage",
      "StartupMode" : "RUNNING",
      "StartupTimeout" : 0,
      "StuckThreadMaxTime" : 600,
      "StuckThreadTimerInterval" : 60,
      "SystemPasswordEncrypted" : "ERROR: Access not allowed for subject: principals=[], on ResourceType: Server Action: read, Target: SystemPasswordEncrypted (class weblogic.management.NoAccessRuntimeException)",
      "TGIOPEnabled" : true,
      "ThreadPoolPercentSocketReaders" : 33,
      "TimedOutRefIsolationTime" : 0,
      "TransactionLogFilePrefix" : "./",
      "TransactionLogFileWritePolicy" : "Direct-Write",
      "TransactionLogJDBCStore" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=TransactionLogJDBCStore" },
      "TunnelingClientPingSecs" : 45,
      "TunnelingClientTimeoutSecs" : 40,
      "TunnelingEnabled" : false,
      "Type" : "Server",
      "UploadDirectoryName" : "./servers/surf1/upload",
      "Use81StyleExecuteQueues" : false,
      "UseConcurrentQueueForRequestManager" : false,
      "UseFusionForLLR" : false,
      "VerboseEJBDeploymentEnabled" : "false",
      "VirtualMachineName" : "surfandconsulting_surf1",
      "WebServer" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=WebServer" },
      "WebService" : { "objectName" : "com.bea:Name=surf1,Server=surf1,Type=WebService" },
      "WeblogicPluginEnabled" : false,
      "XMLEntityCache" : null,
      "XMLRegistry" : null
    }
}
[/JavaScript]

Deployment: List all Deployments

http://localhost:7001/jolokia/search/com.bea:*,Type=ApplicationRuntime

[JavaScript]
{ "request" : { "mbean" : "com.bea:Type=ApplicationRuntime,*",
 "type" : "search"
 },
 "status" : 200,
 "timestamp" : 1334780578,
 "value" : [ "com.bea:Name=mejb,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=consoleapp,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=bea_wls_management_internal2,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=emergencyDB,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=bea_wls_deployment_internal,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=SubmitOrder,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=wls-management-services,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=bea_wls_internal,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=jolokia,ServerRuntime=AdminServer,Type=ApplicationRuntime",
 "com.bea:Name=bea_wls9_async_response,ServerRuntime=AdminServer,Type=ApplicationRuntime"
 ]
 }[/JavaScript]

Deployment: List Deployments Details for an Application with the Name SubmitOrder

[JavaScript]http://localhost:7001/jolokia/read/com.bea:ServerRuntime=*,Name=SubmitOrder,Type=ApplicationRuntime[/JavaScript]

[JavaScript]
{ "request" : { "mbean" : "com.bea:Name=SubmitOrder,ServerRuntime=*,Type=ApplicationRuntime",
      "type" : "read"
    },
  "status" : 200,
  "timestamp" : 1334777478,
  "value" : { "com.bea:Name=SubmitOrder,ServerRuntime=AdminServer,Type=ApplicationRuntime" : { "ActiveVersionState" : 2,
          "ApplicationName" : "SubmitOrder",
          "ApplicationVersion" : null,
          "ClassRedefinitionRuntime" : null,
          "CoherenceClusterRuntime" : null,
          "ComponentRuntimes" : [ { "objectName" : "com.bea:ApplicationRuntime=SubmitOrder,Name=AdminServer_/SubmitOrder,ServerRuntime=AdminServer,Type=WebAppComponentRuntime" } ],
          "EAR" : false,
          "HealthState" : { "critical" : false,
              "mBeanName" : null,
              "mBeanType" : null,
              "reasonCode" : [  ],
              "state" : 0,
              "subsystemName" : null
            },
          "KodoPersistenceUnitRuntimes" : [  ],
          "LibraryRuntimes" : null,
          "MaxThreadsConstraintRuntimes" : [  ],
          "MinThreadsConstraintRuntimes" : [  ],
          "Name" : "SubmitOrder",
          "OptionalPackageRuntimes" : [  ],
          "Parent" : { "objectName" : "com.bea:Name=AdminServer,Type=ServerRuntime" },
          "PersistenceUnitRuntimes" : [  ],
          "QueryCacheRuntimes" : [  ],
          "RequestClassRuntimes" : [  ],
          "Type" : "ApplicationRuntime",
          "WorkManagerRuntimes" : [ { "objectName" : "com.bea:ApplicationRuntime=SubmitOrder,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime" } ],
          "WseeRuntimes" : [ { "objectName" : "com.bea:ApplicationRuntime=SubmitOrder,Name=SubmitOrder!EntryWSProdService,ServerRuntime=AdminServer,Type=WseeRuntime" },
              { "objectName" : "com.bea:ApplicationRuntime=SubmitOrder,Name=SubmitOrder!EntryWSService,ServerRuntime=AdminServer,Type=WseeRuntime" }
            ],
          "WseeV2Runtimes" : [  ]
        } }
}
[/JavaScript]

JDBC Data Source: Retrieve Settings for emergencyDB Data Source with Target Managed Server surf1 Running at localhost:7003

[JavaScript]http://localhost:7003/jolokia/read/com.bea:Name=emergencyDB,ServerRuntime=surf1,Type=JDBCDataSourceRuntime?ignoreErrors=true[/JavaScript]

[JavaScript]
{ "request" : { "mbean" : "com.bea:Name=emergencyDB,ServerRuntime=surf1,Type=JDBCDataSourceRuntime",
      "type" : "read"
    },
  "status" : 200,
  "timestamp" : 1334823502,
  "value" : { "ActiveConnectionsAverageCount" : 0,
      "ActiveConnectionsCurrentCount" : 0,
      "ActiveConnectionsHighCount" : 1,
      "ConnectionDelayTime" : 44,
      "ConnectionsTotalCount" : 1,
      "CurrCapacity" : 1,
      "CurrCapacityHighCount" : 1,
      "DatabaseProductName" : "Apache Derby",
      "DatabaseProductVersion" : "10.6.1.0 - (938214)",
      "DeploymentState" : 2,
      "DriverName" : "Apache Derby Network Client JDBC Driver",
      "DriverVersion" : "10.6.1.0 - (938214)",
      "Enabled" : true,
      "FailedReserveRequestCount" : 0,
      "FailuresToReconnectCount" : 0,
      "HighestNumAvailable" : 1,
      "HighestNumUnavailable" : 1,
      "JDBCDriverRuntime" : { "objectName" : "com.bea:Name=surfandconsulting_surf1_org.apache.derby.jdbc.ClientDriver,ServerRuntime=surf1,Type=JDBCDriverRuntime" },
      "LastTask" : null,
      "LeakedConnectionCount" : 0,
      "ModuleId" : "emergencyDB",
      "Name" : "emergencyDB",
      "NumAvailable" : 1,
      "NumUnavailable" : 0,
      "Parent" : { "objectName" : "com.bea:Name=surf1,Type=ServerRuntime" },
      "PrepStmtCacheAccessCount" : 0,
      "PrepStmtCacheAddCount" : 0,
      "PrepStmtCacheCurrentSize" : 0,
      "PrepStmtCacheDeleteCount" : 0,
      "PrepStmtCacheHitCount" : 0,
      "PrepStmtCacheMissCount" : 0,
      "Properties" : "ERROR: MBean getAttribute failed: weblogic.common.resourcepool.ResourcePermissionsException: User \"\" does not have permission to perform operation \"admin\" on resource \"emergencyDB\" of module \"null\" of application \"null\" of type \"ConnectionPool\" (class javax.management.MBeanException)",
      "ReserveRequestCount" : 1,
      "State" : "Running",
      "Type" : "JDBCDataSourceRuntime",
      "VersionJDBCDriver" : "org.apache.derby.jdbc.ClientDriver",
      "WaitSecondsHighCount" : 0,
      "WaitingForConnectionCurrentCount" : 0,
      "WaitingForConnectionFailureTotal" : 0,
      "WaitingForConnectionHighCount" : 0,
      "WaitingForConnectionSuccessTotal" : 0,
      "WaitingForConnectionTotal" : 0,
      "WorkManagerRuntimes" : null
    }
}
[/JavaScript]

JDBC Connection Pool: Find all JMS Servers with Target set to Managed Server Running at localhost:7003

[JavaScript]

http://localhost:7003/jolokia/read/com.bea:Name=emergencyDB,ServerRuntime=surf1,Type=JDBCConnectionPoolRuntime?ignoreErrors=true[/JavaScript]

which returns the JMS server surfJMS:

[JavaScript]
{ "request" : { "mbean" : "com.bea:Name=emergencyDB,ServerRuntime=surf1,Type=JDBCConnectionPoolRuntime",
      "type" : "read"
    },
  "status" : 200,
  "timestamp" : 1334823105,
  "value" : { "ActiveConnectionsAverageCount" : 0,
      "ActiveConnectionsCurrentCount" : 0,
      "ActiveConnectionsHighCount" : 0,
      "ConnectionDelayTime" : 44,
      "ConnectionLeakProfileCount" : 0,
      "ConnectionsTotalCount" : 1,
      "CurrCapacity" : 1,
      "DeploymentState" : 2,
      "Enabled" : true,
      "FailuresToReconnectCount" : 0,
      "HighestNumAvailable" : 1,
      "HighestNumUnavailable" : 0,
      "LeakedConnectionCount" : 0,
      "MaxCapacity" : 15,
      "ModuleId" : "emergencyDB",
      "Name" : "emergencyDB",
      "NumAvailable" : 1,
      "NumUnavailable" : 0,
      "Parent" : { "objectName" : "com.bea:Name=surf1,Type=ServerRuntime" },
      "PoolState" : true,
      "Properties" : "ERROR: MBean getAttribute failed: weblogic.common.resourcepool.ResourcePermissionsException: User \"\" does not have permission to perform operation \"admin\" on resource \"emergencyDB\" of module \"null\" of application \"null\" of type \"ConnectionPool\" (class javax.management.MBeanException)",
      "State" : "Running",
      "StatementProfileCount" : 0,
      "Type" : "JDBCConnectionPoolRuntime",
      "VersionJDBCDriver" : "org.apache.derby.jdbc.ClientDriver",
      "WaitSecondsHighCount" : 0,
      "WaitingForConnectionCurrentCount" : 0,
      "WaitingForConnectionHighCount" : 0,
      "WorkManagerRuntimes" : null
    }
}
[/JavaScript]

JMS: Find all JMS Servers with Target set to Managed Server Running at localhost:7003

http://localhost:7003/jolokia/search/com.bea:Type=JMSServerRuntime,*

which returns the JMS server surfJMS:

[JavaScript]
{
   "timestamp":1334776840,
   "status":200,
   "request":{
      "mbean":"com.bea:Type=JMSServerRuntime,*",
      "type":"search"
   },
   "value":[
      "com.bea:Name=surfJMS,ServerRuntime=surf1,Type=JMSServerRuntime"
   ]
}
[/JavaScript]

JMS: Runtime Properties for a Particular Queue

Request the MessagesCurrentCount, MessagesHighCount and MessagesReceivedCount for a queue with the name jms/ShippingRequestQueue (note the “!” is used in the URL to escape the “/” character in the queue name) in a JMS module with the name surfJMSModule (note that JMS module and queue name are typically separated in the MBean Name attribute with a “!” which has to be escaped by another “!”).

[JavaScript]http://localhost:7003/jolokia/read/com.bea:JMSServerRuntime=surfJMS,Name=surfJMSModule!!jms!/ShippingRequestQueue,ServerRuntime=surf1,Type=JMSDestinationRuntime/MessagesCurrentCount,MessagesHighCount,MessagesReceivedCount[/JavaScript]

which returns the following JSON structure.

[JavaScript]
{
   "request":{
      "attribute":[
         "MessagesCurrentCount",
         "MessagesHighCount",
         "MessagesReceivedCount"
      ],
      "mbean":"com.bea:JMSServerRuntime=surfJMS,Name=surfJMSModule!jms/ShippingRequestQueue,ServerRuntime=surf1,Type=JMSDestinationRuntime",
      "type":"read"
   },
   "status":200,
   "timestamp":1334775025,
   "value":{
      "MessagesCurrentCount":0,
      "MessagesHighCount":1,
      "MessagesReceivedCount":3
   }
}

[/JavaScript]

The output for the queue monitoring shows that there was a total of 3 messages sent, with a maximum of 1 in the queue (because of a deployed receiver retrieving the messages immediately) and currently 0 message in the queue.

Filed Under: oracle Tagged With: availability, best practice, monitoring, weblogic, wls12c

Spicing up WebLogic RESTful Management with Jolokia

April 19, 2012 by frank 1 Comment

[This posting will be part of my upcoming WebLogic 12c book]

Management and monitoring of WebLogic resources is a day-to-day challenge for many administrators. Programming a management solution by writing JMX code in Java is a low-level and time consuming process which can be best avoided using the Jython based WebLogic scripting tool (WLST).

WLST is a higher level, domain specific language (DSL) especially developed to address management issues. A few lines of WLST code typically encapsulate hundreds of lines of Java code using JMX. Although WLST is compact and easy to write it still suffers from the “J” in JMX: Every time you execute a WLST script to monitor some attribute on the application server a JVM has to be started on the client side for WLST. This overhead can become quite substantial if you are monitoring many servers and retrieving attributes at regular intervals. In particular for monitoring systems such as Nagios (or the newer Shinken) the WLST approach is not suitable.

WebLogic 12c RESTful Management Service

One of the distinguishing new features of WebLogic 12c is its RESTful management service which can be enabled by a simple click under DOMAIN / Configuration / General / Advanced / Enable RESTful Management Services followed by a server restart.

Once enabled, you can access the most important WebLogic 12c runtime values using a simple URL syntax from your web browser or a Unix command line tool such as curl. For example you can retrieve runtime data of the administration server with fast HTTP request instead of spawning a JVM process for WLST:

http://localhost:7001/management/tenant-monitoring/servers/AdminServer/

Jolokia

Basics

WebLogic 12c RESTful management is an easy and convinient start, yet there is more possible. The open source framework Jolokia was named after of one of the the hottest chilis on the planet. Jolokia is the kind of chili that most people agree better not to eat (it’s still useful, e.g. to produce military grade tear gas and marine paint).

(Image source: courtesy of R. Huss / Jolokia)

The project is best described by the its developer:

“Jolokia is an agent based approach for remote JMX access. It is an alternative to standard JSR 160 connectors. The communication between client and agent goes over HTTP (either GET or POST), where the request and response payload is represented in JSON. The Jolokia protocol supports the following operations:

- Reading and writing JMX attributes

- Execution of JMX operations

- Searching for MBean Names by pattern

- Listing of MBean Meta-data like supported attributes, operations and notifications”

Installation

The installation process is well described at the Jolokia site. You can simply get the jolokia.war file from the Jolokia download site or the Maven central repository, nothing else is needed for WebLogic.

It’s even possible to use it without any deployment at all. Starting with JDK6 you can run it as an JVM agent, a technique typically used for profilers etc. So your java call starting WebLogic has the following format:

java -javaagent:agent.jar ...

If you like to install the whole Jolokia project (including the JMX shell described later, the Nagios plugin, a Spring Roo addon etc) on a UNIX system with Perl and CPAN already installed it can be as easy as:

[root@ccloud ~]# perl -MCPAN -e shell cpan shell -- CPAN exploration and modules installation (v1.9800)

cpan[1]> install JMX:Jmx4Perl
[... you have to confirm the installation of several dependencies ...]

cpan[2]> exit

After building the Jolokia module it can download the jolokia.war which is then deployed to WebLogic. In case you are playing with an admin server running in development only move it to the DOMAIN_NAME/autodeploy directory otherwise deploy it via the administration console to all servers in the domain:

[bash]
oracle@ccloud [~]$ jolokia download

* Loading Jolokia meta data from http://www.jolokia.org/jolokia.meta
* Good PGP signature (EF101165)
* Using Jolokia 1.0.3 for Jmx4Perl 1.04
* Downloading war agent version 1.0.3 from repository http://labs.consol.de/maven/repository
* Saved ./jolokia.war
* Good PGP signature (EF101165)
[/bash]

Feature Comparison: Jolokia and WebLogic 12c RESTful Management Service

Now, how does Jolokia compare to what you already have out of the box in WebLogic 12c? Here is a short overview:

	WebLogic 12c Management Service	Jolokia
Installation	Not required: enable from admin console	Simple installation: deploy prebuilt jolokia.war
Availability	Only for WebLogic 12c	Multi vendor, multi version support: WebLogic 9 to 12c Most other application servers Mule agent JVM agent OSGi agent
Approach	Central (available on admin server only)	Can be distributed (target on every server in domain)
Access syntax	Easy, proprietary URL format	Standard MBean name
Return format	HTML, JSON, XML	JSON
Accessible Data	Servers, Clusters, Deployed Apps, DataSources Not supported: JMS	All accessable MBeans and attributes
Security	Requires authentication with admin or monitor role	Standard web based security (can be restricted to certain IPs, roles, HTTPprotocols, etc.)
Documentation	Oracle documentation	Very good online documentation
Access for single attributes only example: HeapMemoryUsage	No	Yes
Access for inner path example HeapMemoryUsage/max	No	Yes
Bulk requests	No	Yes (One HTTPrequest for different MBeans is translated into multiple JMX requests within WebLogic JVM and yet one result is returned)
Additional utilities		JMX shell j4psh Nagios plugin check_jmx4perl

Jolokia Request

The GET URL for a Jolokia read request has the following format:

<base-url>/read/<mbean name>/<attribute name>/<inner path>

To get started just paste the following URL into your browser or use the Unix command line utility curl (assuming you have the admin server running at localhost:7001). It will retrieve the configured ListenPort attributes:

http://localhost:7001/jolokia/read/com.bea:*,Type=Server/ListenPort

which will return the following result for a WebLogic domain with 3 managed servers configured:

[JavaScript]
{ "request" : { "attribute" : "ListenPort",
      "mbean" : "com.bea:Type=Server,*",
      "type" : "read"
    },
  "status" : 200,
  "timestamp" : 1334776908,
  "value" : { "com.bea:Name=AdminServer,Type=Server" : { "ListenPort" : 7001 },
      "com.bea:Name=surf1,Type=Server" : { "ListenPort" : 7003 },
      "com.bea:Name=surf2,Type=Server" : { "ListenPort" : 7005 },
      "com.bea:Name=surf3,Type=Server" : { "ListenPort" : 7007 }
    }
}
[/JavaScript]

Your response will be displayed in a single line. To get the format above you can past the result into one of the many online JSON formatters (or pipe the curl output to an appropriate utility).

Also I recommend to have a look at some examples about how to use Jolokia from Java, Perl and JavaScript.

Now continue reading part II of this posting with examples showing how to use Jolokia to retrieve monitoring values for servers, deployments, JDBC and JMS.

Filed Under: oracle Tagged With: monitoring, RESTful, weblogic, wls12c

WebLogic Stuck Threads: Creating, Understanding and Dealing with them

April 10, 2012 by frank 23 Comments

Using the time off during the bank holidays over Easter I spent some time coding and looking into more unknown details of WebLogic stuck thread behavior. (Actually I started to write this posting because I was told by my doctor to keep my mouth shut for some days, but that’s another story…).

My personal task was to answer some of the most common questions I’ve encountered while consulting and running WebLogic 12c workshops. As often with my postings, this article is not meant to explain the basic concept of thread pools or workmanagers. I recommend to read the Oracle WebLogic 12c documentation about stuck thread handling first which explains how you can deal with stuck threads by configuring a workmanager.

Also there are some excellent details about stuck threads (including WLST scripting and monitoring) to be found at the Middleware Magic site - a site run by a group of really knowledgeable guys.

Now, typically customers tell me that they “observe some stuck threads”, “sometimes”, but often they are “not sure what caused them” and typically they “don’t know what exact state these thread are in” and in addition nobody seems to know if “the stuck threads ever clear up again without rebooting”. I am a pragmatic guy. I enjoy having little applications or tools to demonstrate and measure how WebLogic is working. Keen to play around with the newest edition of Netbeans (I used to be an Eclipse guy) and EJB 3.1 in WLS12c I built a small application to easily test WebLogic stuck thread settings and countermeasures.

Here are some more details about the StuckThreadForFree application:

The application allows you to create threads which are busy or which are waiting long enough to be detected as “stuck” by WebLogic.
This little application will only work with WLS12c. I intentionally avoided JSF, so a plain JSP page is used to set your parameters. The JSP is calling a simple Servlet which in a for loop is calling an asynchronous business method of an injected stateless session bean. @Asynchronous and no-interface session beans are only available in EJB3.1 so you have to run it on WLS12c. Unlike in previous versions, the EJB is directly packaged into the .war file for deployment.
Every call to the stateless session bean is serialized by the EJB container, so every EJB method is executing in its own thread.
Depending on which method was called on the EJB is either waiting n seconds using Thread.sleep() or calculating some trigonometric function for n seconds. Both methods will cause stuck threads.
There is zero configuration in the deployment descriptors for the EJB! Only context-root for the web part is set (which could be avoided as well).
Building the StuckThreadForFree app with Netbeans was a smooth ride and a real pleasure.
The app is provided as is. You can have it for free, yet there is no guarantee for anything but it shouldn’t cause any problems either. Better don’t run it on your production system.
It’s just a hack. It demonstrates what it should, nothing else.

DOWNLOAD: for your conveninience you can download the StuckThreadForFree.war from here and follow the example yourself (here is the link to the whole Netbeans project). After downloading you can easily deploy it to WebLogic. To follow the example it’s good enough to run it the admin server. Then you can start with the following URL:

http://localhost:7001/StuckThreadForFree

Now, lets use the app to answer some typical questions.

What are hogging threads? When do threads become hogged? After what period of time?

According to the Oracle doc hogging threads “.. will either be declared as stuck after the configured timeout or will return to the pool before that. The self-tuning mechanism will backfill if necessary.”

So how long does it take for them to become hogged? Nobody (including Google) seemed to know. Trust me I did some research and asked plenty of colleagues about this. Here is the answer:

If you run the application with 3 threads / 100 seconds / Thread.sleep() and immediately switch to the WebLogic 12c admin console Admin Server / Monitoring / Threads you will observe the following:

So interestingly hogging threads are detected right away! In my case it took about 2 seconds (I had to hit reload once).

So WebLogic transitions into FAILED state when a certain number of stuck threads are detected, right?

That’s a common misconception! The default configuration of WLS 12c (I also checked for WLS 11 = 10.3.3) is Stuck Thread Count = 0, which means the server “never transitions into FAILED server irrespective of the number of stuck threads”. You will only see the FAILED state only when you set the value to a positive number of threads!

Once the server transitions into FAILED, you can define if WLS should be shut down (and restarted by WLS nodemanager) or suspended.

Remember: WLS will not transition into FAILED state when StuckThreadCount is set to zero. Only the health runtime value is set to Warning (but this will be cleared if the hogging thread conditions clears) as shown below:

What exactly causes a stuck thread? What state does a thread have to be in to be marked as stuck?

In general there is a number of different thread states in Java: NEW, RUNNABLE, BLOCKED, WAITING, TIMED_WAITING, TERMINATED.

But which state has a thread to be in to be marked as stuck later? If you run the StuckThreadForFree application and create a stack trace with WebLogic admin console under Server / ServerName / Monitoring / Threads you can observe that the thread state is ACTIVE/TIMED_WAITING when using the Thread.sleep() method to block it:

"[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" TIMED_WAITING
            	java.lang.Thread.sleep(Native Method)
            	com.munzandmore.stuckthread.LongRunningEJB.threadSleep(LongRunningEJB.java:26)
            	com.munzandmore.stuckthread.LongRunningEJB_x9v26k_NoIntfViewImpl.__WL_invoke(Unknown Source)

when using the calc() method to keep the threads busy they are state ACTIVE/RUNNABLE :

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE
            	com.munzandmore.stuckthread.LongRunningEJB.threadCalc(LongRunningEJB.java:40)
            	com.munzandmore.stuckthread.LongRunningEJB_x9v26k_NoIntfViewImpl.__WL_invoke(Unknown Source)
            	weblogic.ejb.container.internal.SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:31)

So both states can become stuck. Also, I am pretty sure I could also show the BLOCKED state when using a monitor lock for synchronization but due to time restrictions this is not included in the app.

Can a stuck thread still do reasonable work?

Absolutely! Just because a thread is marked as stuck it doesn’t mean it is frozen or unusable. Imagine you wanted to calculate PI, you are creating PDFs, distance maps, mapping the human genome or you have deployed some JCA adapter talking to MQ-Series, SAP or PeopleSoft which is internally using a Thread.sleep() method call. All of this is are reasonable usages likely to occur in the wild.

Do stuck threads ever dissapear? Can they be cleared somehow? Are they stuck forever?

First of all you cannot get rid of a stuck thread by simply “killing it”. You cannot cancel or kill any thread in Java. However, stuck threads automatically will disappear if the condition clears up which caused them to be marked as stuck (e.g. the sleep period is over or the calculation is done).

To prove the point, switch to the WebLogic admin console and under Server / ServerName / Configuration set StuckThreadCount to 3 and StuckThreadTime to 60 seconds then restart the server and run the StuckThreadForFree app to create 3 threads running for 120 seconds using the Thread.sleep() method (the other method will work as well, there is no difference, but keeping 3 threads busy by doing math proves to be a fan test of your machine as well):

In the WebLogic log file you will find three entries logging the stuck thread state after a while:

<05.04.2012 10:55 Uhr MESZ> <Critical> <WebLogicServer> <BEA-000385> <Server health failed. Reason: health of
critical service 'Thread Pool' failed>
<05.04.2012 10:55 Uhr MESZ> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '4' for queue: 'webl
ogic.kernel.Default (self-tuning)' has been busy for "85" seconds working on the request "Workmanager: default
, Version: 1, Scheduled=false, Started=true, Started time: 85443 ms
", which is more than the configured time (StuckThreadMaxTime) of "60" seconds. Stack trace:
 java.lang.Thread.sleep(Native Method)

After waiting about one minute you will observe that WebLogic is transitioning into FAILED state as configured:

Wait another minute, then check the thread states under Server / ServerName / Monitoring / Threads which reveals the following:

So once the condition causing the stuck threads is cleared also the stuck threads will disappear again! Stuck threads are not stuck forever. Phew!

When should I use StuckThreadCount in the admin console or a Workmanager stuck-thread setting then?

Very good question. Use StuckThreadCount from the WebLogic admin console or with a <work-manager-shutdown-trigger> definition moving the application into ADMIN mode if you can react on the FAILED state.

Do not use StuckThreadCount if the threads might be doing something useful and you cannot react on the situation anyway. Obviously transitioning into FAILED state and restarting WLS with the nodemanager is counterproductive if you threads are doing something useful.

More?

The following posting shows how simple tools like ps, top and jcmd can track down the exact line of Java code causing a thread to use a high amount of CPU. Exactly the same StuckThreadForFree application is used as here.

Filed Under: oracle Tagged With: availability, performance, weblogic, wls12c

WebLogic Feature Timeline: What is new in WLS 12, 11, 10, 9?

February 24, 2012 by frank 11 Comments

To whom it may concern, here is a quick overview of the new features as they were included in WebLogic Server versions 9 to 12c. Destilled for my first WebLogic 12c workshop running a fortnight ago and including some links to the WebLogic documentation and some other useful resources.

happy weekend then … !

Download WebLogic Feature Timeline (PDF): What is new in WebLogic 12c? WebLogic 11? WebLogic 10? WebLogic 9?

Update (05-March-2012):I just uploaded a newer version after adding valuable input from Steve and Jan. (Due to other changes, those of you linking to the PDF directly need to update their link this time).

Filed Under: oracle Tagged With: weblogic, wls12c, workshop

localhost is mostly wrong (btw that’s why it’s called Distributed System)

February 4, 2012 by frank Leave a Comment

According to a post of Oracle’s A-Team, you can accelerate the startup of WebLogic when configuring “localhost” as ListenAddress. My comment on their posting isn’t enabled yet, so I will share some details here: Actually, that’s a good point, but only half of the picture (especially when looking at all the non-integrated WLS). Here is my response.

Don’t forget to mention that configuring “localhost” causes WebLogic stop doing what you bought it for: accepting requests from any client. Makes sense, doesn’t it?, but some developers don’t get this at first (no offence, it’s typically not their business). Using “localhost” for the JDBC URL is another common mistake in the first version of quite a few configurations I have seen.

As a rule of thumb:

Using “localhost” is almost always wrong in a distributed system configuration.

You are correct, it’s important to put a fixed entry for the listen address (for all manged servers and the admin server as well, and better don’t forget the node manager ). Unlike your private Mac or PC, multi-project, multi-domain machines may have dozens of IP addresses. WLS shouldn’t listen to all available addresses on the machine. So use the IP of your machine. If you are using DHCP you can also put your (qualified) hostname if it can be resolved. This will even work for JDeveloper “integrated” development WLS .

Filed Under: oracle

WebLogic Deployment: Answers to Frequently Asked Questions

February 2, 2012 by frank Leave a Comment

Here is a number of answers to many question that typically pop up when explaining WebLogic deployment concepts

What is the difference between staged deployment and non-staged deployment?

You maybe want to read the official Oracle documentation regarding staged deployment first which gives a great overview. This Q&A cannot replace reading the manuals.

Anyway, here is short answer: Staged deployment will copy your deployment units onto the deployment servers, place them into the $WL_HOME/servers/SERVER_NAME/stage directory and deploy them from there.

Good, so when does staged deployment make sense?

In short: When the target servers cannot access the source file of your deployment unit use staged deployment, e.g. because there is no shared file system.

What happens if I use staged deployment although I have shared domain directory for every server?

You will unnecessarily multiply the deployment units in every servers stage directory. The more servers you have, the more time and space it will cost you.

So if I use staged deployment and then delete the deployment, will the be deleted from the stage directory?

I just tried this with WebLogic 12c because I heard complaints about this not working in previous version. So far I could only reproduce it on one machine, but then it is working.

I don’t get it. When I look at the deployment wizzard in WebLogic 12c it doesn’t make sense?!

That’s correct. It doesn’t make sense. The deployment screen is very confusing (you might say “incorrect”) since the horizontal separators are wrong. See the screenshot below for how they are (in red) and how they should be to make sense (in green). However, this glitch is not new. You can observe the same for at least for WebLogic 10 and 11. Sometimes I wonder if anybody at all wonders…

Production Redeployment

First of all I recommend to browse the official Oracle documentation about production redeployment. For short step-by-step instructions have a look at Jay’s article in Middleware Magic.

Can I add the version number for production redeployment in the admin server console?

To the best of knowledge, no. For some reason you can only see the version number during deployment, but you cannot set it. You can set it in the MANIFEST.MF file (recommended by Oracle) file or during deployment with weblogic.Deployer:

[code]

java weblogic.Deployer -adminurl http://localhost:7001 -user weblogic -password weblogic

-deploy -name purchaseSurfWear.war -source /apps -targets surfCluster

-stage -appversion 2b

[/code]
or

use the WebLogic Scripting Tool WLST deploy() command and add the -archiveVersion under [options]

[java]deploy( appName , path , [ targets ], [ stageMode], [planPath], [options])
[/java]

Should I deploy to a cluster or to all servers in the cluster? What is the difference?

If you deploy to a cluster and later extend the cluster by adding another server, then the deployment will be available automatically to the new server (without redeploying or changing the target settings of the deployment).

Can I delete the source file of an application after deploying, since staged deployment should copy the deployment unit onto the target server?

In short: no, you cannot. One obvious reason is the functionality described above.

Here is a quote from the Oracle documentation: “After you have deployed your application, do not change or remove the original source files for the application. The WebLogic Server deployment framework uses the files to manage deployed applications. Changing or removing the original source files may result in unexpected application behavior.”

Homework for Oracle / My wishlist for deployment features in WebLogic 13f:

- Change the admin console so that a version number can be added during deployment.

- To confuse people less, get the horizontal lines in the deployment wizzard of the WebLogic 12c admin console right as described above.

Follow @frankmunz

Filed Under: oracle

WebLogic 12c: Node Manager Best Practices

January 31, 2012 by frank 9 Comments

During the last couple of years (and the last couple of WebLogic versions) I collected a number of best practices regarding WebLogic nodemanager. All of them hold true for WebLogic 12c as well. This posting is not a step-by-step beginners guide and it will not save you from attending some training or studying the Oracle documentation regarding node manager yourself. Anyway, here are some suggestions, check if the apply for your environments:

Node Manager Best Practices

At first, take a decision to start servers with or without NM. Note, that is not absolutely necessary. You can always start your servers with the scripts generated by the config wizzard. I personally know rather big companies building lovely cars who took the decsicion not to use node manager.
Would I use nodemanger myself? For an “average” project: yes! Only after configuring node manager you can use the WebLogic admin console to start and stop managed servers and node manager will restart you failed servers as well. However, if you consider restarting you servers automatically because of out-of-memory problems, better read this article about “surviving generations” to understand how to track down memory leaks and fix them. Anyway, you still want to use node manager.
Make sure you understand that nodemanger will use default values to start your servers unless you specify them yourself in the admin console under server startup parameters.
Make sure you always start your servers with same startup parameters! This is really important. You end up in deep trouble if you don’t. Believe me.
Imagine somebody is starting a managed server using the admin console and the provided values there. Next day somebody else starts a server using the provided scripts (which – at least in real life – will never be identical to the startup values configured in the admin console). Now depending on the way the server was started it will behave differently and show erratic behaviour or not.
Document and communicate the usage of node manager. Write it down in the operations manual. If you ever hire me as a consultant for some performance tuning it helps to know if you are actually using node manager or not.
Don’t forget to enroll new machines for NM usingnmEnroll()
A good way to overcome the potential problem with startup parameters configured in admin console is to use:
[java][file: nodemanger.properties]startScriptEnabled=true
stopScriptEnabled=true
[/java]Then node manager will use the generated start script and you do not need to configure startup values in the admin server console.
If you are not using SSL for your domain the default option for node manager to use encrypted communication does not make that much sense for you. Disable it. On the admin server site switcht to “plain text” for node manager communication and in the node manager.properties located in WL_HOME/common/nodemanager set
[java]
secureListener=false
[/java]
If you decide to use SSL for the node manager communication, get correct certificates! The demo certificate will not work in a distributed system. Make sure the hostnames in the certificates are correct. If they are not correct, you may want to consider disabling host name verification on the admin server (which is the client for the node manager).
Remember that node manager is not part of the domain. Still you can check the node manager status and and see the logs directly from the WebLogic admin console.

Some homework for Oracle Here is my personal wishlist for WebLogic Server 13f:

Enable plain text communication for node manager as default. Why should it be SSL?
Set start/stopScriptEnabled as default. This will cause less confusion.

Any settings you would like to share? Something to argue? Let me know.

Follow @frankmunz

Filed Under: oracle Tagged With: admin, best practice, node manager, weblogic, wls12c, workshop

WebLogic 12c Proxy Plugin for Web Servers is missing (kinda solved)

January 15, 2012 by frank 1 Comment

According to the documentation for the brand new WebLogic 12c the WLS mod_wl proxy plugin for web servers should be located under the following location:

$WL_SERVER/server/plugin

and this is where the trouble starts:

This directory does not exists for WebLogic 12c.
You can download the plugins seperately from here. Yet there is no version for WebLogic 12c either.
You must not simply use an lower plugin version (e.g. the one that comes with WebLogic 11g) because lower versions of the plugin e.g. 10.3.5 are not guaranteed to work (only higher versions). Now, this is between you me: yes they do work typically. Yet I remember at least one situation when I spent over a day debugging an issue because an older plugin version was used. I am pretty sure you can imagine serveral way to spend your time better.
The behavior for the Apache mod_wl 10.x plugins is known to be rather strange after a a managed server drops out of the dynamic server list (clustered or not).

Here are some possible solutions:

I briefly checked the 10.3.5 plugin for Apache 2.2 with a WebLogic 12c cluster and it seems to work okay so far. Actually even the glitch with the failing load balancing after a server drops out of a cluster has improved. Probably that’s the best solution currently, but remember it’s at your own risk.

Follow up with my OTN thread and some valuable input from René and Sandeep.

For a more critical production environment I recommend to open a support ticket with Oracle. If you do so, please post an update here and share your findings.

So how are you using WebLogic 12 in production? If you do, how is your clustering set up?

UPDATE / Jan 2013:

Thanks to input from my friend Rene I am aware now that Oracle officially recommends to use the latest 10.3.x plugin even for WebLogic 12c. Still it is causing a lot of confusion and the WLS 12.1.1 documentation is wrongly stating that the plugin is part of the WebLogic distribution.

Select “Generally Available System Configurations”:

http://www.oracle.com/technetwork/middleware/ias/downloads/wls-1211x-certmatrix-1395408.xls

Filed Under: oracle Tagged With: cluster, oracle weblogic, plugin, weblogic, wls12c

Cloud Computing Workshop 2011: Oracle, Rackspace and Amazon

December 20, 2011 by frank Leave a Comment

This year I really kept the best until the end! Last week I was running a 2-day cloud computing workshop with a 2-hour hands-on management presentation the night before the workshop for Contribute in Belgium. Contribute is an Oracle Platinum partner and being surrounded by Oracle Fusion Middleware experts, DBAs, application architects and senior level management the technical level of the workshop was very high with many interesting discussions.

We covered Oracle Public Cloud (OPC), Amazon Web Services (AWS) and Rackspace (RS). OPC is not available yet, but the overall functionality including its limitations for the first release is more or less known and quite interesting compared to let’s say running WebLogic on AWS.

To prove the point I was running WebLogic 12c on AWS cloud with 30GB of heap on a high-mem 4xl instance with 8 cores. Proving the point cost me a bit more than US$2.

Typically I expected that the more tech savvy audience prefers AWS over Rackspace, yet this time people were impressed by the easy setup of Rackspace and the way they handled a minor problem with their web console file-upload feature during a live chat session.

Among hundreds of other details we looked at the I/O performance. The performance of Amazon’s EBS is known to be interesting (you may want to read this as ‘difficult’). See Adrian’s posting for a thorough explanation, some benchmarks here, and some more details there.

The out-of-the-box performance looking at Rackspace Cloud is more consistent and there is a surprisingly high throughput which is almost independent of the data size. Here is some data comparing a local laptop disk, to the disks attached to the Rackspace Cloud servers to my brand new consumer SSD (not sure if a 512 GB SSD still qualifies as ‘consumer’). All numbers refer to a READ-benchmark with increasing data size.

Laptop HD (500GB SATA): 80 MB/s

Laptop SSD (Crucial m4): 281 MB/s

Rackspace (SAN): 302 MB/s

I am only posting the screenshot for one of the Rackspace I/O measurements since quality isn’t perfect. There is some older data with graphs available in a previous post of mine.

Now I am still curious about the dip on the left part of the graph which is consistent over several instances and measurements. Any comments?

Filed Under: Announcement, CloudComputing, oracle Tagged With: aws, cloud computing, course, oracle public cloud, rackspace cloud, weblogic, workshop

WebLogic 12c is announced!

November 17, 2011 by frank Leave a Comment

WebLogic 12c will be out soon. Launch event is Dec 1st 2011. http://bit.ly/uge9cu .

Certainly you can expect a “WebLogic 12c unboxed” report here once it will be out.

My current estimate is that I will be able to provide custom WebLogic 12c training courses around January 2012.

Filed Under: Announcement, oracle Tagged With: weblogic

Review: WebLogic Server 11g Administration Handbook by S. Alapati and an Overview of other WLS 11g books

November 9, 2011 by frank 4 Comments

“So which WebLogic book do you recommend?“

During the last couple of years I must have been delivering more than a dozen WebLogic server workshops for all kind of audience. Naturally the question that comes up at the end is: “So which WebLogic book do you recommend?”.

Only a few years ago…

To be honest, for a long time there wasn’t anything that I could recommend at all. This situation changed when Patrick and Nyberg published Professional Oracle WebLogic Server. And their book certainly doesn’t deserves the not so good reviews on Amazon. For quite a while it was the best book available about WebLogic server – certainly not because it was the only one. Written by Patrick, who was (or still is?) VP in Oracle’s Fusion Middleware Development, there is a least some additional content to be found beyond of what is covered in the offical Oracle documentation (the first technical description of cluster unicast, how to build singleton services in a cluster for developers). The book targets developers as well as admins and it’s useful for architects. But then, it was written when WebLogic 9/10 was hot.
Summary: Technically sound, but outdated now. Doesn’t deserve the bad reviews on Amazon.

… now

Nowadays, while all the world seems to be chit-chatting about clouds only (and I personally can’t wait for WebLogic 12c to be released) there are 2 new WebLogic 11g books out.

I have to admit that I only browsed Michel Schildmeijer’s book Oracle Weblogic Server 11gR1 PS2: Administration Essentials. Yet it seems an interesting book for those who are getting started with WebLogic server and those who are looking for a gentle step-by-step instruction to get most important aspects configured. On a first glance it does exactly what it says on the tin: administration essentials. Markus Eisele has written an insightful and detailed review about the book that I pretty much agree with.

Summary: Too many screenshots, not much content beyond that, outdated since it doesn’t cover WebLogic 12c.

WebLogic Server 11g Administration Handbook

However, there is a another book out now which caught my attention. Oracle WebLogic Server 11g Administration Handbook written by Sam Alapati and published in Sept 2011.

Content:

Myself I don’t like reviews that repeat the content of a book over and over again but for your convinience there is a TOC available at Amazon. Interesting enough, the book covers roughly the topics that you could learn about in a 5 day WebLogic admin training course. Actually it reads as if somebody knowlegeable enough took the content of the official Oracle WebLogic training course and converted it into a book. The book is up to date and mentions the unique features of WebLogic 11g such as the new dashboard that replaces the old console extension of WebLogic diagnostic framework.

Style:

From the way the book is written you can tell that Alapati has a deep understanding of the topics covered. There is no fluff, everything is technically sound. Still it is easy enough to read.

Recommendation:

Do I recommend it? Yes.

Summary

what I liked:

+ covers all the content of a 5 day WebLogic admin training in enough detail.

+ it makes a great companion book for the WebLogic administration certification or for somebody with a Fusion Application background.

+ precise writing style yet still easy to read. You can tell the author knows very well what he is writing about.

minor things I did not like:

- it only repeats the content of a basic 5 day WebLogic admin training, there is not much advanced content beyond that.

- it explains how to configure stuff but it does not deal with best practices if you are running hundreds or thousands of WebLogic servers (no, just scripting is not enough then!).

- reading a book written by an Oracle ACE I was expecting to learn about limitations and showstoppers as well. But maybe I am personally expecting too much here.

- there is a kindle version now. It’s stupidly more expensive than the printed version.

My summary summary condensed: 5* (on an Amazon scale from 1 to 5*)

The new kid on the block

Just for completeness: I now published a book about WebLogic Server 12c. When I wrote this blog posting it didn’t even plan to write a WebLogic book. So no shameless plug intended. Sincerely! Please judge yourself, it contains all the stuff that Oracle doesn’t tell you about

WebLogic in the Cloud

I saved the real shameless plug for the end. So feel free to stop reading here! However, if you believe that cloud computing is purely a marketing hype you may want to check out the Oracle Middleware and Cloud Computing book for a critical review of what is possible today without vaporware. The book explains why the cloud is like a fully programmable data center at your finger tips and how to build up highly available, globally distributed, load balanced instances and deploy your applications. All this just for a few cents.

I am happy to report that I received the Oracle Technologist of the Year award for publishing this book.

Filed Under: oracle, publishing Tagged With: book, weblogic

Artikel: Oracle WebLogic Server und Fusion Middleware in der Cloud

October 17, 2011 by frank Leave a Comment

German only. Heute exklusiv auf deutsch ein Artikel den ich für das DOAG Magazin im Früjahr 2011 geschrieben habe:
Download: Oracle Fusion Middleware und WebLogic Server in der Cloud (PDF)

Cloud Dienste oder Fusion Middleware Features?
Was zeichnet eine echte Cloud aus?
Architektur Blueprint für die AWS Cloud und Java EE Anwendungen.

Teile des Artikels sowie zahlreiche Grafiken sind aus meinem “Middleware and Cloud Computing” entommen. Viel Spaß beim Lesen!

Filed Under: oracle, publishing Tagged With: aws, cloud computing, ec2, load balancing, oracle service bus, weblogic

Usage of Oracle Exadata, Exalogic, Oracle Fusion Middleware and Enterprise Manager within Oracle

October 12, 2011 by frank Leave a Comment

Oracle is eating its own dog food.

Here is an interesting slide set from OOW11 about how Oracle is using it’s own hardware and software.

Still a lot of Oracle appserver to see instead of WebLogic..

Filed Under: oracle Tagged With: database, load balancing, weblogic

Oracle / Amazon Cloud Computing Workshop in the Netherlands

September 22, 2011 by frank Leave a Comment

Did you miss the Sydney event? I will offer an even better Oracle / Amazon Cloud Computing workshop with Transfer Solutions (together with René van Wijk) in the Netherlands.

Enjoy two and a half days filled with real Cloud Computing, hands-on labs, and a 2h management overview live demos on the Amazon cloud.

Learn about features and showstoppers – all in plain English!
See here for an updated version of this workshop.

Filed Under: oracle

Complete Silence. WebLogic on fanless, ultra-small, low energy hardware.

September 12, 2011 by frank Leave a Comment

Imagine you are trying out some Oracle stuff at home: No humming sound, no high pitch fan noise, no distraction (assuming your kids are somewhere else and your SO is walking out the dog). Nothing. Complete silence while you are reading manuals and working with WebLogic Server 11g.

Since a long time ago I wanted to build that system. A fanless, eco-friendly and incredibly cheap computer running WebLogic. A real computer that makes no noise at all.

I started with a Shuttle XS35GTV2 barebone. It’s preassembled, based on an Intel Atom 525, but it comes without HD and RAM. I added a max of 4 GB SDDO 1333 RAM (204 pin) and a HD that was lying around from an old laptop.Total hardware cost including HD and RAM was about 250€ which should roughly translate to a bit more than 250 US$ (not if you calculate the exchange rate but when buying it overseas). Replace the HD with an SSD to avoid the remaining noise.

Fig1: Shuttle with 2,5 SATA HD and RAM added.

Fig 2: Shuttle assembled.

The Shuttle is running a dual core processor with hyperthreading enabled so UNIX top displays 4 cores.I installed Ubuntu 11.04 on it, booting from an USB-stick, without any problems at all. One reason why I selected the Shuttle in favor of others is that it comes with 5x USB, a WLAN, gigabit Ethernet, card reader and a discrete ION2 graphics card with HDMI output. You can guess I will use it as a media player as well.

WebLogic performance is alright (don’t expect miracles though). It’s certainly not blasting fast but it does keep up with most training environments or virtualized systems.

It’s lovely to have a fanless UNIX server at home.

Filed Under: oracle

Oracle Fusion Middleware and AWS Cloud Services

August 18, 2011 by frank 6 Comments

I yesterday gave a presentation at InSync2011 about Oracle’s cloud computing strategy, AWS cloud services and it’s current limitations. Amazon reacted quickly and updated their offering. We have been waiting for that. Keep going!

Filed Under: CloudComputing, oracle Tagged With: aws, weblogic

WebLogic JMS with SAF and JMS bridges or SQS : Legacy Integration in the Cloud with Oracle WebLogic, WebSphere and OSB / Apache Camel

July 18, 2011 by frank Leave a Comment

An interesting question popped up on my Oracle Middleware and Cloud Computing book site which I like to answer here for the benefits of all the others puzzling at similar integration questions. In the context of using JMS as an integration technology I’d like to summarize the usage scenarios for Oracle WebLogic JMS Store-and-Forward and JMS-bridges (both are included in WebLogic server).

Hi Frank – [ ...] We have a requirement to build 2-way asynchronous integration between an application running on WLS in AWS and a legacy J2EE app running on IBM WebSphere in our Data Centre. From your excepts my understanding is that SQS is intended for use only between AWS apps – is this correct ? I think we need to be looking at a full JMS solution for our integration – perhaps using WLS JMS Store-And-Forward – Thanks, Peter D

Hi Peter,

Based on your comment I cannot go into great detail or even provide a solid architecture that anwsers you question (one that will save you from more reading) but here are some important points to consider:

- Amazon’s SQS is not restricted to be only used from AWS instances. SQS is purely based on web services (or language bindings that encapsulate those WS calls) so you can use it from any computer. E.g. you can read or write to SQS queues from remote.

- WLS Store-and-Forward (SAF) can only couple WLS instances of the same version and does not bridge to other JMS providers. You cannot use SAF to transfer from WLS JMS to IBM MQSeries (or whatever Websphere might use). JMS is a pretty bad integration technology which requires to have the right messaging classes in your classpath. E.g. when writing messages from Websphere to a WebLogic JMS queue you are required to have the WLS JMS classes in Websphere classpath.

- You can use the WebLogic’s JMS bridge to solve the somehow messy classpath issues. WLS JMS bridge has to be deployed as JCA adapter (still the jar file from the other provider is required but it is not used in custom code). The bridge will automatically forward from e.g. WLS JMS to MQSeries and even supports transcations. However there is no support to bridge between WLS JMS and AWS SQS.

- Unlike let’s say Oracle Service Bus, if you are looking at Apache Camel there is support to convert incoming JMS messages to outgoing SQS. Note to Oracle’s product manager of OSB: we would appreciate to have SQS as a supported transport protocol or possibly as an SOA Suite JCA adapter. Thanks for considering it

regards,

Frank

Filed Under: CloudComputing, oracle Tagged With: aws, cloud computing, oracle service bus, service bus, sqs, weblogic

Detect your Memory Leaks by counting Surviving Generations: Size matters!

June 28, 2011 by frank 9 Comments

The sunshine motivates me to run more these days. Just back from a beautiful run in Englischer Garten I am still sweating but I am also inspired by Rene’s article about JRockit that I read last night. It’s a very detailed article and well done – I tempted to write ‘as good as usual’ but I do understand how much work it is!

Java Memory Leaks

Most likely there is no training I do on behalf of Oracle without the subject “memory leaks”. So while still cooling down from my run, let me add a few points about memory leaks.

- Size matters. At the end it is all about size! Wasting 20 bytes due to a mem leak is bad. Yet 20 bytes don’t affect your system. I bet you will neither notice nor detect it.

Wasting 20 bytes in a loop that runs a million times makes you probably notice the leak. Running the same loop with the leak 10 million times might crash your appserver.

- Fast growth is not the issue. Slow growth is bad enough. Imagine you lose 20 bytes every minute. It’s a bit like a slow puncture tyre which constantly loses a bit of air. At the very end you sitting there with a flat tyre (Let me assume you life in a country with a mild climate and friendly people who will assist you to get the tyre changed quickly) . The same happens with your WebLogic heap. It’s not a lot of memory that you lose per time but it is the total size of the memory that is wasted over time. At the end your server will fail after three weeks (and WebLogic nodemanager restarts it for you ).

Don’t get me wrong. I am a big fan of JRockit, the team behind JRockit and its superb tools. Yet for a long time I wondered why JRockit’s memory leak tools still tries to detect “growth”.

Often I like to demo a 20 line Java program with 2 methods (mine is derived from the IBM page here, so you can get an idea). One method is consuming a lot of memory. The other one is losing 20 bytes every now and then. Of course the memory leak is within the second method, but JRockit tools detects the first method (which just happens to be mem intensive but correct).

Good news is: there is a better approach! It’s the kind of tools I love to explain in my workshops. Now it’s within the Oracle eco-system anyway but I used to spread the word long before.

Visual VM / Netbeans

VisualVM comes with your Sun JDK6 for free, originates from Netbeans, look for jvisualvm in JAVA_HOME/bin. It incorporates a memory profiler that uses a metric called “surviving generations” (or short “generations”). So what is a surviving generation?

Surviving generations:

The number of different ages for all objects allocated on the JVM heap since the profiling session started.

Age of object:

The age of the object is the number of garbage collections the object has survived.

Now the surviving generation metric is as good as it gets IMHO. A high number of SG tells that there is a high number of objects created which are never garbage collected (which is the definition of a memory leak, isn’t it?).

VisualVM displays the surviving generations metric, so it is easy to hunt them down.

To conclude:

Does it detect the slowly growing memory leak in the example above? Yes, it does!
VisualVM is only part of the Sun JDK, not of JRockit. Since we are all one happy family now I expect it survives the merging of both JDKs (Oracle is doing pretty well with these descisions usually. Hello Oracle …). Just in case it doesn’t survive I will keep posting references to this blog entry .
Read more about JDK, WebLogic and monitoring (also the most important non-Oracle monitoring solutions) in my Oracle Middleware book.
I am regularly running a customized high-end Tuning, Sizing and Monitoring WebLogic workshop myself (preferably in Munich or Sydney). Drop me an email if you are interested.
I am off for a shower and the second coffee of the day. Enjoy the sunshine!

A live update on this from Oracle Open World 2011 as of Oct 3rd 2011. I just spoke to the tech lead of JRockit, Marcus Hirt, asked him about the support of surviving generations in the Jrockit Mission Control tooling for the merged JDK 7. His answer (with a very sympathic grin in his face) was: “Stay tuned…”.

Follow @frankmunz

Filed Under: oracle Tagged With: Java

WebLogic with Jolokia

General: List Configuration Details of Managed Server with Name surf1

Deployment: List all Deployments

Deployment: List Deployments Details for an Application with the Name SubmitOrder

JDBC Data Source: Retrieve Settings for emergencyDB Data Source with Target Managed Server surf1 Running at localhost:7003

JDBC Connection Pool: Find all JMS Servers with Target set to Managed Server Running at localhost:7003

JMS: Find all JMS Servers with Target set to Managed Server Running at localhost:7003

JMS: Runtime Properties for a Particular Queue

WebLogic 12c RESTful Management Service

Jolokia

Basics

Installation

Feature Comparison: Jolokia and WebLogic 12c RESTful Management Service

Jolokia Request

What are hogging threads? When do threads become hogged? After what period of time?

So WebLogic transitions into FAILED state when a certain number of stuck threads are detected, right?

What exactly causes a stuck thread? What state does a thread have to be in to be marked as stuck?

Can a stuck thread still do reasonable work?

Do stuck threads ever dissapear? Can they be cleared somehow? Are they stuck forever?

When should I use StuckThreadCount in the admin console or a Workmanager stuck-thread setting then?

More?

Using “localhost” is almost always wrong in a distributed system configuration.

Here is a number of answers to many question that typically pop up when explaining WebLogic deployment concepts

What is the difference between staged deployment and non-staged deployment?

Good, so when does staged deployment make sense?

What happens if I use staged deployment although I have shared domain directory for every server?

So if I use staged deployment and then delete the deployment, will the be deleted from the stage directory?

I don’t get it. When I look at the deployment wizzard in WebLogic 12c it doesn’t make sense?!

Production Redeployment

Can I add the version number for production redeployment in the admin server console?

Should I deploy to a cluster or to all servers in the cluster? What is the difference?

Can I delete the source file of an application after deploying, since staged deployment should copy the deployment unit onto the target server?

Homework for Oracle / My wishlist for deployment features in WebLogic 13f:

Node Manager Best Practices

Some homework for Oracle Here is my personal wishlist for WebLogic Server 13f:

UPDATE / Jan 2013:

Während der DOAG2011 Konferenz können Sie ein Examplar des Oracle Cloud Computing Buches gewinnen:

“So which WebLogic book do you recommend?“

Only a few years ago…

… now

WebLogic Server 11g Administration Handbook

Content:

Style:

Recommendation:

Summary

what I liked:

minor things I did not like:

My summary summary condensed: 5* (on an Amazon scale from 1 to 5*)

The new kid on the block

WebLogic in the Cloud

Java Memory Leaks

Visual VM / Netbeans

Tags and Clouds

WebLogic 12c Book

Twitter

Cloud Book