IOPS Quality of Service for Amazon EBS Cloud Storage

Define the pipe!

You can now have EBS optimized EC2 instances with e.g.  500 or 1000 Mbit/sec throughput to EBS.

According to AWS a standard EBS volume will cope with 100 IOPS on average allowing some burts. As I mentioned before there was plenty of complaints and discussion about EBS performance workarounds though. Here is the news: AWS is offering provisioned IOPS EBS volumes. You can create up to 20 TB of provisioned IOPS volumes with a total of 10,000 IOPS per AWS account (but apply to extend your AWS account for more).

Seems like a lovely idea for a db data file, doesn’t it?

Does that make you run your EBS I/O benchmark again? Let me know about the results!

Simple Custom JMX MBeans with WebLogic 12c and Spring

Using the Spring framework bundled with WebLogic 12 the programming of JMX MBeans becomes as easy as writing a simple POJO within a web application. After coding the MBean as a POJO all that is left to do is creating the Spring configuration file and editing the web.xml deployment decriptor:

- Firstly you have to add the Spring <listener> to the web.xml deployment descriptor with the web.xml to get Spring started.

- Secondly create the applicationContext.xml for Spring in the WEB-INF subdirectory. Declare the Spring bean and let Spring export it to the platform MBean server.

- Thirdly deploy the web application and test it with JConsole, WLST or j4psh.

For your convinience I already built an example. You can download the deployable SimpleJMX.war file or the complete NetBeans project.

You can see the application in action, including the involved files and the deployment in the following screencast:

How to access custom MBeans from WLST is shown in the second part:

The third part is showing how to retrieve MBean attributes from a JSP:

Amazon Web Services (AWS) EC2 Autoscaling

 

How many blog posts have you read that were written on a ferry? Well this is post makes it one more. Live from famous Manly ferry since I am currently offering a number of Amazon and Oracle Cloud Computing Workshops in Australia.

Elasticity is a key criteria for cloud computing. Autoscaling is automated elasticity. Interesting enough is widely unknown how this will be implemented on the Oracle Public Cloud for the Java Service.

Autoscaling on AWS can only be configured with lengthy commands from the command line but not from the web cased AWS console. Getting all the parameters right can be tricky, so here is one of the easiest examples and a hands-on screen cast using it:

 

 


as-create-launch-config surfLaunch --region ap-southeast-1 --key access --image-id ami-b83374ea --instance-type t1.micro
as-create-auto-scaling-group surfScale --launch-configuration surfLaunch --region ap-southeast-1 -availability-zones ap-southeast-1a,ap-southeast-1b --min-size 2 --max-size 4
as-describe-auto-scaling-groups --headers --region ap-southeast-1

as-describe-auto-scaling-instances --headers --region ap-southeast-1
as-put-scaling-policy ScaleDown -auto-scaling-group surfScale --adjustment=-1 --type ChangeInCapacity --region ap-southeast-1
as-put-scaling-policy ScaleUp -auto-scaling-group surfScale --adjustment=+1 --type ChangeInCapacity --region ap-southeast-1

 

as-execute-policy ScaleUp --auto-scaling-group surfScale --region ap-southeast-1

as-describe-scaling-activities -g surfScale --region ap-southeast-1 --show-long

 

as-execute-policy ScaleUp --auto-scaling-group surfScale --region ap-southeast-1

 

mon-put-metric-alarm HighCPUAlarm --comparison-operator GreaterThanThreshold --evaluation-periods 1 --metric-name CPUUtilization --namespace "AWS/EC2" --period 600 --statistic Average --threshold 80 --alarm-actions "XXX" --dimensions "AutoScalingGroupName=surfScale" --region ap-southeast-1

 

mon-put-metric-alarm LowCPUAlarm --comparison-operator LessThanThreshold --evaluation-periods 1 --metric-name CPUUtilization --namespace "AWS/EC2" --period 600 --statistic Average --threshold 20 --alarm-actions "XXX" --dimensions "AutoScalingGroupName=surfScale" --region ap-southeast-1

 

REMOVE ALL:

as-update-auto-scaling-group surfScale --min-size 0 --max-size 0 --region ap-southeast-1
as-delete-auto-scaling-group surfScale --region ap-southeast-1

as-delete-launch-config surfLaunch --region ap-southeast-1

 

The comments section is open now for your suggestions how this will be done with the Oracle Public Cloud.

Another Myth Debunked: More than 200 Continuous Redeployments with WebLogic

The other day I overheard the myth that you simply cannot redeploy several times with WebLogic because inevitably you would end up having an out of memory error. Since I am currently writing on an WebLogic 12c book I decided to verify that.

Certainly this depends a lot on your heap settings and what your application is doing, eg. it is easier to get it wrong if you are writing your own classloaders etc. I certainly admit that these problems, if they occur, are particularly hard to sort out.

Hands-on Test

More important you should detect them early. So go and test it! It’s a simple 10 line WLST script. I have done it for you to verifiy if there is a general problem with redeployment in WebLogic.

I redeployed the MedRec application over 200 times. MedRec comes with the WebLogic 12c installation (only if you decide to install the samples). It consists of  over 20 EJBs, a web part and a few web services.

The sample domain can be found in the WL_HOME/samples/domains/medrec directory.

Download

A little WLST script that you can download does the trick (it’s a quick hack, you can certainly beautify and add error handling to it!). Download it (it comes zipped) and unzip it to the domain directory. It even contains comments about how to run it.

The script deploys the MedRec application (so undeploy it first!), waits for 30 seconds, undeploys, waits again and deploys again. At the same time it prints the number of loaded classes, heap and non-heap runtime values.

Results

I was running this experiment for 200 redeployments. Apart from the output of the program, I recorded some data plots with jconsole.

The non-heap usage after 200 redeployments is shown on the following screenshot.

Everything was fine. I guess I could also do 1000 redeployments. I just stopped it because I got bored. I believe it would go on forever and I needed to shut down my computer.

 

Also here is the heap usage:

 

So test your own application deployment! It’s an easy test. IMHO it should be run for every important application. Add it to your tests when load testing the application.

Also I recommend to create some load for the app while it’s deployed. You may want shorten the interval between the deployments to a few seconds to get more deployments done faster.

Let me know if it breaks in your case ;)

Also stay informed about other exciting topics covered in the my WebLogic 12c book. I just created a Youtube channel with many WebLogic 12c screencasts (more to come every week).

 

WebLogic Startup Mode: Development or Production? The (hopefully) Complete List of Differences

Let me quickly summarize the all differences between development and production startup mode for WebLogic 12c and previous versions. Information about it is scattered across the Oracle documentation and several blog postings but non of those I checked were complete.

Differences

Startup Mode

Development                     Production

Auto deployment
(to admin server only)

Yes No

Automatically created
boot.properties for Admin server only under
$DOMAIN_HOME/servers/$SERVER_NAME/security 

Yes No

Nodemanger username and password

default admin
credentials
randomly
generated

SSL demo certificates cause warning in log files

No Yes

Log file are rotated at server startup

Yes No

On demand deployment of internal applications as default

Yes No

Availability of web service test client

Yes No

Availability of Classloader Analysis Tool

Yes No

Availability of Fast Swap

Yes No

Change Center Enabled

No Yes

Default start parameters
when using Sun JDK and the default
startWebLogic script 

java
-client
-Xms256m
-Xmx512m
-XX:CompileThreshold=
8000
-XX:PermSize=128m
-XX:MaxPermSize=256m

java
-server
-Xms256m
-Xmx512m
-XX:MaxPermSiz=256m

The following settings in the weblogic.xml deployment descriptors default to -1:

pageCheckSeconds

servlet-reload-check-secs

servlet-reload-check

resource-reload-check-secs

No Yes

 

 

Differences in previous WebLogic versions:

Up to WebLogic 11:

Default JDK used for new domain

Sun JRockit

Up to WebLogic 8 only:
Default number of execute threads

15 25

Up to WebLogic 8 only:
maximum number of JDBC connections default setting

15 25

 

Configuration

There is different ways to configure the WebLogic startup mode:

  1. The startup mode can be configured with the admin console, as shown on the screenshot below (requires a restart). Once it’s set to production mode it cannot be reverted with the admin console.
  2. Supply the -Dweblogic.ProductionModeEnabled=true switch as startup parameter
  3. Set the PRODUCTION_MODE variable in the setDomainEnv script in the directory DOMAIN_NAME/bin.

What you should know

Some issues to be aware of:

  • The differences between the two settings depend on the WebLogic version. This can be a bit confusing when you are running different environments with different versions. See the table above: For WebLogic 8 the maximum number of connections in a JDBC connection pool defaults to 25 in production mode (but 15 in development mode); for WebLogic 12c it defaults to 15 in both cases.
  • You cannot revert to development mode from the WebLogic admin console. It’s like the child-proof lock in the back of your car. I understand that is makes sense to protect the children in the back seat your car from themselves but you are are logged with the admin role – still you cannot revert to development mode using the admin console.
  • Autodeployment is not used that much anymore since it only deploys to the admin server – so it doesn’t make sense for a production domain. What if you are a developer? Then your IDE is deploying to the admin server, that is correct – but your IDE is typically not using autodeployment.
  • Note, that there will be a randomly set user and password for nodemanager when creating your domain in production mode with config.sh (see comments from Jacco).
  • You cannot disable the change center in production mode

 

Something I forgot? I appreciate your feedback.

 


[A more detailed discussion with directions will be part of my upcoming WebLogic 12c book]

WebLogic 12c: Node Manager Best Practices

During the last couple of years (and the last couple of WebLogic versions) I collected a number of best practices  regarding WebLogic nodemanager. All of them hold true for WebLogic 12c as well. This posting is not a step-by-step beginners guide and it will not save you from attending some training or studying the Oracle documentation regarding node manager yourself. Anyway, here are some suggestions, check if the apply for your environments:

Node Manager Best Practices

 

  • At first, take a decision to start servers with or without NM. Note, that  is not absolutely necessary. You can always start your servers with the scripts generated by the config wizzard. I personally know rather big companies building lovely cars who took the decsicion not to use node manager.
  • Would I use nodemanger myself? For an “average” project: yes! Only after configuring node manager you can use the WebLogic admin console to start and stop managed servers and node manager will restart you failed servers as well. However, if you consider restarting you servers automatically because of out-of-memory problems, better read this article about “surviving generations”  to understand how to track down memory leaks and fix them. Anyway, you still want to use node manager.
  • Make sure you understand that nodemanger will use default values to start your servers unless you specify them yourself in the admin console under server startup parameters.
  • Make sure you always start your servers with same startup parameters! This is really important. You end up in deep trouble if you don’t. Believe me.
    Imagine somebody is starting a managed server using the admin console and the provided values there. Next day somebody else starts a server using the provided scripts (which – at least in real life – will never be identical to the startup values configured in the admin console). Now depending on the way the server was started it will behave differently and show erratic behaviour or not.
  • Document and communicate the usage of node manager. Write it down in the operations manual. If you ever hire me as a consultant for some performance tuning it helps to know if you are actually using node manager or not.
  • Don’t forget to enroll new machines for NM usingnmEnroll()
  • A good way to overcome the potential problem with  startup parameters configured in admin console is to use:
    [file: nodemanger.properties]startScriptEnabled=true
    stopScriptEnabled=true
    

    Then node manager will use the generated start script and you do not need to configure startup values in the admin server console.

  • If you are not using SSL for your domain the default option for node manager to use encrypted communication does not make that much sense for you. Disable it. On the admin server site switcht to “plain text” for node manager communication and in the node manager.properties located in WL_HOME/common/nodemanager set
    secureListener=false
    
  • If you decide to use SSL for the node manager communication, get correct certificates! The demo certificate will not work in a distributed system. Make sure the hostnames in the certificates are correct. If they are not correct, you may want to consider disabling host name verification on the admin server (which is the client for the node manager).
  • Remember that node manager is not part of the domain. Still you can check the node manager status and and see the logs directly from the WebLogic admin console.

Some homework for Oracle ;) Here is my personal wishlist for WebLogic Server 13f:

  1. Enable plain text communication for node manager as default. Why should it be SSL?
  2. Set start/stopScriptEnabled as default. This will cause less confusion.

Any settings you would like to share? Something to argue? Let me know.