Apache Kafka on AWS

I recently collected some resources about running Apache Kafka on AWS (with or without using Amazon Managed Streaming for Apache Kafka) after recording a webcast about it.

You can find the Medium article here with all the links and ressources.

If you are only interested in the slide set about Apache Kafka on AWS, it’s on speakerdeck:

Serverless Docker Containers with AWS Fargate / ECS

I took Fargate on ECS for a quick spin the other day. My idea was to use reveal.js for serving slides about Docker in a Docker container – something I have shown last year at OUGN 2017 in Norway. To make this Docker container run serverless on AWS Fargate I created a new task and specified the container as shown in the webcast.

I guess you can just follow my webcast to create an example yourself – the only edits I did was removing the parts where you wait for the provisioning. A few things to point out:

  • I recommend to get started with the ECS quick start wizzard.
  • It does help if you understand about ECS tasks and services.
  • The Docker image I used: fmunz/slidesougn17
  • Surprisingly the hardest part is to find the correct IP to be able to connect to the running container. Note that there is obviously no EC2 instance and the ENI is attached to the ECS task. So you can find the IP under Task -> public IP4.

Have (serverless) fun!

Fn Project in Public Clouds (aka Serverless on IaaS ?)

Fn in Public Clouds (IaaS)

Fn project is a cloud agnostic FaaS platform and a common question is how to use Fn in public clouds. Similar to the local installation that we used in the Oracle blog posting (link soon), it can also be installed on any public cloud IaaS. For most IaaS it is enough to pass the installation command directly to the creation of a compute instance as so called user-data. User data is commands that are acted upon when the instance is provisioned. Also when running Fn in a public cloud, don’t forget to enable access rules for Fn server allowing port 8080 – depending on requirements – either from your own IP or all public IP addresses.

Once Fn server is running on your favourite cloud provider, you could deploy the recommendation engine mock example mentioned in the posting above in two different ways.

Deploy Your Fn Function in the Cloud

# example 1 (for teaching purpose only, in production use approach below)
# note: run these commands on the cloud instance

$ fn apps create advtravel
$ fn routes create advtravel /fn-recommend DOCKER_ID/recommend:0.0.2

 

Another, probably even more useful way to deploy the function is to set the FN_API_URL environment variable locally, point it to the remote cloud instance, and run the local Fn deploy command against the remote cloud instance.

# example 2 (easier, what you'd do in real life)
$ export FN_API_URL=URLofRemoteCloudInstance
$ fn deploy --app advtravel 

Note, that with the commands above you never had copy over the function or the container image to the cloud instance. When the function will be invoked the first time, Fn will pull the Docker container from the registry, store it locally, and then simply run the function.

Test Fn in the Cloud

Once the Fn is running in the cloud and your application is deployed you can access the application from a local machine using the command-line or Postman. The invocation is the same as in the local example, just replace localhost with the public IP address of your cloud instance:

$ curl -X POST --data @testdata/syd.json PUBLIC_IP:8080/r/advtravel/fn-recommend 

Real FaaS?

Obviously, when running Fn project on an IaaS you do not get the true pay per invocation benefit as for a FaaS implemented by the cloud provider as PaaS. You will get automated scalability to some degree, since it is built into the load balancer Fn LB. At the end running Fn on IaaS is only serverless from a user perspective.

It will be interesting to see if a cloud platform (most likely Oracle, since Fn Project is driven a lot by Oracle) will provide a proper FaaS service with pay per invocation and automated scalability that is compatible with open source Fn Project.

Fn Cloud Demo

A recorded live demo from Devoxx conference about deploying Fn on IaaS can be seen here.

Get the demo app used in webcast from here.

Show EC2 Instances across all Regions

A note to myself (might be useful to you as well):

for region in `aws ec2 describe-regions --output text | cut -f3`
do
     echo -e "\nListing Instances in region:'$region'..."
     aws ec2 describe-instances --region $region
done

Originally from Stackoverflow

Cloud, Microservices and Container Workshop in South Africa!

Lot’s of people are talking about these topics nowadays. Heaps of slides and samples are available for download, lots of presentations can simply be streamed from youtube.

In Johannesburg we were working with these solutions hands-on: I delivered a 3 day Cloud, Microservices and Containers workshop on behalf of Oracle.

Find attached some impressions from the smart and fun group of devs and architects I was working with.