I recently discovered a free web-based management console for the Kafka message broker called Kafka Manager. Although I have only spent half-an-hour playing with it so far, it looks reasonable.

I recently had to process multiple messages in parallel from a single RabbitMQ queue. The official docs are sadly lacking in this area, so I did some investigation and found out some interesting things.

While on the subject of RabbitMQ, I have also written a brief summary of RabbitMQ Exchanges and Queues.

As you may have noticed, I’ve been doing a lot with Kafka Connect recently.

In order to test a custom Kafka Connect connector and connector configurations, I needed a suitable environment with the services installed. I set this up via Vagrant, and have now documented how this is done - mostly for myself in case I need to do something similar later, but maybe it is also useful to you..

Recently I posted some information about using Kafka Connect to import data into Kafka message-broker topics. That article included a brief review of some alternative tools for solving the same problem - eg Flume, Logstash and NiFi.

Since then I have spent some more time looking into NiFi in more detail. If you are interested in “data integration” (whether from/into Kafka or not) then my notes might be of interest.

Some links to information about thread-safety in Java…

I’ve already written about the Apache Kafka scalable and high-performance message broker. It is a fine tool, and now very widely used.

Kafka Connect is another component of the Apache Kafka project, dedicated to importing data into Kafka from external systems or exporting data from Kafka into external systems. I am currently setting up ETL (extract, transform, load) for a client using kafka-connect and have written up some notes on it here.

Just two links to interesting articles about correctly using the class java.util.Optional which was added to Java in version 1.8..

• Nicolai Parlog/codefx: Java 8’s New Type Optional - an interesting blog series (particularly part 3)
• Oracle Technetwork: Tired of Null Pointer Exceptions?

A while ago, I received a bunch of books on “big data” themes, one of which was about something called the Lambda Architecture. It presents a design-pattern useful for IT systems that have large amounts of persistent data representing “current state” - stored data like users, devices or bank-accounts which is mutable (updateable). I found it interesting but too complex for the issues I needed to deal with at the time.

Recently one of my work colleagues mentioned something called the Kappa Architecture which seemed to be related. I did some reading on that topic too, and found it to be a simplification of Lambda which is applicable to many more cases. Here are my notes on the topics for those interested.