As is obvious from this blog, I am a software developer who mostly implements software in Java (though I have used several other languages in the past). However over the last couple of months I’ve been using the Scala programming language for the first time. I’ve made some notes of the things I’ve learnt about Scala from a Java developer’s point of view - for those who might be interested, here are some notes on Scala for Java developers, and here are some more advanced notes on Scala’s pattern-matching.
The Kafka Connect JDBCSourceConnector reads from a relational database and outputs each row as a message in a kafka topic.
The config-file supports specifying the data to read as either a table-name (
table.whitelist) or a custom query (
query). Unfortunately, the documentation states clearly that when option
query is used, and “incremental load” is also enabled then the query must not include any “where” component as the connector will itself add a where-clause and this will result in invalid SQL syntax.
Is there anything that can be done about this? For most databases, the answer is yes…
I recently had to create a single Git repo (company internal) holding a mirror of several other projects (from Github). While Git can do this, its default behaviour is to mix all tag names into the same namespace, leading to rather confusing results. My solution is documented here.
One of the big features for the upcoming Java version 9 is supposed to be the Jigsaw (aka Java Platform Modular System or JPMS) module framework. However it has been controversial over its whole development cycle - and now that the release is coming up, some non-Oracle groups on the advisory board are intending to vote against the release of Java 9 due to flaws in Jigsaw.
During a recent code-review I noticed a work colleague had used method
parallelStream in Java code, and realized I didn’t know much about this method or the fork/join framework it relies on. The results of my research on the Java fork/join framework can be found here.
I recently discovered a free web-based management console for the Kafka message broker called Kafka Manager. Although I have only spent half-an-hour playing with it so far, it looks reasonable.
I recently had to process multiple messages in parallel from a single RabbitMQ queue. The official docs are sadly lacking in this area, so I did some investigation and found out some interesting things.
While on the subject of RabbitMQ, I have also written a brief summary of RabbitMQ Exchanges and Queues.
As you may have noticed, I’ve been doing a lot with Kafka Connect recently.
In order to test a custom Kafka Connect connector and connector configurations, I needed a suitable environment with the services installed. I set this up via Vagrant, and have now documented how this is done - mostly for myself in case I need to do something similar later, but maybe it is also useful to you..