Smoke testing Kafka in HDP

Assuming that you have a vanilla HDP, or the HDP sandbox, or have installed a cluster with Ambari and added Kafka, then the following may help you to smoke test the behaviour of Kafka. Obviously if you’ve configured Kafka or Zookeeper to be running on different ports, this isn’t going to help you much, and it also assumes that you are testing on one of the cluster boxes, and a ton of other assumptions.

The following assumes that you have found and changed to the Kafka installation directory – for default Ambari or HDP installations, this is probably under /usr/hdp, but your mileage may vary. To begin with, you might need to pre-create a testing topic:

bin/kafka-topics.sh
    --zookeeper localhost:2181 \
    --create --replication-factor 1 \
    --partitions 1 \
     --topic test

then in one terminal window, run a simple consumer:

bin/kafka-console-consumer.sh \
     --zookeeper localhost:2181 \
     --topic test \
     --from-beginning

Note that this is reading from the beginning of the topic, if you want to just tail the recent entries, omit the --from-beginning instruction. Finally, in another terminal window, open a dummy producer:

bin/kafka-console-producer.sh \
    --broker-list localhost:6667 \
    --topic test

There is an annoying asymmetry here – the consumer and most other utilities look to ZooKeeper to find the brokers, but the dummy producer requires an explicit pointer to one or more of the brokers. On this consumer window, type stuff, and you should see it echoed realtime in the consumer window. When finished, ^C out of the producer and consumer, and consider your work done.

Lies, Damned Lies and Programmers

I recently came across a really nice set – not directly related – of articles dealing with various profound errors that programmers and system designers fall into when dealing with names and addresses.

The TL;DR if you don’t read these: names and addresses are hard and most things you believe about them are wrong.

Let’s start with Falsehoods Programmers Believe About Names. Without even trying the author lists 40 things we believe about names that are just plain wrong.

In a similar vein, Falsehoods programmers believe about addresses, which particularly speaks to me. One of the fundamental errors about addresses is to think they identify a location. This is incorrect. An address might identify a location, but it is fundamentally a description which instructs a postman how to deliver a letter or parcel. Substitute pizza operative, Amazon driver or writ server as desired.

Even without getting into the weirdness around the actual shape of the planet, Falsehoods programmers believe about geography touches on place names.

And as a bonus: Falsehoods programmers believe about time – computers prove to be pretty bad clocks, and working out a calendar is very complicated.

A Demonstration NiFi Cluster

In order to explore NiFi clustering, and NiFi site-to-site protocol, I decided that I could use a minimal installation – as I’m really just exploring the behaviour of NiFi itself, I don’t need to have any Hadoop environment running as well. To this end, my thought was that I could get the flexibility to just play around that I need by building a minimal Centos/7 virtual machine, running in VirtualBox. The plan was to have little more than a Java 8 SDK and NiFi installed on this, and then I would clone copies of it which would be modified to be independent nodes in a cluster. At the time of writing this is still in progress, but I thought it was worth capturing some information about how I proceeded to get my VM prepared.

There are a handful of requirements for this VM:

  1. It needs a static IP (so that I can assign different static IPs to the clones, later)
  2. It needs to be able to reach out to the broader internet, in order to pull down OS updates and similar
  3. I need to be able to ssh to it from my desktop
  4. Different instances of the VM need to be able to reach each other easily
  5. A Java 8 JVM is needed
Continue reading “A Demonstration NiFi Cluster”

List-o-mania

I have, once again, felt stuck, spinning my wheels in the mud. There is an unpleasant, and possibly vicious, cycle at play here in my head: my planning falls apart, I feel like I am not getting anything done, my anxiety spikes, I cannot plan cogently. Repeat and repeat and repeat like some damned overwrought Philip Glass piece. I am trying to look at this dispassionately, because if I can understand how this happens, maybe I can head it off next time.

There are a few factors – health, political chaos, and too many months of uncertainty at work. Having a work and personal phone, and a work and personal computer, and disconnected accounts across both is really not helping either – I keep dropping things between the various calendars and todo lists, which has been exacerbated in the last few months by traveling. You would think that separating work and non-work would be easy. I can partition off my 37.5 hours and leave it at work, can’t I? Well, no. Because I’m trying to juggle calendars and waking hours and mental effort between work and non-work, and I cannot just turn off my brain at the end of the working day. Increasingly I feel like I would do very well if I cloned myself at least twice, so that different instances of myself could live full and uncomplicated lives. And I really resent the 3+ hours tied up each day in commuting, even while I know other people are doing the same or worse.

Continue reading “List-o-mania”

Thibault – Chapter 6

On Attacks and Counters In the Straight Line

Zachary, in the preceding chapter, saw how easy it was for Alexander to defend the simple attacks at First Instance. He asks Alexander to give him those attacks so that he can practice. Alexander, being a jerk, tries to win the drill by introducing the subjection, to which Zachary responds with a variety of “oh my god, you have your SWORD in my FACE” reactions.

Continue reading “Thibault – Chapter 6”

Two-factor in the middle of the night

Wherever possible I have been enabling two-factor authentication and similar protections. Not that I am paranoid, it’s just that I am paranoid. One of these I have had in play for a long time is protection on my Google account. So it’s somewhat comforting to get an unexpected SMS message from Google in the middle of the night sending me an unexpected authorisation code. Because it means whoever just tried to access my account could not.

Lock your doors people. A simple username and password combination, particularly on anything critical, is effectively useless.

Thibault – Chapter 5

On Attacks at the First Instance, and Feints

This is the first chapter where Thibault leaves off his purely theoretical discussion and begins actual paired exercises. Poor Zachary comes off rather the worse for wear here, as he launches a variety of simple thrusts straight down the diameter (with one exception) from the first instance i.e. the distance he minutely detailed in the previous chapter. At least part of the point of this chapter is to setup the reasons for the actions and plays in the next chapter, as well as illustrate that the straight line is sufficient preparation and fortification against these attacks. Having said that though, to use his own words:

…fortified against all manner of feints, assured against all attacks…always making the (counter) with small movements, which have more force than showiness, and making the execution with as much force and assurance as possible, in opposition to common practice. If you say to me that it is not likely that anyone may easily reach such perfection in demonstrating all these effects, I answer that nothing commendable can ordinarily be acquired without great labour.

Continue reading “Thibault – Chapter 5”

If it looks like a Duck

Wikipedia has a good article on the probable source of what is now known as the Duck Test

If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.

In recent weeks, in light of what is happening in the US, and the direction various European states are moving, I was going to write a rant about using “Nazi” as a short hand descriptor. But I won’t.

Until recently, I was never particularly comfortable with any modern reactionary being labelled as a “Nazi”. Even “neo-Nazi” is something I was not keen on. My reasoning behind this is a little circuitous, and I know that some folk won’t agree with me, so bear with me a little. Over my lifetime I have seen a change in the way that “Nazis” were presented in popular western culture, and how they are present in the zeitgeist. Going back to the 70’s, I recall that we started to see movies (other than movies about WW2) where villains were identified as Nazis – I am thinking of things like Marathon Man and The Boys From Brazil, where the villain was a remnant representive of an ancient and terrible evil. Fast forward to the 80’s, and we see a more light-hearted and comic-book evil in Raiders Of The Lost Ark, or the 90’s with Doom and other video games where Nazis have transformed into “abstract Bad Guys”. Most presentations of Nazis in popular culture now are as arbitrary mooks that nobody can object to seeing punched, shot, or bombed.

This is not a new idea of mine, if you dig around you will find far better analyses of this process of abstraction, and indeed trivialisation. I encourage you to do so. So until recently, I would rather that real-world current evil bastards not be labelled with something that has become a trivial insult. I would until recently have said “look, they are not actual Nazis, and you are using that label as a lazy short hand for evil bell ends who should be locked away on an island for our own safety”

Except I can’t. Not any more. If it looks like a Nazi, salutes like a Nazi, marches like a Nazi, talks like a Nazi, I think we should treat it like a Nazi. Particularly since it is evident that in the US there are significant numbers of people who are literally draping themselves with the symbology and styling of the historical Nazis.

So my plea to you now is: If you want to label someone as Nazi, please double check your reasons for doing so. We owe it to the people who fought the Nazis last time around to take the label seriously. Mostly for myself I will be trying to use “racist arsehole”, “nationalistic bell-end” and “evil twat”, because I do not want to award these pricks with any of the weight that the very specific historical name carries.

Thibault – Interlude and context

A brief interlude resulting from a conversation over rather a nice stout, while we explore Chapter 5 with sword in hand. It is worth thinking about the context for Thibault’s life. Think about some of the other people and masters:

WhoBirthDeath
Thibault15741627
Narvaez157?1640
Fabris15441618
Giganti155?1622
Swetnam????1621
Carranza15391600
Shakespeare15641616
Ben Jonson15721637
Elizabeth I15581603

It is also worth noting that he was roughly contemporaneous with Luis Díaz de Viedma, who we have been looking at with LLA in recent months.

His late life is framed against the Thirty Years War, and the years when he was (probably) studying La Verdadera Destreza saw the publication of Don Quixote, Guy Fawkes’ adventure underneath Parliament, the establishment of the Jamestown, Virginia, and the Dutch East India company beginning to trade.

It is far too easy, and common, for you and I, the modern reader, to view a historical individual or event as a discrete and disconnected entity, forgetting or not observing what is contemporaneous. Take 1969 for example: Nixon became POTUS, Armstrong stepped onto the moon, the Beatles are photographed crossing Abbey Road, and Monty Python’s Flying Circus began to be broadcast.

The cultural milieu of Thibault needs to be taken into account – what is the context of sword play? What contemporary and near-contemporary masters influenced him? What sort of people would have he been teaching, and why? There are direct clues to some of this in his book, even the simple differentiation between “arresting with courtesy”, a thrust, and ramming it through the opponent’s eye suggests the student will be playing, but may expect to be fighting in earnest.

Some entirely context-less links for your additional consideration:

Henry VI, Part 2, Act 4, Scene 2

The first thing we do, let’s kill all the lawyers.

I’ve heard back from my contact at Blue Point regarding the defunct electric car chargers. On 4th of August I was assured by Berkeley’s “Head of Estates” that matters were going to be soon resolved, and that a quick trip through Legal would see it all sorted. The ball is still in Berkeley’s court, and appears to have been left to lie. Because Brexit. Or Trump. Or the wrong sort of rain. Or something.

It is now 280 days since we first lodged a formal report of these units being dead, and I know they were dead for at least 6 months prior to that. That is 9 months and 6 days. Had the date of our first formal contact been the date of an imagined conception, then mother and child would now be safely at home and posting cute pictures to Facebook. I do hope that these units can be repaired before this imagined child is taking it’s first steps, or indeed entering school.

I am left with the inescapable and pervasive sense that Berkeley group would prefer to appear to be taking action on community facilities and environmentally sound improvements rather than actually taking action.