Bootstrapping AWS with Terraform and CodeCommit

A rough model that I’ve been working on and thinking about recently is for the AWS account (or accounts) be put together so that there’s a “bastion” or “bootstrap” instance that can be used to build out the rest of the environment. There is a certain chicken-and-egg around this, particularly if you want to use AWS resources and services to bootstrap this up.

I’m going to talk (at length) about a solution I’ve recently gotten sorted out. This has a certain number of pre-requisites that I’ll outline before getting into how it all hangs together. The key thing around this is to limit as far as possible any manual tinkering about, and script up as much as possible so that it is both repeatable and able to be exposed to standard sorts of code cutting practices.

One caveat around what I’m presenting – the Terraform state is stored locally to where we are running Terraform, which is not best practice. Ideally we’d be tucking it away in something like S3, which I will probably cover at a later point.

Workshop, Mark II

I’ve moved my workshop to a new location, which has the advantage of security, lower cost, and a far more pleasant location. Also, apparently it’s a studio now, if only I could either monetise it or adopt the life of a penniless bohemian starving artist.

Creating a custom Kylo Sandbox

I had a need – or desire – to build a VM with a certain version of NiFi on it, and a handful of other Hadoop-type services, to act as a local sandbox. As I’ve mentioned before, I do find it slightly more convenient to use a single VM for a collection of services, rather than a collection of Docker images, mainly because it allows me to open the bonnet of the box and get my hands dirty fiddling with the insides of the machine. Since I wanted to be picky about what was getting installed, I opted to start from scratch rather than re-using the HDP or Kylo sandboxes.

The only real complication was that I realised that I also wanted to drop Kylo on this sandbox, which happened after I’d already gone down the route of getting NiFi installed. This was entertaining as it revealed various ways in which the documentation and scripts around installing Kylo have some inadvertent hard-wired assumptions about where and how NiFi is installed that I needed to work around.

Smoke testing Kafka in HDP

Assuming that you have a vanilla HDP, or the HDP sandbox, or have installed a cluster with Ambari and added Kafka, then the following may help you to smoke test the behaviour of Kafka. Obviously if you’ve configured Kafka or Zookeeper to be running on different ports, this isn’t going to help you much, and it also assumes that you are testing on one of the cluster boxes, and a ton of other assumptions.

The following assumes that you have found and changed to the Kafka installation directory – for default Ambari or HDP installations, this is probably under /usr/hdp, but your mileage may vary. To begin with, you might need to pre-create a testing topic:

bin/kafka-topics.sh
    --zookeeper localhost:2181 \
    --create --replication-factor 1 \
    --partitions 1 \
     --topic test

then in one terminal window, run a simple consumer:

bin/kafka-console-consumer.sh \
     --zookeeper localhost:2181 \
     --topic test \
     --from-beginning

Note that this is reading from the beginning of the topic, if you want to just tail the recent entries, omit the --from-beginning instruction. Finally, in another terminal window, open a dummy producer:

bin/kafka-console-producer.sh \
    --broker-list localhost:6667 \
    --topic test

There is an annoying asymmetry here – the consumer and most other utilities look to ZooKeeper to find the brokers, but the dummy producer requires an explicit pointer to one or more of the brokers. On this consumer window, type stuff, and you should see it echoed realtime in the consumer window. When finished, ^C out of the producer and consumer, and consider your work done.

Lies, Damned Lies and Programmers

I recently came across a really nice set – not directly related – of articles dealing with various profound errors that programmers and system designers fall into when dealing with names and addresses.

The TL;DR if you don’t read these: names and addresses are hard and most things you believe about them are wrong.

Let’s start with Falsehoods Programmers Believe About Names. Without even trying the author lists 40 things we believe about names that are just plain wrong.

In a similar vein, Falsehoods programmers believe about addresses, which particularly speaks to me. One of the fundamental errors about addresses is to think they identify a location. This is incorrect. An address might identify a location, but it is fundamentally a description which instructs a postman how to deliver a letter or parcel. Substitute pizza operative, Amazon driver or writ server as desired.

Even without getting into the weirdness around the actual shape of the planet, Falsehoods programmers believe about geography touches on place names.

And as a bonus: Falsehoods programmers believe about time – computers prove to be pretty bad clocks, and working out a calendar is very complicated.

A Demonstration NiFi Cluster

In order to explore NiFi clustering, and NiFi site-to-site protocol, I decided that I could use a minimal installation – as I’m really just exploring the behaviour of NiFi itself, I don’t need to have any Hadoop environment running as well. To this end, my thought was that I could get the flexibility to just play around that I need by building a minimal Centos/7 virtual machine, running in VirtualBox. The plan was to have little more than a Java 8 SDK and NiFi installed on this, and then I would clone copies of it which would be modified to be independent nodes in a cluster. At the time of writing this is still in progress, but I thought it was worth capturing some information about how I proceeded to get my VM prepared.

There are a handful of requirements for this VM:

It needs a static IP (so that I can assign different static IPs to the clones, later)
It needs to be able to reach out to the broader internet, in order to pull down OS updates and similar
I need to be able to ssh to it from my desktop
Different instances of the VM need to be able to reach each other easily
A Java 8 JVM is needed

List-o-mania

I have, once again, felt stuck, spinning my wheels in the mud. There is an unpleasant, and possibly vicious, cycle at play here in my head: my planning falls apart, I feel like I am not getting anything done, my anxiety spikes, I cannot plan cogently. Repeat and repeat and repeat like some damned overwrought Philip Glass piece. I am trying to look at this dispassionately, because if I can understand how this happens, maybe I can head it off next time.

There are a few factors – health, political chaos, and too many months of uncertainty at work. Having a work and personal phone, and a work and personal computer, and disconnected accounts across both is really not helping either – I keep dropping things between the various calendars and todo lists, which has been exacerbated in the last few months by traveling. You would think that separating work and non-work would be easy. I can partition off my 37.5 hours and leave it at work, can’t I? Well, no. Because I’m trying to juggle calendars and waking hours and mental effort between work and non-work, and I cannot just turn off my brain at the end of the working day. Increasingly I feel like I would do very well if I cloned myself at least twice, so that different instances of myself could live full and uncomplicated lives. And I really resent the 3+ hours tied up each day in commuting, even while I know other people are doing the same or worse.

Continue reading “List-o-mania”

Thibault – Chapter 6

On Attacks and Counters In the Straight Line

Zachary, in the preceding chapter, saw how easy it was for Alexander to defend the simple attacks at First Instance. He asks Alexander to give him those attacks so that he can practice. Alexander, being a jerk, tries to win the drill by introducing the subjection, to which Zachary responds with a variety of “oh my god, you have your SWORD in my FACE” reactions.

Continue reading “Thibault – Chapter 6”

Two-factor in the middle of the night

Wherever possible I have been enabling two-factor authentication and similar protections. Not that I am paranoid, it’s just that I am paranoid. One of these I have had in play for a long time is protection on my Google account. So it’s somewhat comforting to get an unexpected SMS message from Google in the middle of the night sending me an unexpected authorisation code. Because it means whoever just tried to access my account could not.

Lock your doors people. A simple username and password combination, particularly on anything critical, is effectively useless.

Thibault – Chapter 5

On Attacks at the First Instance, and Feints

This is the first chapter where Thibault leaves off his purely theoretical discussion and begins actual paired exercises. Poor Zachary comes off rather the worse for wear here, as he launches a variety of simple thrusts straight down the diameter (with one exception) from the first instance i.e. the distance he minutely detailed in the previous chapter. At least part of the point of this chapter is to setup the reasons for the actions and plays in the next chapter, as well as illustrate that the straight line is sufficient preparation and fortification against these attacks. Having said that though, to use his own words:

…fortified against all manner of feints, assured against all attacks…always making the (counter) with small movements, which have more force than showiness, and making the execution with as much force and assurance as possible, in opposition to common practice. If you say to me that it is not likely that anyone may easily reach such perfection in demonstrating all these effects, I answer that nothing commendable can ordinarily be acquired without great labour.

Continue reading “Thibault – Chapter 5”