Category: Technology

Things Technological

Doing More With Less (Part 1 of N)

In recent weeks I have been massively overhauling the monitoring and alerting infrastructure. Most of the low-level box checks are easily handled by CloudWatch, and some of the more sophisticated trip-wires can be handled by looking for patterns in our logs, collated by LogStash and exported to Loggly. In either case, I have trip wires handing off to PagerDuty to do the actual alerting. This appeals to my preference for strong separation of concerns – LogStash/Loggly are good at collating logs, CloudWatch is good at triggering events off metrics, and PagerDuty knows how to navigate escalation paths and how to send outgoing messages to which poor benighted bastard – generally and almost always me – has to be woken at 1:00 AM.

One hole in the new scheme was a simple reachability test for some of our web end points. These are mostly simple enough that a positive response is a reliable indicator that the service is working, so sophisticated monitoring is not needed (yet). I looked around at the various offerings akin to Pingdom, and wondered if there was a cheaper way of doing it. Half an hour with the (excellent) API documentation from PagerDuty, and I’ve got a series of tiny shell scripts being executed via RunDeck.

#!/bin/bash
if [ $(curl -sL -w "%{http_code}\\n" "http://some.host.com/api/status" -o /dev/null) -ne 200 ]
then
  echo "Service not responding, raising PagerDuty alert"

  curl -H "Content-type: application/json" -X POST \
    -d '{
    "service_key": "66c69479d8b4a00c609245f656d443f1",
    "event_type": "trigger",
    "description": "Service on http://some.host.com/api/status is not responding with HTTP 200",
    "client": "Infra RunDeck",
    "client_url": "http://our.rundeck.com"
    }' https://events.pagerduty.com/generic/2010-04-15/create_event.json
fi

This weekend I hope to replace the remaining staff with a series of cunning shell scripts. Meanwhile the above script saves us potentially hundreds of pounds a year in monitoring costs.

First World Problems

So we have installed Rocki units in three rooms in the flat – the lounge, the bedroom, and the library. I just started playing a Clannad album from my laptop to the speakers in the lounge. Much to my bewilderment, a moment later a different album started playing in the bedroom.

I thought it may have been coming from my phone, so shut that down. Kept going. From my iPad? Shut that down. From my partner’s phone? Shut that down. Something bizarrely broken with AirFoil on my laptop? Shut down the laptop. Has someone managed to hack our network and is pranking us for lulz? Is it the NSA? MI5?

Nope. One of the cats had sat on the stereo remote control in the bedroom and started playing a CD.

I need a simpler life.

Passwords definitely considered broken

So we have news of yet another major slurping-up of poorly secured credential sets. A column at the Guardian talks about all the usual measures that can be taken to more-or-less protect your multiple identities, but once again misses the two subtle and deeply geeky issues that underly this breach.

Continue reading “Passwords definitely considered broken”

Singletons considered harmful

Ok, I know it’s not a new observation, but the Singleton pattern must be one of the most overused, and abused, patterns that the Gang Of Four described.

This is on my mind this week as I’m working on a body of code that has way too many Singletons. I must emphasise that ultimately it’s my problem, not the original author’s, as I dropped the ball over a year ago and did not review the design and implementation. The problem has come home to haunt me as I introduced just one change too many and all the tests began to fail.

Particularly in this case, while looking at test coverage I wondered why a pretty important piece of life cycle management wasn’t being traversed in tests. Which led me to have a close look and realise that it was buggy, and failing out right at the start of execution during tests. So I fixed that, and all the tests threw up because the Singleton in question was no longer in the expected state.

My main gripe with Singletons is that they run headlong into one of the cardinal rules of unit testing: all tests should be entirely independent of each other. The problem with a Singleton – particularly one that has some sort of lifecycle – is that suddenly tests are connected by the internal state of an object that may not even be the unit under test. Which leads to unstable tests prone to mystery failures. And unstable tests lead to a lack of confidence in the validity of the code.

Now, I’m going to need to articulate this to other coders to head off any repeat of this problem, so it’s worth my while to hand wave about when Singletons are appropriate, and when other techniques are better.

To begin with, I often see Singletons introduced to provide static pieces of code. I strongly suspect that this is because the coder does not understand how static methods and attributes work, or simply forgets. Probably the biggest single clue that these cases should not be implemented as Singletons is that they have no persistent state.

When talking it through with the team, both zoomed in to that idea from two different directions with little prompting: by thinking about the code construct (the Singleton pattern) instead of thinking about the data, it is way too easy to not see that the Singleton pattern gives the data state a different scope and different life cycle to other code.

In the space I’m mainly playing in, it’s fairly common to have a bunch of threads handling incoming requests from some external agency, all in kind of similar ways. This transactional model, if inverted to be data centric, can be summarised as: accept data, map it onto an output state, and throw away any working state in preparation for the next request. In Java terms the scope of all data is local to the thread. The data state of the Singleton, however, is at a higher level – an application or service level. Thus objection one: Singletons cause data states at different levels of abstraction or different levels of management to be promiscuously mixed.

This immediately leads to objection two: Singletons easily cause cross-thread side effects, as they bind threads together in non-obvious ways. This problem can be lessened if the Singleton provides read-only state, in which case it might be better done using static attributes, and if the potential side effects are well documented and described.

Objection three is somewhat more of an aesthetic gripe. The common ways in which Singletons are usually implemented in Java, apart from not being as thread-safe as they appear to the naive eye, beaks the doctrine of Separation of Concerns. The Singleton class has two responsibilities, not just one, which is a very bad smell: it is responsible for whatever it’s purpose in life is, and it’s responsible for making sure it’s alone in the universe.

There are a variety of ways of getting around this bad smell. A lot of runtime containers – be it simply the JVM firing up with some single instance of a class providing main(), or Spring or a web application server taking care of the “only one” behaviour behind the scenes – provide a trustable context for which you can say “if I make just one of these objects, and put it in that context, there will only be one of them”. In the case of the examples above, as well, it means that we have the instance of the object in some sort of “application” or “service” scope, with a life cycle that can be tied to the broader context.

At a bare minimum, if you cannot identify or obtain access to the application context, you should aim to separate out the two concerns – provide a class that does stuff, and a class that holds a single instance of that do-stuff class. While adding a little bit of extra boiler plate code, this simple change suddenly means you can test the two behaviours independently and that you can have thread-local instances injected in the scope of your independent unit tests.

And a final objection, primarily aesthetic. There are a bunch of different ways to build a Singleton in Java. Not all of them are thread safe, and it’s annoyingly difficult to do lazy instantiation in a thread safe manner, particularly if you want there to be exactly one run through a costly process. The ugliness arises because generally the methods to be thread safe are clunky kinds of fiddles that require the coder to think about the behaviour of the JVM instead the behaviour of their code. There’s that separation of concerns biting us in they arse again.

I do not think the pattern is to be universally avoided though. It’s highly probable that the application or service scope is stateful, and has a well defined life cycle. Like it or not, the life cycle state is a single piece of information that needs to exist at a different level of abstraction to the per-thread state (unless you are fortunate enough to be able to think entirely at a thread level, and there genuinely is no application level state).

As an example, I’ve fallen into the habit of using a roughly MVC architectural pattern. Sometime I will go into this in detail, but for now simply accept that it’s a handy simple framework to hang more complex behaviour of, while encouraging the decomposition of the code into easily testable parts. In my case, the ‘view’ is often provided as servlets, often with a RESTful design, and not necessarily provided by a single class. It’s pretty common for me thus to not have an accessible application level context without using Spring or similar. In these instances, I tend to use the Controller layer to hold the application-level state, and manage the application-level lifecycle. Of course, this is easily abused as well, as without paying attention you can find all sorts of pieces of code dialling home to the controller layer or object, but at least by separating the singleton aspects from the controller aspects, you can make the opportunity to not bind tests together.

Let me leave you with a thought experiment: if I have a simple web application with just a single servlet class, does that servlet class provide a single-instance application level context?

Journalled Out

I’ve been thinking in recent days that I could use something journal-ish. There are two aspects to this thinking. For one, I tend to accumulate documents and links to things that will probably be useful someday, or I want to remember short-term, but they get smeared everywhere. Bookmarks across several machines and browsers, text documents tucked into folders optimistically labelled ‘to-do’ or ‘in progress’, stuff in various note-taking applications. All of which leads to a definite sense of mental clutter which I really want to eliminate. I have identified that one of the things that makes me anxious is physical and mental clutter, a sense of being overwhelmed by Stuff To Take Care Of Right Now.

It would be nice just to declare mental bankruptcy, throw all this in the bin, tear off my clothes, and run naked into the woods to live as a wild man, feeding on berries and roots. Regrettably while this simple life has certain attractions – not the least being an opportunity to dispense gnomic wisdom and entirely fabricated home-spun philosophy to unsuspecting passers-by – it does not appear to be paid particularly well anymore. Besides, brambles, briars and badgers are not a good match for running naked through the woods at my age.

Initially I’ve been thinking about something like Day One, which has the attraction of being somewhat insulated against future obsolecence (as far as I can tell, the data is stored in individual PLIST files), as well as having a frictionless interface. That’s important. The benefit of pencil and paper is that it’s always on. The disadvantages for me are that I cannot read my own handwriting, and generally cannot fit a usefully large notebook in my pocket. Also, so much of what I need to refer to comes with a URL or an image associated with it, there’s friction arising from needing to manually link together disparate data repositories.

The elephant in the room for all of this (see what I did there) is of course Evernote. I was startled to discover how many apps I already have on phone, iPad and desktop natively link to Evernote, and the environment Evernote occupies is rich and varied. Which makes me a little nervous: if I went this way, would I then still have different bits of data scattered across multiple interfaces? Additionally, even though they appear to be an honest and reliable company, the product still revolves around having my data on servers for a ‘free’ service.

Sigh. Thinking is in progress.

A Certain Quality

Java is not the best of languages. There are plenty of languages better for particular niches or uses, and it’s littered with annoyances and prone to abuses. So are C, COBOL and Fortran. But it’s good enough almost always, and the environment that has grown up around it has made it a useful language for building reasonably performant web-facing server products. One thing that is a standout though is the ease with which Java can reflect on itself and examine itself at runtime.

This has opened the door for a number of community led tools that allow us to declare quality standards, and automatically monitor and control adherence to those standards. These are powerful ideas: coders can relax and focus on the task at hand, secure in the knowledge that the surrounding infrastructure will maintain the quality of the code. It’s like a writer with a word processor and a good editor: spelling errors will get sorted out immediately, and somewhere down the track the grammar and prose will get beaten into shape.

There are now a good mix of static and dynamic analysis frameworks out there, and I’ve settled on Findbugs, Checkstyle and Jacoco as the core. PMD is in the mix as well, but more as a backstop for the other tools. The thing that appeals to me about these three is that the analysis they will do, and the standards they mandate, can be declared via the same Maven POM as the rest of the build definition – and in the IDE as well – so that quality control is baked in at the lowest level of development activity.

Because these are declared quality standards, it means that our Jenkins CI tool can use the same declaration to pass or fail a build – code that does not meet required standards cannot progress out of development, and Jenkins provides visibility of the current level of code quality. Jenkins is not so good, though, at showing longer term trends, which is where Sonar comes in. I was delighted to discover that Sonar had become freely available as SonarQube, as it’s a fantastic tool for seeing at a glance if there are quality trends that need to be addressed, and for expressing complex code quality issues in a cogent fashion.

The tool chain then is trivially simple for the developer to use. Maven and the IDE on the desktop tell her immediately if there are code quality issues to address before committing. On commit, the Jenkins CI build is a gatekeeper that will not allow code that does not meet certain basic criteria to pass. Finally Sonar gets to look at the code and see how it is progressing over time.

I am pleased with this tool chain for two reasons. First, code quality is an integral part of the developers daily experience, rather than something bolted on that happens later and is somebody else’s problem. Quality becomes a habit. Second, the process is entirely transparent and visible. The hard code quality metrics are right there for all to see (for certain values of “all”, they do require authentication to examine) and are visibly impartial and objective, not subjective. If I commit something dumb, it’s not a person telling me he thinks I’m wrong. The quality of my work is not only my responsibility, I have objective benchmarks to measure it against.

This sort of toolchain exemplifies in my mind a mature approach to technology by automating standard procedures, and automating whatever does not need human intervention. It’s madness to repeat any process that can be automated, more than once or twice, and the time and cost saving of automated quality control compared to manual quality control is enormous. The drawback is that setting up – and to some extent maintaining – the tool chain is non-trivial, and there is a risk that the cost of this setup and maintenance can deter enhancement or rectification of flaws in the toolchain. An interesting implication of this is that the elements of this tool chain – Jenkins, Sonar and so forth – should be treated as production environments, even though they are used to support development. This is a distinction frequently lost: this stuff needs to be backed up and cared for with as much love and attention as any other production infrastructure.

Now, not everyone appreciates the dogmatism and rather strong opinions about style implicit in the toolchain, particularly arising from Checkstyle. Part of the point of Checkstyle, Findbugs and PMD is that, like it or not, they do express the common mean generally accepted best practices that have arisen from somewhat over 15 years of community work on and with Java. They’re not my rules, they’re the emergent rules from the zeitgeist. There are really two responses if these tools persistently complain about something you habitually do in code, that one thing that you always do that they always complain about. You can relax or modify the rules, build in local variations. Or you can stop and think, and acknowledge, that maybe, just maybe, your way of doing things is not the best.

They are, after all, fallible automated rules expressed through fallible software. They are not always going to get it right. But the point of the alerts and warnings from these tools is not to force the coder to do something, but to encourage her to notice the things they are pointing out, encourage her to think about what she is doing, encourage her to think about quality as part of her day-to-day hammering on the keyboard. I’d rather see fewer, more beautiful lines of code, than lots of lines of code. It’s not a race.

I find it interesting that being able to objectively measure code quality has tended to improve code quality. Observation changed the thing being observed (is that where heisenbugs arise?). There’s not a direct relationship between the measuring tools and the code quality. Rather what seems to have happened is that by using the toolchain to specify certain fixed metrics that must be attained by the code in order for that code to ‘pass’ and be built into release artefacts, then the code changes made to attain the metrics have tended to push the code to cleaner, simpler, more maintainable code. I am aware that there are still knots of complexity, and knots of less than beautiful architecture, both of which I hope to clean up over the next year, but the point is not that those problem areas exist, but that they are visible and there’s going to be an objective indication of when they’ve been eradicated.

There seems to be a lower rate of defects reaching the QA team as well, although I don’t have a good handle on that numerically – when I first started noticing it, I neglected to come up with a way of measuring it, and now it’s going to be hard to work it out from the Jira records. (The lesson of course being: measure early, measure often.) In general the defects that seem to be showing up are now functional and design problems, not simply buggy code, or else the sorts of performance or concurrency problems that really only show up under production-like load which are difficult and expensive to test for at the development stage as a matter of day-to-day development.

There is a big caveat attached to this toolchain though. I’m a fan of an approach that can be loosely hand-waved as design-by-contract. There’s value in expressing exposed functional end-points – at whatever level of the code or system you pick – in terms of statements about what input will be accepted, what the relationship between input and output is, what side-effects the invocation has, and so forth. Black box coding. As an approach it fits neatly against TDD and encourages loose coupling and separation of concern. All very good things. In practical terms, however, it depends on two things: trust that the documentation is correct and the contract matches the implementation, and trust that the implementation has been tested and verified against the contract. If those two things can be trusted, then the developer can just use the implementation as a black box, and not have to either delve into the implementation code, nor build redundant data sanitisation or error handling. At the moment, there’s no automated means to perform this sort of contract validation. The best option at this point seems to be peer code reviews, and a piece of 2×4 with nails in it (1), but that’s expensive and resource intensive.

The bottom line reason for investing in a tool chain like this – and make no mistake, it’s potentially expensive to set up and maintain – is that if you have a typical kind of team structure, it’s easy for the developers to overwhelm the QA team with stuff to be tested. The higher your code quality, and the more dumb-ass errors you can trap at the development stage, the less likely it is that defects will get past your harried QA guys.

(1) It’s like I always say, you get more with a kind word and a two-by-four than with just a kind word. – Marcus Cole

Java 7 JDK on Mac OS X

This is one of the things that Apple should be kicked in the shin for. There is no excuse for continuing to completely foul up Java installation on Mac OS X

If you are like me, and trying to figure out how to get the Java 7 JDK installed on the latest build, here is the key: http://stackoverflow.com/a/19737307

The trick for me is probably the trick for you:
1) download the JDK from Oracle
2) run the downloaded DMG to install
3) modify your .profile or .bashrc or wherever you have it to include

JAVA_HOME=$(/usr/libexec/java_home)
export JAVA_HOME

4) make another cup of coffee and curse.