Category: Web

Matters related to the Web

Robots. They are coming to take your content.

I am in the process of revising my site, and discovered for whatever reason that I had an empty robots.txt file present. I know it is only a voluntary ‘standard’, but as far as I know all the major players do respect it. As the overwhelming proportion of users use a search engine that respects the standard, it does form a useful way of shaping what shows up in the general public eye.

I can never remember the syntax though, so for your reference and my recollection – http://www.robotstxt.org

Addendum: I was not familiar with the semi-standard for site maps so I’ve added that as well to see what the effect will be.

Addendum:Ritta Blens has pointed me to another very useful tool for testing the structure of a robots.txt file: https://www.websiteplanet.com/webtools/robots-txt/

Doing More With Less (Part 1 of N)

In recent weeks I have been massively overhauling the monitoring and alerting infrastructure. Most of the low-level box checks are easily handled by CloudWatch, and some of the more sophisticated trip-wires can be handled by looking for patterns in our logs, collated by LogStash and exported to Loggly. In either case, I have trip wires handing off to PagerDuty to do the actual alerting. This appeals to my preference for strong separation of concerns – LogStash/Loggly are good at collating logs, CloudWatch is good at triggering events off metrics, and PagerDuty knows how to navigate escalation paths and how to send outgoing messages to which poor benighted bastard – generally and almost always me – has to be woken at 1:00 AM.

One hole in the new scheme was a simple reachability test for some of our web end points. These are mostly simple enough that a positive response is a reliable indicator that the service is working, so sophisticated monitoring is not needed (yet). I looked around at the various offerings akin to Pingdom, and wondered if there was a cheaper way of doing it. Half an hour with the (excellent) API documentation from PagerDuty, and I’ve got a series of tiny shell scripts being executed via RunDeck.

#!/bin/bash
if [ $(curl -sL -w "%{http_code}\\n" "http://some.host.com/api/status" -o /dev/null) -ne 200 ]
then
  echo "Service not responding, raising PagerDuty alert"

  curl -H "Content-type: application/json" -X POST \
    -d '{
    "service_key": "66c69479d8b4a00c609245f656d443f1",
    "event_type": "trigger",
    "description": "Service on http://some.host.com/api/status is not responding with HTTP 200",
    "client": "Infra RunDeck",
    "client_url": "http://our.rundeck.com"
    }' https://events.pagerduty.com/generic/2010-04-15/create_event.json
fi

This weekend I hope to replace the remaining staff with a series of cunning shell scripts. Meanwhile the above script saves us potentially hundreds of pounds a year in monitoring costs.

Passwords definitely considered broken

So we have news of yet another major slurping-up of poorly secured credential sets. A column at the Guardian talks about all the usual measures that can be taken to more-or-less protect your multiple identities, but once again misses the two subtle and deeply geeky issues that underly this breach.

Continue reading “Passwords definitely considered broken”

Deserialising Lists with Jersey

I very much like the way in which Jackson and Jersey interact to make building a RESTful interface with objects transported as JSON really, really simple.

As an example, if we have on the server side a class like this:

@Path("/things")
public final class ThingService {
  @GET
  @Path("/thing")
  @Produces(MediaType.APPLICATION_JSON)
  public final Thing getThing(@PathParam("thingId") final int thingId) {
    return dataLayer.fetchThingById(thingId);
  }
}

then consuming the service is joyfully simple (note that this is a slightly fudged about example, and in reality more sophisticated construction of the Client instances would be recommended)

public final class ThingClient {
  private final transient Client client;
  private final transient WebTarget baseTarget;

  public ThingClient(final String serviceUrl) {
    ClientConfig cc = new ClientConfig().register(new JacksonFeature());
    client = ClientBuilder.newClient(cc);
    baseTarget = client.target(serviceUrl);
  }

  public final Thing getThing(final int thingId) {
    return WebTarget target = baseTarget.path("thing")
    .path(Integer.toString(thingId))
    .request(MediaType.APPLICATION_JSON)
    .get()
    .readEntity(Thing.class);
  }
}

Of course, it’s very likely your service will have a bunch of different end points, so you’ll want to pull some of the repeated boiler plate out into separate methods, perhaps something like this (where goodStatus checks that we’ve got some 2xx response from the server, and parseResponse constructs a suitable exception to throw if we got an error response):

public final class ThingClient {
  private final transient Client client;
  private final transient WebTarget baseTarget;

  public ThingClient(final String serviceUrl) {
    ClientConfig cc = new ClientConfig().register(new JacksonFeature());
    client = ClientBuilder.newClient(cc);
    baseTarget = client.target(serviceUrl);
  }

  public final Thing getThing(final int thingId) {
    WebTarget target = baseTarget
    .path("thing")
    .path(Integer.toString(thingId));
    return fetchObjectFromTarget(Thing.class, target);
  }

  public final OtherThing getOtherThing(final int otherId) {
    WebTarget target = baseTarget
    .path("otherThing")
    .path(Integer.toString(otherId));
    return fetchObjectFromTarget(OtherThing.class, target);
  }

  private <T> T fetchObjectFromTarget(final Class<T> returnType, final WebTarget target) {
    Response response = fetchResponse(resourceWebTarget);
    if (goodStatus(response)) {
      return response.readEntity(returnType);
    } else {
      throw parseResponse(response);
    }
  }

  private Response fetchResponse(final WebTarget target) {
    return target.request(MediaType.APPLICATION_JSON).get();
  }
}

This allows us to have a nice consonance between the client and the server, and you can even muck about and ensure the two are kept in line by deriving them from the same interface or base classes.

The one annoyance in this picture is really a matter of documentation. How do you consume a collection of objects?

Declaring the collection service is equally trivial 

@Path("/things")
public final class ThingService {
  @GET
  @Path("/thing")
  @Produces(MediaType.APPLICATION_JSON)
  public final Thing getThing(@PathParam("thingId") final int thingId) {
    return dataLayer.fetchThingById(thingId);
  }

  @GET
  @Path("/all")
  @Produces(MediaType.APPLICATION_JSON)
  public final List<Thing> getAllThings() {
    return dataLayer.fetchThings();
  }
}

however the Jersey documentation is… opaque… when it comes to consuming this on the client side. It turns out that this is where the GenericType comes into play (at line 26)

public final class ThingClient {
  private final transient Client client;
  private final transient WebTarget baseTarget;

  public ThingClient(final String serviceUrl) {
    ClientConfig cc = new ClientConfig().register(new JacksonFeature());
    client = ClientBuilder.newClient(cc);
    baseTarget = client.target(serviceUrl);
  }

  public final Thing getThing(final int thingId) {
    WebTarget target = baseTarget.path("thing").path(Integer.toString(thingId));
    return fetchObjectFromTarget(Thing.class, target);
  }

  public final List<Thing> getThings() {
    WebTarget target = baseTarget.path("all");
    return fetchListFromTarget(Thing.class, target);
  }

  private <T> List<T> fetchListFromTarget(final Class<T> returnType, final WebTarget target)  {
    Response response = fetchResponse(resourceWebTarget);
    if (goodStatus(response)) {
      return response.readEntity(new GenericType<List<T>>() {});
    } else {
      throw parseResponse(response);
    }
  }

  private <T> T fetchObjectFromTarget(final Class<T> returnType,
    final WebTarget target) {
    Response response = fetchResponse(resourceWebTarget);
    if (goodStatus(response)) {
      return response.readEntity(returnType);
    } else {
      throw parseResponse(response);
    }
  }

  private Response fetchResponse(final WebTarget target) {
    return target.request(MediaType.APPLICATION_JSON).get();
  }
}

The documentation for GenericType is not great, but essentially it indicates an automagic wrap and unwrap of the collection.

(By the way, a tip of the hat to John Yeary for identifying this solution a few years ago).

When Typography Goes Bad

So I’ve just spent a very confused 15 minutes trying to figure out why something that cannot possibly go wrong was breaking. I’m working to get DynamoDBLocal up and running.

It all looks very simple: download the tar ball, unpack it, and execute the Java invocation:
java –Djava.library.path=. -jar DynamoDBLocal.jar

Hmm. That’s odd:
Error: Could not find or load main class –Djava.library.path=.
Looking at the blog comments, a good number of folk have found the same problem, and come up with suspiciously complex solutions. Let’s see what happens if I re-type the command rather than cutting-and-pasting:
java -Djava.library.path=. -jar DynamoDBLocal.jar

Success:
2013-10-15 10:14:47.024:INFO:oejs.Server:jetty-8.y.z-SNAPSHOT
2013-10-15 10:14:47.220:INFO:oejs.AbstractConnector:Started SelectChannelConnector@0.0.0.0:8000

Can you see it? No? I missed it as well: the ‘-‘ in front of the D is an em-dash, not a minus sign (ie ASCII character 45)

Toward a vision of Sustainable Server Programming

For a number of years I’ve been thinking that I should write down some of the ideas I’ve had and lessons I’ve learned over way too many years banging on a keyboard. In my head this has has always been centered around the vague label “sustainable server development”. Let me try to peel away some layers of that, and make the label a little less vague.

We will begin with “server”. A substantial amount of what I’ve written over the past decade or so I would handwave label as “server side”. But what do I mean by that? For me a “server” is a program intended to be mainly headless, running unattended for considerable periods of time, probably on remote hardware, and providing a well defined service. By preference for me a server runs under some form of Unix. Really the landscape has collapsed to three platforms now: some form of Unix, Windows, and the widely variable set of options for software embedded in specialist hardware (although these days, that is very frequently a Unix variant as well). I’ve done a little bit against Windows, and always found the experience frustratingly complex and ultimately unrewarding. Unix was built from the ground up to provide many of the facilities needed for “server side” coding (I’m particularly thinking of a robust security model, abstracted hardware and networking facilities, and sophisticated multi-processing facilities), and provides the coder with access to multiple layers of the stack between her code and the hardware in ways that Windows makes difficult. Bringing that back to a statement: for me a “server” is a headless, service-oriented piece of code running under Unix, and required to be robust, performant and reliable.

So. “sustainable”. Like any coin, this has two sides (I know, technically all coins have at least three faces): “sustainable server” and “sustainable development”. I believe the two are really linked, and hope over a series of articles to illustrate this. When I talk about a “sustainable server”, I mean something that has been built to minimise hassle and surprise for administrators and for code maintainers. When I talk about “sustainable development” I mean approaches that make building and maintaining robust, reliable and performant code a pleasant and simple 9-to-5 job, rather than a heroic nightmare of late nights, pizza and caffeine.

I am not a fan of heroic coding. There is plenty of verified clinical evidence that amply demonstrates that a tired and stressed coder is a bad coder: some clinical studies suggest that a few days disturbed sleep has the same effects on cognition to being seriously inebriated. We have a culture that is proving very hard to break, where a mad hack over a sleepless week resulting in partially completed, un-documented, un-maintainable code is an effort to be applauded (and repeated) rather than treated as an unwelcome and undesired exception. While the coder is subject to a variety of lunacies from project managers and product owners, we are our own worst enemies if we keep committing to unhealthy and irrational death marches. A calm and rational approach to developing server side services should make it unnecessary: most of the problems to be solved in this space have been solved before, and we’ve got a lot of historical precedents to call on. Most of the time, none of this has to be considered hard or complex, so please just go take a cold shower and a walk around the block, and calm down.

Let me point to an example outside the coding world. Watch a carpenter, or a blacksmith, at work. There’s no sense of rush or panic or urgency. The craftsman knows how long each part of the process takes, has learned from the past, and is happy to re-use established patterns. She gives herself time to deal with the hard parts of the problem by knocking away the simple parts efficiently. And most relevantly: if a project manager rushes in and says “this needs to be done in half the time”, the response is not “oh, in that case we’d better order pizzas because it will be a long night.”

The key elements of what I would classify as ‘good’ server software are as follows:

1) Clarity of purpose. The software does one thing, provides one well defined service;
2) Predictability. The software behaves in a well defined and documented fashion;
3) Robustness. The server should be resilient and gracefully adapt to environmental changes, cope with malformed or malicious requests, and not collapse under extreme load;
4) Manageability. Administrators should be able to monitor and configure the service with ease, and should be able to programatically manage the service state;
5) Performant. Requests should be responded to within 50ms – 100ms or better under all conditions. In particular performance should not degrade unpredictably under load.

In my experience a lot of coders – and managers of coders – have the idea that setting these goals as a minimum base requirement are unrealistic and expensive. Twaddle and nonsense. Let me point to two exemplars, both available as FOSS and both initially built out by small teams: Varnish and the core Apache server. In both cases, these are not part of the base operating system, they are services run on a server. In both cases, all the goals above are amply met. And in both cases, there is a development and maintenance infrastructure around the code which is palpably sustainable and effective.

Varnish is a particularly fine example. There were no surprises or complexities in installing it and running it. It worked as expected ‘out of the box’ without intensive configuration. It’s very easy to monitor and manage. It does one thing, extremely well, and does it in the fashion described, documented and expected. And most importantly it just runs and runs and runs without intervention or alarm.

Lets make all server software that good, and knock off work at 5pm, enjoy our weekends, take up hobbies and stop these panicked head-long rushes into the night. Our partners, family, waistlines and hearts will thank us for it.

All in a twitter.

There has been some talk already regarding the use of Twitter as Brisbane sank beneath the waves. Unfortunately all the talk I’ve seen so far has limited itself to merely cheering that the service was marevelous (for example, some of the talk over at The Drum), without examining what worked and what did not.

As I tap away at this on the train, I note that John Birmingham has touched on the subject, and his comments are certainly accurate and pertinent. I definitely echo his thoughts on the essential uselessness of traditional broadcast media through all of this. The offerings from the free-to-air television services were worse than useless, and the commercial radio stations carried forward as if nothing was happening. I say “worse than useless” because all that I saw from the FTA television services was distorted, often inaccurate and out of date, and carried an air of desperately attempting to manufacture panic and crisis.

There was a particular gulf between the representations of what areas have been affected. If you watched any of the three commercial stations, you would gather that the only flood affected areas were Toowoomba, the Lockyer Valley, Rosalie, Milton and West End. If you watched the ABC you knew that Rocklea and Fairfield were trashed. If you monitored Twitter and other social media, you saw people on the ground with phones desperately broadcasting that areas like Fig Tree Pocket and Goodna were essentially destroyed, and can we please stop talking about the Three Monkeys Cafe in West End?

Of course, I no longer have any expectation that traditional broadcast media can be either informative or effective. And I include our apallingly bad newspaper of record here – the joke in Brisbane goes “Is that true, or did you read it in the Courier Mail?” Direct dealings with representatives of the broadcast and print media here over the last ten years or so have consistently emphasised that they will not travel more than a few kilometers from the center of town, and absolutely will not seek anything other than a single image or 15 second film grab that can be used as a sting. [refer channel 9 drinking game here].

What interested me most over the past week has been how various “official” Twitter voices have used the service. There were some marked and intriguing differences. Individual users definitely grok Twitter – a constellation of different #hashtags coalesced to one or two within about 24 hours, and the crowd mainly acted to filter out spam and emphasise important and useful information. There was a constant background hum of spam and attempted scams in the feed, but I noticed whenever an important message was submitted from one of several voices of authority (and a tip of the hat to John Birmingham here, he carries a lot of weight on line), the crowd spontaneously amplified the message and ensured it was being heard: the flow was usually from Twitter to other social services like Facebook and LiveJournal, and even back onto the comments pages on web sites for the traditional media outlets.

Three particular accounts interested me: the 612 ABC channel, the Queensland Police channel, and my bete noir, the TransLink SEQ channel. A parenthetical aside here as well, I use the word ‘channel’ in the sense of water (and information) flow, not in the sense of a TV or Radio channel.

Someone at 612 has understood Twitter right from the beginning, although it’s pretty obvious when their official operator is working, and not working, as the rate of messaging fluctuates wildly over the day. The bulk of their messages are snippets of information, or direct questions requesting feedback or information. Occasionally they will point off to their own website for further interaction, usually to pages used to gather information rather than distribute, and occasionally point off at other resources.

The QPS channel historically was of mixed quality, and their direction zig-zagged over the week before settling into a solid pattern: messages were well #hashtagged, important information was emphasised and repeated, messages about deeper background information held on other sites had sufficient summary information to allow the reader to tell whether they needed to go to the external site.

TransLink, by contrast, was an example of how not to use the service. There was every indication that they were explicitly refusing to respond to direct messages or any sort of feedback, and virtually all their messages contained no content and directed readers to their web site. Of course on Tuesday as the CBD was to all intents and purposes evacuated, the web site melted down, and it was unusable for much of the week. I will refrain from pointing out the flaws of their site, here and now, but may come back to it. The height of their lunacy on Tuesday was when many, many people were asking if the rumour that public transport was halting at 2PM was true, and the *only* response in return was to keep repeating that they had a page with service statuses on it.

Energex and the Main Roads department had similar problems with their websites failing under load, and in retrospect this is an argument for the QPS media unit using Facebook to distribute further information: the chance of failure of Facebook as a web destination is far lower.

The twitter stream from TransLinkSEQ is particularly interesting for the relative lack of information:

Through the morning, we had the following:

  • All CityCat & CityFerry suspended. Check the web for connecting buses. Leave extra time. More info http://www.translink.com.au
  • Due to heavy rain delays to some bus services, diversions on some routes. Check service status http://www.translink.com.au
  • Caboolture Line inbound and outbound services delayed up to 15mins due to signal fault. http://alturl.com/2thz8
  • Caboolture bus services majorly affected by flooding. http://alturl.com/b2brf
  • North Coast Line delays up to 40mins due to track/signal faults. Effects
  • Caboolture line, delays up to 15mins. http://alturl.com/y99ap
  • Rosewood-Ipswich train services suspended due to water on tracks at Rosewood. Buses arranged. http://alturl.com/c6yvq
  • All CityCat & CityFerry services expected to be out of action all day due to strong river currents –> http://twurl.nl/7bwxnl
  • Caboolture bus services cancelled. Visit http://translink.com.au for more.
  • All Kangaroo buses cancelled. Visit http://translink.com.au for more.

After about 12pm there were wide spread rumours – and a lot of direct questions were sent to TransLink about this – that public transport in the CBD was to be suspended at 2pm. This was what they broadcast in that period:

  • For more information on flood and weather affected services – http://twurl.nl/jct4cl
  • For information on the current status of flood affected services please refer to our website – http://twurl.nl/6z52j0
  • TransLink advises there are delays and disruptions on parts of the network. Services continue to run where possible.
  • Public Transport continues to run where possible – for latest disruption information see http://www.translink.com.au

At no point did they respond to the simple question “are services halting at 2pm”. The only rebuttal of that rumour came from the QPS Media service. After about 3pm they changed their message, and seemed to understand that people were understandably cranky:

  • Services are running throughout this afternoon. Expect delays & some cancellations. Check the website for service status info.
  • Our call centre is receiving a high number of calls, causing delays in answering. Check website for info to help us manage the call volume.
  • Trains are not operating to schedule this evening due to flooding. Services are still operating on all lines -> http://twurl.nl/z2i223
  • All train services at reduced frequency until further notice, some services have been suspended. Find out more –>http://twurl.nl/7c7esj
  • All train services suspended until 6am Wed 12 Jan. An hourly train timetable will then be in place, until further notice.

It’s no surprise that their website melted down after midday – note that virtually all their messages contained no useful information and just redirected to the website.
Successful use of Twitter as a meaningful and important information and communication tool recognised a handful of very key features of the service that distinguish it from many other services:

  • it is more like a broadcast service than an an asynchronous service like a web page;
  • messages should be considered ephemeral and only made meaningful by currency;
  • the tiny messages mean that it is accessible through an extremely broad range of mobile devices;
  • a very significant number of users use Twitter via mobile devices;
  • the infrastructure has evolved and been designed to support a staggeringly large number of simultaneous requests;
  • relevant information-rich messages are spread further and live longer than information-poor messages;
  • the service is inherently a two-way information flow, and questions and criticisms that flow back are indicators of errors or inadequacies in the outgoing flow;

I am hoping that organisations involved in this disaster take stock of how they used these services, and how these services can and should be used. The big lesson that can be learned here is that significantly more people have mobile phones with internet access than have battery powered radios.

Moving…

I’m back to updating this site, and will soon publish a link on the main site page to it. For the time being, I’m transferring pieces from Deviant Art, backdating entries to roughly correspond to the publication date. It will be interesting to see how long this takes, as I’m testing the use of ecto again, and it will take me a little time to adjust. Not to mention that I will halt every so often from boredom…