Retry and Recover

Coding against eventually consistent systems

Eventual consistency is a key concept in most distributed systems. Lets consider a distributed storage system, we can say it is eventually consistent if the data that we write/update will eventually be available to all the users at some point if not immediately. When we write an application that depends on such a system, our application shouldn't suck at intermediate failures and should be able to quickly recover from transient issues. Amazon S3 is a good candidate for a storage service that is eventually consistent. So lets write a very simple piece of code which,

1. Creates a bucket in Amazon S3
2. Adds an item to the bucket

If we write something like the following, we are in deep trouble.

AmazonS3 s3 = new AmazonS3Client(awsCredentials);
String bucket = "awesome_bucket";
s3.createBucket(bucket);
s3.putObject(bucket, "my_key", file);

The call to createBucket() doesn't guarentee the availability of bucket immediately. As a result,

s3.putObject() may fail with a BucketNotFoundException and that's bad.

A better code would be,

AmazonS3 s3 = new AmazonS3Client(awsCredentials);
String bucket = "awesome_bucket";
boolean bucketExists = false;
int tries = 0;
while (tries < MAX_RETRIES) {
    tries++;
    s3.createBucket(bucket);
    if ((bucketExists = s3.doesBucketExist(bucket))) {
        break;
    }
    Thread.sleep(1000);
}
if (bucketExists) {
    s3.putObject(bucket, "my_key", file);
}

Let's go one step further and come up with a nice abstraction. A functor could've made things much simpler. Anyways,

public void eventually(Runnable retryableTask, 
        Callable<Boolean> validatorTask,
        long bailOutTimeInMs) throws Exception {

    int noOpInMs = 1000;
    Exception retryableTaskException;
    while (bailOutTimeInMs > 0) {
        try {
            retryableTask.run();
        } catch (RuntimeException e) {
            retryableTaskException = e;
        }
        if (taskValidator.call()) {
            return;
        }
        Thread.sleep(noOpInMs);
        bailOutTimeInMs -= noOpInMs;
        noOpInMs *= 1.2;
    }
    throw retryableTaskException;
}

Basically the above wrapper takes in a retryable task and a validator task and sleeps between two tries with a geometric back off timer. If validator task returns success, then we return.

Lets try our example using the wrapper,

eventually(
    new Runnable() {
        @Override
        public void run() {
            s3.createBucket(bucket);
        }
    },
    new Callable() {
        @Override
        public Boolean call() {
            return s3.doesBucketExist(bucket);
        }
    });

This way, we can submit any task that needs to be retried conditionally. If we want to make the validator return an object, we can use generics and pass a Callable <T> validatorTask.

What have we done ?

• We made sure that our service is resilient to intermittent inconsistencies caused by dependent services that are eventually consistent.
• Written a template to which we can submit retry-validate logic so that we don't end up writing while loops and conditional breaks all over our business logic and trash the code.

~Surya