In a distributed system we must ensure that failure situations across the network are handled properly without impacting the robustness nor performance of the whole system negatively. A technical error handling can often be done with libraries, like Hystrix, resilience4j, feign or others. Also Spring AMQP cares about and offers different strategies to deal with errors. But it is not always possible to handle failures in one standardized and automated way. At least, we need to decide what failure resolution strategy from one of those libraries is chosen in case of an error.

Error Handling with Asynchronous AMQP Messaging

Spring AMQP with RabbitMQ is used for the asynchronous messaging infrastructure. A publisher sends a message to an AMQP Exchange where a consumer process has connected a AMQP Queue to. A message is sent to the Exchange and published to one or more connected Queues. If a consumer fails to process the queued message there are several possibilities how to react: Re-Queue, Throw-Away or Forward.

Re-Queuing failed Messages In case of re-queueing the message is left in the queue and executed again. How often this happens and with what interval can be configured.

Throw-Away failed Messages Simply omit the error occurred and be silent could also be a consumer strategy. In this case the message is acknowledged to RabbitMQ and is gone from the queue - but without handling the message as demanded.

Forward failed Messages A consumer could also decided to forward a message it could not process to someone else or to just forward it into a message store without knowing about the consumers of that store - there might actually be no consumer. This pattern in basically called Dead-Letter Queueing and also RabbitMQ offers this technique.

The list of strategies is by for sure not complete but enough for what we need in OpenWMS.org. So here is a short comparison of these three strategies in relation to the use in OpenWMS.org:

So when to use what kind of strategy and what configuration to apply is often a decision specific to the use case.

Strategies to re-queue Messages

By default, Spring AMQP does not acknowledge a message to RabbitMQ that has failed to process. That means a listener is called over and over again. This behaviour can simply be stopped by throwing an org.springframework.amqp.AmqpRejectAndDontRequeueException within the application listener code like it is shown in Strategies to Throw-Away Messages.

But with Spring AMQP the handling is also configurable: How often a message is re-queued and the repeat interval with many other options. …​

A typical Spring AMQP RabbitTemplate configuration of a service looks like:

@Bean
RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) {
    var rabbitTemplate = new RabbitTemplate(connectionFactory);
    var backOffPolicy = new ExponentialBackOffPolicy();
    backOffPolicy.setMultiplier(2);
    backOffPolicy.setMaxInterval(15000);
    backOffPolicy.setInitialInterval(500);
    var retryTemplate = new RetryTemplate();
    retryTemplate.setBackOffPolicy(backOffPolicy);
    rabbitTemplate.setRetryTemplate(retryTemplate);
    rabbitTemplate.setMessageConverter(jsonConverter());
    return rabbitTemplate;
}

Use Cases This is often done in use cases where the consumer expects the message is going to be repeated by the sender after a period of time.

Strategies to Throw-Away Messages

Use Cases For sure we can throw away messages, silently acknowledge them, where we know the publisher expects no answer and repeats the message in case no answer is returned. So we depend on the frequency of the publisher how often the message is repeated. For example, the publisher may be a PLC that is configured to resend the initial message request every 150 milliseconds. In this case, this could have an impact on the overall performance. Whereas an PLC may also be configured to repeat the message only after e.g. 30 seconds what has not that much impact on performance.

Configuration A typical message listener consuming the message and throws it away on any exceptions looks like this:

@RabbitListener(queues = "${owms.commands.common.tu.queue-name}")
public void onCommand(@Payload TUCommand command) {
    try {
        handler.handle(command);
    } catch (Exception ex) {
        throw new AmqpRejectAndDontRequeueException(ex.getMessage(), ex);
    }
}

The listener delegates to a message handler that is responsible to execute the business logic. In case of any errors the listener throws an AmqpRejectAndDontRequeueException to signal the Spring AMQP runtime to reject the message. So it is not processed anymore.

Strategies to Forward Messages