Posts

How to correct clock skew in AWS

Every request to an AWS service is signed using the current timestamp, that is the client time at which your application makes the request. For security reasons and to protect against potential replay attacks, AWS requires that the difference between this timestamp and the AWS server time is less than 5 minute.

A request must reach AWS servers within five minutes of the timestamp in the request, otherwise AWS denies it (some docs state 15 minutes as limit). This time difference between clocks on different nodes of a network is called clock skew. In some situations client time could be significantly out of sync, i.e. on mobile devices, in VMs or in Docker containers. Calls to AWS from such clients could fail, but the triggered exception is not necessarily clear enough to uncover the underlying problem; these are a few examples:

Depending on your scenario, several tips or solutions can be found; here is a not exhaustive list that covers some common scenarios:

  • standard Linux server: check system clock and configure the ntp service. If you are using an Amazon EC2 instance just follow this AWS guide.
  • VirtualBox VMs: modify the default VB –timesync-set-threshold value (20 mins) to something lower, i.e. 1 min, for more information check this post.
  • Docker containers: there are several issues mainly related to the sleeping or hibernation of the host both on Mac and on Windows, on this stackoverflow question you can find some tips

There are other scenarios where you won’t have administrator privileges, i.e. on mobile devices, on-premises installations or similar; in this cases your last chance is to fix the problem programmatically. AWS SDKs tackle this issue in different ways: