Windows time service doesn't synchronize properly

So, I've spent quite some time to fix it, but to no avail...

Here is the problem: we synchronize time with swisstime.ethz.ch. We do it once a day (modified SpecialPollInterval to 8640 from the default one.)

Every day it goes out of sync, although it says that it successfully synchronized time. When we do it manually it works fine. One theory was that SpecialPollTimeRemaining might get corrupted, so I tried removing it. It worked fine for couple days, but then again it started going out of sync.

I am on Windows Server 2008 R2 Standard

Thanks for your help.