A few weeks ago we noticed that one of our applications had wrong timestamps in the log files by late for an hour.
At first I thought the timezone setting in the OS could be wrong but our ops people confirmed it is actually correct. Then I thought the JVM settings on that server must have something wrong. But I compared the JVM directories on the problematic server to a good server, they are exactly the same. This puzzles me. Then I wrote a simple Java app to print out all system properties and found that on the “bad” server, the default timezone is “GMT-08:00″ instead of “US/Pacific”.
After searching online, it turned out this is actually a bug in Java.
In short, in some cases JVM can get a wrong timezone from the machine, even if all the settings are right. What happened is when JVM starts, it looks for a matched file in directory /usr/share/zoneinfo/ to “/etc/localtime”. If it finds one, it uses the path to that file as the default timezone. The problem is, on different machine, JVM can traverse the directory in a different way; and even worse, for one “localtime” file, there can be multiple matches in the zoneinfo directory. For example, for our /etc/localtime, I found following matches:
/usr/share/zoneinfo/US/Pacific
/usr/share/zoneinfo/SystemV/PST8PDT
/usr/share/zoneinfo/America/Los_Angeles
/usr/share/zoneinfo/posix/US/Pacific
/usr/share/zoneinfo/posix/SystemV/PST8PDT
/usr/share/zoneinfo/posix/America/Los_Angeles
If we are lucky the “US/Pacific” happens to be the first match, we are in good shape; if the file “posix/SystemV/PST8PDT” matches first, Java will find it a invalid timezone. And the JVM stops looking further once the first match is found. So in this case it will use a default timezone, which is something like “GMT”, which of course doesn’t have day light saving adjustment. That’s why the timestamp was offset by one hour.
If you want to know more about this bug, check out following links:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6456628
https://bugs.launchpad.net/debian/+source/sun-java5/+bug/49068
The problem is complex, but the solution is simple: just specify a system property when you start your Java app:
-Duser.timezone=US/Pacific
Because JVM always tries to use that system property before it goes down to its default system value search. I added this property to the wrapper.conf file, restarted the application and everything is fine now. Maybe you should do the same thing just in case one day in a new server, a JVM goes through the directory in a slightly different way …
Entries (RSS)