Received: (qmail 19741 invoked by uid 2012); 1 Apr 1998 18:34:10 -0000 Message-Id: <19980401183410.19740.qmail@hyperreal.org> Date: 1 Apr 1998 18:34:10 -0000 From: Robert Mela Reply-To: rob_mela@zd.com To: apbugs@hyperreal.org Subject: Max log file size is 2,147,483,616 (2^31 - 32) bytes X-Send-Pr-Version: 3.2 >Number: 2026 >Category: mod_log-any >Synopsis: Max log file size is 2,147,483,616 (2^31 - 32) bytes >Confidential: no >Severity: non-critical >Priority: medium >Responsible: apache >State: closed >Class: sw-bug >Submitter-Id: apache >Arrival-Date: Wed Apr 1 10:40:01 PST 1998 >Last-Modified: Wed May 20 04:31:20 PDT 1998 >Originator: rob_mela@zd.com >Organization: >Release: 1.2.5 >Environment: Solaris 2.5.1 gcc 2.8.1 >Description: After log files reach 2,147,483,616 bytes no more hits are logged. The server otherwise seems to perform normally. The server in question only getting about 5 million hits per day, so this only becomes a problem if our nightly rollover/ restart process does not run. >How-To-Repeat: >Fix: I know there are routines for using 64-bit file access. I'll see if these can be used and if Apache can be compiled to use them. We may hit the 2 gig limit in a few months, but terabyte limits would be the best answer. Workarounds include using the existing facility for logging to a subprocess which handles log rotation automatically. Still, I'd prefer not to introduce a second process into the system. %0 >Audit-Trail: From: Marc Slemko To: Robert Mela Cc: apbugs@hyperreal.org Subject: Re: mod_log-any/2026: Max log file size is 2,147,483,616 (2^31 - 32) bytes Date: Wed, 1 Apr 1998 11:54:38 -0700 (MST) On 1 Apr 1998, Robert Mela wrote: > >Environment: > Solaris 2.5.1 > gcc 2.8.1 > >Description: > After log files reach 2,147,483,616 bytes no more hits are logged. The > server otherwise seems to perform normally. > > The server in question only getting about 5 million hits per day, so this > only becomes a problem if our nightly rollover/ restart process does not > run. AFAIK, Solaris 2.5.x can not handle regular files larger than two gigs. Things like llseek() which do 64-bit stuff are only valid for things like device files. This should work fine on Solaris 2.6 without any changes to the Apache code. Are you logging more than a standard common log format? The sizes seem big for the number of hits using CLF. I would also suggest that for this volume of logs, if you want to do anything with them, logging to a pipe and having a program automatically put them into something (eg. database, binary format, etc.) more suited to this volume. Unfortunately, I'm not sure I can recommend you try that with 1.2.x because piped logs aren't "reliable". This doesn't mean they don't work fine, but if the process dies it won't get restarted, etc. Recent 1.3 betas do add that feature, to make piped logging a very attractive solution. From: Dean Gaudet To: Robert Mela Cc: apbugs@hyperreal.org Subject: Re: mod_log-any/2026: Max log file size is 2,147,483,616 (2^31 - 32) bytes Date: Wed, 1 Apr 1998 12:38:35 -0800 (PST) This isn't really an apache limitation at all, or it shouldn't be. Doublecheck that you don't have your filesize limits at 2Gb -- that's what nailed me previously on a solaris box. BTW, you could also consider hourly rotation using SIGUSR1 which won't interrupt transfers. This is how I handle the load on other similarly high volume sites. Dean From: Marc Slemko To: Rob Mela Cc: Apache bugs database Subject: Re: mod_log-any/2026: Max log file size is 2,147,483,616 (2^31 - 32) bytes Date: Wed, 1 Apr 1998 14:13:26 -0700 (MST) On Wed, 1 Apr 1998, Rob Mela wrote: > > > I am using my own log format, and I'm pretty sure it is longer than the > CLF. I have some options there for saving space. But, the 2 gig limit was > reached after 8 or 9 million hits, which occurred only as a result of one > night's log rotation and server restart not running. > > I'd thought about two things: > > - logging to a pipe to gzip or a program that does file rotations > - hacking the log module to log directly to a database > - hacking the log module to do automatic log rotations > > I'd prefer to avoid pipes or a database. I see the simplest solutions as > the least error-prone and most recoverable --and a direct file write seems > the simplest. > > My first choice is automatic rotations. Fewer moving parts, and no need to > restart the server. I don't like restarting the server during the day > (approx 90 hits per second), since I think a lot of people will be getting > timeouts (it takes a minute or two to get up to speed again). Rotate your logs more often using SIGUSR1. See http://www.apache.org/docs/stopping.html for details. It doesn't cause the same disruption that a HUP does, but it means you have to be more careful about touching the old logs afterwards for a while since hits will still get written to them for a bit until all the old children exit. 1.2.x shouldn't take that long to get up to speed iff your StartServers is high enough; if not, then it will because it will only start a new process once every second. 1.3 does exponential spawning, so behaves much better under such situations. There is a rotatelogs in the support directory that can run from a pipe, but it looks a bit pokey for high volumes and you still have potential issues with piped logs and 1.2. In 1.3, using piped logs is really a very good solution and is very reliable and flexible. From: Dean Gaudet To: Marc Slemko Cc: apbugs@apache.org, Rob Mela Subject: Re: mod_log-any/2026: Max log file size is 2,147,483,616 (2^31 - 32) bytes Date: Wed, 1 Apr 1998 13:53:08 -0800 (PST) On 1 Apr 1998, Marc Slemko wrote: > 1.2.x shouldn't take that long to get up to speed iff your StartServers is > high enough; if not, then it will because it will only start a new process > once every second. 1.3 does exponential spawning, so behaves much better > under such situations. After a SIGUSR1 1.2 will come back to speed very quickly -- it does a one-for-one respawn for every child that exits until it meets up with the StartServers. It can be really quick. > There is a rotatelogs in the support directory that can run from a pipe, > but it looks a bit pokey for high volumes and you still have potential > issues with piped logs and 1.2. In 1.3, using piped logs is really a very > good solution and is very reliable and flexible. At 90 hits/s I bet it'd run into the pipe-size buffer limitation... I've been meaning to write a socketpair() patch so that we can crank the buffer up to 64k or something. Dean State-Changed-From-To: open-closed State-Changed-By: brian State-Changed-When: Wed May 20 04:31:20 PDT 1998 State-Changed-Why: Apparently the user's problems are fixed. >Unformatted: [In order for any reply to be added to the PR database, ] [you need to include in the Cc line ] [and leave the subject line UNCHANGED. This is not done] [automatically because of the potential for mail loops. ]