Received: (qmail 83782 invoked by uid 501); 30 Jul 2001 15:36:23 -0000 Message-Id: <20010730153623.83781.qmail@apache.org> Date: 30 Jul 2001 15:36:23 -0000 From: Tom Grindrod Reply-To: tom@omg.org To: submit@bugz.apache.org Subject: Apache keeping files open to long X-Send-Pr-Version: 3.110 >Number: 8107 >Category: config >Synopsis: Apache keeping files open to long >Confidential: no >Severity: serious >Priority: medium >Responsible: apache >State: closed >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: apache >Arrival-Date: Mon Jul 30 08:40:01 PDT 2001 >Closed-Date: Mon Jul 30 08:51:02 PDT 2001 >Last-Modified: Wed Aug 1 09:00:01 PDT 2001 >Originator: tom@omg.org >Release: 1.3.12 >Organization: >Environment: Redhat linux 2.2.19-7.0.1, gcc compiler, Apache version 1.3.12 Compaq ProLiant ML330 Single Pentium 3 with a gig of ram. >Description: each apache process is opening 150+ files multiplied by 60-80 connections. Most of the file are log files for virtual hosts. Apache will continue to grab file descriptors no matter how high I set file-max in linux and it never releases them, so I am constanly running out of files and have to reboot. Is there a fix for this, it is killing our web site. Is it possible not to log for virtal host?? If this is possible I can remove the logging entries as a temp solution. Still the big problem is Apache needs to release the files when there are not many processes running, else I will continue to have this issue and need to reboot daily which is not acceptable. >How-To-Repeat: Doesn't show a problem on URLS. Errors show up on the system itself. OR the machine slows to a crawl and needs to be booted. >Fix: I have tried increasing max-files and max-inodes etc... , but that just gives me an extra day before Apache eats all the file descriptors up and I have to boot. >Release-Note: >Audit-Trail: State-Changed-From-To: open-closed State-Changed-By: slive State-Changed-When: Mon Jul 30 08:51:02 PDT 2001 State-Changed-Why: Apache should not be "leaking" file descriptors. If this is actually happening, I suggest upgrading to the most recent release of Apache 1.3 and seeing if that fixes your problem. However, I suspect your problem is more along the lines discussed here: http://httpd.apache.org/docs/misc/FAQ.html#fdlim Thanks for using Apache! From: "Tom Grindrod" To: Cc: Subject: RE: config/8107: Apache keeping files open to long Date: Mon, 30 Jul 2001 15:31:57 -0400 What I did was get rid of most of my access logs for the virtual hosts and consolidate my error logs, but the size of the apache process files continues to grow. Below is an example of all the FD's associated w/one apache process on my system. The thing that seems to be growing is the lines with "socket" in them. This appears to be the problem. They just keep growing and none are going away. How can I clean up these?? This is not the OS, I already spent two weeks working with RedHat before discovering problems with the apache process files. A little help PLEASE??? Here is on Apache FD [root@petros fd]# ls -l total 0 lr-x------ 1 root root 64 Jul 30 14:57 0 -> /dev/null l-wx------ 1 root root 64 Jul 30 14:57 1 -> /dev/null lr-x------ 1 root root 64 Jul 30 14:57 10 -> pipe:[46297] lrwx------ 1 root root 64 Jul 30 14:57 11 -> socket:[41882] lrwx------ 1 root root 64 Jul 30 14:57 12 -> socket:[41926] lrwx------ 1 root root 64 Jul 30 14:57 13 -> socket:[41939] lrwx------ 1 root root 64 Jul 30 14:57 14 -> socket:[41956] l-wx------ 1 root root 64 Jul 30 14:57 15 -> /usr/local/apache/ logs/error_log l-wx------ 1 root root 64 Jul 30 14:57 16 -> /usr/local/apache/ logs/secure_error_log l-wx------ 1 root root 64 Jul 30 14:57 17 -> /home/httpd/html/s urvey/logs/survey-error_log l-wx------ 1 root root 64 Jul 30 14:57 18 -> /usr/local/apache/ logs/doc-error_log l-wx------ 1 root root 64 Jul 30 14:57 19 -> /usr/local/apache/ logs/www.corba.org-error_log l-wx------ 1 root root 64 Jul 30 14:57 2 -> /home/apache_logs/e rror_log l-wx------ 1 root root 64 Jul 30 14:57 20 -> /home/apache_logs/ error_log lrwx------ 1 root root 64 Jul 30 14:57 21 -> socket:[27285] lrwx------ 1 root root 64 Jul 30 14:57 22 -> socket:[27286] l-wx------ 1 root root 64 Jul 30 14:57 23 -> /usr/local/apache/ logs/ssl_engine_log l-wx------ 1 root root 64 Jul 30 14:57 24 -> /usr/local/apache/ logs/ssl_mutex.2293 l-wx------ 1 root root 64 Jul 30 14:57 25 -> /usr/local/apache/ logs/ip_log l-wx------ 1 root root 64 Jul 30 14:57 26 -> /home/apache_logs/ access_log l-wx------ 1 root root 64 Jul 30 14:57 27 -> /usr/local/apache/ logs/secure_access_log l-wx------ 1 root root 64 Jul 30 14:57 28 -> /usr/local/apache/ logs/ssl_request_log l-wx------ 1 root root 64 Jul 30 14:57 29 -> /home/httpd/html/s urvey/logs/survey-access_log l-wx------ 1 root root 64 Jul 30 14:57 30 -> /usr/local/apache/ logs/doc-access_log l-wx------ 1 root root 64 Jul 30 14:57 31 -> /usr/local/apache/ logs/www.corba.org-access_log l-wx------ 1 root root 64 Jul 30 14:57 32 -> /home/apache_logs/ access_log l-wx------ 1 root root 64 Jul 30 14:57 33 -> /home/apache_logs/ access_log l-wx------ 1 root root 64 Jul 30 14:57 34 -> /usr/local/apache/ logs/www.corba.org-access_log l-wx------ 1 root root 64 Jul 30 14:57 35 -> /usr/local/apache/ logs/httpd.lock.2294 (deleted) l-wx------ 1 root root 64 Jul 30 14:57 36 -> /usr/local/apache/ logs/ssl_mutex.2293 lrwx------ 1 root root 64 Jul 30 14:57 37 -> socket:[41972] lrwx------ 1 root root 64 Jul 30 14:57 38 -> socket:[41987] lrwx------ 1 root root 64 Jul 30 14:57 39 -> socket:[42491] lrwx------ 1 root root 64 Jul 30 14:57 4 -> socket:[29892] lrwx------ 1 root root 64 Jul 30 14:57 40 -> socket:[58347] lr-x------ 1 root root 64 Jul 30 14:57 41 -> pipe:[54554] l-wx------ 1 root root 64 Jul 30 14:57 42 -> pipe:[54554] lrwx------ 1 root root 64 Jul 30 14:57 43 -> socket:[61241] lrwx------ 1 root root 64 Jul 30 14:57 44 -> socket:[61505] lrwx------ 1 root root 64 Jul 30 14:57 45 -> socket:[61493] lrwx------ 1 root root 64 Jul 30 14:57 5 -> socket:[30471] lrwx------ 1 root root 64 Jul 30 14:57 6 -> socket:[30518] lrwx------ 1 root root 64 Jul 30 14:57 7 -> socket:[30540] lrwx------ 1 root root 64 Jul 30 14:57 8 -> socket:[30565] lrwx------ 1 root root 64 Jul 30 14:57 9 -> socket:[34981] This is a log of the file-max increasing at the same time. Mon Jul 30 14:20:00 EDT 2001 1590 19 4096 Mon Jul 30 14:25:00 EDT 2001 1611 16 4096 Mon Jul 30 14:30:00 EDT 2001 1631 26 4096 Mon Jul 30 14:35:01 EDT 2001 1656 23 4096 Mon Jul 30 14:40:00 EDT 2001 1812 111 4096 Mon Jul 30 14:45:00 EDT 2001 1844 87 4096 Mon Jul 30 14:50:00 EDT 2001 1844 74 4096 Mon Jul 30 14:55:01 EDT 2001 1871 83 4096 Mon Jul 30 15:00:00 EDT 2001 1871 125 4096 Mon Jul 30 15:05:00 EDT 2001 1871 88 4096 [root@petros fd]#-----Original Message----- From: slive@apache.org [mailto:slive@apache.org] Sent: Monday, July 30, 2001 11:51 AM To: apache-bugdb@apache.org; slive@apache.org; tom@omg.org Subject: Re: config/8107: Apache keeping files open to long [In order for any reply to be added to the PR database, you need] [to include in the Cc line and make sure the] [subject line starts with the report component and number, with ] [or without any 'Re:' prefixes (such as "general/1098:" or ] ["Re: general/1098:"). If the subject doesn't match this ] [pattern, your message will be misfiled and ignored. The ] ["apbugs" address is not added to the Cc line of messages from ] [the database automatically because of the potential for mail ] [loops. If you do not include this Cc, your reply may be ig- ] [nored unless you are responding to an explicit request from a ] [developer. Reply only with text; DO NOT SEND ATTACHMENTS! ] Synopsis: Apache keeping files open to long State-Changed-From-To: open-closed State-Changed-By: slive State-Changed-When: Mon Jul 30 08:51:02 PDT 2001 State-Changed-Why: Apache should not be "leaking" file descriptors. If this is actually happening, I suggest upgrading to the most recent release of Apache 1.3 and seeing if that fixes your problem. However, I suspect your problem is more along the lines discussed here: http://httpd.apache.org/docs/misc/FAQ.html#fdlim Thanks for using Apache! From: To: Tom Grindrod Cc: Subject: RE: config/8107: Apache keeping files open to long Date: Mon, 30 Jul 2001 12:17:21 -0700 (PDT) As I mentioned, the next step is to upgrade to the latest version of Apache 1.3. I hate to pass the buck, but I suspect this problem is caused by a third-party module, since we know that Apache doesn't leak descriptors in general. It appears that you are using an SSL module which is one place to look. From: Marc Slemko To: Tom Grindrod Cc: Apache bugs database Subject: RE: config/8107: Apache keeping files open to long Date: Mon, 30 Jul 2001 12:34:28 -0700 (PDT) On 30 Jul 2001 slive@apache.org wrote: > The following reply was made to PR config/8107; it has been noted by GNATS. > > From: > To: Tom Grindrod > Cc: > Subject: RE: config/8107: Apache keeping files open to long > Date: Mon, 30 Jul 2001 12:17:21 -0700 (PDT) > > As I mentioned, the next step is to upgrade to the latest > version of Apache 1.3. > > I hate to pass the buck, but I suspect this problem is > caused by a third-party module, since we know that > Apache doesn't leak descriptors in general. It appears > that you are using an SSL module which is one place > to look. I definitely agree with this... it is very unlikely that it is Apache itself causing this problem, since someone else would be seeing it if it were. You may want to run lsof on some of the httpd processes to get a better idea of exactly what these sockets are, which can help you figure out what is the real cause. From: "Tom Grindrod" To: "'Marc Slemko'" Cc: Subject: RE: config/8107: Apache keeping files open to long Date: Mon, 30 Jul 2001 16:22:20 -0400 I ran lsof and below a partial listing. Anyone know how to interpret this so as to find what is creating the socket??? Looks French to me. httpd 2333 2294 root 10u unix 0xe5e57480 65946 socket httpd 2335 2294 root 9u unix 0xe5adabc0 66478 socket httpd 2336 2294 root 37u unix 0xe6011600 49563 socket httpd 2337 2294 root 5u unix 0xebb0c840 37746 socket httpd 2341 2294 root 11u unix 0xe5bed3c0 78367 socket httpd 2342 2294 root 7u unix 0xe5adbc40 55195 socket -----Original Message----- From: Marc Slemko [mailto:marcs@znep.com] Sent: Monday, July 30, 2001 3:34 PM To: Tom Grindrod Cc: Apache bugs database Subject: RE: config/8107: Apache keeping files open to long On 30 Jul 2001 slive@apache.org wrote: > The following reply was made to PR config/8107; it has been noted by GNATS. > > From: > To: Tom Grindrod > Cc: > Subject: RE: config/8107: Apache keeping files open to long > Date: Mon, 30 Jul 2001 12:17:21 -0700 (PDT) > > As I mentioned, the next step is to upgrade to the latest > version of Apache 1.3. > > I hate to pass the buck, but I suspect this problem is > caused by a third-party module, since we know that > Apache doesn't leak descriptors in general. It appears > that you are using an SSL module which is one place > to look. I definitely agree with this... it is very unlikely that it is Apache itself causing this problem, since someone else would be seeing it if it were. You may want to run lsof on some of the httpd processes to get a better idea of exactly what these sockets are, which can help you figure out what is the real cause. From: "Tom Grindrod" To: "'Marc Slemko'" Cc: Subject: RE: config/8107: Apache keeping files open to long Date: Tue, 31 Jul 2001 10:16:43 -0400 I ran lsof with the -c option ( allows you to choose a process to look at to see what it has open). Then I grepped for one Apache process that had over a hundred files open. Below is the output. The sockets seem to refer to a second machine that I have mounted to it called emerald. The web server petros is mounted to emerald. It seems to have many ports open pointing to emerald via sockets. Not sure what this means, but I feel I am getting closer. Doesn't look like a module problem though. Any thoughts?? Thanks -Tom -----Original Message----- From: Marc Slemko [mailto:marcs@znep.com] Sent: Monday, July 30, 2001 9:57 PM To: Tom Grindrod Cc: apbugs@apache.org Subject: RE: config/8107: Apache keeping files open to long On Mon, 30 Jul 2001, Tom Grindrod wrote: > I ran lsof and below a partial listing. Anyone know how to interpret this so > as to find what is creating the socket??? Looks French to me. grumble. silly linux. you could try installing a fresh versoin of lsof, since it "should" be able to do that itself on many OSes, including Linux I thought... Or see if netstat -p outputs anything useful. Or see if you can match up any of the numbers in /proc/net/tcp with the "e5e57480", etc. in 0xe5e57480, then deciper the addresses and ports from their hex representation.... But I'm afraid I'm not a linux expert... > > > > httpd 2333 2294 root 10u unix 0xe5e57480 65946 socket > httpd 2335 2294 root 9u unix 0xe5adabc0 66478 socket > httpd 2336 2294 root 37u unix 0xe6011600 49563 socket > httpd 2337 2294 root 5u unix 0xebb0c840 37746 socket > httpd 2341 2294 root 11u unix 0xe5bed3c0 78367 socket > httpd 2342 2294 root 7u unix 0xe5adbc40 55195 socket > > -----Original Message----- > From: Marc Slemko [mailto:marcs@znep.com] > Sent: Monday, July 30, 2001 3:34 PM > To: Tom Grindrod > Cc: Apache bugs database > Subject: RE: config/8107: Apache keeping files open to long > > > On 30 Jul 2001 slive@apache.org wrote: > > > The following reply was made to PR config/8107; it has been noted by > GNATS. > > > > From: > > To: Tom Grindrod > > Cc: > > Subject: RE: config/8107: Apache keeping files open to long > > Date: Mon, 30 Jul 2001 12:17:21 -0700 (PDT) > > > > As I mentioned, the next step is to upgrade to the latest > > version of Apache 1.3. > > > > I hate to pass the buck, but I suspect this problem is > > caused by a third-party module, since we know that > > Apache doesn't leak descriptors in general. It appears > > that you are using an SSL module which is one place > > to look. > > I definitely agree with this... it is very unlikely that it is > Apache itself causing this problem, since someone else would be > seeing it if it were. You may want to run lsof on some of the httpd > processes to get a better idea of exactly what these sockets are, > which can help you figure out what is the real cause. > > > From: Marc Slemko To: Tom Grindrod Cc: apbugs@apache.org Subject: RE: config/8107: Apache keeping files open to long Date: Tue, 31 Jul 2001 08:58:53 -0700 (PDT) On Tue, 31 Jul 2001, Tom Grindrod wrote: > I ran lsof with the -c option ( allows you to choose a process to look at to > see what it has open). Then I grepped for one Apache process that had over a > hundred files open. Below is the output. The sockets seem to refer to a > second machine that I have mounted to it called emerald. The web server > petros is mounted to emerald. It seems to have many ports open pointing to > emerald via sockets. Not sure what this means, but I feel I am getting > closer. Doesn't look like a module problem though. Any thoughts?? I don't see any output below... The port numbers being used would possibly tell a lot more. But since Apache has no reason to know about or even think about opening such sockets, there isn't much I can say... could well be some module or some OS library. Look very carefully at every dependency that you have between the two machines to see what could be happening...x > > Thanks > -Tom > > -----Original Message----- > From: Marc Slemko [mailto:marcs@znep.com] > Sent: Monday, July 30, 2001 9:57 PM > To: Tom Grindrod > Cc: apbugs@apache.org > Subject: RE: config/8107: Apache keeping files open to long > > > On Mon, 30 Jul 2001, Tom Grindrod wrote: > > > I ran lsof and below a partial listing. Anyone know how to interpret this > so > > as to find what is creating the socket??? Looks French to me. > > grumble. silly linux. > > you could try installing a fresh versoin of lsof, since it "should" be > able to do that itself on many OSes, including Linux I thought... > > Or see if netstat -p outputs anything useful. > > Or see if you can match up any of the numbers in /proc/net/tcp with the > "e5e57480", etc. in 0xe5e57480, then deciper the addresses and ports from > their hex representation.... > > But I'm afraid I'm not a linux expert... > > > > > > > > > httpd 2333 2294 root 10u unix 0xe5e57480 65946 socket > > httpd 2335 2294 root 9u unix 0xe5adabc0 66478 socket > > httpd 2336 2294 root 37u unix 0xe6011600 49563 socket > > httpd 2337 2294 root 5u unix 0xebb0c840 37746 socket > > httpd 2341 2294 root 11u unix 0xe5bed3c0 78367 socket > > httpd 2342 2294 root 7u unix 0xe5adbc40 55195 socket > > > > -----Original Message----- > > From: Marc Slemko [mailto:marcs@znep.com] > > Sent: Monday, July 30, 2001 3:34 PM > > To: Tom Grindrod > > Cc: Apache bugs database > > Subject: RE: config/8107: Apache keeping files open to long > > > > > > On 30 Jul 2001 slive@apache.org wrote: > > > > > The following reply was made to PR config/8107; it has been noted by > > GNATS. > > > > > > From: > > > To: Tom Grindrod > > > Cc: > > > Subject: RE: config/8107: Apache keeping files open to long > > > Date: Mon, 30 Jul 2001 12:17:21 -0700 (PDT) > > > > > > As I mentioned, the next step is to upgrade to the latest > > > version of Apache 1.3. > > > > > > I hate to pass the buck, but I suspect this problem is > > > caused by a third-party module, since we know that > > > Apache doesn't leak descriptors in general. It appears > > > that you are using an SSL module which is one place > > > to look. > > > > I definitely agree with this... it is very unlikely that it is > > Apache itself causing this problem, since someone else would be > > seeing it if it were. You may want to run lsof on some of the httpd > > processes to get a better idea of exactly what these sockets are, > > which can help you figure out what is the real cause. > > > > > > > > From: "Tom Grindrod" To: "'Marc Slemko'" Cc: Subject: RE: config/8107: Apache keeping files open to long Date: Tue, 31 Jul 2001 12:43:20 -0400 This may help.. netstat -n -a -udp | grep httpd | grep 10971 This command shows the sockets open on my web server for a single process and the calling machine which is a named server. Could this be a virtual host issue? Here is the listing for one Apache process. ( there are a hundred more lines like this that correspond to an open stream. 192.67.184.65:53 is the name server. 192.67.184.5 is the web server, so it is opening ports constantly for the named server it lookes like. Port 53 is the udp domain port which maps to our domain: omg.org. I do have 60 or so virstual hosts set up in Apache and wonder if this is somehow related. I don't agree that this in not an Apache problem. I only have this problem with Apache processes and this is not related to any module. Clearly it is something with apache doing a domain lookup. Still I don't know why they don't go away or how apache determines that. Perhaps there is a way to time out these calls to the name server? Below this output, is the output from isof that I forgot so send in my last mail. I don't mean to harp, but this really looks like apache to me. Please send Ideas/commnents. Thanks for helping out. udp 0 0 192.67.184.5:3255 192.67.184.65:53 ESTABLISHED 10971/httpd udp 0 0 192.67.184.5:2897 192.67.184.65:53 ESTABLISHED 10971/httpd udp 0 0 192.67.184.5:2354 192.67.184.65:53 ESTABLISHED 10971/httpd udp 0 0 192.67.184.5:2078 192.67.184.65:53 ESTABLISHED 10971/httpd udp 0 0 192.67.184.5:3231 192.67.184.65:53 ESTABLISHED ISOF output (forgot to send in last mail) I look at the contents of one of the Apache process directories that is growing. cd /proc/10971/fd and get a listing of what is open. i.e : lrwx------ 1 root root 64 Jul 31 09:33 60 -> socket:[218140] I now run isof -c http | grep 218140 to find the corresponing entry. httpd 10971 root 60u IPv4 218140 UDP petros.omg.org:3255->emerald.omg.org:domain -----Original Message----- From: Marc Slemko [mailto:marcs@znep.com] Sent: Tuesday, July 31, 2001 11:59 AM To: Tom Grindrod Cc: apbugs@apache.org Subject: RE: config/8107: Apache keeping files open to long On Tue, 31 Jul 2001, Tom Grindrod wrote: > I ran lsof with the -c option ( allows you to choose a process to look at to > see what it has open). Then I grepped for one Apache process that had over a > hundred files open. Below is the output. The sockets seem to refer to a > second machine that I have mounted to it called emerald. The web server > petros is mounted to emerald. It seems to have many ports open pointing to > emerald via sockets. Not sure what this means, but I feel I am getting > closer. Doesn't look like a module problem though. Any thoughts?? I don't see any output below... The port numbers being used would possibly tell a lot more. But since Apache has no reason to know about or even think about opening such sockets, there isn't much I can say... could well be some module or some OS library. Look very carefully at every dependency that you have between the two machines to see what could be happening...x > > Thanks > -Tom > > -----Original Message----- > From: Marc Slemko [mailto:marcs@znep.com] > Sent: Monday, July 30, 2001 9:57 PM > To: Tom Grindrod > Cc: apbugs@apache.org > Subject: RE: config/8107: Apache keeping files open to long > > > On Mon, 30 Jul 2001, Tom Grindrod wrote: > > > I ran lsof and below a partial listing. Anyone know how to interpret this > so > > as to find what is creating the socket??? Looks French to me. > > grumble. silly linux. > > you could try installing a fresh versoin of lsof, since it "should" be > able to do that itself on many OSes, including Linux I thought... > > Or see if netstat -p outputs anything useful. > > Or see if you can match up any of the numbers in /proc/net/tcp with the > "e5e57480", etc. in 0xe5e57480, then deciper the addresses and ports from > their hex representation.... > > But I'm afraid I'm not a linux expert... > > > > > > > > > httpd 2333 2294 root 10u unix 0xe5e57480 65946 socket > > httpd 2335 2294 root 9u unix 0xe5adabc0 66478 socket > > httpd 2336 2294 root 37u unix 0xe6011600 49563 socket > > httpd 2337 2294 root 5u unix 0xebb0c840 37746 socket > > httpd 2341 2294 root 11u unix 0xe5bed3c0 78367 socket > > httpd 2342 2294 root 7u unix 0xe5adbc40 55195 socket > > > > -----Original Message----- > > From: Marc Slemko [mailto:marcs@znep.com] > > Sent: Monday, July 30, 2001 3:34 PM > > To: Tom Grindrod > > Cc: Apache bugs database > > Subject: RE: config/8107: Apache keeping files open to long > > > > > > On 30 Jul 2001 slive@apache.org wrote: > > > > > The following reply was made to PR config/8107; it has been noted by > > GNATS. > > > > > > From: > > > To: Tom Grindrod > > > Cc: > > > Subject: RE: config/8107: Apache keeping files open to long > > > Date: Mon, 30 Jul 2001 12:17:21 -0700 (PDT) > > > > > > As I mentioned, the next step is to upgrade to the latest > > > version of Apache 1.3. > > > > > > I hate to pass the buck, but I suspect this problem is > > > caused by a third-party module, since we know that > > > Apache doesn't leak descriptors in general. It appears > > > that you are using an SSL module which is one place > > > to look. > > > > I definitely agree with this... it is very unlikely that it is > > Apache itself causing this problem, since someone else would be > > seeing it if it were. You may want to run lsof on some of the httpd > > processes to get a better idea of exactly what these sockets are, > > which can help you figure out what is the real cause. > > > > > > > > From: Marc Slemko To: Tom Grindrod Cc: apbugs@apache.org Subject: RE: config/8107: Apache keeping files open to long Date: Tue, 31 Jul 2001 10:33:04 -0700 (PDT) On Tue, 31 Jul 2001, Tom Grindrod wrote: > This may help.. > > netstat -n -a -udp | grep httpd | grep 10971 > This command shows the sockets open on my web server for a single process > and the calling machine which is a named server. Could this be a virtual > host issue? Here is the listing for one Apache process. ( there are a > hundred more lines like this that correspond to an open stream. > 192.67.184.65:53 is the name server. 192.67.184.5 is the web server, so it > is opening ports constantly for the named server it lookes like. Port 53 is > the udp domain port which maps to our domain: omg.org. I do have 60 or so > virstual hosts set up in Apache and wonder if this is somehow related. I > don't agree that this in not an Apache problem. I only have this problem > with Apache processes and this is not related to any module. Clearly it is > something with apache doing a domain lookup. Still I don't know why they > don't go away or how apache determines that. Perhaps there is a way to time Apache doesn't determine that. Apache never opens a connection of any sort to a nameserver itself, it only uses standard library calls to resolve names. Apache has no control over what bugs there may or may not be in libraries that it uses. > out these calls to the name server? Below this output, is the output from > isof that I forgot so send in my last mail. I don't mean to harp, but this > really looks like apache to me. Please send Ideas/commnents. Thanks for > helping out. > > udp 0 0 192.67.184.5:3255 192.67.184.65:53 > ESTABLISHED > 10971/httpd > udp 0 0 192.67.184.5:2897 192.67.184.65:53 > ESTABLISHED > 10971/httpd > udp 0 0 192.67.184.5:2354 192.67.184.65:53 > ESTABLISHED > 10971/httpd > udp 0 0 192.67.184.5:2078 192.67.184.65:53 > ESTABLISHED > 10971/httpd > udp 0 0 192.67.184.5:3231 192.67.184.65:53 > ESTABLISHED > > > > ISOF output (forgot to send in last mail) > I look at the contents of one of the Apache process directories that is > growing. > cd /proc/10971/fd and get a listing of what is open. > i.e : > lrwx------ 1 root root 64 Jul 31 09:33 60 -> > socket:[218140] > I now run isof -c http | grep 218140 to find the corresponing entry. > httpd 10971 root 60u IPv4 218140 UDP > petros.omg.org:3255->emerald.omg.org:domain > > > > > > -----Original Message----- > From: Marc Slemko [mailto:marcs@znep.com] > Sent: Tuesday, July 31, 2001 11:59 AM > To: Tom Grindrod > Cc: apbugs@apache.org > Subject: RE: config/8107: Apache keeping files open to long > > > On Tue, 31 Jul 2001, Tom Grindrod wrote: > > > I ran lsof with the -c option ( allows you to choose a process to look at > to > > see what it has open). Then I grepped for one Apache process that had over > a > > hundred files open. Below is the output. The sockets seem to refer to a > > second machine that I have mounted to it called emerald. The web server > > petros is mounted to emerald. It seems to have many ports open pointing to > > emerald via sockets. Not sure what this means, but I feel I am getting > > closer. Doesn't look like a module problem though. Any thoughts?? > > I don't see any output below... > > The port numbers being used would possibly tell a lot more. But since > Apache has no reason to know about or even think about opening such > sockets, there isn't much I can say... could well be some module or some > OS library. Look very carefully at every dependency that you have between > the two machines to see what could be happening...x > > > > > Thanks > > -Tom > > > > -----Original Message----- > > From: Marc Slemko [mailto:marcs@znep.com] > > Sent: Monday, July 30, 2001 9:57 PM > > To: Tom Grindrod > > Cc: apbugs@apache.org > > Subject: RE: config/8107: Apache keeping files open to long > > > > > > On Mon, 30 Jul 2001, Tom Grindrod wrote: > > > > > I ran lsof and below a partial listing. Anyone know how to interpret > this > > so > > > as to find what is creating the socket??? Looks French to me. > > > > grumble. silly linux. > > > > you could try installing a fresh versoin of lsof, since it "should" be > > able to do that itself on many OSes, including Linux I thought... > > > > Or see if netstat -p outputs anything useful. > > > > Or see if you can match up any of the numbers in /proc/net/tcp with the > > "e5e57480", etc. in 0xe5e57480, then deciper the addresses and ports from > > their hex representation.... > > > > But I'm afraid I'm not a linux expert... > > > > > > > > > > > > > > httpd 2333 2294 root 10u unix 0xe5e57480 65946 socket > > > httpd 2335 2294 root 9u unix 0xe5adabc0 66478 socket > > > httpd 2336 2294 root 37u unix 0xe6011600 49563 socket > > > httpd 2337 2294 root 5u unix 0xebb0c840 37746 socket > > > httpd 2341 2294 root 11u unix 0xe5bed3c0 78367 socket > > > httpd 2342 2294 root 7u unix 0xe5adbc40 55195 socket > > > > > > -----Original Message----- > > > From: Marc Slemko [mailto:marcs@znep.com] > > > Sent: Monday, July 30, 2001 3:34 PM > > > To: Tom Grindrod > > > Cc: Apache bugs database > > > Subject: RE: config/8107: Apache keeping files open to long > > > > > > > > > On 30 Jul 2001 slive@apache.org wrote: > > > > > > > The following reply was made to PR config/8107; it has been noted by > > > GNATS. > > > > > > > > From: > > > > To: Tom Grindrod > > > > Cc: > > > > Subject: RE: config/8107: Apache keeping files open to long > > > > Date: Mon, 30 Jul 2001 12:17:21 -0700 (PDT) > > > > > > > > As I mentioned, the next step is to upgrade to the latest > > > > version of Apache 1.3. > > > > > > > > I hate to pass the buck, but I suspect this problem is > > > > caused by a third-party module, since we know that > > > > Apache doesn't leak descriptors in general. It appears > > > > that you are using an SSL module which is one place > > > > to look. > > > > > > I definitely agree with this... it is very unlikely that it is > > > Apache itself causing this problem, since someone else would be > > > seeing it if it were. You may want to run lsof on some of the httpd > > > processes to get a better idea of exactly what these sockets are, > > > which can help you figure out what is the real cause. > > > > > > > > > > > > > > > From: "Tom Grindrod" To: "'Marc Slemko'" Cc: Subject: RE: config/8107: Apache keeping files open to long Date: Wed, 1 Aug 2001 09:52:46 -0400 So, Are you saying there are bugs in the Apache libraries or in the OS libraries?? Apache has to request a lookup if I have a virtual host xyz.com in my httpd.conf file. How does it do that? I would assume via calls to the OS or the name server. How else would it get the IP address for the virtual host?? -----Original Message----- From: Marc Slemko [mailto:marcs@znep.com] Sent: Tuesday, July 31, 2001 1:33 PM To: Tom Grindrod Cc: apbugs@apache.org Subject: RE: config/8107: Apache keeping files open to long On Tue, 31 Jul 2001, Tom Grindrod wrote: > This may help.. > > netstat -n -a -udp | grep httpd | grep 10971 > This command shows the sockets open on my web server for a single process > and the calling machine which is a named server. Could this be a virtual > host issue? Here is the listing for one Apache process. ( there are a > hundred more lines like this that correspond to an open stream. > 192.67.184.65:53 is the name server. 192.67.184.5 is the web server, so it > is opening ports constantly for the named server it lookes like. Port 53 is > the udp domain port which maps to our domain: omg.org. I do have 60 or so > virstual hosts set up in Apache and wonder if this is somehow related. I > don't agree that this in not an Apache problem. I only have this problem > with Apache processes and this is not related to any module. Clearly it is > something with apache doing a domain lookup. Still I don't know why they > don't go away or how apache determines that. Perhaps there is a way to time Apache doesn't determine that. Apache never opens a connection of any sort to a nameserver itself, it only uses standard library calls to resolve names. Apache has no control over what bugs there may or may not be in libraries that it uses. > out these calls to the name server? Below this output, is the output from > isof that I forgot so send in my last mail. I don't mean to harp, but this > really looks like apache to me. Please send Ideas/commnents. Thanks for > helping out. > > udp 0 0 192.67.184.5:3255 192.67.184.65:53 > ESTABLISHED > 10971/httpd > udp 0 0 192.67.184.5:2897 192.67.184.65:53 > ESTABLISHED > 10971/httpd > udp 0 0 192.67.184.5:2354 192.67.184.65:53 > ESTABLISHED > 10971/httpd > udp 0 0 192.67.184.5:2078 192.67.184.65:53 > ESTABLISHED > 10971/httpd > udp 0 0 192.67.184.5:3231 192.67.184.65:53 > ESTABLISHED > > > > ISOF output (forgot to send in last mail) > I look at the contents of one of the Apache process directories that is > growing. > cd /proc/10971/fd and get a listing of what is open. > i.e : > lrwx------ 1 root root 64 Jul 31 09:33 60 -> > socket:[218140] > I now run isof -c http | grep 218140 to find the corresponing entry. > httpd 10971 root 60u IPv4 218140 UDP > petros.omg.org:3255->emerald.omg.org:domain > > > > > > -----Original Message----- > From: Marc Slemko [mailto:marcs@znep.com] > Sent: Tuesday, July 31, 2001 11:59 AM > To: Tom Grindrod > Cc: apbugs@apache.org > Subject: RE: config/8107: Apache keeping files open to long > > > On Tue, 31 Jul 2001, Tom Grindrod wrote: > > > I ran lsof with the -c option ( allows you to choose a process to look at > to > > see what it has open). Then I grepped for one Apache process that had over > a > > hundred files open. Below is the output. The sockets seem to refer to a > > second machine that I have mounted to it called emerald. The web server > > petros is mounted to emerald. It seems to have many ports open pointing to > > emerald via sockets. Not sure what this means, but I feel I am getting > > closer. Doesn't look like a module problem though. Any thoughts?? > > I don't see any output below... > > The port numbers being used would possibly tell a lot more. But since > Apache has no reason to know about or even think about opening such > sockets, there isn't much I can say... could well be some module or some > OS library. Look very carefully at every dependency that you have between > the two machines to see what could be happening...x > > > > > Thanks > > -Tom > > > > -----Original Message----- > > From: Marc Slemko [mailto:marcs@znep.com] > > Sent: Monday, July 30, 2001 9:57 PM > > To: Tom Grindrod > > Cc: apbugs@apache.org > > Subject: RE: config/8107: Apache keeping files open to long > > > > > > On Mon, 30 Jul 2001, Tom Grindrod wrote: > > > > > I ran lsof and below a partial listing. Anyone know how to interpret > this > > so > > > as to find what is creating the socket??? Looks French to me. > > > > grumble. silly linux. > > > > you could try installing a fresh versoin of lsof, since it "should" be > > able to do that itself on many OSes, including Linux I thought... > > > > Or see if netstat -p outputs anything useful. > > > > Or see if you can match up any of the numbers in /proc/net/tcp with the > > "e5e57480", etc. in 0xe5e57480, then deciper the addresses and ports from > > their hex representation.... > > > > But I'm afraid I'm not a linux expert... > > > > > > > > > > > > > > httpd 2333 2294 root 10u unix 0xe5e57480 65946 socket > > > httpd 2335 2294 root 9u unix 0xe5adabc0 66478 socket > > > httpd 2336 2294 root 37u unix 0xe6011600 49563 socket > > > httpd 2337 2294 root 5u unix 0xebb0c840 37746 socket > > > httpd 2341 2294 root 11u unix 0xe5bed3c0 78367 socket > > > httpd 2342 2294 root 7u unix 0xe5adbc40 55195 socket > > > > > > -----Original Message----- > > > From: Marc Slemko [mailto:marcs@znep.com] > > > Sent: Monday, July 30, 2001 3:34 PM > > > To: Tom Grindrod > > > Cc: Apache bugs database > > > Subject: RE: config/8107: Apache keeping files open to long > > > > > > > > > On 30 Jul 2001 slive@apache.org wrote: > > > > > > > The following reply was made to PR config/8107; it has been noted by > > > GNATS. > > > > > > > > From: > > > > To: Tom Grindrod > > > > Cc: > > > > Subject: RE: config/8107: Apache keeping files open to long > > > > Date: Mon, 30 Jul 2001 12:17:21 -0700 (PDT) > > > > > > > > As I mentioned, the next step is to upgrade to the latest > > > > version of Apache 1.3. > > > > > > > > I hate to pass the buck, but I suspect this problem is > > > > caused by a third-party module, since we know that > > > > Apache doesn't leak descriptors in general. It appears > > > > that you are using an SSL module which is one place > > > > to look. > > > > > > I definitely agree with this... it is very unlikely that it is > > > Apache itself causing this problem, since someone else would be > > > seeing it if it were. You may want to run lsof on some of the httpd > > > processes to get a better idea of exactly what these sockets are, > > > which can help you figure out what is the real cause. > > > > > > > > > > > > > > > From: Marc Slemko To: Tom Grindrod Cc: apbugs@apache.org Subject: RE: config/8107: Apache keeping files open to long Date: Wed, 1 Aug 2001 08:51:08 -0700 (PDT) On Wed, 1 Aug 2001, Tom Grindrod wrote: > So, Are you saying there are bugs in the Apache libraries or in the OS > libraries?? Apache has to request a lookup if I have a virtual host xyz.com > in my httpd.conf file. How does it do that? I would assume via calls to the > OS or the name server. How else would it get the IP address for the virtual > host?? The OS libraries. > > -----Original Message----- > From: Marc Slemko [mailto:marcs@znep.com] > Sent: Tuesday, July 31, 2001 1:33 PM > To: Tom Grindrod > Cc: apbugs@apache.org > Subject: RE: config/8107: Apache keeping files open to long > > > On Tue, 31 Jul 2001, Tom Grindrod wrote: > > > This may help.. > > > > netstat -n -a -udp | grep httpd | grep 10971 > > This command shows the sockets open on my web server for a single process > > and the calling machine which is a named server. Could this be a virtual > > host issue? Here is the listing for one Apache process. ( there are a > > hundred more lines like this that correspond to an open stream. > > 192.67.184.65:53 is the name server. 192.67.184.5 is the web server, so it > > is opening ports constantly for the named server it lookes like. Port 53 > is > > the udp domain port which maps to our domain: omg.org. I do have 60 or so > > virstual hosts set up in Apache and wonder if this is somehow related. I > > don't agree that this in not an Apache problem. I only have this problem > > with Apache processes and this is not related to any module. Clearly it is > > something with apache doing a domain lookup. Still I don't know why they > > don't go away or how apache determines that. Perhaps there is a way to > time > > Apache doesn't determine that. Apache never opens a connection of any > sort to a nameserver itself, it only uses standard library calls to resolve > names. Apache has no control over what bugs there may or may not be in > libraries that it uses. > > > out these calls to the name server? Below this output, is the output from > > isof that I forgot so send in my last mail. I don't mean to harp, but this > > really looks like apache to me. Please send Ideas/commnents. Thanks for > > helping out. > > > > udp 0 0 192.67.184.5:3255 192.67.184.65:53 > > ESTABLISHED > > 10971/httpd > > udp 0 0 192.67.184.5:2897 192.67.184.65:53 > > ESTABLISHED > > 10971/httpd > > udp 0 0 192.67.184.5:2354 192.67.184.65:53 > > ESTABLISHED > > 10971/httpd > > udp 0 0 192.67.184.5:2078 192.67.184.65:53 > > ESTABLISHED > > 10971/httpd > > udp 0 0 192.67.184.5:3231 192.67.184.65:53 > > ESTABLISHED > > > > > > > > ISOF output (forgot to send in last mail) > > I look at the contents of one of the Apache process directories that is > > growing. > > cd /proc/10971/fd and get a listing of what is open. > > i.e : > > lrwx------ 1 root root 64 Jul 31 09:33 60 -> > > socket:[218140] > > I now run isof -c http | grep 218140 to find the corresponing entry. > > httpd 10971 root 60u IPv4 218140 UDP > > petros.omg.org:3255->emerald.omg.org:domain > > > > > > > > > > > > -----Original Message----- > > From: Marc Slemko [mailto:marcs@znep.com] > > Sent: Tuesday, July 31, 2001 11:59 AM > > To: Tom Grindrod > > Cc: apbugs@apache.org > > Subject: RE: config/8107: Apache keeping files open to long > > > > > > On Tue, 31 Jul 2001, Tom Grindrod wrote: > > > > > I ran lsof with the -c option ( allows you to choose a process to look > at > > to > > > see what it has open). Then I grepped for one Apache process that had > over > > a > > > hundred files open. Below is the output. The sockets seem to refer to a > > > second machine that I have mounted to it called emerald. The web server > > > petros is mounted to emerald. It seems to have many ports open pointing > to > > > emerald via sockets. Not sure what this means, but I feel I am getting > > > closer. Doesn't look like a module problem though. Any thoughts?? > > > > I don't see any output below... > > > > The port numbers being used would possibly tell a lot more. But since > > Apache has no reason to know about or even think about opening such > > sockets, there isn't much I can say... could well be some module or some > > OS library. Look very carefully at every dependency that you have between > > the two machines to see what could be happening...x > > > > > > > > Thanks > > > -Tom > > > > > > -----Original Message----- > > > From: Marc Slemko [mailto:marcs@znep.com] > > > Sent: Monday, July 30, 2001 9:57 PM > > > To: Tom Grindrod > > > Cc: apbugs@apache.org > > > Subject: RE: config/8107: Apache keeping files open to long > > > > > > > > > On Mon, 30 Jul 2001, Tom Grindrod wrote: > > > > > > > I ran lsof and below a partial listing. Anyone know how to interpret > > this > > > so > > > > as to find what is creating the socket??? Looks French to me. > > > > > > grumble. silly linux. > > > > > > you could try installing a fresh versoin of lsof, since it "should" be > > > able to do that itself on many OSes, including Linux I thought... > > > > > > Or see if netstat -p outputs anything useful. > > > > > > Or see if you can match up any of the numbers in /proc/net/tcp with the > > > "e5e57480", etc. in 0xe5e57480, then deciper the addresses and ports > from > > > their hex representation.... > > > > > > But I'm afraid I'm not a linux expert... > > > > > > > > > > > > > > > > > > > httpd 2333 2294 root 10u unix 0xe5e57480 65946 socket > > > > httpd 2335 2294 root 9u unix 0xe5adabc0 66478 socket > > > > httpd 2336 2294 root 37u unix 0xe6011600 49563 socket > > > > httpd 2337 2294 root 5u unix 0xebb0c840 37746 socket > > > > httpd 2341 2294 root 11u unix 0xe5bed3c0 78367 socket > > > > httpd 2342 2294 root 7u unix 0xe5adbc40 55195 socket > > > > > > > > -----Original Message----- > > > > From: Marc Slemko [mailto:marcs@znep.com] > > > > Sent: Monday, July 30, 2001 3:34 PM > > > > To: Tom Grindrod > > > > Cc: Apache bugs database > > > > Subject: RE: config/8107: Apache keeping files open to long > > > > > > > > > > > > On 30 Jul 2001 slive@apache.org wrote: > > > > > > > > > The following reply was made to PR config/8107; it has been noted by > > > > GNATS. > > > > > > > > > > From: > > > > > To: Tom Grindrod > > > > > Cc: > > > > > Subject: RE: config/8107: Apache keeping files open to long > > > > > Date: Mon, 30 Jul 2001 12:17:21 -0700 (PDT) > > > > > > > > > > As I mentioned, the next step is to upgrade to the latest > > > > > version of Apache 1.3. > > > > > > > > > > I hate to pass the buck, but I suspect this problem is > > > > > caused by a third-party module, since we know that > > > > > Apache doesn't leak descriptors in general. It appears > > > > > that you are using an SSL module which is one place > > > > > to look. > > > > > > > > I definitely agree with this... it is very unlikely that it is > > > > Apache itself causing this problem, since someone else would be > > > > seeing it if it were. You may want to run lsof on some of the httpd > > > > processes to get a better idea of exactly what these sockets are, > > > > which can help you figure out what is the real cause. > > > > > > > > > > > > > > > > > > > > > > > > >Unformatted: [In order for any reply to be added to the PR database, you need] [to include in the Cc line and make sure the] [subject line starts with the report component and number, with ] [or without any 'Re:' prefixes (such as "general/1098:" or ] ["Re: general/1098:"). If the subject doesn't match this ] [pattern, your message will be misfiled and ignored. The ] ["apbugs" address is not added to the Cc line of messages from ] [the database automatically because of the potential for mail ] [loops. If you do not include this Cc, your reply may be ig- ] [nored unless you are responding to an explicit request from a ] [developer. Reply only with text; DO NOT SEND ATTACHMENTS! ]