[Bug 62907] New: high virtual address space per httpd process

[b***@apache.org]

https://bz.apache.org/bugzilla/show_bug.cgi?id=62907

Bug ID: 62907
Summary: high virtual address space per httpd process
Product: Apache httpd-2
Version: 2.4.37
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: All
Assignee: ***@httpd.apache.org
Reporter: ***@gmail.com
Target Milestone: ---

Hi all

we have observed a huge virtual address space (~20 GB) per httpd process on one
of our productive systems (Red Hat 6.5, amd64).

Trying to reproduce the situation have not yet succeeded, nevertheless we can
observe a quite high address space on a test system with Fedora 29 and standard
httpd (installed from Red Hat repository).

[***@localhost ~]# cat /etc/fedora-release
Fedora release 29 (Twenty Nine)

[***@localhost ~]# uname -a
Linux localhost.localdomain 4.18.17-300.fc29.x86_64 #1 SMP Mon Nov 5 17:56:16
UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[***@localhost ~]# /usr/sbin/httpd -V
Server version: Apache/2.4.37 (Fedora)
Server built: Nov 7 2018 11:56:38
Server's Module Magic Number: 20120211:83
Server loaded: APR 1.6.5, APR-UTIL 1.6.1
Compiled using: APR 1.6.5, APR-UTIL 1.6.1
Architecture: 64-bit
Server MPM: event
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="/run/httpd/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

The high address space seems to be related to the memory pool creation per
thread of glbic. At least it can be shown that the space decreases when the
number of threads are decreased. For example, with standard configuration from
Red Hat following can be observed

[***@localhost ~]# ps -o comm,pid,ppid,vsz,rss -C httpd
COMMAND PID PPID VSZ RSS
httpd 29227 1 38768 13744
httpd 29228 29227 48940 7220
httpd 29229 29227 1106848 9324
httpd 29230 29227 1237980 9324
httpd 29231 29227 1106848 9324

[***@localhost ~]# cat /proc/29228/status | grep Threads
Threads: 1

[***@localhost ~]# cat /proc/29229/status | grep Threads
Threads: 65

Note the difference of VSZ between process 29228 (one thread) and 29229 (65
threads). Reducing the number of threads per process with following settings...

H2MaxWorkers 1
ThreadsPerChild 1
ThreadLimit 1
MaxRequestWorkers 1
ServerLimit 2

...leads to these results

[***@localhost ~]# ps -o comm,pid,ppid,vsz,rss -C httpd
COMMAND PID PPID VSZ RSS
httpd 30693 1 38432 13528
httpd 30694 30693 48940 7216
httpd 30695 30693 417804 8436

[***@localhost ~]# cat /proc/30695/status | grep Threads
Threads: 21

However, limiting the number of memory arenas by exporting MALLOC_ARENA_MAX
(e.g. to 2) does not show any difference.

I know that virtual address space is not likely to run out on a 64-bit machine.
Anyway I'd like to ask for your opinion how the address space can grow up to 20
GB?

[b***@apache.org]

https://bz.apache.org/bugzilla/show_bug.cgi?id=62907

Stefan Eissing <***@eissing.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO

--- Comment #1 from Stefan Eissing <***@eissing.org> ---
Use "ulimit -s" to see what the default stack size per thread is on your
system. On my ubuntu 64bit it is 8192. So every thread, regardless of what else
is allocated for it, will have a 8MB stack.

With 50 threads per process and 3 worker children, that would sum up to 1.2 GB
on virtual stack space alone.

See also:
https://unix.stackexchange.com/questions/127602/default-stack-size-for-pthreads#280865

Does this match your observations?