Сервер убивает процесс mongod, где утечка или как сократить использование памяти?

Рейтинг: 0Ответов: 0Опубликовано: 22.02.2023

Устал уже с mongodb, регулярно крашится, че только не делаю, дошло до того что каждый день ночью кроном перезапускаю процесс, но и этого теперь похоже мало.... Мысли посещают о переезде на какую-нибудь другую DB, подальше от этого головняка. Вот что ей надо?

Сервер отведен специально под mongodb, SSD, 16GB оперативки, с учетом данных хранимых и прочего добра, для DB должно хватать, к тому же ранее все работало и на 4GB, но по мере релизов новых версий mongodb, начал увеличивать до 8, теперь до 16, ну в общем все идет к тому что нужно будет создавать собственный ДЦ под мелкий сайт чтобы mongod работал как надо и то не уверен в том что и это поможет, сожрет все что не дай. Накипело....

Feb 22 13:14:21 MongoDB kernel: [3162488.269559] Out of memory: Kill process 9652 (mongod) score 983 or sacrifice child
Feb 22 13:14:21 MongoDB kernel: [3162488.271773] Killed process 9652 (mongod) total-vm:50837620kB, anon-rss:15290160kB, file-rss:0kB, shmem-rss:0kB
Feb 22 13:14:23 MongoDB kernel: [3162490.417750] oom_reaper: reaped process 9652 (mongod), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Feb 22 13:14:24 MongoDB systemd[1]: mongod.service: Main process exited, code=killed, status=9/KILL
Feb 22 13:14:24 MongoDB systemd[1]: mongod.service: Failed with result 'signal'.
Feb 22 13:14:40 MongoDB crontab[9771]: (root) END EDIT (root)
Feb 22 13:14:47 MongoDB systemd[1]: Started MongoDB Database Server.
Feb 22 13:15:01 MongoDB CRON[12088]: (root) CMD (/bin/bash -c 'if ! pgrep mongod; then /usr/sbin/service mongod restart; fi')
Feb 22 13:15:01 MongoDB CRON[12087]: (CRON) info (No MTA installed, discarding output)
Feb 22 13:16:02 MongoDB CRON[12148]: (root) CMD (/bin/bash -c 'if ! pgrep mongod; then /usr/sbin/service mongod restart; fi')
Feb 22 13:16:02 MongoDB CRON[12147]: (CRON) info (No MTA installed, discarding output)
Feb 22 13:17:02 MongoDB CRON[12186]: (root) CMD (/bin/bash -c 'if ! pgrep mongod; then /usr/sbin/service mongod restart; fi')
Feb 22 13:17:02 MongoDB CRON[12187]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb 22 13:17:02 MongoDB CRON[12185]: (CRON) info (No MTA installed, discarding output)
Feb 22 13:18:02 MongoDB CRON[12219]: (root) CMD (/bin/bash -c 'if ! pgrep mongod; then /usr/sbin/service mongod restart; fi')
Feb 22 13:18:02 MongoDB CRON[12218]: (CRON) info (No MTA installed, discarding output)
Feb 22 13:18:40 MongoDB kernel: [3162747.048622] conn85 invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
Feb 22 13:18:41 MongoDB kernel: [3162747.051837] conn85 cpuset=/ mems_allowed=0
Feb 22 13:18:41 MongoDB kernel: [3162747.053057] CPU: 3 PID: 12027 Comm: conn85 Not tainted 4.19.0-22-amd64 #1 Debian 4.19.260-1
Feb 22 13:18:41 MongoDB kernel: [3162747.054841] Hardware name: Hetzner vServer, BIOS 20171111 11/11/2017
Feb 22 13:18:41 MongoDB kernel: [3162747.056324] Call Trace:
Feb 22 13:18:41 MongoDB kernel: [3162747.057378]  dump_stack+0x66/0x81
Feb 22 13:18:41 MongoDB kernel: [3162747.058382]  dump_header+0x6b/0x283
Feb 22 13:18:41 MongoDB kernel: [3162747.059337]  ? ___ratelimit+0xa1/0x100
Feb 22 13:18:41 MongoDB kernel: [3162747.060422]  oom_kill_process.cold.30+0xb/0x1cf
Feb 22 13:18:41 MongoDB kernel: [3162747.061515]  ? oom_badness+0x23/0x140
Feb 22 13:18:41 MongoDB kernel: [3162747.062672]  out_of_memory+0x1a5/0x450
Feb 22 13:18:41 MongoDB kernel: [3162747.063605]  __alloc_pages_slowpath+0xbd8/0xcb0
Feb 22 13:18:41 MongoDB kernel: [3162747.064698]  __alloc_pages_nodemask+0x28b/0x2b0
Feb 22 13:18:41 MongoDB kernel: [3162747.065793]  filemap_fault+0x333/0x780
Feb 22 13:18:41 MongoDB kernel: [3162747.066891]  ? alloc_set_pte+0xf7/0x550
Feb 22 13:18:41 MongoDB kernel: [3162747.068196]  ? filemap_map_pages+0x1ed/0x3a0
Feb 22 13:18:41 MongoDB kernel: [3162747.069534]  ext4_filemap_fault+0x2c/0x40 [ext4]
Feb 22 13:18:41 MongoDB kernel: [3162747.070839]  __do_fault+0x36/0x170
Feb 22 13:18:41 MongoDB kernel: [3162747.071955]  __handle_mm_fault+0xdb6/0x11b0
Feb 22 13:18:41 MongoDB kernel: [3162747.073021]  handle_mm_fault+0xd6/0x200
Feb 22 13:18:41 MongoDB kernel: [3162747.074015]  __do_page_fault+0x249/0x4f0
Feb 22 13:18:41 MongoDB kernel: [3162747.074930]  ? __x64_sys_poll+0x9c/0x140
Feb 22 13:18:41 MongoDB kernel: [3162747.075823]  ? async_page_fault+0x8/0x30
Feb 22 13:18:41 MongoDB kernel: [3162747.076675]  async_page_fault+0x1e/0x30
Feb 22 13:18:41 MongoDB kernel: [3162747.077523] RIP: 0033:0x56294739e040
Feb 22 13:18:41 MongoDB kernel: [3162747.078618] Code: Bad RIP value.
Feb 22 13:18:41 MongoDB kernel: [3162747.079402] RSP: 002b:00007f44ea78e1c8 EFLAGS: 00010283
Feb 22 13:18:41 MongoDB kernel: [3162747.080363] RAX: 000056294739e040 RBX: 00007f44ea78e2c0 RCX: 000000000000001f
Feb 22 13:18:41 MongoDB kernel: [3162747.081773] RDX: 00005629473ac7d0 RSI: 00007f44ea78e170 RDI: 00005629518e0910
Feb 22 13:18:41 MongoDB kernel: [3162747.082978] RBP: 00007f44ea78e240 R08: 0000000000000010 R09: 0000563315462280
Feb 22 13:18:41 MongoDB kernel: [3162747.084202] R10: 0000000000000002 R11: 000056297ecb1020 R12: 0000000000000000
Feb 22 13:18:41 MongoDB kernel: [3162747.085604] R13: 00007f44ea78e1f0 R14: 0000563499899910 R15: 00007f44ea78e1e0
Feb 22 13:18:41 MongoDB kernel: [3162747.086918] Mem-Info:
Feb 22 13:18:41 MongoDB kernel: [3162747.088379] active_anon:3575213 inactive_anon:326690 isolated_anon:2272
Feb 22 13:18:41 MongoDB kernel: [3162747.088379]  active_file:453 inactive_file:383 isolated_file:50
Feb 22 13:18:41 MongoDB kernel: [3162747.088379]  unevictable:87 dirty:2 writeback:70 unstable:0
Feb 22 13:18:41 MongoDB kernel: [3162747.088379]  slab_reclaimable:13163 slab_unreclaimable:12172
Feb 22 13:18:41 MongoDB kernel: [3162747.088379]  mapped:919 shmem:118 pagetables:24318 bounce:0
Feb 22 13:18:41 MongoDB kernel: [3162747.088379]  free:34444 free_pcp:2010 free_cma:0
Feb 22 13:18:41 MongoDB kernel: [3162747.100889] Node 0 active_anon:14300852kB inactive_anon:1306560kB active_file:1812kB inactive_file:1728kB unevictable:348kB isolated(anon):9088kB isolated(file):200kB mapped:3676kB dirty:8kB writeback:280kB shmem:472kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 180224kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Feb 22 13:18:41 MongoDB kernel: [3162747.115977] Node 0 DMA free:15908kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Feb 22 13:18:41 MongoDB kernel: [3162747.126361] lowmem_reserve[]: 0 1938 15615 15615 15615
Feb 22 13:18:41 MongoDB kernel: [3162747.128318] Node 0 DMA32 free:62784kB min:8380kB low:10472kB high:12564kB active_anon:979796kB inactive_anon:939584kB active_file:1172kB inactive_file:1260kB unevictable:20kB writepending:136kB present:2080624kB managed:2015056kB mlocked:20kB kernel_stack:304kB pagetables:9548kB bounce:0kB free_pcp:3212kB local_pcp:984kB free_cma:0kB
Feb 22 13:18:41 MongoDB kernel: [3162747.135887] lowmem_reserve[]: 0 0 13676 13676 13676
Feb 22 13:18:41 MongoDB kernel: [3162747.138010] Node 0 Normal free:58920kB min:59132kB low:73912kB high:88692kB active_anon:13320968kB inactive_anon:366912kB active_file:1076kB inactive_file:1944kB unevictable:328kB writepending:0kB present:14286848kB managed:14009324kB mlocked:328kB kernel_stack:5792kB pagetables:87724kB bounce:0kB free_pcp:2688kB local_pcp:948kB free_cma:0kB
Feb 22 13:18:41 MongoDB kernel: [3162747.151623] lowmem_reserve[]: 0 0 0 0 0
Feb 22 13:18:41 MongoDB kernel: [3162747.153398] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
Feb 22 13:18:41 MongoDB kernel: [3162747.156689] Node 0 DMA32: 44*4kB (UME) 62*8kB (UE) 38*16kB (UEH) 624*32kB (UMH) 54*64kB (UMEH) 64*128kB (UMEH) 32*256kB (UMEH) 20*512kB (UMEH) 11*1024kB (UME) 0*2048kB 0*4096kB = 62592kB
Feb 22 13:18:41 MongoDB kernel: [3162747.159879] Node 0 Normal: 184*4kB (UME) 41*8kB (UME) 11*16kB (U) 6*32kB (UM) 892*64kB (UM) 2*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 58776kB
Feb 22 13:18:41 MongoDB kernel: [3162747.162898] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Feb 22 13:18:41 MongoDB kernel: [3162747.165350] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 22 13:18:41 MongoDB kernel: [3162747.167272] 53800 total pagecache pages
Feb 22 13:18:41 MongoDB kernel: [3162747.168714] 52551 pages in swap cache
Feb 22 13:18:41 MongoDB kernel: [3162747.169989] Swap cache stats: add 1629195157, delete 1631014593, find 652052405/1085309763
Feb 22 13:18:41 MongoDB kernel: [3162747.172422] Free swap  = 0kB
Feb 22 13:18:41 MongoDB kernel: [3162747.175210] Total swap = 33554428kB
Feb 22 13:18:41 MongoDB kernel: [3162747.176469] 4095866 pages RAM
Feb 22 13:18:41 MongoDB kernel: [3162747.177583] 0 pages HighMem/MovableOnly
Feb 22 13:18:41 MongoDB kernel: [3162747.178791] 85794 pages reserved
Feb 22 13:18:41 MongoDB kernel: [3162747.180067] 0 pages hwpoisoned
Feb 22 13:18:41 MongoDB kernel: [3162747.182076] Tasks state (memory values in pages):
Feb 22 13:18:41 MongoDB kernel: [3162747.185780] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Feb 22 13:18:41 MongoDB kernel: [3162747.190475] [    341]     0   341    12030       98   126976     1303             0 systemd-journal
Feb 22 13:18:41 MongoDB kernel: [3162747.194521] [    353]     0   353     5490       14    69632      230         -1000 systemd-udevd
Feb 22 13:18:41 MongoDB kernel: [3162747.197443] [    439]   101   439    23274        2    81920      230             0 systemd-timesyn
Feb 22 13:18:41 MongoDB kernel: [3162747.199770] [    512]     0   512     2374        8    53248      317             0 dhclient
Feb 22 13:18:41 MongoDB kernel: [3162747.201852] [    703]     0   703     1607       22    53248       38             0 qemu-ga
Feb 22 13:18:41 MongoDB kernel: [3162747.203979] [    704]     0   704    55943       28    77824      937             0 rsyslogd
Feb 22 13:18:41 MongoDB kernel: [3162747.205938] [    706]     0   706     4884       40    86016      215             0 systemd-logind
Feb 22 13:18:41 MongoDB kernel: [3162747.208128] [    707]   104   707     2336      132    61440      103          -900 dbus-daemon
Feb 22 13:18:41 MongoDB kernel: [3162747.209895] [    720]     0   720     2134       19    53248       50             0 cron
Feb 22 13:18:41 MongoDB kernel: [3162747.211478] [    725]     0   725     1371        6    45056       39             0 atd
Feb 22 13:18:41 MongoDB kernel: [3162747.213936] [    732]     0   732     1336        0    53248       37             0 agetty
Feb 22 13:18:41 MongoDB kernel: [3162747.216316] [    735]     0   735      661        0    45056       31             0 agetty
Feb 22 13:18:41 MongoDB kernel: [3162747.218847] [    736]     0   736     3453       20    69632      196         -1000 sshd
Feb 22 13:18:41 MongoDB kernel: [3162747.220959] [  24701]     0 24701     5605      126    86016      417             0 proftpd
Feb 22 13:18:41 MongoDB kernel: [3162747.222834] [  30206]     0 30206     3715        1    73728      283             0 sshd
Feb 22 13:18:41 MongoDB kernel: [3162747.224653] [  30213]     0 30213     5260        0    81920      322             0 systemd
Feb 22 13:18:41 MongoDB kernel: [3162747.226485] [  30214]     0 30214    42897        1    90112      670             0 (sd-pam)
Feb 22 13:18:41 MongoDB kernel: [3162747.228232] [  30229]     0 30229     1929        1    53248      320             0 bash
Feb 22 13:18:41 MongoDB kernel: [3162747.229746] [  11898]   106 11898 12664260  3846990 98906112  8345287             0 mongod
Feb 22 13:18:41 MongoDB kernel: [3162747.231474] Out of memory: Kill process 11898 (mongod) score 985 or sacrifice child
Feb 22 13:18:41 MongoDB kernel: [3162747.233585] Killed process 11898 (mongod) total-vm:50657040kB, anon-rss:15387964kB, file-rss:0kB, shmem-rss:0kB
Feb 22 13:18:42 MongoDB kernel: [3162748.963857] oom_reaper: reaped process 11898 (mongod), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Feb 22 13:18:42 MongoDB systemd[1]: mongod.service: Main process exited, code=killed, status=9/KILL
Feb 22 13:18:42 MongoDB systemd[1]: mongod.service: Failed with result 'signal'.
Feb 22 13:19:01 MongoDB CRON[12267]: (root) CMD (/bin/bash -c 'if ! pgrep mongod; then /usr/sbin/service mongod restart; fi')
Feb 22 13:19:01 MongoDB systemd[1]: Started MongoDB Database Server.

enter image description here

free -m

              total        used        free      shared  buff/cache   available
Mem:          15664       10856        2077          13        2730        4515
Swap:         32767         169       32598

mongostat

insert query update delete getmore command dirty  used flushes vsize   res qrw arw net_in net_out conn                time
     8    15     *0     *0       0    23|0  0.5% 79.9%       0 11.8G 10.3G 0|0 2|0   128k    195k   87 Feb 22 14:30:55.141
     7    10     *0     *0       0    10|0  0.5% 80.0%       0 11.8G 10.3G 0|0 2|0  49.4k    134k   87 Feb 22 14:30:56.138
     5     9     *0     *0       0    16|0  0.5% 80.0%       0 11.8G 10.3G 0|0 2|0  87.9k    152k   87 Feb 22 14:30:57.143
     8    16     *0     *0       0    16|0  0.5% 80.0%       0 11.8G 10.3G 0|0 3|0   127k    136k   87 Feb 22 14:30:58.138
    12    24     *0     *0       0    30|0  0.5% 80.0%       0 11.8G 10.3G 1|0 2|0   203k    244k   87 Feb 22 14:30:59.137
     9    16     *0     *0       0    18|0  0.5% 80.0%       0 11.8G 10.3G 0|0 3|0   108k    228k   87 Feb 22 14:31:00.140
     7     9     *0     *0       0    10|0  0.5% 79.9%       0 11.8G 10.3G 0|0 3|0  68.4k    149k   87 Feb 22 14:31:01.137
     6    12     *0     *0       0    15|0  0.5% 79.9%       0 11.8G 10.3G 0|0 1|0   118k    159k   87 Feb 22 14:31:02.137
     7    22     *0      1       0   142|0  0.5% 80.0%       0 11.8G 10.3G 0|0 8|1   110k    194k   98 Feb 22 14:31:03.134
     9    13     *0     *0       0    12|0  0.5% 80.0%       0 11.8G 10.3G 0|0 5|1  83.6k   96.7k   95 Feb 22 14:31:04.138
insert query update delete getmore command dirty  used flushes vsize   res qrw arw net_in net_out conn                time
     7    20     *0     *0       0    21|0  0.5% 79.9%       0 11.8G 10.3G 0|0 6|1   106k    194k   94 Feb 22 14:31:05.135
     3     8     *0     *0       0    13|0  0.5% 79.9%       0 11.8G 10.3G 0|0 3|0  70.1k    140k   90 Feb 22 14:31:06.139
     5    14     *0     *0       0    14|0  0.5% 80.0%       0 11.8G 10.3G 0|0 5|0  62.2k    122k   90

/etc/mongod.conf

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
#  engine:
  wiredTiger:
    engineConfig:
      cacheSizeGB: 7

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log
  logRotate: reopen

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0
  #bindIp: 127.0.0.1


# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

#security:
#  authorization: 'enabled'

#operationProfiling:

#replication:

#sharding:

## Enterprise-Only Options:

#auditLog:

#snmp:

Индекс, индексы больная тема, но они вроде как есть, выборка проходит быстро и больших проблем не замечал. В базе порядка 7 таблиц.

В чем дело может быть? Может можно как то ограничить использование памяти mongod или это не поможет? С чем может быть связана утечка памяти? Тяжелое запросы aggregate так же использую с allowDiskUse(true)

Ну и в сторону какой DB смотреть для переезда?

mongodb
Источник: Stack Overflow на русском

Ответы

Ответов пока нет.