From 3c8d3258df046a6e3c95e7ad7169867a247ff690 Mon Sep 17 00:00:00 2001 From: Qiumiao Zhang Date: Wed, 20 Mar 2024 19:36:10 +0800 Subject: [PATCH] backport patches from upstream Signed-off-by: Qiumiao Zhang (cherry picked from commit b83e0b1c2ac17ebf5b21196f8a2dea501b196072) --- ...x-potential-segfault-on-busy-systems.patch | 59 ++++ ...rtup-issue-on-modern-systemd-systems.patch | 188 ++++++++++ ...rectory-s-fd-leak-in-case-of-inotify.patch | 46 +++ ...ctorys-fd-leak-in-case-of-inotify-on.patch | 197 +++++++++++ ...gfix-reload-on-HUP-did-not-work-when.patch | 326 ++++++++++++++++++ ...kup-tables-fix-static-analyzer-issue.patch | 57 +++ ...msg-cannot-be-parsed-parser-chain-is.patch | 38 ++ ...epreceated-method-SSLv23_method-with.patch | 110 ++++++ ...ubsystem-handle-data-race-gracefully.patch | 36 ++ ...sending-was-not-implemented-properly.patch | 155 +++++++++ ...tcp-send-error-not-properly-reported.patch | 42 +++ rsyslog.spec | 29 +- 12 files changed, 1282 insertions(+), 1 deletion(-) create mode 100644 backport-core-bugfix-potential-segfault-on-busy-systems.patch create mode 100644 backport-fix-startup-issue-on-modern-systemd-systems.patch create mode 100644 backport-imfile-fix-ext-directory-s-fd-leak-in-case-of-inotify.patch create mode 100644 backport-imfile-tests-ext-directorys-fd-leak-in-case-of-inotify-on.patch create mode 100644 backport-lookup-tables-bugfix-reload-on-HUP-did-not-work-when.patch create mode 100644 backport-lookup-tables-fix-static-analyzer-issue.patch create mode 100644 backport-mmnormalize-bugfix-if-msg-cannot-be-parsed-parser-chain-is.patch create mode 100644 backport-openssl-Replaced-depreceated-method-SSLv23_method-with.patch create mode 100644 backport-tcp-net-subsystem-handle-data-race-gracefully.patch create mode 100644 backport-tcpflood-bugfix-TCP-sending-was-not-implemented-properly.patch create mode 100644 backport-tcpflood-bugfix-plain-tcp-send-error-not-properly-reported.patch diff --git a/backport-core-bugfix-potential-segfault-on-busy-systems.patch b/backport-core-bugfix-potential-segfault-on-busy-systems.patch new file mode 100644 index 0000000..6accf15 --- /dev/null +++ b/backport-core-bugfix-potential-segfault-on-busy-systems.patch @@ -0,0 +1,59 @@ +From 1dfa5f9d836d2153fb76bcbbb235cf5bfdff538b Mon Sep 17 00:00:00 2001 +From: "Konstantin J. Chernov" <> +Date: Mon, 17 Apr 2023 13:22:29 +0200 +Subject: [PATCH] core bugfix: potential segfault on busy systems + +This was discovered by Konstantin J. Chernov in a practicaly deployment. +Here, msg object tag processing caused sporadic segfaults. We did not +hear from similiar cases, but there clearly is potential for problems +because a mutex lock had insufficient range, thus leading to a potential +race. + +The patch is directly from Konstantin J. Chernov, thanks for that. + +Please note that the mutex lock could be minimized as it is not strictly +needed for the pM == NULL case, but this cause is extremely exotic +and the resulting code would be harder to understand. Thus we opt +to do the locking on funtion level (as usual). + +Descriptiond edited by Rainer Gerhards + +closes: https://github.com/rsyslog/rsyslog/issues/5110 + +Reference:https://github.com/rsyslog/rsyslog/commit/1dfa5f9d836d2153fb76bcbbb235cf5bfdff538b +Conflict:NA +--- + runtime/msg.c | 8 +++++++- + 1 file changed, 7 insertions(+), 1 deletion(-) + +diff --git a/runtime/msg.c b/runtime/msg.c +index 03511d3f82..b35bc1dfd2 100644 +--- a/runtime/msg.c ++++ b/runtime/msg.c +@@ -2552,12 +2552,15 @@ tryEmulateTAG(smsg_t *const pM, const sbool bLockMutex) + void ATTR_NONNULL(2,3) + getTAG(smsg_t * const pM, uchar **const ppBuf, int *const piLen, const sbool bLockMutex) + { ++ if(bLockMutex == LOCK_MUTEX) ++ MsgLock(pM); ++ + if(pM == NULL) { + *ppBuf = UCHAR_CONSTANT(""); + *piLen = 0; + } else { + if(pM->iLenTAG == 0) +- tryEmulateTAG(pM, bLockMutex); ++ tryEmulateTAG(pM, MUTEX_ALREADY_LOCKED); + if(pM->iLenTAG == 0) { + *ppBuf = UCHAR_CONSTANT(""); + *piLen = 0; +@@ -2566,6 +2569,9 @@ getTAG(smsg_t * const pM, uchar **const ppBuf, int *const piLen, const sbool bLo + *piLen = pM->iLenTAG; + } + } ++ ++ if(bLockMutex == LOCK_MUTEX) ++ MsgUnlock(pM); + } + + diff --git a/backport-fix-startup-issue-on-modern-systemd-systems.patch b/backport-fix-startup-issue-on-modern-systemd-systems.patch new file mode 100644 index 0000000..f26ffa0 --- /dev/null +++ b/backport-fix-startup-issue-on-modern-systemd-systems.patch @@ -0,0 +1,188 @@ +From 061f186488a9dbe4c3c5d94742b52403c84e0676 Mon Sep 17 00:00:00 2001 +From: Rainer Gerhards +Date: Tue, 26 Sep 2023 14:38:44 +0200 +Subject: [PATCH] fix startup issue on modern systemd systems + +When we startup AND are told to auto-background ourselfs, we must +close all unneeded file descriptors. Not doing this has some +security implications. Traditionally, we do this by iterating +over all possible file descriptor values. This is fairly compatible, +because we need no OS-specific method. However, modern systemd configs +tend to not limit the number of fds, so there are potentially 2^30(*) +fds to close. While this is OKish, it takes some time and makes +systemd think that rsyslog did not properly start up. + +We have now solved this by using the /proc filesystem to obtain our +currently open fds. This works for Linux, as well as Cygwin, NetBSD, +FreeBDS and MacOS. Where not available,and close_range() is available +on the (build) platform, we try to use it. If that fails as well, we +fall back to the traditional method. In our opionion, this fallback +is unproblematic, as on these platforms there is no systemd and in +almost all cases a decent number of fds to close. + +Very special thanks go out to Brennan Kinney, who clearly described +the issue to us on github and also provided ample ways to solve it. +What we did is just implement what we think is the best fit from +rsyslog's PoV. + +(*) Some details below on the number of potentially to close fds. + This is directly from a github posting from Brennan Kinney. +Just to clarify, by default since systemd v240 (2018Q4), that +should be `1024:524288` limit. As in the soft limit is the expected +`1024`. + +The problem is other software shipping misconfiguration in systemd +services that overrides this to something silly like +`LimitNOFILE=infinity`. +- Which will map to the sysctl `fs.nr_open` (_a value systemd + v240 also raises from `2^20` to 2^30`, some distro like Debian are + known to opt-out via patch for the `fs.nr_open` change_). +- With the biggest issue there being that the soft limit was also + set to `infinity` instead of their software requesting to raise + the soft limit to a higher value that the hard limit permits. + `infinity` isn't at all sane though. +- The known source of this misconfiguration is container software such + as Docker and `containerd` (_which would often sync with the + systemd `.service` config from the Docker daemon `dockerd.service`_). + +closes https://github.com/rsyslog/rsyslog/issues/5158 + +Reference:https://github.com/rsyslog/rsyslog/commit/144cc03d90293cc09c4c100dd941226b421709e2 +Conflict:NA +--- + configure.ac | 4 +-- + tools/rsyslogd.c | 68 +++++++++++++++++++++++++++++++++++++++++++----- + 2 files changed, 63 insertions(+), 9 deletions(-) + +diff --git a/configure.ac b/configure.ac +index 6ff0955..49bb316 100644 +--- a/configure.ac ++++ b/configure.ac +@@ -200,7 +200,7 @@ AC_CHECK_HEADERS([malloc.h],[],[],[ + #endif + ] + ]) +-AC_CHECK_HEADERS([fcntl.h locale.h netdb.h netinet/in.h paths.h stddef.h stdlib.h string.h sys/file.h sys/ioctl.h sys/param.h sys/socket.h sys/time.h sys/stat.h unistd.h utmp.h utmpx.h sys/epoll.h sys/prctl.h sys/select.h getopt.h]) ++AC_CHECK_HEADERS([fcntl.h locale.h netdb.h netinet/in.h paths.h stddef.h stdlib.h string.h sys/file.h sys/ioctl.h sys/param.h sys/socket.h sys/time.h sys/stat.h unistd.h utmp.h utmpx.h sys/epoll.h sys/prctl.h sys/select.h getopt.h linux/close_range.h]) + + # Checks for typedefs, structures, and compiler characteristics. + AC_C_CONST +@@ -233,7 +233,7 @@ AC_TYPE_SIGNAL + AC_FUNC_STAT + AC_FUNC_STRERROR_R + AC_FUNC_VPRINTF +-AC_CHECK_FUNCS([flock recvmmsg basename alarm clock_gettime gethostbyname gethostname gettimeofday localtime_r memset mkdir regcomp select setsid socket strcasecmp strchr strdup strerror strndup strnlen strrchr strstr strtol strtoul uname ttyname_r getline malloc_trim prctl epoll_create epoll_create1 fdatasync syscall lseek64 asprintf]) ++AC_CHECK_FUNCS([flock recvmmsg basename alarm clock_gettime gethostbyname gethostname gettimeofday localtime_r memset mkdir regcomp select setsid socket strcasecmp strchr strdup strerror strndup strnlen strrchr strstr strtol strtoul uname ttyname_r getline malloc_trim prctl epoll_create epoll_create1 fdatasync syscall lseek64 asprintf close_range]) + AC_CHECK_FUNC([setns], [AC_DEFINE([HAVE_SETNS], [1], [Define if setns exists.])]) + AC_CHECK_TYPES([off64_t]) + +diff --git a/tools/rsyslogd.c b/tools/rsyslogd.c +index bd19a5b..458b8f6 100644 +--- a/tools/rsyslogd.c ++++ b/tools/rsyslogd.c +@@ -3,7 +3,7 @@ + * because it was either written from scratch by me (rgerhards) or + * contributors who agreed to ASL 2.0. + * +- * Copyright 2004-2022 Rainer Gerhards and Adiscon ++ * Copyright 2004-2023 Rainer Gerhards and Adiscon + * + * This file is part of rsyslog. + * +@@ -27,6 +27,7 @@ + #include + #include + #include ++#include + #include + #include + #ifdef ENABLE_LIBLOGGING_STDLOG +@@ -38,6 +39,9 @@ + # include + #include + #endif ++#if defined(HAVE_LINUX_CLOSE_RANGE_H) ++# include ++#endif + + #include "rsyslog.h" + #include "wti.h" +@@ -358,6 +362,36 @@ finalize_it: + RETiRet; + } + ++ ++ ++/* note: this function is specific to OS'es which provide ++ * the ability to read open file descriptors via /proc. ++ * returns 0 - success, something else otherwise ++ */ ++static int ++close_unneeded_open_files(const char *const procdir, ++ const int beginClose, const int parentPipeFD) ++{ ++ DIR *dir; ++ struct dirent *entry; ++ ++ dir = opendir(procdir); ++ if (dir == NULL) { ++ dbgprintf("closes unneeded files: opendir failed for %s\n", procdir); ++ return 1; ++ } ++ ++ while ((entry = readdir(dir)) != NULL) { ++ const int fd = atoi(entry->d_name); ++ if(fd >= beginClose && (((fd != dbgGetDbglogFd()) && (fd != parentPipeFD)))) { ++ close(fd); ++ } ++ } ++ ++ closedir(dir); ++ return 0; ++} ++ + /* prepares the background processes (if auto-backbrounding) for + * operation. + */ +@@ -403,12 +437,32 @@ prepareBackground(const int parentPipeFD) + } + #endif + +- /* close unnecessary open files */ +- const int endClose = getdtablesize(); +- close(0); +- for(int i = beginClose ; i <= endClose ; ++i) { +- if((i != dbgGetDbglogFd()) && (i != parentPipeFD)) { +- aix_close_it(i); /* AIXPORT */ ++ /* close unnecessary open files - first try to use /proc file system, ++ * if that is not possible iterate through all potentially open file ++ * descriptors. This can be lenghty, but in practice /proc should work ++ * for almost all current systems, and the fallback is primarily for ++ * Solaris and AIX, where we do expect a decent max numbers of fds. ++ */ ++ close(0); /* always close stdin, we do not need it */ ++ ++ /* try Linux, Cygwin, NetBSD */ ++ if(close_unneeded_open_files("/proc/self/fd", beginClose, parentPipeFD) != 0) { ++ /* try MacOS, FreeBSD */ ++ if(close_unneeded_open_files("/proc/fd", beginClose, parentPipeFD) != 0) { ++ /* did not work out, so let's close everything... */ ++ const int endClose = getdtablesize(); ++# if defined(HAVE_CLOSE_RANGE) ++ if(close_range(beginClose, endClose, 0) != 0) { ++ dbgprintf("errno %d after close_range(), fallback to loop\n", errno); ++# endif ++ for(int i = beginClose ; i <= endClose ; ++i) { ++ if((i != dbgGetDbglogFd()) && (i != parentPipeFD)) { ++ aix_close_it(i); /* AIXPORT */ ++ } ++ } ++# if defined(HAVE_CLOSE_RANGE) ++ } ++# endif + } + } + seedRandomNumberForChild(); +-- +2.19.1 + diff --git a/backport-imfile-fix-ext-directory-s-fd-leak-in-case-of-inotify.patch b/backport-imfile-fix-ext-directory-s-fd-leak-in-case-of-inotify.patch new file mode 100644 index 0000000..f6ad261 --- /dev/null +++ b/backport-imfile-fix-ext-directory-s-fd-leak-in-case-of-inotify.patch @@ -0,0 +1,46 @@ +From fcf72d6ac5662e3cee536eddbae1b4fa9f1aff8a Mon Sep 17 00:00:00 2001 +From: Sergey Kacheev +Date: Mon, 5 Dec 2022 12:36:19 +0700 +Subject: [PATCH] imfile: fix ext directory's fd leak in case of inotify on + symlink + +Reference:https://github.com/rsyslog/rsyslog/commit/fcf72d6ac5662e3cee536eddbae1b4fa9f1aff8a +Conflict:NA +--- + plugins/imfile/imfile.c | 15 +++++++++------ + 1 file changed, 9 insertions(+), 6 deletions(-) + +diff --git a/plugins/imfile/imfile.c b/plugins/imfile/imfile.c +index e922261889..5febd6db67 100644 +--- a/plugins/imfile/imfile.c ++++ b/plugins/imfile/imfile.c +@@ -852,10 +852,13 @@ detect_updates(fs_edge_t *const edge) + * the old file in case a process is still writing into it until the FILE_DELETE_DELAY + * is reached OR the inode has changed (see elseif below). In most cases, the + * delay will never be reached and the file will be closed when the inode has changed. ++ * Directories are deleted without delay. + */ +- if (act->time_to_delete + FILE_DELETE_DELAY < ttNow) { +- DBGPRINTF("detect_updates obj gone away, unlinking: '%s', ttDelete: %lds, ttNow:%ld\n", +- act->name, ttNow - (act->time_to_delete + FILE_DELETE_DELAY), ttNow); ++ sbool is_file = act->edge->is_file; ++ if (!is_file || act->time_to_delete + FILE_DELETE_DELAY < ttNow) { ++ DBGPRINTF("detect_updates obj gone away, unlinking: " ++ "'%s', ttDelete: %lds, ttNow:%ld isFile: %d\n", ++ act->name, ttNow - (act->time_to_delete + FILE_DELETE_DELAY), ttNow, is_file); + act_obj_unlink(act); + restart = 1; + } else { +@@ -1038,9 +1041,9 @@ act_obj_destroy(act_obj_t *const act, const int is_deleted) + act_obj_t *target_act; + for(target_act = act->edge->active ; target_act != NULL ; target_act = target_act->next) { + if(target_act->source_name && !strcmp(target_act->source_name, act->name)) { +- DBGPRINTF("act_obj_destroy: unlinking slink target %s of %s " +- "symlink\n", target_act->name, act->name); +- act_obj_unlink(target_act); ++ DBGPRINTF("act_obj_destroy: detect_updates for parent of target %s of %s symlink\n", ++ target_act->name, act->name); ++ detect_updates(target_act->edge->parent->root->edges); + break; + } + } diff --git a/backport-imfile-tests-ext-directorys-fd-leak-in-case-of-inotify-on.patch b/backport-imfile-tests-ext-directorys-fd-leak-in-case-of-inotify-on.patch new file mode 100644 index 0000000..eff1992 --- /dev/null +++ b/backport-imfile-tests-ext-directorys-fd-leak-in-case-of-inotify-on.patch @@ -0,0 +1,197 @@ +From e8ac82e09f930bf99421cc323c24a9dbf215f9da Mon Sep 17 00:00:00 2001 +From: Sergey Kacheev +Date: Wed, 2 Nov 2022 03:45:31 +0700 +Subject: [PATCH] imfile tests: ext directory's fd leak in case of inotify on + symlink + +Reference:https://github.com/rsyslog/rsyslog/commit/e8ac82e09f930bf99421cc323c24a9dbf215f9da +Conflict:NA +--- + tests/Makefile.am | 2 + + tests/README | 12 ++-- + tests/diag.sh | 33 ++++++++++ + tests/imfile-symlink-ext-tmp-dir-tree.sh | 80 ++++++++++++++++++++++++ + 4 files changed, 123 insertions(+), 4 deletions(-) + create mode 100755 tests/imfile-symlink-ext-tmp-dir-tree.sh + +diff --git a/tests/Makefile.am b/tests/Makefile.am +index 396e159..1063079 100644 +--- a/tests/Makefile.am ++++ b/tests/Makefile.am +@@ -1499,6 +1499,7 @@ TESTS += \ + imfile-rename.sh \ + imfile-symlink.sh \ + imfile-symlink-multi.sh \ ++ imfile-symlink-ext-tmp-dir-tree.sh \ + imfile-logrotate.sh \ + imfile-logrotate-async.sh \ + imfile-logrotate-multiple.sh \ +@@ -2479,6 +2480,7 @@ EXTRA_DIST= \ + imfile-rename.sh \ + imfile-symlink.sh \ + imfile-symlink-multi.sh \ ++ imfile-symlink-ext-tmp-dir-tree.sh \ + imfile-logrotate.sh \ + imfile-logrotate-async.sh \ + imfile-logrotate-copytruncate.sh \ +diff --git a/tests/README b/tests/README +index f0d3bb9..63a15d4 100644 +--- a/tests/README ++++ b/tests/README +@@ -41,15 +41,19 @@ make check + + Running named tests + =================== +-make testname.sh.log ++make testname.log + + For example, to run the imfile-basic.sh test, use + +-make imfile-basic.sh.log ++ make imfile-basic.log + +-Test output is in imfile-basic.sh.log ++Test output is in imfile-basic.log + +-To re-run the test, first remove imfile-basic.sh.log then make again ++To re-run the test, first remove imfile-basic.log then make again ++ ++Or an alternative option is to run ++ ++ make check TESTS='imfile-basic.sh' + + * Using gdb to debug rsyslog during a test run + +diff --git a/tests/diag.sh b/tests/diag.sh +index 8bab35b..14966e0 100755 +--- a/tests/diag.sh ++++ b/tests/diag.sh +@@ -889,6 +889,39 @@ check_journal_testmsg_received() { + fi; + } + ++# checks that among the open files found in /proc//fd/* ++# there is or is not, depending on the calling mode, ++# a link with the specified suffix in the target name ++check_fd_for_pid() { ++ local pid="$1" mode="$2" suffix="$3" target seen ++ seen="false" ++ for fd in $(echo /proc/$pid/fd/*); do ++ target="$(readlink -m "$fd")" ++ if [[ "$target" != *$RSYSLOG_DYNNAME* ]]; then ++ continue ++ fi ++ if ((i % 10 == 0)); then ++ echo "INFO: check target='$target'" ++ fi ++ if [[ "$target" == *$suffix ]]; then ++ seen="true" ++ if [[ "$mode" == "exists" ]]; then ++ echo "PASS: check fd for pid=$pid mode='$mode' suffix='$suffix'" ++ return 0 ++ fi ++ fi ++ done ++ if [[ "$seen" == "false" ]] && [[ "$mode" == "absent" ]]; then ++ echo "PASS: check fd for pid=$pid mode='$mode' suffix='$suffix'" ++ return 0 ++ fi ++ echo "FAIL: check fd for pid=$pid mode='$mode' suffix='$suffix'" ++ if [[ "$mode" != "ignore" ]]; then ++ return 1 ++ fi ++ return 0 ++} ++ + # wait for main message queue to be empty. $1 is the instance. + # we run in a loop to ensure rsyslog is *really* finished when a + # function for the "finished predicate" is defined. This is done +diff --git a/tests/imfile-symlink-ext-tmp-dir-tree.sh b/tests/imfile-symlink-ext-tmp-dir-tree.sh +new file mode 100755 +index 0000000..df15f54 +--- /dev/null ++++ b/tests/imfile-symlink-ext-tmp-dir-tree.sh +@@ -0,0 +1,80 @@ ++#!/bin/bash ++# This test creates multiple symlinks (all watched by rsyslog via wildcard) ++# chained to target files via additional symlinks and checks that all files ++# are recorded with correct corresponding metadata (name of symlink ++# matching configuration). ++# This is part of the rsyslog testbench, released under ASL 2.0 ++. "${srcdir:=.}"/diag.sh init ++. "$srcdir"/diag.sh check-inotify ++ ++# #define FILE_DELETE_DELAY 5 /* how many seconds to wait before finally deleting a gone file */ ++export RSYSLOG_DEBUG="debug nologfuncflow noprintmutexaction nostdout" ++export RSYSLOG_DEBUGLOG="log" ++export TEST_TIMEOUT=30 ++ ++# generate input files first. Note that rsyslog processes it as ++# soon as it start up (so the file should exist at that point). ++generate_conf ++add_conf ' ++# comment out if you need more debug info: ++global( debug.whitelist="on" debug.files=["imfile.c"] ++ workDirectory="./'"$RSYSLOG_DYNNAME"'.work" ++) ++module(load="../plugins/imfile/.libs/imfile" mode="inotify") ++input(type="imfile" File="./'"$RSYSLOG_DYNNAME"'.links/*.log" Tag="file:" ++ Severity="error" Facility="local7" addMetadata="on") ++template(name="outfmt" type="list") { ++ constant(value="HEADER ") ++ property(name="msg" format="json") ++ constant(value=", filename: ") ++ property(name="$!metadata!filename") ++ constant(value=", fileoffset: ") ++ property(name="$!metadata!fileoffset") ++ constant(value="\n") ++} ++if $msg contains "msgnum:" then ++ action( type="omfile" file="'"$RSYSLOG_DYNNAME.out/$RSYSLOG_OUT_LOG"'" template="outfmt") ++' ++ ++mkdir "$RSYSLOG_DYNNAME".links "$RSYSLOG_DYNNAME".work "$RSYSLOG_DYNNAME".out ++ ++printf '\ncreating %s\n' "$RSYSLOG_DYNNAME".targets/container-1/logs/0.log ++mkdir -p "$RSYSLOG_DYNNAME".targets/container-1/logs ++./inputfilegen -m 1 >"$RSYSLOG_DYNNAME".targets/container-1/logs/0.log ++ls -l "$RSYSLOG_DYNNAME".targets/container-1/logs/0.log ++ln -sv "$PWD/$RSYSLOG_DYNNAME".targets/container-1/logs/0.log "$PWD/$RSYSLOG_DYNNAME".links/container-1.log ++printf '%s generated link %s\n' "$(tb_timestamp)" "container-1" ++ls -l "$RSYSLOG_DYNNAME".links/container-1.log ++ ++# Start rsyslog now ++startup ++ ++PID=$(cat "$RSYSLOG_PIDBASE".pid) ++echo "Rsyslog pid $RSYSLOG_PIDBASE.pid=$PID" ++if [[ "$PID" == "" ]]; then ++ error_exit 1 ++fi ++ ++echo "INFO: check files" ++# wait until this files has been opened ++check_fd_for_pid "$PID" exists "container-1/logs/0.log" ++check_fd_for_pid "$PID" exists "container-1/logs" ++ ++echo "INFO: remove watched files" ++rm -vr "$RSYSLOG_DYNNAME".targets/container-1 ++rm -v "$RSYSLOG_DYNNAME".links/container-1.log ++ ++until check_fd_for_pid "$PID" absent "container-1/logs (deleted)"; do ++ if ((_wait_for_absent++ > TEST_TIMEOUT)); then ++ error_exit 1 ++ fi ++ echo "INFO: trigger fd unlinking" ++ ./inputfilegen -m 1 >"$RSYSLOG_DYNNAME".links/gogogo.log ++ ./msleep 1000 ++ rm -v "$RSYSLOG_DYNNAME".links/gogogo.log ++ ./msleep 10 ++done ++ ++shutdown_when_empty ++wait_shutdown ++exit_test +-- +2.19.1 + diff --git a/backport-lookup-tables-bugfix-reload-on-HUP-did-not-work-when.patch b/backport-lookup-tables-bugfix-reload-on-HUP-did-not-work-when.patch new file mode 100644 index 0000000..f720c13 --- /dev/null +++ b/backport-lookup-tables-bugfix-reload-on-HUP-did-not-work-when.patch @@ -0,0 +1,326 @@ +From b6b4f25eda025d3f5a5fcbf2be395bfade03d788 Mon Sep 17 00:00:00 2001 +From: Rainer Gerhards +Date: Wed, 2 Aug 2023 10:45:09 +0200 +Subject: [PATCH] lookup tables bugfix: reload on HUP did not work when + backgrounded + +Lookup tables were only reloaded on HUP if the -n option was given +and rsyslog no backgrounded. This patch fixes the issue. + +closes: https://github.com/rsyslog/rsyslog/issues/4813 + +Reference:https://github.com/rsyslog/rsyslog/commit/b6b4f25eda025d3f5a5fcbf2be395bfade03d788 +Conflict:NA +--- + runtime/lookup.c | 68 +++++++++++++++++++------- + runtime/lookup.h | 3 +- + runtime/rsconf.c | 3 +- + tests/Makefile.am | 2 + + tests/diag.sh | 14 ++++++ + tests/lookup_table-hup-backgrounded.sh | 51 +++++++++++++++++++ + tools/rsyslogd.c | 5 +- + 7 files changed, 126 insertions(+), 20 deletions(-) + create mode 100755 tests/lookup_table-hup-backgrounded.sh + +diff --git a/runtime/lookup.c b/runtime/lookup.c +index 66fb09d51a..d9ee53b7ad 100644 +--- a/runtime/lookup.c ++++ b/runtime/lookup.c +@@ -1,7 +1,7 @@ + /* lookup.c + * Support for lookup tables in RainerScript. + * +- * Copyright 2013-2018 Adiscon GmbH. ++ * Copyright 2013-2023 Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * +@@ -75,6 +75,7 @@ lookupTableReloader(void *self); + static void + lookupStopReloader(lookup_ref_t *pThis); + ++ + /* create a new lookup table object AND include it in our list of + * lookup tables. + */ +@@ -83,24 +84,12 @@ lookupNew(lookup_ref_t **ppThis) + { + lookup_ref_t *pThis = NULL; + lookup_t *t = NULL; +- int initialized = 0; + DEFiRet; + + CHKmalloc(pThis = calloc(1, sizeof(lookup_ref_t))); + CHKmalloc(t = calloc(1, sizeof(lookup_t))); +- CHKiConcCtrl(pthread_rwlock_init(&pThis->rwlock, NULL)); +- initialized++; /*1*/ +- CHKiConcCtrl(pthread_mutex_init(&pThis->reloader_mut, NULL)); +- initialized++; /*2*/ +- CHKiConcCtrl(pthread_cond_init(&pThis->run_reloader, NULL)); +- initialized++; /*3*/ +- CHKiConcCtrl(pthread_attr_init(&pThis->reloader_thd_attr)); +- initialized++; /*4*/ + pThis->do_reload = pThis->do_stop = 0; + pThis->reload_on_hup = 1; /*DO reload on HUP (default)*/ +- CHKiConcCtrl(pthread_create(&pThis->reloader, &pThis->reloader_thd_attr, +- lookupTableReloader, pThis)); +- initialized++; /*5*/ + + pThis->next = NULL; + if(loadConf->lu_tabs.root == NULL) { +@@ -115,7 +104,38 @@ lookupNew(lookup_ref_t **ppThis) + *ppThis = pThis; + finalize_it: + if(iRet != RS_RET_OK) { +- LogError(errno, iRet, "a lookup table could not be initialized: " ++ LogError(errno, iRet, "a lookup table could not be initialized"); ++ free(t); ++ free(pThis); ++ } ++ RETiRet; ++} ++ ++/* activate a lookup table entry once rsyslog is ready to do so */ ++static rsRetVal ++lookupActivateTable(lookup_ref_t *pThis) ++{ ++ DEFiRet; ++ int initialized = 0; ++ ++ DBGPRINTF("lookupActivateTable called\n"); ++ CHKiConcCtrl(pthread_rwlock_init(&pThis->rwlock, NULL)); ++ initialized++; /*1*/ ++ CHKiConcCtrl(pthread_mutex_init(&pThis->reloader_mut, NULL)); ++ initialized++; /*2*/ ++ CHKiConcCtrl(pthread_cond_init(&pThis->run_reloader, NULL)); ++ initialized++; /*3*/ ++ CHKiConcCtrl(pthread_attr_init(&pThis->reloader_thd_attr)); ++ initialized++; /*4*/ ++ pThis->do_reload = pThis->do_stop = 0; ++ CHKiConcCtrl(pthread_create(&pThis->reloader, &pThis->reloader_thd_attr, ++ lookupTableReloader, pThis)); ++ initialized++; /*5*/ ++ ++ ++finalize_it: ++ if(iRet != RS_RET_OK) { ++ LogError(errno, iRet, "a lookup table could not be activated: " + "failed at init-step %d (please enable debug logs for details)", + initialized); + /* Can not happen with current code, but might occur in the future when +@@ -128,8 +148,6 @@ lookupNew(lookup_ref_t **ppThis) + if (initialized > 2) pthread_cond_destroy(&pThis->run_reloader); + if (initialized > 1) pthread_mutex_destroy(&pThis->reloader_mut); + if (initialized > 0) pthread_rwlock_destroy(&pThis->rwlock); +- free(t); +- free(pThis); + } + RETiRet; + } +@@ -846,8 +864,8 @@ lookupTableReloader(void *self) + if (pThis->do_stop) { + break; + } else if (pThis->do_reload) { +- pThis->do_reload = 0; + lookupDoReload(pThis); ++ pThis->do_reload = 0; + } else { + pthread_cond_wait(&pThis->run_reloader, &pThis->reloader_mut); + } +@@ -868,6 +886,22 @@ lookupDoHUP(void) + } + } + ++/* activate lookup table system config ++ * most importantly, this means tarting the lookup table reloader thread in the ++ * right process space - it is a difference if we fork or not! ++ */ ++void ++lookupActivateConf(void) ++{ ++ DBGPRINTF("lookup tables: activate config \n"); ++ lookup_ref_t *luref; ++ for(luref = runConf->lu_tabs.root ; luref != NULL ; luref = luref->next) { ++ DBGPRINTF("lookup actiate: processing %p\n", luref); ++ lookupActivateTable(luref); ++ } ++ DBGPRINTF("lookup tables: activate done\n"); ++} ++ + uint + lookupPendingReloadCount(void) + { +diff --git a/runtime/lookup.h b/runtime/lookup.h +index ccfe80d12c..7c0aa28904 100644 +--- a/runtime/lookup.h ++++ b/runtime/lookup.h +@@ -1,6 +1,6 @@ + /* header for lookup.c + * +- * Copyright 2013 Adiscon GmbH. ++ * Copyright 2013-2023 Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * +@@ -111,5 +111,6 @@ void lookupDoHUP(void); + rsRetVal lookupReload(lookup_ref_t *pThis, const uchar *stub_value_if_reload_fails); + uint lookupPendingReloadCount(void); + rsRetVal lookupClassInit(void); ++void lookupActivateConf(void); + + #endif /* #ifndef INCLUDED_LOOKUP_H */ +diff --git a/runtime/rsconf.c b/runtime/rsconf.c +index ae297e3b82..a7fb272c5e 100644 +--- a/runtime/rsconf.c ++++ b/runtime/rsconf.c +@@ -2,7 +2,7 @@ + * + * Module begun 2011-04-19 by Rainer Gerhards + * +- * Copyright 2011-2022 Adiscon GmbH. ++ * Copyright 2011-2023 Adiscon GmbH. + * + * This file is part of the rsyslog runtime library. + * +@@ -1038,6 +1038,7 @@ activate(rsconf_t *cnf) + + CHKiRet(dropPrivileges(cnf)); + ++ lookupActivateConf(); + tellModulesActivateConfig(); + startInputModules(); + CHKiRet(activateActions()); +diff --git a/tests/Makefile.am b/tests/Makefile.am +index 4f0df94328..a9713e0008 100644 +--- a/tests/Makefile.am ++++ b/tests/Makefile.am +@@ -480,6 +480,7 @@ TESTS += \ + incltest_dir_empty_wildcard.sh \ + linkedlistqueue.sh \ + lookup_table.sh \ ++ lookup_table-hup-backgrounded.sh \ + lookup_table_no_hup_reload.sh \ + key_dereference_on_uninitialized_variable_space.sh \ + array_lookup_table.sh \ +@@ -2893,6 +2894,7 @@ EXTRA_DIST= \ + rscript_re_match.sh \ + rscript_re_match-dbl_quotes.sh \ + lookup_table.sh \ ++ lookup_table-hup-backgrounded.sh \ + lookup_table_no_hup_reload.sh \ + lookup_table_no_hup_reload-vg.sh \ + lookup_table_rscript_reload.sh \ +diff --git a/tests/diag.sh b/tests/diag.sh +index c8f58ff0fa..493177ad1c 100755 +--- a/tests/diag.sh ++++ b/tests/diag.sh +@@ -130,6 +130,20 @@ skip_platform() { + + } + ++# function to skip a test if TSAN is enabled ++# This is necessary as TSAN does not properly handle thread creation ++# after fork() - which happens regularly in rsyslog if backgrounding ++# is activated. ++# $1 is the reason why TSAN is not supported ++# note: we depend on CFLAGS to properly reflect build options (what ++# usually is the case when the testbench is run) ++skip_TSAN() { ++ if [[ "$CFLAGS" == *"sanitize=thread"* ]]; then ++ printf 'test incompatible with TSAN because of %s\n' "$1" ++ exit 77 ++ fi ++} ++ + + # a consistent format to output testbench timestamps + tb_timestamp() { +diff --git a/tests/lookup_table-hup-backgrounded.sh b/tests/lookup_table-hup-backgrounded.sh +new file mode 100755 +index 0000000000..bc43c4584f +--- /dev/null ++++ b/tests/lookup_table-hup-backgrounded.sh +@@ -0,0 +1,51 @@ ++#!/bin/bash ++# test for lookup-table and HUP based reloading of it ++# added 2015-09-30 by singh.janmejay ++# This file is part of the rsyslog project, released under ASL 2.0 ++. ${srcdir:=.}/diag.sh init ++skip_TSAN ++generate_conf ++add_conf ' ++lookup_table(name="xlate" file="'$RSYSLOG_DYNNAME'.xlate.lkp_tbl" reloadOnHUP="on") ++ ++template(name="outfmt" type="string" string="- %msg% %$.lkp%\n") ++ ++set $.lkp = lookup("xlate", $msg); ++ ++action(type="omfile" file=`echo $RSYSLOG_OUT_LOG` template="outfmt") ++' ++cp -f $srcdir/testsuites/xlate.lkp_tbl $RSYSLOG_DYNNAME.xlate.lkp_tbl ++export RSTB_DAEMONIZE="YES" ++startup ++injectmsg 0 3 ++wait_queueempty ++content_check "msgnum:00000000: foo_old" ++content_check "msgnum:00000001: bar_old" ++assert_content_missing "baz" ++cp -f $srcdir/testsuites/xlate_more.lkp_tbl $RSYSLOG_DYNNAME.xlate.lkp_tbl ++issue_HUP ++await_lookup_table_reload ++injectmsg 0 3 ++wait_queueempty ++content_check "msgnum:00000000: foo_new" ++content_check "msgnum:00000001: bar_new" ++content_check "msgnum:00000002: baz" ++cp -f $srcdir/testsuites/xlate_more_with_duplicates_and_nomatch.lkp_tbl $RSYSLOG_DYNNAME.xlate.lkp_tbl ++issue_HUP ++await_lookup_table_reload ++injectmsg 0 10 ++echo doing shutdown ++shutdown_when_empty ++echo wait on shutdown ++wait_shutdown ++content_check "msgnum:00000000: foo_latest" ++content_check "msgnum:00000001: quux" ++content_check "msgnum:00000002: baz_latest" ++content_check "msgnum:00000003: foo_latest" ++content_check "msgnum:00000004: foo_latest" ++content_check "msgnum:00000005: baz_latest" ++content_check "msgnum:00000006: foo_latest" ++content_check "msgnum:00000007: baz_latest" ++content_check "msgnum:00000008: baz_latest" ++content_check "msgnum:00000009: quux" ++exit_test +diff --git a/tools/rsyslogd.c b/tools/rsyslogd.c +index 02fa1028ea..08bd5fd895 100644 +--- a/tools/rsyslogd.c ++++ b/tools/rsyslogd.c +@@ -1861,7 +1861,6 @@ parseAndSubmitMessage(const uchar *const hname, const uchar *const hnameIP, cons + */ + DEFFUNC_llExecFunc(doHUPActions) + { +- dbgprintf("doHUP called\n"); + actionCallHUPHdlr((action_t*) pData); + return RS_RET_OK; /* we ignore errors, we can not do anything either way */ + } +@@ -1882,6 +1881,7 @@ doHUP(void) + { + char buf[512]; + ++ DBGPRINTF("doHUP: doing modules\n"); + if(ourConf->globals.bLogStatusMsgs) { + snprintf(buf, sizeof(buf), + "[origin software=\"rsyslogd\" " "swVersion=\"" VERSION +@@ -1893,8 +1893,11 @@ doHUP(void) + + queryLocalHostname(); /* re-read our name */ + ruleset.IterateAllActions(ourConf, doHUPActions, NULL); ++ DBGPRINTF("doHUP: doing modules\n"); + modDoHUP(); ++ DBGPRINTF("doHUP: doing lookup tables\n"); + lookupDoHUP(); ++ DBGPRINTF("doHUP: doing errmsgs\n"); + errmsgDoHUP(); + } + diff --git a/backport-lookup-tables-fix-static-analyzer-issue.patch b/backport-lookup-tables-fix-static-analyzer-issue.patch new file mode 100644 index 0000000..2f3e68d --- /dev/null +++ b/backport-lookup-tables-fix-static-analyzer-issue.patch @@ -0,0 +1,57 @@ +From 8d36cbd29d8db7091e13dd8d720b744c7399acf2 Mon Sep 17 00:00:00 2001 +From: Rainer Gerhards +Date: Wed, 2 Aug 2023 13:20:47 +0200 +Subject: [PATCH] lookup tables: fix static analyzer issue + +If something goes really wrong, a lookup table's name would not +be set. That could lead to a NULL pointer access. HOWEVER, this +would require serious bugs in config parameter parsing, as the +lookup table name is a required parameter and the parser will +error out if not set. + +So the bug is mostly cosmetic - but it does not hurt to handle +this case, of course. + +Reference:https://github.com/rsyslog/rsyslog/commit/8d36cbd29d8db7091e13dd8d720b744c7399acf2 +Conflict:NA +--- + runtime/lookup.c | 15 +++++++++++---- + 1 file changed, 11 insertions(+), 4 deletions(-) + +diff --git a/runtime/lookup.c b/runtime/lookup.c +index d9ee53b7ad..44085818c2 100644 +--- a/runtime/lookup.c ++++ b/runtime/lookup.c +@@ -1037,11 +1037,18 @@ lookupTableDefProcessCnf(struct cnfobj *o) + "param '%s'\n", modpblk.descr[i].name); + } + } ++ const uchar *const lu_name = lu->name; /* we need a const to keep TSAN happy :-( */ ++ const uchar *const lu_filename = lu->filename; /* we need a const to keep TSAN happy :-( */ ++ if(lu_name == NULL || lu_filename == NULL) { ++ iRet = RS_RET_INTERNAL_ERROR; ++ LogError(0, iRet, "internal error: lookup table name not set albeit being mandatory"); ++ ABORT_FINALIZE(iRet); ++ } + #ifdef HAVE_PTHREAD_SETNAME_NP +- thd_name_len = ustrlen(lu->name) + strlen(reloader_prefix) + 1; ++ thd_name_len = ustrlen(lu_name) + strlen(reloader_prefix) + 1; + CHKmalloc(reloader_thd_name = malloc(thd_name_len)); + strcpy(reloader_thd_name, reloader_prefix); +- strcpy(reloader_thd_name + strlen(reloader_prefix), (char*) lu->name); ++ strcpy(reloader_thd_name + strlen(reloader_prefix), (char*) lu_name); + reloader_thd_name[thd_name_len - 1] = '\0'; + #if defined(__NetBSD__) + pthread_setname_np(lu->reloader, "%s", reloader_thd_name); +@@ -1051,9 +1058,9 @@ lookupTableDefProcessCnf(struct cnfobj *o) + pthread_setname_np(lu->reloader, reloader_thd_name); + #endif + #endif +- CHKiRet(lookupReadFile(lu->self, lu->name, lu->filename)); ++ CHKiRet(lookupReadFile(lu->self, lu_name, lu_filename)); + LogMsg(0, RS_RET_OK, LOG_INFO, "lookup table '%s' loaded from file '%s'", +- lu->name, lu->filename); ++ lu_name, lu->filename); + + finalize_it: + #ifdef HAVE_PTHREAD_SETNAME_NP diff --git a/backport-mmnormalize-bugfix-if-msg-cannot-be-parsed-parser-chain-is.patch b/backport-mmnormalize-bugfix-if-msg-cannot-be-parsed-parser-chain-is.patch new file mode 100644 index 0000000..79991f0 --- /dev/null +++ b/backport-mmnormalize-bugfix-if-msg-cannot-be-parsed-parser-chain-is.patch @@ -0,0 +1,38 @@ +From f2bac98f2ada404fb8ed9bc10ff13bfb72366abd Mon Sep 17 00:00:00 2001 +From: Rainer Gerhards +Date: Fri, 9 Jun 2023 14:37:26 +0200 +Subject: [PATCH] mmnormalize bugfix: if msg cannot be parsed, parser chain is + stopped + +When an parser is not able to parse a message, it should indicate this +to rsyslog core, which then activates the next parser(s) inside the +configured parser chain. + +Unfortunatley, mmnormalize always tells core "success", and so no +other parsers are activated. + +closes https://github.com/rsyslog/rsyslog/issues/5148 + +Reference:https://github.com/rsyslog/rsyslog/commit/f2bac98f2ada404fb8ed9bc10ff13bfb72366abd +Conflict:NA +--- + plugins/pmnormalize/pmnormalize.c | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +diff --git a/plugins/pmnormalize/pmnormalize.c b/plugins/pmnormalize/pmnormalize.c +index e1a89841bc..4d3ad76e49 100644 +--- a/plugins/pmnormalize/pmnormalize.c ++++ b/plugins/pmnormalize/pmnormalize.c +@@ -234,10 +234,11 @@ CODESTARTparse2 + "json: %s\n", r, fjson_object_to_json_string(json)); + } + fjson_object_put(json); ++ ABORT_FINALIZE(RS_RET_COULD_NOT_PARSE); + } else { + iRet = MsgSetPropsViaJSON_Object(pMsg, json); + } +- ++finalize_it: + ENDparse2 + + diff --git a/backport-openssl-Replaced-depreceated-method-SSLv23_method-with.patch b/backport-openssl-Replaced-depreceated-method-SSLv23_method-with.patch new file mode 100644 index 0000000..21388d9 --- /dev/null +++ b/backport-openssl-Replaced-depreceated-method-SSLv23_method-with.patch @@ -0,0 +1,110 @@ +From 68216c794e8eb97f0c2d8f791bb58f332943b6ae Mon Sep 17 00:00:00 2001 +From: Andre lorbach +Date: Fri, 28 Jul 2023 14:58:50 +0200 +Subject: [PATCH] openssl: Replaced depreceated method SSLv23_method with + TLS_method + +In OpenSSL 1.1.0 and higher, SSLv23_method causes some errors +in TLS handshake from time to time. As this method is depreceated +since 1.1.0, I have replaced it with the follow up method +TLS_method which is the most generic one. + +It fixes the random test failures in tests like +- sndrcv_tls_ossl_anon_rebind.sh + +Also added some debug output in OpenSSL error handling, which is +useful when analysing debug files. + +closes: ./sndrcv_tls_ossl_anon_rebind.sh + +Reference:https://github.com/rsyslog/rsyslog/commit/8d8fe80d871b07ab14f44e4fddb68445601b66b5 +Conflict:NA +--- + runtime/nsd_ossl.c | 19 +++++++++++++++++-- + runtime/nsdsel_ptcp.c | 3 +++ + tests/tcpflood.c | 6 +++++- + 3 files changed, 25 insertions(+), 3 deletions(-) + +diff --git a/runtime/nsd_ossl.c b/runtime/nsd_ossl.c +index 45b0e03..ba62b7d 100644 +--- a/runtime/nsd_ossl.c ++++ b/runtime/nsd_ossl.c +@@ -192,10 +192,19 @@ void osslLastSSLErrorMsg(int ret, SSL *ssl, int severity, const char* pszCallSou + int iSSLErr = 0; + if (ssl == NULL) { + /* Output Error Info*/ +- dbgprintf("osslLastSSLErrorMsg: Error in '%s' with ret=%d\n", pszCallSource, ret); ++ DBGPRINTF("osslLastSSLErrorMsg: Error in '%s' with ret=%d\n", pszCallSource, ret); + } else { + /* if object is set, get error code */ + iSSLErr = SSL_get_error(ssl, ret); ++ /* Output Debug as well */ ++ DBGPRINTF("osslLastSSLErrorMsg: %s Error in '%s': '%s(%d)' with ret=%d, errno=%d, sslapi='%s'\n", ++ (iSSLErr == SSL_ERROR_SSL ? "SSL_ERROR_SSL" : ++ (iSSLErr == SSL_ERROR_SYSCALL ? "SSL_ERROR_SYSCALL" : "SSL_ERROR_UNKNOWN")), ++ pszCallSource, ERR_error_string(iSSLErr, NULL), ++ iSSLErr, ++ ret, ++ errno, ++ pszOsslApi); + + /* Output error message */ + LogMsg(0, RS_RET_NO_ERRCODE, severity, +@@ -1309,8 +1318,12 @@ osslInit_ctx(nsd_ossl_t *const pThis) + bHaveExtraCAFiles = 1; + } + +- /* Create main CTX Object */ ++ /* Create main CTX Object. Use SSLv23_method for < Openssl 1.1.0 and TLS_method for all newer versions! */ ++#if OPENSSL_VERSION_NUMBER < 0x10100000L + pThis->ctx = SSL_CTX_new(SSLv23_method()); ++#else ++ pThis->ctx = SSL_CTX_new(TLS_method()); ++#endif + if(bHaveExtraCAFiles == 1) { + while((extraCaFile = strsep(&extraCaFiles, ","))) { + if(SSL_CTX_load_verify_locations(pThis->ctx, extraCaFile, NULL) != 1) { +@@ -1575,6 +1588,8 @@ osslHandshakeCheck(nsd_ossl_t *pNsd) + "SSL_do_handshake"); + ABORT_FINALIZE(RS_RET_NO_ERRCODE /*RS_RET_RETRY*/); + } else { ++ dbgprintf("osslHandshakeCheck: OpenSSL Client handshake failed with %d " ++ "- Aborting handshake.\n", resErr); + osslLastSSLErrorMsg(res, pNsd->ssl, LOG_ERR, "osslHandshakeCheck Client", + "SSL_do_handshake"); + LogMsg(0, RS_RET_NO_ERRCODE, LOG_WARNING, +diff --git a/runtime/nsdsel_ptcp.c b/runtime/nsdsel_ptcp.c +index 7a95dfc..2558f09 100644 +--- a/runtime/nsdsel_ptcp.c ++++ b/runtime/nsdsel_ptcp.c +@@ -158,6 +158,9 @@ IsReady(nsdsel_t *const pNsdsel, nsd_t *const pNsd, const nsdsel_waitOp_t waitOp + } + + const short revent = pThis->fds[idx].revents; ++ if (revent & POLLNVAL) { ++ DBGPRINTF("ndssel_ptcp: revent & POLLNVAL is TRUE, something is wrong, revent = %d", revent); ++ } + assert(!(revent & POLLNVAL)); + switch(waitOp) { + case NSDSEL_RD: +diff --git a/tests/tcpflood.c b/tests/tcpflood.c +index f08bdad..0797af8 100644 +--- a/tests/tcpflood.c ++++ b/tests/tcpflood.c +@@ -1195,8 +1195,12 @@ initTLS(void) + ERR_load_BIO_strings(); + ERR_load_crypto_strings(); + +- /* Create main CTX Object */ ++ /* Create main CTX Object. Use SSLv23_method for < Openssl 1.1.0 and TLS_method for all newer versions! */ ++#if OPENSSL_VERSION_NUMBER < 0x10100000L + ctx = SSL_CTX_new(SSLv23_method()); ++#else ++ ctx = SSL_CTX_new(TLS_method()); ++#endif + + if(tlsCAFile != NULL && SSL_CTX_load_verify_locations(ctx, tlsCAFile, NULL) != 1) { + printf("tcpflood: Error, Failed loading CA certificate" +-- +2.33.0 + diff --git a/backport-tcp-net-subsystem-handle-data-race-gracefully.patch b/backport-tcp-net-subsystem-handle-data-race-gracefully.patch new file mode 100644 index 0000000..e448177 --- /dev/null +++ b/backport-tcp-net-subsystem-handle-data-race-gracefully.patch @@ -0,0 +1,36 @@ +From b73ccfb4be883862f1405bd40deca5a111f0c0a2 Mon Sep 17 00:00:00 2001 +From: Rainer Gerhards +Date: Mon, 2 Oct 2023 09:32:07 +0200 +Subject: [PATCH] tcp net subsystem: handle data race gracefully + +It may happen that a socket file descriptor has been closed either +while setting up poll() et al or while being inside the system call. +This was previously treated as error and caused abort in debug +builds. However, it was essentially ignored in production builds. + +This has now been fixed and now is always gracefully ignored. This +most importantly fixes some flakes in CI runs (which were caused +by this situation). + +Reference:https://github.com/rsyslog/rsyslog/commit/b73ccfb4be883862f1405bd40deca5a111f0c0a2 +Conflict:NA +--- + runtime/nsdsel_ptcp.c | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +diff --git a/runtime/nsdsel_ptcp.c b/runtime/nsdsel_ptcp.c +index 2558f09df6..d77c729809 100644 +--- a/runtime/nsdsel_ptcp.c ++++ b/runtime/nsdsel_ptcp.c +@@ -159,9 +159,9 @@ IsReady(nsdsel_t *const pNsdsel, nsd_t *const pNsd, const nsdsel_waitOp_t waitOp + + const short revent = pThis->fds[idx].revents; + if (revent & POLLNVAL) { +- DBGPRINTF("ndssel_ptcp: revent & POLLNVAL is TRUE, something is wrong, revent = %d", revent); ++ DBGPRINTF("ndssel_ptcp: revent & POLLNVAL is TRUE, we had a race, ignoring, revent = %d", revent); ++ *pbIsReady = 0; + } +- assert(!(revent & POLLNVAL)); + switch(waitOp) { + case NSDSEL_RD: + *pbIsReady = revent & POLLIN; diff --git a/backport-tcpflood-bugfix-TCP-sending-was-not-implemented-properly.patch b/backport-tcpflood-bugfix-TCP-sending-was-not-implemented-properly.patch new file mode 100644 index 0000000..5fec84e --- /dev/null +++ b/backport-tcpflood-bugfix-TCP-sending-was-not-implemented-properly.patch @@ -0,0 +1,155 @@ +From 5050249a1ea69ed6ac6b953a7bd722a71b09f9f7 Mon Sep 17 00:00:00 2001 +From: Rainer Gerhards +Date: Thu, 27 Jul 2023 18:15:07 +0200 +Subject: [PATCH] tcpflood bugfix: TCP sending was not implemented properly + +Note: tcpflood is a testbench tool. This bug could lead to testbench +false positives. No way it can affect production deployments. + +The tcpflood tool did improperly assume that a TCP sendto() call +would send messages of any size in a single shot. This is not the +case. It has now been corrected to proper behavior. + +As a side-activity, some int variables which acutally needed to be +size_t have been fixed as well. + +Reference:https://github.com/rsyslog/rsyslog/commit/5050249a1ea69ed6ac6b953a7bd722a71b09f9f7 +Conflict:NA +--- + tests/tcpflood.c | 52 +++++++++++++++++++++++++++++++++++------------- + 1 file changed, 38 insertions(+), 14 deletions(-) + +diff --git a/tests/tcpflood.c b/tests/tcpflood.c +index 7dce5c99a4..ea91b4623f 100644 +--- a/tests/tcpflood.c ++++ b/tests/tcpflood.c +@@ -266,7 +266,7 @@ static enum { TP_UDP, TP_TCP, TP_TLS, TP_RELP_PLAIN, TP_RELP_TLS } transport = T + + /* forward definitions */ + static void initTLSSess(int); +-static int sendTLS(int i, char *buf, int lenBuf); ++static int sendTLS(int i, char *buf, size_t lenBuf); + static void closeTLSSess(int __attribute__((unused)) i); + + #ifdef ENABLE_RELP +@@ -579,7 +579,7 @@ void closeConnections(void) + * of constructing test messages. -- rgerhards, 2010-03-31 + */ + static void +-genMsg(char *buf, size_t maxBuf, int *pLenBuf, struct instdata *inst) ++genMsg(char *buf, size_t maxBuf, size_t *pLenBuf, struct instdata *inst) + { + int edLen; /* actual extra data length to use */ + char extraData[MAX_EXTRADATA_LEN + 1]; +@@ -650,7 +650,7 @@ genMsg(char *buf, size_t maxBuf, int *pLenBuf, struct instdata *inst) + *pLenBuf = snprintf(buf, maxBuf, "%s%c", MsgToSend, frameDelim); + } + if (octateCountFramed == 1) { +- snprintf(payloadLen, sizeof(payloadLen), "%d ", *pLenBuf); ++ snprintf(payloadLen, sizeof(payloadLen), "%zd ", *pLenBuf); + payloadStringLen = strlen(payloadLen); + memmove(buf + payloadStringLen, buf, *pLenBuf); + memcpy(buf, payloadLen, payloadStringLen); +@@ -661,6 +661,29 @@ genMsg(char *buf, size_t maxBuf, int *pLenBuf, struct instdata *inst) + finalize_it: /*EMPTY to keep the compiler happy */; + } + ++ ++static int ++sendPlainTCP(int socknum, char *buf, size_t lenBuf, int *ret_errno) ++{ ++ size_t lenSent; ++ int r, err; ++ ++ lenSent = 0; ++ while(lenSent != lenBuf) { ++ r = send(sockArray[socknum], buf, lenBuf, 0); ++ if(r > 0) { ++ lenSent += r; ++ } else { ++ err = errno; ++ goto finalize_it; ++ } ++ } ++ ++finalize_it: ++ return lenSent; ++} ++ ++ + /* send messages to the tcp connections we keep open. We use + * a very basic format that helps identify the message + * (via msgnum:: e.g. msgnum:00000001:). This format is suitable +@@ -673,8 +696,8 @@ int sendMessages(struct instdata *inst) + { + unsigned i = 0; + int socknum; +- int lenBuf; +- int lenSend = 0; ++ size_t lenBuf; ++ size_t lenSend = 0; + char *statusText = ""; + char buf[MAX_EXTRADATA_LEN + 1024]; + char sendBuf[MAX_SENDBUF]; +@@ -722,8 +745,7 @@ int sendMessages(struct instdata *inst) + exit(1); + } + } +- lenSend = send(sockArray[socknum], buf, lenBuf, 0); +- error_number = errno; ++ lenSend = sendPlainTCP(socknum, buf, lenBuf, &error_number); + } else if(transport == TP_UDP) { + lenSend = sendto(udpsock, buf, lenBuf, 0, &udpRcvr, sizeof(udpRcvr)); + error_number = errno; +@@ -771,8 +793,10 @@ int sendMessages(struct instdata *inst) + printf("\r%5.5u\n", i); + fflush(stdout); + test_rs_strerror_r(error_number, errStr, sizeof(errStr)); +- printf("send() failed \"%s\" at socket %d, index %u, msgNum %lld\n", +- errStr, sockArray[socknum], i, inst->numSent); ++ printf("send() failed \"%s\" at socket %d, index %u, msgNum %lld, " ++ "lenSend %zd, lenBuf %zd\n", ++ errStr, sockArray[socknum], i, inst->numSent, lenSend, ++ lenBuf); + fflush(stderr); + + return(1); +@@ -792,7 +816,7 @@ int sendMessages(struct instdata *inst) + lenSend = sendTLS(socknum, sendBuf, offsSendBuf); + if(lenSend != offsSendBuf) { + fprintf(stderr, "tcpflood: error in send function causes potential " +- "data loss lenSend %d, offsSendBuf %d\n", ++ "data loss lenSend %zd, offsSendBuf %d\n", + lenSend, offsSendBuf); + } + offsSendBuf = 0; +@@ -1375,9 +1399,9 @@ initTLSSess(int i) + + + static int +-sendTLS(int i, char *buf, int lenBuf) ++sendTLS(int i, char *buf, size_t lenBuf) + { +- int lenSent; ++ size_t lenSent; + int r, err; + + lenSent = 0; +@@ -1525,7 +1549,7 @@ initTLSSess(int i) + + + static int +-sendTLS(int i, char *buf, int lenBuf) ++sendTLS(int i, char *buf, size_t lenBuf) + { + int lenSent; + int r; +@@ -1552,7 +1576,7 @@ static void initTLS(void) {} + static void exitTLS(void) {} + static void initTLSSess(int __attribute__((unused)) i) {} + static int sendTLS(int __attribute__((unused)) i, char __attribute__((unused)) *buf, +- int __attribute__((unused)) lenBuf) { return 0; } ++ size_t __attribute__((unused)) lenBuf) { return 0; } + static void closeTLSSess(int __attribute__((unused)) i) {} + # endif + diff --git a/backport-tcpflood-bugfix-plain-tcp-send-error-not-properly-reported.patch b/backport-tcpflood-bugfix-plain-tcp-send-error-not-properly-reported.patch new file mode 100644 index 0000000..05c0ab5 --- /dev/null +++ b/backport-tcpflood-bugfix-plain-tcp-send-error-not-properly-reported.patch @@ -0,0 +1,42 @@ +From c8a2969580935121440f220af03d74ab1dfa5c11 Mon Sep 17 00:00:00 2001 +From: Rainer Gerhards +Date: Wed, 13 Sep 2023 09:33:40 +0200 +Subject: [PATCH] tcpflood bugfix: plain tcp send error not properly reported + +The error code when plain tcp sending failed was improperly returned, +resulting in no meaningful error message. + +Note: tcpflood is a testbench tool, not part of production rsyslog. + +Reference:https://github.com/rsyslog/rsyslog/commit/c8a2969580935121440f220af03d74ab1dfa5c11 +Conflict:NA +--- + tests/tcpflood.c | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +diff --git a/tests/tcpflood.c b/tests/tcpflood.c +index cff4a59dba..12781f55ff 100644 +--- a/tests/tcpflood.c ++++ b/tests/tcpflood.c +@@ -665,10 +665,10 @@ genMsg(char *buf, size_t maxBuf, size_t *pLenBuf, struct instdata *inst) + + + static int +-sendPlainTCP(int socknum, char *buf, size_t lenBuf, int *ret_errno) ++sendPlainTCP(const int socknum, const char *const buf, const size_t lenBuf, int *const ret_errno) + { + size_t lenSent; +- int r, err; ++ int r; + + lenSent = 0; + while(lenSent != lenBuf) { +@@ -676,7 +676,7 @@ sendPlainTCP(int socknum, char *buf, size_t lenBuf, int *ret_errno) + if(r > 0) { + lenSent += r; + } else { +- err = errno; ++ *ret_errno = errno; + goto finalize_it; + } + } diff --git a/rsyslog.spec b/rsyslog.spec index c7b15be..9f2d6bc 100644 --- a/rsyslog.spec +++ b/rsyslog.spec @@ -7,7 +7,7 @@ Name: rsyslog Version: 8.2210.0 -Release: 7 +Release: 8 Summary: The rocket-fast system for log processing License: (GPLv3+ and ASL 2.0) URL: http://www.rsyslog.com/ @@ -38,6 +38,17 @@ Patch6005: backport-bugfix-prevent-pot.-segfault-when-switchung.patch Patch6006: backport-core-bugfix-using-uuid-msg-prop-can-deadloc.patch Patch6007: backport-GNUTls-Driver-Fix-memory-leaks-in-gtlsInitC.patch Patch6008: backport-outchannel-eleminate-type-cast-for-compatibility-rea.patch +Patch6009: backport-imfile-tests-ext-directorys-fd-leak-in-case-of-inotify-on.patch +Patch6010: backport-imfile-fix-ext-directory-s-fd-leak-in-case-of-inotify.patch +Patch6011: backport-core-bugfix-potential-segfault-on-busy-systems.patch +Patch6012: backport-mmnormalize-bugfix-if-msg-cannot-be-parsed-parser-chain-is.patch +Patch6013: backport-openssl-Replaced-depreceated-method-SSLv23_method-with.patch +Patch6014: backport-lookup-tables-bugfix-reload-on-HUP-did-not-work-when.patch +Patch6015: backport-lookup-tables-fix-static-analyzer-issue.patch +Patch6016: backport-tcpflood-bugfix-TCP-sending-was-not-implemented-properly.patch +Patch6017: backport-tcpflood-bugfix-plain-tcp-send-error-not-properly-reported.patch +Patch6018: backport-fix-startup-issue-on-modern-systemd-systems.patch +Patch6019: backport-tcp-net-subsystem-handle-data-race-gracefully.patch BuildRequires: gcc autoconf automake bison dos2unix flex pkgconfig python3-docutils libtool BuildRequires: libgcrypt-devel libuuid-devel zlib-devel krb5-devel libnet-devel gnutls-devel @@ -503,6 +514,22 @@ done %{_mandir}/man1/rscryutil.1.gz %changelog +* Thu Mar 21 2024 zhangqiumiao - 8.2210.0-8 +- Type:bugfix +- CVE:NA +- SUG:NA +- DESC:tcp net subsystem: handle data race gracefully + fix startup issue on modern systemd systems + tcpflood bugfix: plain tcp send error not properly reported + tcpflood bugfix: TCP sending was not implemented properly + lookup tables: fix static analyzer issue + lookup tables bugfix: reload on HUP did not work when backgrounded + openssl: Replaced depreceated method SSLv23_method with TLS_method + mmnormalize bugfix: if msg cannot be parsed, parser chain is stopped + core bugfix: potential segfault on busy systems + imfile: fix ext directory's fd leak in case of inotify on symlink + imfile tests: ext directory's fd leak in case of inotify on symlink + * Mon Dec 25 2023 zhangqiumiao - 8.2210.0-7 - Type:bugfix - ID:NA -- Gitee