From: Michael Orlitzky Date: Thu, 24 Aug 2017 11:01:21 +0000 (-0400) Subject: Upstream source. X-Git-Tag: 1.40.04~13 X-Git-Url: https://gitweb.michael.orlitzky.com/?a=commitdiff_plain;h=af564700c95b53f3c2c6f111c1a8b5fd32575728;p=postfix-logwatch.git Upstream source. --- af564700c95b53f3c2c6f111c1a8b5fd32575728 diff --git a/Bugs b/Bugs new file mode 100644 index 0000000..5f5d9cb --- /dev/null +++ b/Bugs @@ -0,0 +1 @@ +- Percentiles reports do not honor max_report_width diff --git a/Changes b/Changes new file mode 100644 index 0000000..c352d7c --- /dev/null +++ b/Changes @@ -0,0 +1,981 @@ +2014-09-03 (version: 1.40.03) + - Fix: Corrected typo in RE for postscreen PREGREET & HANGUP fix was lost in merge. + Thanks: Michael Orlitzky + +2014-08-28 (version: 1.40.02) + - Fix: Commented out some code which was accidentally left enabled, preventing the + script from running. + +2014-08-01 (version: 1.40.01) + - Fix: Correct TLS matching for Postfix 2.11 and later. Thanks: John Wilcock + - Fix: Ignore cfg_get_str debug lines. Thanks: Bas Mevissen + - Fix: Ignore sacl_check:... lines. Thanks: Jonathan Herbach + +2012-01-11 (version: 1.40.00) + - Change: License is now the MIT/X-Consortium License: + http://www.opensource.org/licenses/mit-license.php + This allows re-inclusion into the logwatch project. + - Fix: Ignore additional debug lines. Thanks: Peter Smrcak + - Fix: Corrected typo in RE for postscreen PREGREET & HANGUP. + +2011-09-26 (version: 1.39.07) + - New: Support 2.9's enable_long_queue_ids via command line option + --[no]long_queue_ids and config file var $postfix_Enable_Long_Queue_Ids. + Default is disabled. There will be issues with existing logs that contain + both formats. + - Fix: Support postscreen's WHITELIST VETO. + - Thanks: Noel Jones + +2011-09-23 (version: 1.39.06) + - New: Support postscreen from postfix 2.8. Updated for minor dnsblog + log changes. Note: tlsproxy messages are ignored for now. + - Change: Section dnsblog primary sort key is now listing site. + - Fix: Some "improper command pipelining" log lines were unmatched + (section: smtpprotocolviolation). + - Fix: Ignore proxy-reject w/no SMTP reply code. Occurs after a queue file + size lmit exceeded warning by smtpd. + - New: Postfix 2.8.1 changes FCRDNS warning messages (section: hostnameverification). + See: http://article.gmane.org/gmane.mail.postfix.user/218112 + - Internal: Removed remaining $re_IP in rblerror section, from v1.39.05. + - Thanks: Noel Jones, Tonio, Jase Thew, Alex, Michael Orlitzky + +2010-11-12 (version: 1.39.05) + - Fix: Check input value to commify() against undef; required for perl 5.12. + Thanks: Michael Orlitzky + - Fix: Eliminate use of $re_IP, as it is too complicated, expensive, and did + not correctly match all valid IPv6 addresses. Note: one remains in + RBL lookup error section, as parsing out IP is nontrivial w/IPv6. + - Internal: Update URL to Sourceforge. + +2010-03-23 (version: 1.39.04) + - Fix: Support Postfix 2.8's reject_rhsbl_reverse_client in RejectRBL. + Thanks: Noel Jones + +2010-03-06 (version: 1.39.03) + - Fix: Support for reject_rhsbl_helo in RejectRBL section. + Thanks: Steve Bytnar + +2010-03-03 (version: 1.39.02) + - Change: Merged section SaslAuthRelay into SaslAuth. The purported SASL + sender is now included after an existing SASL_username. + +2010-03-02 (version: 1.39.01) + - New: Allow enabling/disabling the summary section in config file, using + var $postfix_Show_Summary and command line option --[no]summary. + Thanks: Benedikt Bohm + +2010-01-11 (version: 1.39.00) + - New: Support 2.7 experimental branch logging: + + smtpd's proxy-reject: placed in standard reject sections. + + smtpd's proxy-accept: ignored. + + postfix/postscreen: new section 'postscreen'. + Note: memory intensive, so limiter is set to 1, showing only counts + of various postscreen result. + + postfix/dnsblog: new section 'dnsblog'. + Note: memory intensive, so limiter is set to 1, showing only a list + of DNSBL'd IPs. Disable this section by setting limiter to 0. + + postfix/verify: currently ignored + Thanks: Stefan Forster, Noel Jones + - New: Support 2.6 access(5) action BCC. New section 'bcced'. + - New: New option --[no]unknown to show/suppress output of hostname of + 'unknown' in formatted ip/hostname pairs. Default: on. + - New: Updated postfix-logwatch.conf files to include instructions on how to + increase log scanning performance when not using policy services such as + policy-spf, etc. See "Performance Note" in the .conf files. + - Change/Fix: Warning sections 'ratelimit and 'concurrenclylimit merged into + a new, generic 'anvil' section, which includes all of anvil's various limit + exceeded messages. If you see errors after updating about these unknown + limiters, be sure to replace these two deprecated limiters with the new + one to your config file(s). + - Change: Use DSN from delivery verification probes as first level + key (section: 'deliverable'). + - Change: Replace databasegeneration section header 'Database file + needs update' with 'Database is older than source file', as per request + on the Postfix mailing list. + - Fix: Handle more, and restructure, SASL authentication failed messages. + Thanks: Stefan + - Fix: Ignore some smtpd cache lines; handle postmaster delay notifications. + Thanks: Gabriele Beltrame + - Fix: Handle more postfwd lines. + - Fix: Prevent useless level 3 ':' output in section 'saslauthfail'. + Thanks: Armin Tüting + - Fix: Corrected some typos in man page. + - Fix: Added missing Error section (how was this missed!). + - Fix: Handle unmatched milter-{reject,hold,discard} lines that do not + include smtpd envelope data. + - Fix: reduce memory footprint for Delays reports (~30Mb per 1Mb log file). + - Fix: performance increases. + +2009-07-14 (version: 1.38.01) + - Fix: RE typo caused policydweight lines to be unmatched. + - Fix: Reduce to-be-ignored RE list searching by moving most common + REs to head of the list. + +2009-07-10 (version: 1.38.00pre6) + - Fix: Handle Softfail and Temperror in pypolicyd-spf (PolicySPF). + postfix/policy-spf. Thanks: Chris Burton. + +2009-07-10 (version: 1.38.00pre5) + - Fix: Comment out extra OnlyService line in postfix-logwatch.conf* files, + leaving as a sample. The duplicate causes logwatch to filter out postgrey, + postfwd, and policyd-spf lines. Also include '-' in the RE to capture + postfix/policy-spf. Thanks: Malte Koestner, Chris Burton. + +2009-07-07 (version: 1.38.00pre4) + - New: Error and diagnostic output from non-postfix programs will be moved + into the WarningsOther section. + - Fix: Added Tarpit whitelisted to postgrey output. + - Fix: Handle additional output for pypolicyd-spf, postgrey, and postfix. + - Fix: Add policyd-spf to postfix_Syslog_Name and OnlyService. + Thanks: Chris Burton + - Fix: logic error which inadvertently incremented Notification Sent counter + on ignored postfix/bounce lines. + - Fix: Add -T to interpreter line (removed inadvertently to use perl debugger). + - Change: Remove 'postgrey' from default syslog_name. This only affects users + without a configuration file, where it is enabled by default. + +2009-07-06 (version: 1.38.00pre3) + - New: Handle pypolicyd-spf. Thanks: Chris Burton + - New: ConnectionInbound section shows IP/hostname. Disabled by default, + for performance reasons. Requested by: Jernej Porenta + - Change: MxErrors changed to DNSErrors, generalizing DNS lookup errors. + A beneficial side affect is the removal of the numerous DNS lookup + informational "warnings" from the top of the Summary section; they + can also, of course, be disabled in the Detailed section. + - Fix: Handle END-OF-MESSAGE stage in reject code. + +2009-07-02 (version: 1.38.00pre2) + - Fix: Remove debug print + - New: Section SMTP protocol violation (smtpprotocolviolation) + - Change: Removed section "toomanyerrors", merging it into new section + smtpprotocolviolation. + - Fix: Handle "improper command pipelining ..." messages in new section + smtpprotocolviolation. + - Fix: Ignore more debug lines and miscellaneous canonicalization. + +2009-06-29 (version: 1.38.00pre1) + - New: Support milter-hold (aka: "quarantine") and milter-discard lines. + Stats are located in "hold" and "discarded" sections, respectively. + Requested by: Gary Casterline; + - Fix: Perl v5.10 introduces tighter taint control in sprintf format + strings. Perform strict data checking on $Opts{'ipaddr_width'} to + untaint it, as it is used directly as an sprintf() field width specifier. + Thanks: Dudi Goldenberg + - Fix: Some Policy-SPF lines were unmatched, due to log format change + in Mail::SPF v2.006, which changed "identify=mfrom" to "identity=mailfrom". + Thanks: Chris Burton + - Fix: Resolved several bugs in the Policy-SPF module: some missing return + return statements, which for certain unmatched log lines would both add to + the Unmatched list and increment a PolicySPF hit counter; a few older + log lines would show blank IP addresses. + - Fix: Postgrey whitelisted lines that did not include a host IP address + were reported as unmatched. + - Fix: Relax email address capture in to=, orig_to=, and from= fields in + log lines. + - Fix: Better diagnostic when limiter level is not specified on + the command line using the --limit option. Eg: + "--limit rejectbody" v. "--limit rejectbody=10" + - Fix: Remove bogus --content_filter option from man page (feature not yet + implemented). Thanks: David DeFranco + - Fix: Better handle "Delivery temporarily suspended: conversation with xxx timed + out..." deferred messages. + - Fix: Handle multi-word XYZ in "blocked using XYZ" in RejectRBL. + Thanks: Sergey Pylinsky + - Fix: Ignore lines from postlog service. + - Change: Detail level now included in Detail title + - Internal: Support variable hash key lengths, but using the special key + sequence of two ASCII bell's ("\a\a"). This removes the existing + requirement that dummy keys must be used at the end of the %Counts + accumulator hash. + - Internal: Test data generation + +2008-10-21 (version: 1.37.08) + - Fix: Saslauth messages were not being detected if the only field in a + smtpd "client=..." line was "sasl_sender=". + +2008-10-20 (version: 1.37.07) + - New: Support "Temporary lookup failure" temporary rejects. New reject + limiter 'XXXrejectlookupfailure', where XXX is any of the set of + reject codes in effect. + - New: Support smtp_body_checks and smtp_*header_checks. + Requested by: Noel Jones + - Fix: Previous fix to support reload log line that includes postfix + version number broke previous postfix reload log lines. + Thanks: Armin Tüting + - Fix: Setting recipient_delimiter caused RE failure, due to missing + \Q \E quoting. Thanks: Armin Tüting + - Fix: BCC action support from 1.37.06 had a "if" vs. "elsif" typo. + Thanks: Noel Jones + - Fix: Allow postgrey filter to handle absent recipient= field. + Thanks: Alexander Kolesnik + - Fix: Limiters were not case insensitive on the command line. + - Fix: Use of bare reject limiters did not set each reply code variant of + the given limiter after using --nodetail, resulting in no output for the + given section. + - Fix: Change Logwatch's OnlyService variable in the postfix-logwatch.conf + file to also capture postfwd and postgrey. Thanks: Alex Schuilenburg + - Fix: Ignore more debug lines. + +2008-08-22 (version: 1.37.06) + - New: Handle message_reject_characters cleanup rejects. New reject + limiter 'XXXrejectcontent', where XXX is any of the set of reject + reply codes in effect. + - New: Support BCC action from 2.6 experimental branch. New limiter + 'bcced'. + - Fix: Ignore "mapping DSN status" lines. + - Fix: Support reload log line including postfix version number. + Postfix snapshot 2.6-2008081 + - Thanks: Noel Jones + - Fix: Ignore more debug lines. + +2008-08-19 (version: 1.37.05) + - New: Initial implementation of Postfwd reporting. + Requested by: Sahil Tandon + - Change: Undeliverable (address verification) section now + grouped by DSN, canonicalized host reply, domainpart, localpart, + and formatted host/hostip. This dramatically cleans up this + section's output for systems that perform many address verifications. + - Fix: Postgrey lines were being ignored due to change made to + implement Ignore_Service. + - Fix: handle "status=undeliverable-but-not-cached" address verification + response, which is coerced into a simple "status=undeliverable". + If there is any value in distinguishing the two status types, let me + known and I'll create a new limiter. Thanks: Gary Casterline + - Fix: missed conversion of keyword "next" to "return" in postfix_postsuper + routine after inline code was converted into the subroutine. + +2008-07-22 (version: 1.37.04) + - Fix: recognize "approximately" in TooManyErrors section messages + starting introduced in Postfix 2.6 20080621. Thanks: Noel Jones + - Fix: Reset to level 0 the inadvertently changed EnvelopeSenders + and EnvelopeSenderDomains (in postfix-logwatch.conf). + +2008-07-18 (version: 1.37.03) + - Change: TimeoutInbound will include byte count if available; + default level changed to 1 to reduce noise. + - Change: some additional canonicalization is performed on + various "host XXX said: yyy" messages in sections like Deferrals. + Much of the "said" verbiage from some large mail houses is redundant + or excessive, and causes additional memory consumption. If this + canonicalization presents problems, I may include an additional level + of detail to present the raw message. Feedback welcome. + - Fix: Supplemental sections could not be re-enabled after using + --nodetail. + - Fix: Handle some additional RejectRBL variants, esp. the + "day old bread" list (dob.sibl.support-intelligence.net). + Thanks: Michael Monnerie + - Fix: Present the remote MTAs reply code/DSN, if available, at the + beginning of canonicalized "host XXX said: yyy" messages. + Thanks: Jorey Bump + - Fix: Provide remote MTAs SMTP reply code/DSN if available + in various Delay, Deferral, etc. reports. Some additional + canonicalization is performed on various flavors of deferral + replies. + - Fix: recognize "approximately" in lost connection byte count + messages starting introduced in Postfix 2.6 20080621. + Thanks: Noel Jones + - Internal: Ignored lines are now placed into a list, instead of + hard coded into the code. In the future, this will allow users + to configure ignore patterns without making code modifications. + In doing this, I discovered and reported a bug in perl (#56202), + but worked around the problem. + - Internal: move postsuper, panic, and fatal message processing into + their own subroutines. + +2008-05-30 (version: 1.37.02) + - Change: Some changes in Delays report. Now includes total delay + time as reported in delivery agent log entry delay=x. Removed + leading numbers in row titles, and cleaned up the title names. + Also, reduced fields to 2 decimal places. + - New: By IP (permanent) Reject report controlled with ByIpRejects + limiter. Disabled by default. + - Fix: Two minor fixes to PolicydWeight module: it was not the required + importing inc_unmatched, and ignore diagnostic "master: ..." lines. + - Fix: Eliminate some extraneous newline output. + - Fix: Ignore more debug log lines. + +2008-05-09 (version: 1.37.01) + - Change: level limiters are no longer unique command line options, + but are now parameters to the single option "--level" or "-l". + This reduces the number of command line options in the help list, + and simplifies the code added to support reject_reply_patterns. + For example, the config file level limiter $postfix_Sent = 2, + the command line option would be --limit sent=2 or -l sent=2, + rather than the previous --sent=2. This also means the --no + variants for level limiters are gone (eg. --nosent; instead use + -l sent=0). There is no change within the configuration files. + Limiters can still be abbreviated so long as they are unambiguous. + - Change: Section connectionlostoverload is removed, by being merged + into connectionlostinbound. + - Change: PrematureEOI is now AttrError, broadened to include errors + reading attributes from services. + - New: Support RFC 4954 Enhanced status codes (postfix 2.5+). + - New: Report "status=deferred (bounce failed)" messages under + Summary section "Bounce failed". + - New: Support postfix 2.6+ check_reverse_client_hostname_access + Includes new reject config variable: RejectUnverifiedClient. + - New: option --line_style specifies how to handle lines lengths + longer than max_report_width. Options are "wrap", "full", or + "truncate" (default). The older --detail >= 11 is equivalent + to line_style=full; line_style=truncate or line_style=wrap has + higher precedence and will dictate how long lines are handled. + Alternative options are --truncate, --wrap, or --full. + - New: The beginnings of a Top N config file is provided in + postfix-logwatch.conf-topn. Suggestions and improvements welcome. + - New: single letter options for some long options; run with + --help to see list. More may be added in the future. + - New: Option --ignore_services (config var $postfix_Ignore_Services) + provides a mechanism to ignore postfix/SERVICE log lines, where + SERVICE is a regular expression pattern. + - Fix: Support Anonymous TLS in TlsServerConnect (postfix 2.5+). + - Fix: Ignore output from "postfix status" command (postfix 2.5+) + - Fix: Ignore more debug log lines. + - Fix: When postsuper held more than one message in a given call, + the Hold count shown in the Summary section indicated how many + messages were held, but the Detail section showed how many + postsuper calls were made. Thanks: Stefan Jakobs + - Fix: Postgrey sender and recipient fields improperly reported + - Fix: Allow ":unknown" as an acceptable port in smtpd's "client=..." + log lines. Occurs in pre-queue content_filter setup. + Thanks: Robert Brooks + - Fix: At detail > 10, log lines were truncated to max_report_width. + - Fix: Documentation cleanup. Thanks: Chris Pepper + - Fix: Relax capture RE for resent-message-id which may contain + < or > chars. Thanks: Stefan Jakobs + - Fix: Reason for deferral/bounce would sometimes be incorrectly + shown as unknown recipient. + - Fix: Better handle the trigger subject (SMTP_NAME_*) in discard, + filter, hold, redirect, and warn actions (eg. Client host, Sender + address, Recipient Address, Client certificate, etc.). + - Change: reduce length of some extended status codes for readability. + - Change: remove hard-coded "postgrey" from the syslog name pattern + matching code, and instead add it to the configuration variable + postfix_Syslog_Name. + - Internal: converted all Section key names to lowercase to avoid + silly case errors due to hash key case differences in Sections, + Opts, and Collecting hashes. + - Internal: debug output is now controlled by keywords. + +2008-05-09 (version: 1.36.13) + Final version skipped + +2008-01-14 (version: 1.36.13pre7) + - New: Support ETRN rejects (option: RejectEtrn). + - Fix: Include optional text in L1 output for header/body checks Hold + messages. + - Fix: Accept "unknown" as an IP in Connection rate limit messages. + Thanks: Stefan Jakobs + - Fix: Improve MxError captures + - Fix: Ignore "sql auxprop plugin ..." messages + - Fix: Ignore more debug_peer_level=2 messages + - Fix: Correct failure to ignore "connect to subsystem..." debug lines + - Fix: Add missing RejectVerify to config file + +2007-12-15 (version: 1.36.13pre6) + - Fix: in BounceLocal and EnvelopeSenderDomains, set null domain + and formatted host to '*unknown'. + - Fix: Move postfix_warning before postfix_cleanup, as some cleanup + warnings were caught as unmatched. + - Fix: An Accepted message is now triggered by smtpd "client=..." + and pickup "uid=..." log entries, instead of qmgr's "from=xxx, + size=nnn, nrcpt=nnn" log lines. A message may have been accepted + during a previous time period, but delivery delays result in + multiple qmgr delivery attempts, resulting in over counting + Accepted messages. Thanks: Stefan Jakobs + - New: Bytes delivered is now broken down by Bytes sent via SMTP, + LMTP, and forwarded, to be orthogonal with message counts. + - Fix: Resent messages no longer reduce the number of messages + accepted - this was a naive attempt at not counting, for example, + messages released and re-queued from a content filter's quarantine. + +2007-11-14 (version: 1.36.13pre5) + - New: Threshold limiting and Top N lists for every level in each + Detail section. Every level in each section can now be limited + with minimum count thresholds and top N lists. See the updated + README file, the comments in the postfix-logwatch.conf file, and + the new postfix-logwatch man page. Requested by: Pavel Urban + - New: Rejects can now be categorized by reject reply code. A new + option/variable "reject_reply_patterns" is a list of reject reply + code regular expressions, which are used for categorizing rejects. + This feature allows, for example, distinguishing 421 transmission + channel closes from 45x errors. (eg. 450 mailbox unavailable, 451 + local processing errors, 452 insufficient storage). The default + list is: "5.. 4.. Warn" which creates three groups of rejects: + permanent rejects, temporary failures, and reject warnings (as in + warn_if_reject). Requested by: Noel Jones + - New: postfix-logwatch man page created (net yet complete) + - New: Support for all access(5) actions. See the "Level Limiter + Options" section in the postfix-logwatch(1) man page, and "Common + access control actions" in postfix-logwatch.conf. + - New: Added envelope senders and envelope sender domains reports. + These are disabled by default. Enable with level limiter options + --envelopesenders 1 and --envelopesenderdomains 1 (or 2 to + also see senders listed under domains). See also the + postfix-logwatch.conf file. Suggested by: Brendan + - Change: Uncomment all variables in the config file. This should + help ensure the variables in the config file stay in sync with + those used in the source. + - Change: Merged sections SenderDelayNotification, DSNDelivered, and + DSNUndelivered into NotificationSent, with sub-sections indicating + the type of notification. This gives the total number of sent + notifications in the Summary section, and the breakdown by type + of notification in the Detailed section. + - Change: Removed "msgs" prefix from several options: msgsdeferred, + msgsdelivered, msgsforwarded, msgsresent, msgssent, and msgssentlmtp + are now deferred, delivered, forwarded, resent, sent and sentlmtp. + The old options are still usable. + - Change: force --help output to 80 chars. + - Change: Taint mode is now on by default in standalone mode. It is + disabled upon installation in logwatch mode, as logwatch fails with + taint mode enabled. + - Change: Set the primary key in tlsserverconnect/tlsclientconnect + options to type/cipher to reduce excessive level 2 output. + - Change: Reported values that cannot be determined or are unavailable + are prefixed with an asterisk (Eg. *unknown, *unspecified). + - Change: Warn section title 'Warn action logged' changed simply to + "Warned", to be consistent with other access/header_ and body_checks. + The option is --warned, but --warn is still acceptable. + - Fix: Significantly reduce memory footprint when detail < 5. + - Fix: Usage and --help now correctly show only detail section + level limiter options that are available. Previously, summary- + only counts were also display as level limiter options. + - Fix: "too many errors after DATA" and "timeout after DATA" may + include a byte count, as in "(348 bytes)". Thanks: John Beaver + - Fix: Ignore postgrey 'delayed ...' lines + - Fix: Allow logwatch --debug option to pass into postfix-logwatch + - Fix: configuration file reading code was not properly warning on + non-existent files + - Fix: "filter" actions were incremented on "redirect" actions + - Fix: Give more room to percentiles in delays percentiles table. + A 5 day delay is 432000.000 seconds, and the previous table width + was not sufficient. + - Fix: --show_sect_vars command line option inadvertently required an + argument; the name has been shortened to --sect_vars/--nosect_vars + and correctly no longer requires an argument. The longer names + --[no]show_sect_vars work as well. Thanks: Noel Jones + - Fix: Handle some unmatched connect to failures in section + ConnectToFailure. Cleanup and consolidate several similar messages; + add additional detail at level 3. + - Fix: Increment postsuper Hold messages by the number of messages + placed on hold. + - Fix: Add "non-ESMTP response ..." messages to SmtpConversationError. + First level is now the general SMTP error description. + - Internal: Converted source to be package-based to allow code sharing + with amavis-logwatch. The single-file executable is auto-generated + from the packages. + - Internal: internal gen_test_log function to create sample log data + for testing. Reads '#TD' comments from within postfix-logwatch. + +2007-10-16 (version: 1.36.13pre4) + - Fix: Handle messages "SSL_connect error to example.com: 0" and + "Cannot start TLS: handshake failure" by coercing into warnings + Thanks: Rob Sterenborg + +2007-10-15 (version: 1.36.13pre3) + - Fix: Handle "postmaster" DSNs + - Fix: Handle "discarding EHLO keywords:" (ignored) + - Internal: create bounce subroutine to handle bounce messages + +2007-10-05 (version: 1.36.13pre2) + - New: Initial support for postgrey (http://postgrey.schweikert.ch/) + Requested by: Sebastian Wolfgarten + +2007-10-05 (version: 1.36.13pre1) + - New: Support for policy-spf software postfix-policyd-spf-perl + Requested by: Nicodemo P. and Rob Sterenborg + +2007-10-05 (version: 1.36.12) + - New: support postfix 2.5 log changes (20071004, 20071003) + - Incompatible Change: Config variable/command line option name change: + * WarningHeader changed to Warn + * MsgsRedirected changed to Redirected + * ConnectionLost split into ConnectionLost{Inbound,Outbound} + - Incompatible Change: Distinguish inbound (smtpd) vs. outbound (smtp), + which replaces ConnectionsLost with ConnectionsLostInbound and + ConnectionsLostOutbound. ConnectionsLostInbound includes number of + bytes received if the connection was lost during DATA. Encouraged by + 2.5 20071003 change in logging. + - New: Support for all header_checks(5)/body_checks(5) actions. See + also postfix-logwatch.conf for "Common access control actions" + - New: Option show_sect_vars shows names of section configuration + variables/command line options in titles of detailed report sections. + This allows easy correlation of corresponding configuration file + variables/command line option for each section. [ Default: 0 ] + - Fix: catch postsuper's pluralized form of Delete message"s". + - New: handle "cannot load Certificate Authority data" as Misc. warning + - Fix: Makefile: no -D option to install in FreeBSD. Use -d instead + - Change: ConnectionLostInbound now defaults to level 1 only + - Internal: Numerous code cleanups, re-factoring and restructuring + - Internal: Test data migrating to include the correct postfix service, as + internal code moves towards using the service name as an initial log + line match qualifier + +2007-09-13 (version: 1.36.11) + - Incompatible Change: All TempRejectXXX and RejectWarnXXX options/config vars + have been renamed for easier identification in the code, and consistency. + Now, all Reject variants look like RejectXXXyyy, where XXX is the + given reject name (Helo, RBL, etc), and yyy is the optional reject type + of "Warn" (warn_if_reject) or "Temp" (4xx temporary rejects). + For example, the previously named RejectRelay, RejectWarnRelay and + TempRejectRelay are now named RejectRelay, RejectRelayWarn and RejectRelayTemp. + See the usage information or the .conf file to see the list. + - Fix: A %Counts accumulator must use the same number of keys consistently. + This error caused some totals to be wildly incorrect, and the fatal perl error: + "Can't use string ("XXX") as a HASH ref while "strict refs" in use ..." + Fortunately, this error only occurred with a specific set of data, which + seemed not too common. + - Fix: remove Temp and Warn variants of RejectHeader and RejectBody - they + don't exist. + - Fix: Add Temp variant of RejectMilter + - New: Add ProcessLimit section for 2.5 stress messages + - New: accommodate postfix patch which also logs HELO name in smtpd's + "QID: client=..." log entries. + - Thanks: Noel Jones + +2007-09-09 (version: 1.36.10) + - New: handle "reject: DATA from ... : Data command rejected: ..." + - New: ignore "fingerprint=20:..." lines + - Thanks Farkas Levente + +2007-09-01 (version: 1.36.9) + - Fix: remove rooted path in md5 file + - Fix: Makefile install-logwatch rule was missing a parenthesis + - Fix: Makefile updates from Till Mass + +2007-08-31 (version: 1.36.8) + - Change: Include GPLv2 license + - Change: Include version number in tarball file name + - Internal: Move CVS log comments to Changes file + +2007-08-31 (version: 1.36.7) + - Fix: capture older postfix RCPT from RBL reject entries. Thanks + Hugo van der Kooij + +2007-08-15 (version: 1.36.6) + - Changed: for sorting purposes, lowercase localpart of rejected email addresses + - Change: Output help and version info on STDOUT for easier pipelines to a pager + - New: option --nodetail zeros out all detail levels, to more easily obtain + only specified detailed reports (eg: --nodetail --rejecthelo 1) will + only show a list of rejected HELOs in the details section. + - New: option --nosummary disables the summary section + - New: detailed section command line arguments can now be specified with the + prefix "no", to set the level to 0 (eg. --nomsgssent is equivalent to + --msgssent 0). + - Internal: split printReports into printSummaryReport and printDetailReport + - Internal: change variable Formats to the more obvious named Sections + +2007-08-03 (version: 1.36.5) + - Changed: rejected addresses collected by domain, then localpart + - New: delay percentiles report. Config vars: show_delays and + delays_percentiles; command line --[no]delays + - Fix: Yes/True and No/False config values weren't being read properly + in standalone + +2007-08-01 (version: 1.36.4) + - New: option --config_file allows specifying a configuration file via + command line. Options in configuration file act as though they were + set on the command line in order, with earlier settings being over- + ridden by most recent settings. Multiple config_file options may + be specified. + - Change: Summary title includes syslog_name in standalone mode + - Change: Remove some extraneous newlines + +2007-07-24 (version: 1.36.3) + - Refine anvil connection rate exceeded messages + +2007-07-13 (version: 1.36.2) + - Support FreeBSD ( precedes hostname in syslog) + - Thanks Clemens Fischer + +2007-07-10 + - Ignore "nss_ldap: reconnected to LDAP server ..." + - Handle "discard: header/body: messages + - Thanks Jay Chandler + +2007-07-03 + - Corrected some minor typos + +2007-06-08 + - Changed titles shown for sender notifications of (non-)delivery + and delay + - Corrected some incorrect config file variable names. Thanks: Nicolas + +2007-06-02 + - Do not strip <> when address is '<>' in Illegal address syntax + +2007-06-01 + - Changed warning output for smtpd messages: + "lost connection after CONNECT from unknown[unknown]". + Previous warning erroneously attributed this to pre-queue content + filter overloads, but the problem indicates a more general + smtpd overload. + +2007-05-31 + - Fix bug which caused config file to be required in standalone mode + +2007-05-30 + - Support delay_reject=no (debian: 426726) + - Ignore additional unmatched TLS debug messages + - Initial support for redirect messages + - Some corrected typos + - Thanks Jusin Pryzby + +2007-05-25 + - Experimental: syntax to limit Top N level 1 output lines. Variables + that control depth levels in detailed reports can be specified as + m.n, where m is the maximum level to output, and n specifies the number + of level 1 items output. Eg: $postfix_MsgsSent=2.10, will output + the top 10 level 1 items, with each item providing 2 levels of detail. + - Protect interpolated recipient addresses in cleanhostreply with \Q \E + e Add zero-width assertions and use strict IP RE in bycount sort subroutine + to match IP addresses more reliably + +2007-05-09 + - Ignore a few more policydweigh child or cache entries + - Escape metacharacters from being interpreted in recipient_delimiter + - Never split mailer-daemon, double-bounce, or when recipient_delimiter + is a "-" (dash) never split owner- or -request localparts + +2007-05-08 + - Support for running in standalone mode (independent of logwatch) + - Renamed script to "postfix-logwatch" to avoid confusion when running + in standalone mode. See the README. + - Add relay=virtual to "local" class for local vs. remote bounces + +2007-05-07 + - Handle and report postfix/policydweight lines (postfix_PolicydWeight) + - Handle and report Host offered STARTTLS lines + - Generalize "maildrop: Unable to create a dot-lock at " messages + - Corrected typo that prevented fatal errors from being output + +2007-04-26 + - Consolidate similar MX errors + - set IP address to 127.0.0.1 when from=local, and reporting both host/hostip + - More cleanup (re-factor common code, replace most global variables with lexicals, + lowercase non-global variable names, shorten variable names, etc.) + - Capture postsuper hold messages + +2007-04-25 + - Support postfix 2.5 TLS message changes (smtpd_tls_loglevel > 0) + - Move 4xx temporary rejects into their own section (experimental feature) + +2007-04-18 + - Allow for hold messages that do not contain a recipient (Thanks John Wilcock) + +2007-03-26 + - Lowercase recipient addresses in several Reject sections + +2007-03-22 + - Accept spf.pobox.com URLs in Reject recipient address + +2007-03-21 + - Handle spf lines from older version of postfix-spf, and spf.pobox.com URLs + +2007-03-20 + - Capture and report postfix-spf lines + - New config variable postfix_PolicySPF + +2007-03-13 + - Capture and report reject_unknown_reverse_client_hostname. + Thanks Michael M. + +2007-03-09 + - Capture and report as config warning: "looking for plugins" NSF error + +2007-03-03 + - fix reject header|body RE to allow "local" as host/ip + - really ignore lines that don't match SyslogName; + - add inc_unmatched subroutine for easier debug of unmatched lines + +2007-02-27 + - Capture and summarize postfix-script output (starts, stops, refresh, etc.) + - Remove redundant chomps + +2007-02-26 + - Fix problem in sort routine where IP addresses were being captured + anywhere in an output line for comparison via pack 'C4' - only + attempt IP comparison if an IP address is the start of an output line + Thanks: Ian + - Provide support for syslog_name in Postfix 2.4 via postfix.conf + variable postfix_Syslog_Name + - Classify PIX workarounds based on type (there are several) + - Change summary output criteria to check for any non-zero Totals + +2007-02-25 + - Do not interpolate log lines into printf; they may contain % chars + +2007-02-17 + - Ensure no output occurs when nothing is captured + +2007-02-15 + - Place recipients and senders in their own keys, instead of combined + +2007-02-14 + - Fix countdown bug in Deliveries, as ncrpt does not account for always_bcc + +2007-02-14 + - Make reject and warn_if_reject distinct sections + - Track Qids to properly report messages and bytes sent / accepted + - Also track messages deferred (each of which may have many deferrals) + - Consider reject VRFY a reject and accumulate in reject totals + - Move header/body reject code in with the rest of the reject code + - Move 'MAIL from' reject code in with the rest of the reject code + - Remove unused variable + - Print row heading separator lines only when appropriate + - Move printing of report headings into printReports + - Change Sent header to Sent via SMTP (orthogonal to Sent via LMTP) + +2007-02-09 + - Better processing of remote server "host...said" replies + - Made maximum report width configurable in postfix.conf + - All lines in report now obey max report width + +2007-02-07 + - Changed all From -> To lines to To <- From. From address is often bogus + and To is more interesting to see. + +2007-02-06 + - Added new configuration variable "postfix_Recipient_Delimiter", which + can be set to match the postfix variable "recipient_delimiter". + When set, allows Delivered and Sent reports to be grouped by + email addresses minus their address extension. + - Liberalized the RE for capturing VFRY rejects + - Reverted change of primary SASL authenticated messages; primary + key is once again User, followed by Method. Unknown is reported + when these keys are not available. + - Created a SASL authenticated relayed messages which hits with + sasl_sender is present in smtpd messages + - Split orig_to email addresses into a subkey. Allows reports + to be grouped by primary "to" address, instead of various aliases. + - Move Host/HostIP into their own keys for Bounce Remote section + - For Pix Workaround section, format HostIP / Host the same as others + - Add 'o' option where missing in REs + - Fix configuration variable importing + +2007-02-03 + - Added VFRY reject section + - Changed primary key for SASL authentication section to IP/hostname, + as any of sasl_{method,sender,user} may be absent + - Added RFC 3463 DSN code messages to bounce/deferred sections + - More parsing of various "host...said" remote server messages + - Add Sent via LMTP section, removing lmtp-delivered mail out of Delivered. + Allows users with lmtp-based content filters to avoid double counting + (but only for message counts; byte counts are still about 50% too large). + Config variable postfix_MsgsSentLmtp controls display + +2007-01-28 + - Added pre-queue content-filter overload section + - Reworked Bounce (local/remote) and Deferred sections + - Fixed several reversed captures of Host and HostIP + - Format Host and HostIP in Numeric hostname section + - Thanks: Mike Horwath + +2007-01-28 + - Made Reason the primary key in Deliverable/Undeliverable 'sendmail -bv' tests + - Modified re_DSN RE to capture DSNs missing 3 number response code + - Enhanced SASL authenticated messages to capture missing log lines + - Added milter-reject section + - Updated 'Reject server configuration error' RE for older postfix versions + - Fix copy/paste error which caused use of incorrect variable in bad size limit + - Add Concurrency limit reached section + - Add "maildrop" to the list of relay's considered local in Bounce section + +2007-01-23 + - Aggregate recipient/sender address verification lines (Thanks Harald Geiger) + - Uppercase message in reject recipient section + +2007-01-22 + - Update REs to allow null sender in sender addresses + +2007-01-19 + - Capture and summarize "triggers FILTER" messages (Thanks Eray Aslan) + - Fix overly permissive Server configuration error RE (Thanks Harald Geiger) + - Add "Server configuration error" rejects to Reject totals + - Add "Insufficient system storage" rejects to Reject totals + - Make RejectSize report consistent with others + +2007-01-17 + - Update IP RE to support IPV6 (Thanks Harald Geiger) + - Aggregate reject recipient address caused by SPF (Thanks Harald Geiger) + +2007-01-16 + - Made Reject HELO/EHLO report consistent with others: demoted Helo=xxx + string (Thanks: Jorey Bump) + - Fixed incorrect TotalRejects summation (typo) + - "Accepted" / "Rejected" summation is no longer based on "Connection"; + now Total = Accepted + Rejected + * Client may or may not close connection properly after reject + * Multiple rejects for single connection can occur + - ConnectToFailure header incorrectly indicated connection was "inbound" + - Included Makefile for ease of installation during this testing phase (Thanks: Jorey Bump) + +2006-12-16 + - Add pcre map warnings + - Reordered headings to produce more obvious correlation with percentage breakdowns + - Second key sort is now the illegal address in illegal address in SMTP command + - More optimization of log entry capturing + - Added Sent category capturing outbound SMTP connections + requires modified /usr/share/logwatch/default.conf/services/postfix.conf or + /etc/logwatch/conf/services/postfix.con + - Added ability to control max print depth on a per section basis + - Renamed Received to Accepted (to indicate accepted for delivery, and better opposite of Reject) + NOTE: Accepted shows less than that of pflogsumm's Received, which incorrectly + increments rejected messages + - Reduce Accepted by Deferred and Resent, which cause double counting + - Removed erroneous a-z in RE for capturing QIDs + - Added Panic section for postfix panic messages + - Fixed bug which failed to increment watchdog timers + - Started work on better debug capability + +2006-12-13 + - Removed extra blank line for Detail <= 5 + - Filter many more debug lines + - Catch and group additional warning and fatal messages + +2006-12-12 +- Made reject header/body output consistent with each other +- Sort reject header/body first by recipient address, for tighter groupings - reject reason is typically random +- Trimmed excess whitespace in reject header/body reasons +- Added extra line between level 1 headings +- Group SASL authenticated messages by sasl_username +- Filter TLS and SASL debug messages (smtpd_tls_loglevel = 2) + +Rewrite by Mike Cappella (MrC) +Revision 1.29 2007/01/27 20:21:46 mrc + - Provide more useful information and summaries + - Provide increasing detail as requested via --detail + - Provide ability to configure per section maximum detail + - Optimize and combine the numerous REs + - Capture and summarize many more log lines + - Pin important errors to top of report + - Sort by hits, IP, and lexically + - Handle IPv6 addresses + - Generalize log line capturing and reporting + - Eliminate excessive copy/paste reporting code + - Requires updated postfix.conf file + - Thanks: Eray Aslan, Jorey Bump, Harald Geiger, Bill Hudacek, + Frederic Jacquet, Geert Janssens, Leon Kolchinsky, Rob Myroon + +Revision 1.28 2006/12/15 06:24:49 bjorn +Filtering "sender non-delivery notification", by Ivana Varekova. + +Revision 1.27 2006/12/15 05:00:41 bjorn +Filter all held message logs, by Hugo van der Kooij. + +Revision 1.26 2006/10/20 16:51:50 bjorn +Additional matching of sasl messages, by Willi Mann. + +Revision 1.25 2006/08/13 21:25:55 bjorn +Updates to work with the Postfix 2.3.x series (due to log format changes), +by Mike Cappella. + +Revision 1.24 2006/03/22 17:43:46 bjorn +Changes by Harald Geiger: +- ignore additional statistics: messages (Postfix 2.2) +- replaced several 5xx Codes by [0-9]+ + (main reason is to make them match on 4xx if in soft_bounce=yes mode) +- a more generic "Client host rejected" reporting +- changed "Messages rejected:" to "Messages rejected from sender:" + +Revision 1.23 2005/12/19 15:47:47 bjorn +Updates from Mike Cappella: + - Catches some of the Unknown Users messages from newer versions of postfix + - Consolidates a couple of REs + - Adds a cumulative total to each of the Unknown users and Header content rejection headers + - Adds a Body content rejection section + +Revision 1.22 2005/11/22 18:30:47 bjorn +Detecting 'virtual alias table', by Kevin Old. + +Revision 1.21 2005/08/23 23:54:38 mike +Fixed typo probably from Roland Hermans -mgt + +Revision 1.20 2005/07/25 22:26:28 bjorn +Added "Sender address" to "554 Service unavailable" regexp, by Who Knows + +Revision 1.19 2005/04/22 13:48:28 bjorn +This patch catches (un)deliverable messages and many more, which were +missing until now on mu new postfix-2.1.*, from Paweł Gołaszewski + +Revision 1.18 2005/04/17 23:12:28 bjorn +Patches from Peter Bieringer and Willi Mann: ignoring more lines and +some blank spaces + +Revision 1.17 2005/02/24 17:08:05 kirk +Applying consolidated patches from Mike Tremaine + +Revision 1.7 2005/02/16 00:43:28 mgt +Added #vi tag to everything, updated ignore.conf with comments, +added emerge and netopia to the tree from Laurent -mgt + +Revision 1.6 2005/02/13 23:50:42 mgt +Tons of patches from Pawel and PLD Linux folks...Thanks! -mgt + +Revision 1.5 2004/10/06 21:42:53 mgt +patches from Pawel quien-sabe -mgt + +Revision 1.4 2004/07/29 19:33:29 mgt +Chmod and removed perl call -mgt + +Revision 1.3 2004/07/10 01:54:35 mgt +sync with kirk -mgt + +Revision 1.13 2004/06/23 15:01:17 kirk +- Added more patches from blues@ds.pg.gda.pl + +Revision 1.12 2004/06/21 14:59:05 kirk +Added tons of patches from Pawe? Go?aszewski" +Thanks, as always! + +Revision 1.11 2004/06/21 13:42:02 kirk +From: Matthew Wise +This is more of a suggestion than a true patch submission. On a busy +postfix server the messages sent by section is really long and not +helpful. This patch finds and lists the top 10 senders by number of +messages. + +Revision 1.10 2004/06/21 13:41:04 kirk +Patch from rod@nayfield.com + +Revision 1.9.1 2004/02/22 16:44:01 rod +Added patch from rod@nayfield.com + +Revision 1.9 2004/02/03 03:25:02 kirk +Added patch from quien-sabe@metaorg.com + +Revision 1.8 2004/02/03 02:45:26 kirk +Tons of patches, and new 'oidentd' and 'shaperd' filters from +Pawe? Go?aszewski" + +Revision 1.7 2003/12/15 18:35:03 kirk +Tons of patches from blues@ds.pg.gda.pl + +Revision 1.6 2003/12/15 18:09:23 kirk +Added standard vi formatting commands at the bottom of all files. +Applied many patches from blues@ds.pg.gda.pl + +Revision 1.5 2003/12/15 17:45:09 kirk +Added clamAV update log filter from lars@spinn.dk + +Revision 1.4 2003/11/26 14:36:30 kirk +Applied patch from blues@ds.pg.gda.pl + +Revision 1.3 2003/11/18 14:04:05 kirk +More patches from blues@ds.pg.gda.pl + +Revision 1.2 2003/11/18 04:02:21 kirk +Patch from blues@ds.pg.gda.pl + +Revision 1.1 2003/11/03 04:49:18 kirk +Added postfix filter from Sven Conrad + +Revision 1.1 2002/03/29 15:32:14 kirk +Added some filters found in RH's release + +Revision ??? 2000/07/12 Simon Liddington +converted from sendmail to postfix Sven Conrad +added unknown users, relay denials + +Revision 1.1 2003/03/21 21:10 sven +Initial revision +filters all postfix/ messages diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..a3c11df --- /dev/null +++ b/LICENSE @@ -0,0 +1,17 @@ +########################################################################## +# Postfix-logwatch: written and maintained by: +# +# Mike "MrC" Cappella +# http://logreporters.sourceforge.net/ +# +# Please send all comments, suggestions, bug reports regarding this +# program/module to the email address above. I will respond as quickly +# as possible. [MrC] +# +####################################################### +### All work since Dec 12, 2006 (logwatch CVS revision 1.28) +### Copyright (c) 2006-2012 Mike Cappella +### +### Covered under the included MIT/X-Consortium License: +### http://www.opensource.org/licenses/mit-license.php +########################################################################## diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..089a7e7 --- /dev/null +++ b/Makefile @@ -0,0 +1,69 @@ +program = postfix-logwatch +program_logwatch = postfix + +src = $(program) \ + $(program).conf \ + $(program).conf-topn \ + $(program).1 \ + $(program).1.html \ + Changes Bugs Makefile README LICENSE + +prefix_logwatch = /etc/logwatch +prefix_standalone = /usr/local + + +scriptdir_logwatch = $(prefix_logwatch)/scripts/services +confdir_logwatch = $(prefix_logwatch)/conf/services + +scriptdir_standalone = $(prefix_standalone)/bin +confdir_standalone = $(prefix_standalone)/etc +mandir_standalone = $(prefix_standalone)/man/man1 + +toolsdir = ../tools + +INSTALL = /usr/bin/install -c + +install: + @echo 'Run "make install-logwatch" to install as a logwatch filter' + @echo 'Run "make install-standalone" to install as a standalone utility' + @echo 'Run "make install-all" to install both' + +install-logwatch: + $(INSTALL) -d -m 0755 $(DESTDIR)$(scriptdir_logwatch) $(DESTDIR)$(confdir_logwatch) || exit 1; + $(INSTALL) -m 0644 $(program) $(DESTDIR)$(scriptdir_logwatch)/$(program_logwatch) || exit 1; + $(INSTALL) -m 0644 $(program).conf $(DESTDIR)$(confdir_logwatch)/$(program_logwatch).conf || exit 1; + # removes taint mode (-T) to run under logwatch + perl -e 'while (<>) { $$.==1 and s/\s+-T$$//; print "$$_";}' -i $(DESTDIR)$(scriptdir_logwatch)/$(program_logwatch) + +install-standalone: + $(INSTALL) -d -m 0755 $(DESTDIR)$(scriptdir_standalone) $(DESTDIR)$(confdir_standalone) || exit 1; + $(INSTALL) -m 0755 $(program) $(DESTDIR)$(scriptdir_standalone)/$(program) || exit 1; + $(INSTALL) -m 0644 $(program).conf $(DESTDIR)$(confdir_standalone)/$(program).conf || exit 1; + $(INSTALL) -m 0644 $(program).1 $(DESTDIR)$(mandir_standalone)/$(program).1 || exit 1; + +install-all: install-logwatch install-standalone + +uninstall-logwatch: + -rm $(DESTDIR)$(scriptdir_logwatch)/$(program_logwatch) $(DESTDIR)$(confdir_logwatch)/$(program_logwatch).conf + +uninstall-standalone: + -rm $(DESTDIR)$(scriptdir_standalone)/$(program) $(DESTDIR)$(confdir_standalone)/$(program).conf \ + $(DESTDIR)$(mandir_standalone)/$(program).1 + +uninstall-all: uninstall-logwatch uninstall-standalone + +PKGDIR = /tmp/$(program)-package + +release: program htmlpage + vers=`egrep 'Version[ ]*=' $(program) | sed "s/.*'\(.*\)';/\1/"` ; \ + echo Preparing version $$vers; \ + rel=$(program)-$$vers ; \ + tar -czvf $${rel}.tgz --group=0 --owner=0 --mode=644 --transform=s",^,$${rel}/," $(src) ; \ + md5sum $${rel}.tgz > $${rel}.tgz.md5 ; \ + chmod 644 $${rel}.tgz $${rel}.tgz.md5 + +program: + $(toolsdir)/build_from_modules $(program) >| $(program); + +htmlpage: + groff -m mandoc -T ascii $(program).1 | $(toolsdir)/man2html -t 'Man page: $(program)(1)' >| $(program).1.html; diff --git a/README b/README new file mode 100644 index 0000000..1327bf1 --- /dev/null +++ b/README @@ -0,0 +1,90 @@ +##### Overview + +The postfix-logwatch(1) utility is a Postfix MTA log parser, that pro- +duces summaries, details, and statistics regarding the operation of +Postfix. + +This utility can be used as a standalone program, or as a Logwatch fil- +ter module to produce Postfix summary and detailed reports from within +Logwatch. + +A key feature of postfix-logwatch is its ability to produce a very wide +range of reports with data grouped and sorted as much as possible to +reduce noise and highlight patterns. Brief summary reports provide a +quick overview of general Postfix operations and message delivery, +calling out warnings that may require attention. Detailed reports pro- +vide easy to scan, hierarchically-arranged and organized information, +with as much or little detail as desired. + +##### Installation: standalone + +The included Makefile will install the postfix-logwatch utility +for you. Run: + + make install-standalone + +to install the utility and its configuration file. Installation +will default to /usr/local/bin and /usr/local/etc, respectively. + +##### Installation: logwatch + +To use postfix-logwatch as a logwatch script, the files: + + postfix-logwatch + postfix-logwatch.conf + +will need to be installed into one of logwatch's known directories. +To avoid overwriting your existing default logwatch filter files, +the enclosed files can be installed into the global logwatch +installation directory, which is typically: + + /etc/logwatch + +The included Makefile can be used to install the files into +/etc/logwatch for you. To install the filter into an existing +logwatch installation, run: + + make install-logwatch + +NOTE: the files postfix-logwatch and postfix-logwatch.conf must be +renamed (by removing the "-logwatch" suffix), for logwatch to +function correctly. The Makefile takes care of this. The +Makefile also disables perl's taint mode (-T) when running under +logwatch. If you install the files manually, be sure to remove +the -T from the first line of the postfix filter. + +For non-standard installations, you will need to determine your +global logwatch directory, and define "prefix" in the attached +Makefile. + +Alternatively, you can manually copy the files to their proper +locations: + + cp postfix-logwatch /etc/logwatch/scripts/services/postfix + cp postfix-logwatch.conf /etc/logwatch/conf/services/postfix.conf + [ remove -T from line 1 of /etc/logwatch/scripts/services/postfix ] + + Optional: + cp postfix-logwatch.1 /usr/local/man/man1/postfix-logwatch.1 + +##### Usage + +The postfix-logwatch utility is used standalone as: + + postfix-logwatch /path/to/maillog + +For brief help: + + postfix-logwatch --help + +To use within logwatch: + + logwatch --service postfix ... + +See the postfix-logwatch(1) man page for complete details, and +see comments in the postfix-logwatch.conf file for additional +information. + +Mike Cappella +mike (at) cappella (dot) us +last updated: 01/11/2012 diff --git a/postfix-logwatch b/postfix-logwatch new file mode 100644 index 0000000..3e4a673 --- /dev/null +++ b/postfix-logwatch @@ -0,0 +1,5305 @@ +#!/usr/bin/perl -T + +########################################################################## +# Postfix-logwatch: written and maintained by: +# +# Mike "MrC" Cappella +# http://logreporters.sourceforge.net/ +# +# Please send all comments, suggestions, bug reports regarding this +# program/module to the email address above. I will respond as quickly +# as possible. [MrC] +# +# Questions regarding the logwatch program itself should be directed to +# the logwatch project at: +# http://sourceforge.net/projects/logwatch/support +# +####################################################### +### All work since Dec 12, 2006 (logwatch CVS revision 1.28) +### Copyright (c) 2006-2012 Mike Cappella +### +### Covered under the included MIT/X-Consortium License: +### http://www.opensource.org/licenses/mit-license.php +### All modifications and contributions by other persons to +### this script are assumed to have been donated to the +### Logwatch project and thus assume the above copyright +### and licensing terms. If you want to make contributions +### under your own copyright or a different license this +### must be explicitly stated in the contribution an the +### Logwatch project reserves the right to not accept such +### contributions. If you have made significant +### contributions to this script and want to claim +### copyright please contact logwatch-devel@lists.sourceforge.net. +########################################################## + +########################################################################## +# The original postfix logwatch filter was written by +# Kenneth Porter, and has had many contributors over the years. +# +# CVS log removed: see Changes file for postfix-logwatch at +# http://logreporters.sourceforge.net/ +# or included with the standalone postfix-logwatch distribution +########################################################################## + +########################################################################## +# +# Test data included via inline comments starting with "#TD" +# + +#use Devel::Size qw(size total_size); + +package Logreporters; +use 5.008; +use strict; +use warnings; +no warnings "uninitialized"; +use re 'taint'; + +our $Version = '1.40.03'; +our $progname_prefix = 'postfix'; + +# Specifies the default configuration file for use in standalone mode. +my $config_file = "/usr/local/etc/${progname_prefix}-logwatch.conf"; + +# support postfix long (2.9+) or short queue ids +my $re_QID_s = qr/[A-Z\d]+/; +my $re_QID_l = qr/(?:NOQUEUE|[bcdfghjklmnpqrstvwxyzBCDFGHJKLMNPQRSTVWXYZ\d]+)/; +our $re_QID; + +our $re_DSN = qr/(?:(?:\d{3})?(?: ?\d\.\d\.\d)?)/; +our $re_DDD = qr/(?:(?:conn_use=\d+ )?delay=-?[\d.]+(?:, delays=[\d\/.]+)?(?:, dsn=[\d.]+)?)/; + +#MODULE: ../Logreporters/Utils.pm +package Logreporters::Utils; + +use 5.008; +use strict; +use re 'taint'; +use warnings; + +BEGIN { + use Exporter (); + use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION); + $VERSION = '1.003'; + @ISA = qw(Exporter); + @EXPORT = qw(&formathost &get_percentiles &get_percentiles2 &get_frequencies &commify &unitize + &get_usable_sectvars &add_section &begin_section_group &end_section_group + &get_version &unique_list); + @EXPORT_OK = qw(&gen_test_log); +} + +use subs qw (@EXPORT @EXPORT_OK); + + +# Formats IP and hostname for even column spacing +# +sub formathost($ $) { + # $_[0] : hostip + # $_[1] : hostname; + + if (! $Logreporters::Config::Opts{'unknown'} and $_[1] eq 'unknown') { + return $_[0]; + } + + return sprintf "%-$Logreporters::Config::Opts{'ipaddr_width'}s %s", + $_[0] eq '' ? '*unknown' : $_[0], + $_[1] eq '' ? '*unknown' : lc $_[1]; +} + +# Add a new section to the end of a section table +# +sub add_section($$$$$;$) { + my $sref = shift; + die "Improperly specified Section entry: $_[0]" if !defined $_[3]; + + my $entry = { + CLASS => 'DATA', + NAME => $_[0], + DETAIL => $_[1], + FMT => $_[2], + TITLE => $_[3], + }; + $entry->{'DIVISOR'} = $_[4] if defined $_[4]; + push @$sref, $entry; +} + +{ +my $group_level = 0; + +# Begin a new section group. Groups can nest. +# +sub begin_section_group($;@) { + my $sref = shift; + my $group_name = shift; + my $entry = { + CLASS => 'GROUP_BEGIN', + NAME => $group_name, + LEVEL => ++$group_level, + HEADERS => [ @_ ], + }; + push @$sref, $entry; +} + +# Ends a section group. +# +sub end_section_group($;@) { + my $sref = shift; + my $group_name = shift; + my $entry = { + CLASS => 'GROUP_END', + NAME => $group_name, + LEVEL => --$group_level, + FOOTERS => [ @_ ], + }; + push @$sref, $entry; +} +} + +# Generate and return a list of section table entries or +# limiter key names, skipping any formatting entries. +# If 'namesonly' is set, limiter key names are returned, +# otherwise an array of section array records is returned. +sub get_usable_sectvars(\@ $) { + my ($sectref,$namesonly) = @_; + my (@sect_list, %unique_names); + + foreach my $sref (@$sectref) { + #print "get_usable_sectvars: $sref->{NAME}\n"; + next unless $sref->{CLASS} eq 'DATA'; + if ($namesonly) { + $unique_names{$sref->{NAME}} = 1; + } + else { + push @sect_list, $sref; + } + } + # return list of unique names + if ($namesonly) { + return keys %unique_names; + } + return @sect_list; +} + +# Print program and version info, preceeded by an optional string, and exit. +# +sub get_version() { + + print STDOUT "@_\n" if ($_[0]); + print STDOUT "$Logreporters::progname: $Logreporters::Version\n"; + exit 0; +} + + +# Returns a list of percentile values given a +# sorted array of numeric values. Uses the formula: +# +# r = 1 + (p(n-1)/100) = i + d (Excel method) +# +# r = rank +# p = desired percentile +# n = number of items +# i = integer part +# d = decimal part +# +# Arg1 is an array ref to the sorted series +# Arg2 is a list of percentiles to use + +sub get_percentiles(\@ @) { + my ($aref,@plist) = @_; + my ($n, $last, $r, $d, $i, @vals, $Yp); + + $last = $#$aref; + $n = $last + 1; + #printf "%6d" x $n . "\n", @{$aref}; + + #printf "n: %4d, last: %d\n", $n, $last; + foreach my $p (@plist) { + $r = 1 + ($p * ($n - 1) / 100.0); + $i = int ($r); # integer part + # domain: $i = 1 .. n + if ($i == $n) { + $Yp = $aref->[$last]; + } + elsif ($i == 0) { + $Yp = $aref->[0]; + print "CAN'T HAPPEN: $Yp\n"; + } + else { + $d = $r - $i; # decimal part + #p = Y[i] + d(Y[i+1] - Y[i]), but since we're 0 based, use i=i-1 + $Yp = $aref->[$i-1] + ($d * ($aref->[$i] - $aref->[$i-1])); + } + #printf "\np(%6.2f), r: %6.2f, i: %6d, d: %6.2f, Yp: %6d", $p, $r, $i, $d, $Yp; + push @vals, $Yp; + } + + return @vals; +} + +sub get_num_scores($) { + my $scoretab_r = shift; + + my $totalscores = 0; + + for (my $i = 0; $i < @$scoretab_r; $i += 2) { + $totalscores += $scoretab_r->[$i+1] + } + + return $totalscores; +} + +# scoretab +# +# (score1, n1), (score2, n2), ... (scoreN, nN) +# $i $i+1 +# +# scores are 0 based (0 = 1st score) +sub get_nth_score($ $) { + my ($scoretab_r, $n) = @_; + + my $i = 0; + my $n_cur_scores = 0; + #print "Byscore (", .5 * @$scoretab_r, "): "; for (my $i = 0; $i < $#$scoretab_r / 2; $i++) { printf "%9s (%d) ", $scoretab_r->[$i], $scoretab_r->[$i+1]; } ; print "\n"; + + while ($i < $#$scoretab_r) { + #print "Samples_seen: $n_cur_scores\n"; + $n_cur_scores += $scoretab_r->[$i+1]; + if ($n_cur_scores >= $n) { + #printf "range: %s %s %s\n", $i >= 2 ? $scoretab_r->[$i - 2] : '', $scoretab_r->[$i], $i+2 > $#$scoretab_r ? '' : $scoretab_r->[$i + 2]; + #printf "n: $n, i: %8d, n_cur_scores: %8d, score: %d x %d hits\n", $i, $n_cur_scores, $scoretab_r->[$i], $scoretab_r->[$i+1]; + return $scoretab_r->[$i]; + } + + $i += 2; + } + print "returning last score $scoretab_r->[$i]\n"; + return $scoretab_r->[$i]; +} + +sub get_percentiles2(\@ @) { + my ($scoretab_r, @plist) = @_; + my ($n, $last, $r, $d, $i, @vals, $Yp); + + #$last = $#$scoretab_r - 1; + $n = get_num_scores($scoretab_r); + #printf "\n%6d" x $n . "\n", @{$scoretab_r}; + + #printf "\n\tn: %4d, @$scoretab_r\n", $n; + foreach my $p (@plist) { + ###print "\nPERCENTILE: $p\n"; + $r = 1 + ($p * ($n - 1) / 100.0); + $i = int ($r); # integer part + if ($i == $n) { + #print "last:\n"; + #$Yp = $scoretab_r->[$last]; + $Yp = get_nth_score($scoretab_r, $n); + } + elsif ($i == 0) { + #$Yp = $scoretab_r->[0]; + print "1st: CAN'T HAPPEN\n"; + $Yp = get_nth_score($scoretab_r, 1); + } + else { + $d = $r - $i; # decimal part + #p = Y[i] + d(Y[i+1] - Y[i]), but since we're 0 based, use i=i-1 + my $ithvalprev = get_nth_score($scoretab_r, $i); + my $ithval = get_nth_score($scoretab_r, $i+1); + $Yp = $ithvalprev + ($d * ($ithval - $ithvalprev)); + } + #printf "p(%6.2f), r: %6.2f, i: %6d, d: %6.2f, Yp: %6d\n", $p, $r, $i, $d, $Yp; + push @vals, $Yp; + } + + return @vals; +} + + + +# Returns a list of frequency distributions given an incrementally sorted +# set of sorted scores, and an incrementally sorted list of buckets +# +# Arg1 is an array ref to the sorted series +# Arg2 is a list of frequency buckets to use +sub get_frequencies(\@ @) { + my ($aref,@blist) = @_; + + my @vals = ( 0 ) x (@blist); + my @sorted_blist = sort { $a <=> $b } @blist; + my $bucket_index = 0; + +OUTER: foreach my $score (@$aref) { + #print "Score: $score\n"; + for my $i ($bucket_index .. @sorted_blist - 1) { + #print "\tTrying Bucket[$i]: $sorted_blist[$i]\n"; + if ($score > $sorted_blist[$i]) { + $bucket_index++; + } + else { + #printf "\t\tinto Bucket[%d]\n", $bucket_index; + $vals[$bucket_index]++; + next OUTER; + } + } + #printf "\t\tinto Bucket[%d]\n", $bucket_index - 1; + $vals[$bucket_index - 1]++; + } + + return @vals; +} + +# Inserts commas in numbers for easier readability +# +sub commify ($) { + return undef if ! defined ($_[0]); + + my $text = reverse $_[0]; + $text =~ s/(\d\d\d)(?=\d)(?!\d*\.)/$1,/g; + return scalar reverse $text; +} + +# Unitize a number, and return appropriate printf formatting string +# +sub unitize($ $) { + my ($num, $fmt) = @_; + my $kilobyte = 2**10; + my $megabyte = 2**20; + my $gigabyte = 2**30; + my $terabyte = 2**40; + + if ($num >= $terabyte) { + $num /= $terabyte; + $fmt .= '.3fT'; + } elsif ($num >= $gigabyte) { + $num /= $gigabyte; + $fmt .= '.3fG'; + } elsif ($num >= $megabyte) { + $num /= $megabyte; + $fmt .= '.3fM'; + } elsif ($num >= $kilobyte) { + $num /= $kilobyte; + $fmt .= '.3fK'; + } else { + $fmt .= 'd '; + } + + return ($num, $fmt); +} + +# Returns a sublist of the supplied list of elements in an unchanged order, +# where only the first occurrence of each defined element is retained +# and duplicates removed +# +# Borrowed from amavis 2.6.2 +# +sub unique_list(@) { + my ($r) = @_ == 1 && ref($_[0]) ? $_[0] : \@_; # accept list, or a list ref + my (%seen); + my (@unique) = grep { defined($_) && !$seen{$_}++ } @$r; + + return @unique; +} + +# Generate a test maillog file from the '#TD' test data lines +# The test data file is placed in /var/tmp/maillog.autogen +# +# arg1: "postfix" or "amavis" +# arg2: path to postfix-logwatch or amavis-logwatch from which to read '#TD' data +# +# Postfix TD syntax: +# TD() log entry +# +sub gen_test_log($) { + my $scriptpath = shift; + + my $toolname = $Logreporters::progname_prefix; + my $datafile = "/var/tmp/maillog-${toolname}.autogen"; + + die "gen_test_log: invalid toolname $toolname" if ($toolname !~ /^(postfix|amavis)$/); + + eval { + require Sys::Hostname; + require Fcntl; + } or die "Unable to create test data file: required module(s) not found\n$@"; + + my $syslogtime = localtime; + $syslogtime =~ s/^....(.*) \d{4}$/$1/; + + my ($hostname) = split /\./, Sys::Hostname::hostname(); + + # # avoid -T issues + # delete @ENV{'IFS', 'CDPATH', 'ENV', 'BASH_ENV'}; + + my $flags = &Fcntl::O_CREAT|&Fcntl::O_WRONLY|&Fcntl::O_TRUNC; + sysopen(FH, $datafile, $flags) or die "Can't create test data file: $!"; + print "Generating test log data file from $scriptpath: $datafile\n"; + + my $id; + @ARGV = ($scriptpath); + if ($toolname eq 'postfix') { + my %services = ( + DEF => 'smtpd', + bQ => 'bounce', + cN => 'cleanup', + cQ => 'cleanup', + lQ => 'local', + m => 'master', + p => 'pickup', + pQ => 'pickup', + ppQ => 'pipe', + pfw => 'postfwd', + pg => 'postgrey', + pgQ => 'postgrey', + ps => 'postsuper', + qQ => 'qmgr', + s => 'smtp', + sQ => 'smtp', + sd => 'smtpd', + sdN => 'smtpd', + sdQ => 'smtpd', + spf => 'policy-spf', + vN => 'virtual', + vQ => 'virtual', + ); + $id = 'postfix/smtp[12345]'; + + while (<>) { + if (/^\s*#TD([a-zA-Z]*[NQ]?)(\d+)?(?:\(([^)]+)\))? (.*)$/) { + my ($service,$count,$qid,$line) = ($1, $2, $3, $4); + + #print "SERVICE: %s, QID: %s, COUNT: %s, line: %s\n", $service, $qid, $count, $line; + + if ($service eq '') { + $service = 'DEF'; + } + die ("No such service: \"$service\": line \"$_\"") if (!exists $services{$service}); + + $id = $services{$service} . '[123]'; + $id = 'postfix/' . $id unless $services{$service} eq 'postgrey'; + #print "searching for service: \"$service\"\n\tFound $id\n"; + if ($service =~ /N$/) { $id .= ': NOQUEUE'; } + elsif ($service =~ /Q$/) { $id .= $qid ? $qid : ': DEADBEEF'; } + + $line =~ s/ +/ /g; + $line =~ s/^ //g; + #print "$syslogtime $hostname $id: \"$line\"\n" x ($count ? $count : 1); + print FH "$syslogtime $hostname $id: $line\n" x ($count ? $count : 1); + } + } + } + else { #amavis + my %services = ( + DEF => 'amavis', + dcc => 'dccproc', + ); + while (<>) { + if (/^\s*#TD([a-z]*)(\d+)? (.*)$/) { + my ($service,$count,$line) = ($1, $2, $3); + if ($service eq '') { + $service = 'DEF'; + } + die ("No such service: \"$service\": line \"$_\"") if (!exists $services{$service}); + $id = $services{$service} . '[123]:'; + if ($services{$service} eq 'amavis') { + $id .= ' (9999-99)'; + } + print FH "$syslogtime $hostname $id $line\n" x ($count ? $count : 1) + } + } + } + + close FH or die "Can't close $datafile: $!"; +} + +1; + +#MODULE: ../Logreporters/Config.pm +package Logreporters::Config; + +use 5.008; +use strict; +use re 'taint'; +use warnings; + + +BEGIN { + use Exporter (); + use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION); + $VERSION = '1.002'; + @ISA = qw(Exporter); + @EXPORT = qw(&init_run_mode &add_option &get_options &init_cmdline &get_vars_from_file + &process_limiters &process_debug_opts &init_getopts_table_common &zero_opts + @Optspec %Opts %Configvars @Limiters %line_styles $fw1 $fw2 $sep1 $sep2 + &D_CONFIG &D_ARGS &D_VARS &D_TREE &D_SECT &D_UNMATCHED &D_TEST &D_ALL + ); +} + +use subs @EXPORT; + +our @Optspec = (); # options table used by Getopts + +our %Opts = (); # program-wide options +our %Configvars = (); # configuration file variables +our @Limiters; + +# Report separator characters and widths +our ($fw1,$fw2) = (22, 10); +our ($sep1,$sep2) = ('=', '-'); + +use Getopt::Long; + + +BEGIN { + import Logreporters::Utils qw(&get_usable_sectvars); +} + +our %line_styles = ( + truncate => 0, + wrap => 1, + full => 2, +); + +sub init_run_mode($); +sub confighash_to_cmdline(\%); +sub get_vars_from_file(\% $); +sub process_limiters(\@); +sub add_option(@); +sub get_options($); +sub init_getopts_table_common(@); +sub set_supplemental_reports($$); +# debug constants +sub D_CONFIG () { 1<<0 } +sub D_ARGS () { 1<<1 } +sub D_VARS () { 1<<2 } +sub D_TREE () { 1<<3 } +sub D_SECT () { 1<<4 } +sub D_UNMATCHED () { 1<<5 } + +sub D_TEST () { 1<<30 } +sub D_ALL () { 1<<31 } + +my %debug_words = ( + config => D_CONFIG, + args => D_ARGS, + vars => D_VARS, + tree => D_TREE, + sect => D_SECT, + unmatched => D_UNMATCHED, + + test => D_TEST, + all => 0xffffffff, +); + +# Clears %Opts hash and initializes basic running mode options in +# %Opts hash by setting keys: 'standalone', 'detail', and 'debug'. +# Call early. +# +sub init_run_mode($) { + my $config_file = shift; + $Opts{'debug'} = 0; + + # Logwatch passes a filter's options via environment variables. + # When running standalone (w/out logwatch), use command line options + $Opts{'standalone'} = exists ($ENV{LOGWATCH_DETAIL_LEVEL}) ? 0 : 1; + + # Show summary section by default + $Opts{'summary'} = 1; + + if ($Opts{'standalone'}) { + process_debug_opts($ENV{'LOGREPORTERS_DEBUG'}) if exists ($ENV{'LOGREPORTERS_DEBUG'}); + } + else { + $Opts{'detail'} = $ENV{'LOGWATCH_DETAIL_LEVEL'}; + # XXX + #process_debug_opts($ENV{'LOGWATCH_DEBUG'}) if exists ($ENV{'LOGWATCH_DEBUG'}); + } + + # first process --debug, --help, and --version options + add_option ('debug=s', sub { process_debug_opts($_[1]); 1}); + add_option ('version', sub { &Logreporters::Utils::get_version(); 1;}); + get_options(1); + + # now process --config_file, so that all config file vars are read first + add_option ('config_file|f=s', sub { get_vars_from_file(%Configvars, $_[1]); 1;}); + get_options(1); + + # if no config file vars were read + if ($Opts{'standalone'} and ! keys(%Configvars) and -f $config_file) { + print "Using default config file: $config_file\n" if $Opts{'debug'} & D_CONFIG; + get_vars_from_file(%Configvars, $config_file); + } +} + +sub get_options($) { + my $pass_through = shift; + #$SIG{__WARN__} = sub { print "*** $_[0]*** options error\n" }; + # ensure we're called after %Opts is initialized + die "get_options: program error: %Opts is emtpy" unless exists $Opts{'debug'}; + + my $p = new Getopt::Long::Parser; + + if ($pass_through) { + $p->configure(qw(pass_through permute)); + } + else { + $p->configure(qw(no_pass_through no_permute)); + } + #$p->configure(qw(debug)); + + if ($Opts{'debug'} & D_ARGS) { + print "\nget_options($pass_through): enter\n"; + printf "\tARGV(%d): ", scalar @ARGV; + print @ARGV, "\n"; + print "\t$_ ", defined $Opts{$_} ? "=> $Opts{$_}\n" : "\n" foreach sort keys %Opts; + } + + if ($p->getoptions(\%Opts, @Optspec) == 0) { + print STDERR "Use ${Logreporters::progname} --help for options\n"; + exit 1; + } + if ($Opts{'debug'} & D_ARGS) { + print "\t$_ ", defined $Opts{$_} ? "=> $Opts{$_}\n" : "\n" foreach sort keys %Opts; + printf "\tARGV(%d): ", scalar @ARGV; + print @ARGV, "\n"; + print "get_options: exit\n"; + } +} + +sub add_option(@) { + push @Optspec, @_; +} + +# untaint string, borrowed from amavisd-new +sub untaint($) { + no re 'taint'; + + my ($str); + if (defined($_[0])) { + local($1); # avoid Perl taint bug: tainted global $1 propagates taintedness + $str = $1 if $_[0] =~ /^(.*)$/; + } + + return $str; +} + +sub init_getopts_table_common(@) { + my @supplemental_reports = @_; + + print "init_getopts_table_common: enter\n" if $Opts{'debug'} & D_ARGS; + + add_option ('help', sub { print STDOUT Logreporters::usage(undef); exit 0 }); + add_option ('gen_test_log=s', sub { Logreporters::Utils::gen_test_log($_[1]); exit 0; }); + add_option ('detail=i'); + add_option ('nodetail', sub { + # __none__ will set all limiters to 0 in process_limiters + # since they are not known (Sections table is not yet built). + push @Limiters, '__none__'; + # 0 = disable supplemental_reports + set_supplemental_reports(0, \@supplemental_reports); + }); + add_option ('max_report_width=i'); + add_option ('summary!'); + add_option ('show_summary=i', sub { $Opts{'summary'} = $_[1]; 1; }); + # untaint ipaddr_width for use w/sprintf() in Perl v5.10 + add_option ('ipaddr_width=i', sub { $Opts{'ipaddr_width'} = untaint ($_[1]); 1; }); + + add_option ('sect_vars!'); + add_option ('show_sect_vars=i', sub { $Opts{'sect_vars'} = $_[1]; 1; }); + + add_option ('syslog_name=s'); + add_option ('wrap', sub { $Opts{'line_style'} = $line_styles{$_[0]}; 1; }); + add_option ('full', sub { $Opts{'line_style'} = $line_styles{$_[0]}; 1; }); + add_option ('truncate', sub { $Opts{'line_style'} = $line_styles{$_[0]}; 1; }); + add_option ('line_style=s', sub { + my $style = lc($_[1]); + my @list = grep (/^$style/, keys %line_styles); + if (! @list) { + print STDERR "Invalid line_style argument \"$_[1]\"\n"; + print STDERR "Option line_style argument must be one of \"wrap\", \"full\", or \"truncate\".\n"; + print STDERR "Use $Logreporters::progname --help for options\n"; + exit 1; + } + $Opts{'line_style'} = $line_styles{lc($list[0])}; + 1; + }); + + add_option ('limit|l=s', sub { + my ($limiter,$lspec) = split(/=/, $_[1]); + if (!defined $lspec) { + printf STDERR "Limiter \"%s\" requires value (ex. --limit %s=10)\n", $_[1],$_[1]; + exit 2; + } + foreach my $val (split(/(?:\s+|\s*,\s*)/, $lspec)) { + if ($val !~ /^\d+$/ and + $val !~ /^(\d*)\.(\d+)$/ and + $val !~ /^::(\d+)$/ and + $val !~ /^:(\d+):(\d+)?$/ and + $val !~ /^(\d+):(\d+)?:(\d+)?$/) + { + printf STDERR "Limiter value \"$val\" invalid in \"$limiter=$lspec\"\n"; + exit 2; + } + } + push @Limiters, lc $_[1]; + }); + + print "init_getopts_table_common: exit\n" if $Opts{'debug'} & D_ARGS; +} + +sub get_option_names() { + my (@ret, @tmp); + foreach (@Optspec) { + if (ref($_) eq '') { # process only the option names + my $spec = $_; + $spec =~ s/=.*$//; + $spec =~ s/([^|]+)\!$/$1|no$1/g; + @tmp = split /[|]/, $spec; + #print "PUSHING: @tmp\n"; + push @ret, @tmp; + } + } + return @ret; +} + +# Set values for the configuration variables passed via hashref. +# Variables are of the form ${progname_prefix}_KEYNAME. +# +# Because logwatch lowercases all config file entries, KEYNAME is +# case-insensitive. +# +sub init_cmdline() { + my ($href, $configvar, $value, $var); + + # logwatch passes all config vars via environment variables + $href = $Opts{'standalone'} ? \%Configvars : \%ENV; + + # XXX: this is cheeze: need a list of valid limiters, but since + # the Sections table is not built yet, we don't know what is + # a limiter and what is an option, as there is no distinction in + # variable names in the config file (perhaps this should be changed). + my @valid_option_names = get_option_names(); + die "Options table not yet set" if ! scalar @valid_option_names; + + print "confighash_to_cmdline: @valid_option_names\n" if $Opts{'debug'} & D_ARGS; + my @cmdline = (); + while (($configvar, $value) = each %$href) { + if ($configvar =~ s/^${Logreporters::progname_prefix}_//o) { + # distinguish level limiters from general options + # would be easier if limiters had a unique prefix + $configvar = lc $configvar; + my $ret = grep (/^$configvar$/i, @valid_option_names); + if ($ret == 0) { + print "\tLIMITER($ret): $configvar = $value\n" if $Opts{'debug'} & D_ARGS; + push @cmdline, '-l', "$configvar" . "=$value"; + } + else { + print "\tOPTION($ret): $configvar = $value\n" if $Opts{'debug'} & D_ARGS; + unshift @cmdline, $value if defined ($value); + unshift @cmdline, "--$configvar"; + } + } + } + unshift @ARGV, @cmdline; +} + +# Obtains the variables from a logwatch-style .conf file, for use +# in standalone mode. Returns an ENV-style hash of key/value pairs. +# +sub get_vars_from_file(\% $) { + my ($href, $file) = @_; + my ($var, $val); + + print "get_vars_from_file: enter: processing file: $file\n" if $Opts{'debug'} & D_CONFIG; + + my $message = undef; + my $ret = stat ($file); + if ($ret == 0) { $message = $!; } + elsif (! -r _) { $message = "Permission denied"; } + elsif ( -d _) { $message = "Is a directory"; } + elsif (! -f _) { $message = "Not a regular file"; } + + if ($message) { + print STDERR "Configuration file \"$file\": $message\n"; + exit 2; + } + + my $prog = $Logreporters::progname_prefix; + open FILE, '<', "$file" or die "unable to open configuration file $file: $!"; + while () { + chomp; + next if (/^\s*$/); # ignore all whitespace lines + next if (/^\*/); # ignore logwatch's *Service lines + next if (/^\s*#/); # ignore comment lines + if (/^\s*\$(${prog}_[^=\s]+)\s*=\s*"?([^"]+)"?$/o) { + ($var,$val) = ($1,$2); + if ($val =~ /^(?:no|false)$/i) { $val = 0; } + elsif ($val =~ /^(?:yes|true)$/i) { $val = 1; } + elsif ($val eq '') { $var =~ s/${prog}_/${prog}_no/; $val = undef; } + + print "\t\"$var\" => \"$val\"\n" if $Opts{'debug'} & D_CONFIG; + + $href->{$var} = $val; + } + } + close FILE or die "failed to close configuration handle for $file: $!"; + print "get_vars_from_file: exit\n" if $Opts{'debug'} & D_CONFIG; +} + +sub process_limiters(\@) { + my ($sectref) = @_; + + my ($limiter, $var, $val, @errors); + my @l = get_usable_sectvars(@$sectref, 1); + + if ($Opts{'debug'} & D_VARS) { + print "process_limiters: enter\n"; + print "\tLIMITERS: @Limiters\n"; + } + while ($limiter = shift @Limiters) { + my @matched = (); + + printf "\t%-30s ",$limiter if $Opts{'debug'} & D_VARS; + # disable all limiters when limiter is __none__: see 'nodetail' cmdline option + if ($limiter eq '__none__') { + $Opts{$_} = 0 foreach @l; + next; + } + + ($var,$val) = split /=/, $limiter; + + if ($val eq '') { + push @errors, "Limiter \"$var\" requires value (ex. --limit limiter=10)"; + next; + } + + # try exact match first, then abbreviated match next + if (scalar (@matched = grep(/^$var$/, @l)) == 1 or scalar (@matched = grep(/^$var/, @l)) == 1) { + $limiter = $matched[0]; # unabbreviate limiter + print "MATCH: $var: $limiter => $val\n" if $Opts{'debug'} & D_VARS; + # XXX move limiters into section hash entry... + $Opts{$limiter} = $val; + next; + } + print "matched=", scalar @matched, ": @matched\n" if $Opts{'debug'} & D_VARS; + + push @errors, "Limiter \"$var\" is " . (scalar @matched == 0 ? "invalid" : "ambiguous: @matched"); + } + print "\n" if $Opts{'debug'} & D_VARS; + + if (@errors) { + print STDERR "$_\n" foreach @errors; + exit 2; + } + + # Set the default value of 10 for each section if no limiter exists. + # This allows output for each section should there be no configuration + # file or missing limiter within the configuration file. + foreach (@l) { + $Opts{$_} = 10 unless exists $Opts{$_}; + } + + # Enable collection for each section if a limiter is non-zero. + foreach (@l) { + #print "L is: $_\n"; + #print "DETAIL: $Opts{'detail'}, OPTS: $Opts{$_}\n"; + $Logreporters::TreeData::Collecting{$_} = (($Opts{'detail'} >= 5) && $Opts{$_}) ? 1 : 0; + } + #print "OPTS: \n"; map { print "$_ => $Opts{$_}\n"} keys %Opts; + #print "COLLECTING: \n"; map { print "$_ => $Logreporters::TreeData::Collecting{$_}\n"} keys %Logreporters::TreeData::Collecting; +} + +# Enable/disable supplemental reports +# arg1: 0=off, 1=on +# arg2,...: list of supplemental report keywords +sub set_supplemental_reports($$) { + my ($onoff,$aref) = @_; + + $Opts{$_} = $onoff foreach (@$aref); +} + +sub process_debug_opts($) { + my $optstring = shift; + + my @errors = (); + foreach (split(/\s*,\s*/, $optstring)) { + my $word = lc $_; + my @matched = grep (/^$word/, keys %debug_words); + + if (scalar @matched == 1) { + $Opts{'debug'} |= $debug_words{$matched[0]}; + next; + } + + if (scalar @matched == 0) { + push @errors, "Unknown debug keyword \"$word\""; + } + else { # > 1 + push @errors, "Ambiguous debug keyword abbreviation \"$word\": (matches: @matched)"; + } + } + if (@errors) { + print STDERR "$_\n" foreach @errors; + print STDERR "Debug keywords: ", join (' ', sort keys %debug_words), "\n"; + exit 2; + } +} + +# Zero the options controlling level specs and those +# any others passed via Opts key. +# +# Zero the options controlling level specs in the +# Detailed section, and set all other report options +# to disabled. This makes it easy via command line to +# disable the entire summary section, and then re-enable +# one or more sections for specific reports. +# +# eg. progname --nodetail --limit forwarded=2 +# +sub zero_opts ($ @) { + my $sectref = shift; + # remaining args: list of Opts keys to zero + + map { $Opts{$_} = 0; print "zero_opts: $_ => 0\n" if $Opts{'debug'} & D_VARS;} @_; + map { $Opts{$_} = 0 } get_usable_sectvars(@$sectref, 1); +} + +1; + +#MODULE: ../Logreporters/TreeData.pm +package Logreporters::TreeData; + +use 5.008; +use strict; +use re 'taint'; +use warnings; +no warnings "uninitialized"; + +BEGIN { + use Exporter (); + use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION); + $VERSION = '1.001'; + @ISA = qw(Exporter); + @EXPORT = qw(%Totals %Counts %Collecting $END_KEY); + @EXPORT_OK = qw(&printTree &buildTree); + +} + +use subs @EXPORT_OK; + +BEGIN { + import Logreporters::Config qw(%line_styles); +} + +# Totals and Counts are the log line accumulator hashes. +# Totals: maintains per-section grand total tallies for use in Summary section +# Counts: is a multi-level hash, which maintains per-level key totals. +our (%Totals, %Counts); + +# The Collecting hash determines which sections will be captured in +# the Counts hash. Counts are collected only if a section is enabled, +# and this hash obviates the need to test both existence and +# non-zero-ness of the Opts{'keyname'} (either of which cause capture). +# XXX The Opts hash could be used .... +our %Collecting = (); + +sub buildTree(\% $ $ $ $ $); +sub printTree($ $ $ $ $); +=pod +[ a:b:c, ... ] + +which would be interpreted as follows: + +a = show level a detail +b = show at most b items at this level +c = minimun count that will be shown +=cut + +sub printTree($ $ $ $ $) { + my ($treeref, $lspecsref, $line_style, $max_report_width, $debug) = @_; + my ($entry, $line); + my $cutlength = $max_report_width - 3; + + my $topn = 0; + foreach $entry (sort bycount @$treeref) { + ref($entry) ne "HASH" and die "Unexpected entry in tree: $entry\n"; + + #print "LEVEL: $entry->{LEVEL}, TOTAL: $entry->{TOTAL}, HASH: $entry, DATA: $entry->{DATA}\n"; + + # Once the top N lines have been printed, we're done + if ($lspecsref->[$entry->{LEVEL}]{topn}) { + if ($topn++ >= $lspecsref->[$entry->{LEVEL}]{topn} ) { + print ' ', ' ' x ($entry->{LEVEL} + 3), "...\n" + unless ($debug) and do { + $line = ' ' . ' ' x ($entry->{LEVEL} + 3) . '...'; + printf "%-130s L%d: topn reached(%d)\n", $line, $entry->{LEVEL} + 1, $lspecsref->[$entry->{LEVEL}]{topn}; + }; + last; + } + } + + # Once the item's count falls below the given threshold, we're done at this level + # unless a top N is specified, as threshold has lower priority than top N + elsif ($lspecsref->[$entry->{LEVEL}]{threshold}) { + if ($entry->{TOTAL} <= $lspecsref->[$entry->{LEVEL}]{threshold}) { + print ' ', ' ' x ($entry->{LEVEL} + 3), "...\n" + unless ($debug) and do { + $line = ' ' . (' ' x ($entry->{LEVEL} + 3)) . '...'; + printf "%-130s L%d: threshold reached(%d)\n", $line, $entry->{LEVEL} + 1, $lspecsref->[$entry->{LEVEL}]{threshold}; + }; + last; + } + } + + $line = sprintf "%8d%s%s", $entry->{TOTAL}, ' ' x ($entry->{LEVEL} + 2), $entry->{DATA}; + + if ($debug) { + printf "%-130s %-60s\n", $line, $entry->{DEBUG}; + } + + # line_style full, or lines < max_report_width + + #printf "MAX: $max_report_width, LEN: %d, CUTLEN $cutlength\n", length($line); + if ($line_style == $line_styles{'full'} or length($line) <= $max_report_width) { + print $line, "\n"; + } + elsif ($line_style == $line_styles{'truncate'}) { + print substr ($line,0,$cutlength), '...', "\n"; + } + elsif ($line_style == $line_styles{'wrap'}) { + my $leader = ' ' x 8 . ' ' x ($entry->{LEVEL} + 2); + print substr ($line, 0, $max_report_width, ''), "\n"; + while (length($line)) { + print $leader, substr ($line, 0, $max_report_width - length($leader), ''), "\n"; + } + } + else { + die ('unexpected line style'); + } + + printTree ($entry->{CHILDREF}, $lspecsref, $line_style, $max_report_width, $debug) if (exists $entry->{CHILDREF}); + } +} + +my $re_IP_strict = qr/\b(25[0-5]|2[0-4]\d|[01]?\d{1,2})\.(25[0-5]|2[0-4]\d|[01]?\d{1,2})\.(25[0-5]|2[0-4]\d|[01]?\d{1,2})\.(25[0-5]|2[0-4]\d|[01]?\d{1,2})\b/; +# XXX optimize this using packed default sorting. Analysis shows speed isn't an issue though +sub bycount { + # Sort by totals, then IP address if one exists, and finally by data as a string + + local $SIG{__WARN__} = sub { print "*** PLEASE REPORT:\n*** $_[0]*** Unexpected: \"$a->{DATA}\", \"$b->{DATA}\"\n" }; + + $b->{TOTAL} <=> $a->{TOTAL} + + || + + pack('C4' => $a->{DATA} =~ /^$re_IP_strict/o) cmp pack('C4' => $b->{DATA} =~ /^$re_IP_strict/o) + + || + + $a->{DATA} cmp $b->{DATA} +} + +# +# Builds a tree of REC structures from the multi-key %Counts hashes +# +# Parameters: +# Hash: A multi-key hash, with keys being used as category headings, and leaf data +# being tallies for that set of keys +# Level: This current recursion level. Call with 0. +# +# Returns: +# Listref: A listref, where each item in the list is a rec record, described as: +# DATA: a string: a heading, or log data +# TOTAL: an integer: which is the subtotal of this item's children +# LEVEL: an integer > 0: representing this entry's level in the tree +# CHILDREF: a listref: references a list consisting of this node's children +# Total: The cummulative total of items found for a given invocation +# +# Use the special key variable $END_KEY, which is "\a\a" (two ASCII bell's) to end a, +# nested hash early, or the empty string '' may be used as the last key. + +our $END_KEY = "\a\a"; + +sub buildTree(\% $ $ $ $ $) { + my ($href, $max_level_section, $levspecref, $max_level_global, $recurs_level, $show_unique, $debug) = @_; + my ($subtotal, $childList, $rec); + + my @treeList = (); + my $total = 0; + + foreach my $item (sort keys %$href) { + if (ref($href->{$item}) eq "HASH") { + #print " " x ($recurs_level * 4), "HASH: LEVEL $recurs_level: Item: $item, type: \"", ref($href->{$item}), "\"\n"; + + ($subtotal, $childList) = buildTree (%{$href->{$item}}, $max_level_section, $levspecref, $max_level_global, $recurs_level + 1, $debug); + + if ($recurs_level < $max_level_global and $recurs_level < $max_level_section) { + # me + children + $rec = { + DATA => $item, + TOTAL => $subtotal, + LEVEL => $recurs_level, + CHILDREF => $childList, + }; + + if ($debug) { + $rec->{DEBUG} = sprintf "L%d: levelspecs: %2d/%2d/%2d/%2d, Count: %10d", + $recurs_level + 1, $max_level_global, $max_level_section, + $levspecref->[$recurs_level]{topn}, $levspecref->[$recurs_level]{threshold}, $subtotal; + } + push (@treeList, $rec); + } + } + else { + if ($item ne '' and $item ne $END_KEY and $recurs_level < $max_level_global and $recurs_level < $max_level_section) { + $rec = { + DATA => $item, + TOTAL => $href->{$item}, + LEVEL => $recurs_level, + #CHILDREF => undef, + }; + if ($debug) { + $rec->{DEBUG} = sprintf "L%d: levelspecs: %2d/%2d/%2d/%2d, Count: %10d", + $recurs_level, $max_level_global, $max_level_section, + $levspecref->[$recurs_level]{topn}, $levspecref->[$recurs_level]{threshold}, $href->{$item}; + } + push (@treeList, $rec); + } + $subtotal = $href->{$item}; + } + + $total += $subtotal; + } + + #print " " x ($recurs_level * 4), "LEVEL $recurs_level: Returning from recurs_level $recurs_level\n"; + + return ($total, \@treeList); +} + +1; + +#MODULE: ../Logreporters/RegEx.pm +package Logreporters::RegEx; + +use 5.008; +use strict; +use re 'taint'; +use warnings; + +BEGIN { + use Exporter (); + use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION); + $VERSION = '1.000'; + @ISA = qw(Exporter); +# @EXPORT = qw($re_IP); + @EXPORT_OK = qw(); +} + +# IPv4 and IPv6 +# See syntax in RFC 2821 IPv6-address-literal, +# eg. IPv6:2001:630:d0:f102:230:48ff:fe77:96e +#our $re_IP = '(?:(?:::(?:ffff:|FFFF:)?)?(?:\d{1,3}\.){3}\d{1,3}|(?:(?:IPv6:)?[\da-fA-F]{0,4}:){2}(?:[\da-fA-F]{0,4}:){0,5}[\da-fA-F]{0,4})'; + +# Modified from "dartware" case at http://forums.dartware.com/viewtopic.php?t=452# +#our $re_IP = qr/(?:(?:(?:(?:[\da-f]{1,4}:){7}(?:[\da-f]{1,4}|:))|(?:(?:[\da-f]{1,4}:){6}(?::[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(?:(?:[\da-f]{1,4}:){5}(?:(?:(?::[\da-f]{1,4}){1,2})|:(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(?:(?:[\da-f]{1,4}:){4}(?:(?:(?::[\da-f]{1,4}){1,3})|(?:(?::[\da-f]{1,4})?:(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(?:(?:[\da-f]{1,4}:){3}(?:(?:(?::[\da-f]{1,4}){1,4})|(?:(?::[\da-f]{1,4}){0,2}:(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(?:(?:[\da-f]{1,4}:){2}(?:(?:(?::[\da-f]{1,4}){1,5})|(?:(?::[\da-f]{1,4}){0,3}:(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(?:(?:[\da-f]{1,4}:){1}(?:(?:(?::[\da-f]{1,4}){1,6})|(?:(?::[\da-f]{1,4}){0,4}:(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(?::(?:(?:(?::[\da-f]{1,4}){1,7})|(?:(?::[\da-f]{1,4}){0,5}:(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(?:%.+)?)|(?:(?:\d{1,3}\.){3}(?:\d{1,3}))/i; + +# IPv4 only +#our $re_IP = qr/(?:\d{1,3}\.){3}(?:\d{1,3})/; + +1; + +#MODULE: ../Logreporters/Reports.pm +package Logreporters::Reports; + +use 5.008; +use strict; +use re 'taint'; +use warnings; +no warnings "uninitialized"; + +BEGIN { + use Exporter (); + use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION); + $VERSION = '1.002'; + @ISA = qw(Exporter); + @EXPORT = qw(&inc_unmatched &print_unmatched_report &print_percentiles_report2 + &print_summary_report &print_detail_report); + @EXPORT_OK = qw(); +} + +use subs @EXPORT_OK; + +BEGIN { + import Logreporters::Config qw(%Opts $fw1 $fw2 $sep1 $sep2 &D_UNMATCHED &D_TREE); + import Logreporters::Utils qw(&commify &unitize &get_percentiles &get_percentiles2); + import Logreporters::TreeData qw(%Totals %Counts &buildTree &printTree); +} + +my (%unmatched_list); + +our $origline; # unmodified log line, for error reporting and debug + +sub inc_unmatched($) { + my ($id) = @_; + $unmatched_list{$origline}++; + print "UNMATCHED($id): \"$origline\"\n" if $Opts{'debug'} & D_UNMATCHED; +} + +# Print unmatched lines +# +sub print_unmatched_report() { + return unless (keys %unmatched_list); + + print "\n\n**Unmatched Entries**\n"; + foreach my $line (sort {$unmatched_list{$b}<=>$unmatched_list{$a} } keys %unmatched_list) { + printf "%8d %s\n", $unmatched_list{$line}, $line; + } +} + +=pod + ****** Summary ******************************************************** + 2 Miscellaneous warnings + + 20621 Total messages scanned ---------------- 100.00% + 662.993M Total bytes scanned 695,198,092 + ======== ================================================ + + 19664 Ham ----------------------------------- 95.36% + 19630 Clean passed 95.19% + 34 Bad header passed 0.16% + + 942 Spam ---------------------------------- 4.57% + 514 Spam blocked 2.49% + 428 Spam discarded (no quarantine) 2.08% + + 15 Malware ------------------------------- 0.07% + 15 Malware blocked 0.07% + + + 1978 SpamAssassin bypassed + 18 Released from quarantine + 1982 Whitelisted + 3 Blacklisted + 12 MIME error + 51 Bad header (debug supplemental) + 28 Extra code modules loaded at runtime +=cut +# Prints the Summary report section +# +sub print_summary_report (\@) { + my ($sections) = @_; + my ($keyname,$cur_level); + my @lines; + + my $expand_header_footer = sub { + my $line = undef; + + foreach my $horf (@_) { + # print blank line if keyname is newline + if ($horf eq "\n") { + $line .= "\n"; + } + elsif (my ($sepchar) = ($horf =~ /^(.)$/o)) { + $line .= sprintf "%s %s\n", $sepchar x 8, $sepchar x 50; + } + else { + die "print_summary_report: unsupported header or footer type \"$horf\""; + } + } + return $line; + }; + + if ($Opts{'detail'} >= 5) { + my $header = "****** Summary "; + print $header, '*' x ($Opts{'max_report_width'} - length $header), "\n\n"; + } + + my @headers; + foreach my $sref (@$sections) { + # headers and separators + die "Unexpected Section $sref" if (ref($sref) ne 'HASH'); + + # Start of a new section group. + # Expand and save headers to output at end of section group. + if ($sref->{CLASS} eq 'GROUP_BEGIN') { + $cur_level = $sref->{LEVEL}; + $headers[$cur_level] = &$expand_header_footer(@{$sref->{HEADERS}}); + } + + elsif ($sref->{CLASS} eq 'GROUP_END') { + my $prev_level = $sref->{LEVEL}; + + # If this section had lines to output, tack on headers and footers, + # removing extraneous newlines. + if ($lines[$cur_level]) { + # squish multiple blank lines + if ($headers[$cur_level] and substr($headers[$cur_level],0,1) eq "\n") { + if ( ! defined $lines[$prev_level][-1] or $lines[$prev_level][-1] eq "\n") { + $headers[$cur_level] =~ s/^\n+//; + } + } + + push @{$lines[$prev_level]}, $headers[$cur_level] if $headers[$cur_level]; + push @{$lines[$prev_level]}, @{$lines[$cur_level]}; + my $f = &$expand_header_footer(@{$sref->{FOOTERS}}); + push @{$lines[$prev_level]}, $f if $f; + $lines[$cur_level] = undef; + } + + $headers[$cur_level] = undef; + $cur_level = $prev_level; + } + + elsif ($sref->{CLASS} eq 'DATA') { + # Totals data + $keyname = $sref->{NAME}; + if ($Totals{$keyname} > 0) { + my ($numfmt, $desc, $divisor) = ($sref->{FMT}, $sref->{TITLE}, $sref->{DIVISOR}); + + my $fmt = '%8'; + my $extra = ' %25s'; + my $total = $Totals{$keyname}; + + # Z format provides unitized or unaltered totals, as appropriate + if ($numfmt eq 'Z') { + ($total, $fmt) = unitize ($total, $fmt); + } + else { + $fmt .= "$numfmt "; + $extra = ''; + } + + if ($divisor and $$divisor) { + # XXX generalize this + if (ref ($desc) eq 'ARRAY') { + $desc = @$desc[0] . ' ' . @$desc[1] x (42 - 2 - length(@$desc[0])); + } + + push @{$lines[$cur_level]}, + sprintf "$fmt %-42s %6.2f%%\n", $total, $desc, + $$divisor == $Totals{$keyname} ? 100.00 : $Totals{$keyname} * 100 / $$divisor; + } + else { + push @{$lines[$cur_level]}, + sprintf "$fmt %-23s $extra\n", $total, $desc, commify ($Totals{$keyname}); + } + } + } + else { + die "print_summary_report: unexpected control..."; + } + } + print @{$lines[0]}; + print "\n"; +} + +# Prints the Detail report section +# +# Note: side affect; deletes each key in Totals/Counts +# after printout. Only the first instance of a key in +# the Section table will result in Detail output. +sub print_detail_report (\@) { + my ($sections) = @_; + my $header_printed = 0; + + return unless (keys %Counts); + +#use Devel::Size qw(size total_size); + + foreach my $sref ( @$sections ) { + next unless $sref->{CLASS} eq 'DATA'; + # only print detail for this section if DETAIL is enabled + # and there is something in $Counts{$keyname} + next unless $sref->{DETAIL}; + next unless exists $Counts{$sref->{NAME}}; + + my $keyname = $sref->{NAME}; + my $max_level = undef; + my $print_this_key = 0; + + my @levelspecs = (); + clear_level_specs($max_level, \@levelspecs); + if (exists $Opts{$keyname}) { + $max_level = create_level_specs($Opts{$keyname}, $Opts{'detail'}, \@levelspecs); + $print_this_key = 1 if ($max_level); + } + else { + $print_this_key = 1; + } + #print_level_specs($max_level,\@levelspecs); + + # at detail 5, print level 1, detail 6: level 2, ... + +#print STDERR "building: $keyname\n"; + my ($count, $treeref) = + buildTree (%{$Counts{$keyname}}, defined ($max_level) ? $max_level : 11, + \@levelspecs, $Opts{'detail'} - 4, 0, $Opts{'debug'} & D_TREE); + + if ($count > 0) { + if ($print_this_key) { + my $desc = $sref->{TITLE}; + $desc =~ s/^\s+//; + + if (! $header_printed) { + my $header = "****** Detail ($max_level) "; + print $header, '*' x ($Opts{'max_report_width'} - length $header), "\n"; + $header_printed = 1; + } + printf "\n%8d %s %s\n", $count, $desc, + $Opts{'sect_vars'} ? + ('-' x ($Opts{'max_report_width'} - 18 - length($desc) - length($keyname))) . " [ $keyname ] -" : + '-' x ($Opts{'max_report_width'} - 12 - length($desc)) + } + + printTree ($treeref, \@levelspecs, $Opts{'line_style'}, $Opts{'max_report_width'}, + $Opts{'debug'} & D_TREE); + } +#print STDERR "Total size Counts: ", total_size(\%Counts), "\n"; +#print STDERR "Total size Totals: ", total_size(\%Totals), "\n"; + $treeref = undef; + $Totals{$keyname} = undef; + delete $Totals{$keyname}; + delete $Counts{$keyname}; + } + #print "\n"; +} + +=pod + +Print out a standard percentiles report + + === Delivery Delays Percentiles =============================================================== + 0% 25% 50% 75% 90% 95% 98% 100% + ----------------------------------------------------------------------------------------------- + Before qmgr 0.01 0.70 1.40 45483.70 72773.08 81869.54 87327.42 90966.00 + In qmgr 0.00 0.00 0.00 0.01 0.01 0.01 0.01 0.01 + Conn setup 0.00 0.00 0.00 0.85 1.36 1.53 1.63 1.70 + Transmission 0.03 0.47 0.92 1.61 2.02 2.16 2.24 2.30 + Total 0.05 1.18 2.30 45486.15 72776.46 81873.23 87331.29 90970.00 + =============================================================================================== + + === Postgrey Delays Percentiles =========================================================== + 0% 25% 50% 75% 90% 95% 98% 100% + ------------------------------------------------------------------------------------------- + Postgrey 727.00 727.00 727.00 727.00 727.00 727.00 727.00 727.00 + =========================================================================================== + + tableref: + data table: ref to array of arrays, first cell is label, subsequent cells are data + title: + table's title + percentiles_str: + string of space or comma separated integers, which are the percentiles + calculated and output as table column data +=cut +sub print_percentiles_report2($$$) { + my ($tableref, $title, $percentiles_str) = @_; + + return unless @$tableref; + + my $myfw2 = $fw2 - 1; + my @percents = split /[ ,]/, $percentiles_str; + + # Calc y label width from the hash's keys. Each key is padded with the + # string "#: ", # where # is a single-digit sort index. + my $y_label_max_width = 0; + for (@$tableref) { + $y_label_max_width = length($_->[0]) if (length($_->[0]) > $y_label_max_width); + } + + # Titles row + my $col_titles_str = sprintf "%-${y_label_max_width}s" . "%${myfw2}s%%" x @percents , ' ', @percents; + my $table_width = length($col_titles_str); + + # Table header row + my $table_header_str = sprintf "%s %s ", $sep1 x 3, $title; + $table_header_str .= $sep1 x ($table_width - length($table_header_str)); + + print "\n", $table_header_str; + print "\n", $col_titles_str; + print "\n", $sep2 x $table_width; + + my (@p, @coldata, @xformed); + foreach (@$tableref) { + my ($title, $ref) = ($_->[0], $_->[1]); + #xxx my @sorted = sort { $a <=> $b } @{$_->[1]}; + + my @byscore = (); + + for my $bucket (sort { $a <=> $b } keys %$ref) { + #print "Key: $title: Bucket: $bucket = $ref->{$bucket}\n"; + # pairs: bucket (i.e. key), tally + push @byscore, $bucket, $ref->{$bucket}; + } + + + my @p = get_percentiles2 (@byscore, @percents); + printf "\n%-${y_label_max_width}s" . "%${fw2}.2f" x scalar (@p), $title, @p; + } + +=pod + foreach (@percents) { + #printf "\n%-${y_label_max_width}s" . "%${fw2}.2f" x scalar (@p), substr($title,3), @p; + printf "\n%3d%%", $title; + foreach my $val (@{shift @xformed}) { + my $unit; + if ($val > 1000) { + $unit = 's'; + $val /= 1000; + } + else { + $unit = ''; + } + printf "%${fw3}.2f%-2s", $val, $unit; + } + } +=cut + + print "\n", $sep1 x $table_width, "\n"; +} + +sub clear_level_specs($ $) { + my ($max_level,$lspecsref) = @_; + #print "Zeroing $max_level rows of levelspecs\n"; + $max_level = 0 if (not defined $max_level); + for my $x (0..$max_level) { + $lspecsref->[$x]{topn} = undef; + $lspecsref->[$x]{threshold} = undef; + } +} + +# topn = 0 means don't limit +# threshold = 0 means no min threshold +sub create_level_specs($ $ $) { + my ($optkey,$gdetail,$lspecref) = @_; + + return 0 if ($optkey eq "0"); + + my $max_level = $gdetail; # default to global detail level + my (@specsP1, @specsP2, @specsP3); + + #printf "create_level_specs: key: %s => \"%s\", max_level: %d\n", $optkey, $max_level; + + foreach my $sp (split /[\s,]+/, $optkey) { + #print "create_level_specs: SP: \"$sp\"\n"; + # original level specifier + if ($sp =~ /^\d+$/) { + $max_level = $sp; + #print "create_level_specs: max_level set: $max_level\n"; + } + # original level specifier + topn at level 1 + elsif ($sp =~ /^(\d*)\.(\d+)$/) { + if ($1) { $max_level = $1; } + else { $max_level = $gdetail; } # top n specified, but no max level + + # force top N at level 1 (zero based) + push @specsP1, { level => 0, topn => $2, threshold => 0 }; + } + # newer level specs + elsif ($sp =~ /^::(\d+)$/) { + push @specsP3, { level => undef, topn => 0, threshold => $1 }; + } + elsif ($sp =~ /^:(\d+):(\d+)?$/) { + push @specsP2, { level => undef, topn => $1, threshold => defined $2 ? $2 : 0 }; + } + elsif ($sp =~ /^(\d+):(\d+)?:(\d+)?$/) { + push @specsP1, { level => ($1 > 0 ? $1 - 1 : 0), topn => $2 ? $2 : 0, threshold => $3 ? $3 : 0 }; + } + else { + print STDERR "create_level_specs: unexpected levelspec ignored: \"$sp\"\n"; + } + } + + #foreach my $sp (@specsP3, @specsP2, @specsP1) { + # printf "Sorted specs: L%d, topn: %3d, threshold: %3d\n", $sp->{level}, $sp->{topn}, $sp->{threshold}; + #} + + my ($min, $max); + foreach my $sp ( @specsP3, @specsP2, @specsP1) { + ($min, $max) = (0, $max_level); + + if (defined $sp->{level}) { + $min = $max = $sp->{level}; + } + for my $level ($min..$max) { + #printf "create_level_specs: setting L%d, topn: %s, threshold: %s\n", $level, $sp->{topn}, $sp->{threshold}; + $lspecref->[$level]{topn} = $sp->{topn} if ($sp->{topn}); + $lspecref->[$level]{threshold} = $sp->{threshold} if ($sp->{threshold}); + } + } + + return $max_level; +} + +sub print_level_specs($ $) { + my ($max_level,$lspecref) = @_; + for my $level (0..$max_level) { + printf "LevelSpec Row %d: %3d %3d\n", $level, $lspecref->[$level]{topn}, $lspecref->[$level]{threshold}; + } +} + + +1; + +#MODULE: ../Logreporters/RFC3463.pm +package Logreporters::RFC3463; + +use 5.008; +use strict; +use re 'taint'; +use warnings; + +BEGIN { + use Exporter (); + use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION); + $VERSION = '1.000'; + @ISA = qw(Exporter); + @EXPORT = qw(&get_dsn_msg); +} + +use subs @EXPORT; + +#------------------------------------------------- +# Enhanced Mail System Status Codes (aka: extended status codes) +# +# RFC 3463 http://www.ietf.org/rfc/rfc3463.txt +# RFC 4954 http://www.ietf.org/rfc/rfc4954.txt +# +# Class.Subject.Detail +# +my %dsn_codes = ( + class => { + '2' => 'Success', + '4' => 'Transient failure', + '5' => 'Permanent failure', + }, + + subject => { + '0' => 'Other/Undefined status', + '1' => 'Addressing status', + '2' => 'Mailbox status', + '3' => 'Mail system status', + '4' => 'Network & routing status', + '5' => 'Mail delivery protocol status', + '6' => 'Message content/media status', + '7' => 'Security/policy status', + }, + + detail => { + '0.0' => 'Other undefined status', + '1.0' => 'Other address status', + '1.1' => 'Bad destination mailbox address', + '1.2' => 'Bad destination system address', + '1.3' => 'Bad destination mailbox address syntax', + '1.4' => 'Destination mailbox address ambiguous', + '1.5' => 'Destination mailbox address valid', + '1.6' => 'Mailbox has moved', + '1.7' => 'Bad sender\'s mailbox address syntax', + '1.8' => 'Bad sender\'s system address', + + '2.0' => 'Other/Undefined mailbox status', + '2.1' => 'Mailbox disabled, not accepting messages', + '2.2' => 'Mailbox full', + '2.3' => 'Message length exceeds administrative limit.', + '2.4' => 'Mailing list expansion problem', + + '3.0' => 'Other/Undefined mail system status', + '3.1' => 'Mail system full', + '3.2' => 'System not accepting network messages', + '3.3' => 'System not capable of selected features', + '3.4' => 'Message too big for system', + + '4.0' => 'Other/Undefined network or routing status', + '4.1' => 'No answer from host', + '4.2' => 'Bad connection', + '4.3' => 'Routing server failure', + '4.4' => 'Unable to route', + '4.5' => 'Network congestion', + '4.6' => 'Routing loop detected', + '4.7' => 'Delivery time expired', + + '5.0' => 'Other/Undefined protocol status', + '5.1' => 'Invalid command', + '5.2' => 'Syntax error', + '5.3' => 'Too many recipients', + '5.4' => 'Invalid command arguments', + '5.5' => 'Wrong protocol version', + '5.6' => 'Authentication Exchange line too long', + + '6.0' => 'Other/Undefined media error', + '6.1' => 'Media not supported', + '6.2' => 'Conversion required & prohibited', + '6.3' => 'Conversion required but not supported', + '6.4' => 'Conversion with loss performed', + '6.5' => 'Conversion failed', + + '7.0' => 'Other/Undefined security status', + '7.1' => 'Delivery not authorized, message refused', + '7.2' => 'Mailing list expansion prohibited', + '7.3' => 'Security conversion required but not possible', + '7.4' => 'Security features not supported', + '7.5' => 'Cryptographic failure', + '7.6' => 'Cryptographic algorithm not supported', + '7.7' => 'Message integrity failure', + }, + + # RFC 4954 + complete => { + '2.7.0' => 'Authentication succeeded', + '4.7.0' => 'Temporary authentication failure', + '4.7.12' => 'Password transition needed', + '5.7.0' => 'Authentication required', + '5.7.8' => 'Authentication credentials invalid', + '5.7.9' => 'Authentication mechanism too weak', + '5.7.11' => 'Encryption required for requested authentication mechanism', + }, +); + +# Returns an RFC 3463 DSN messages given a DSN code +# +sub get_dsn_msg ($) { + my $dsn = shift; + my ($msg, $class, $subject, $detail); + + return "*DSN unavailable" if ($dsn =~ /^$/); + + unless ($dsn =~ /^(\d)\.((\d{1,3})\.\d{1,3})$/) { + print "Error: not a DSN code $dsn\n"; + return "Invalid DSN"; + } + + $class = $1; $subject = $3; $detail = $2; + + #print "DSN: $dsn, Class: $class, Subject: $subject, Detail: $detail\n"; + + if (exists $dsn_codes{'class'}{$class}) { + $msg = $dsn_codes{'class'}{$class}; + } + if (exists $dsn_codes{'subject'}{$subject}) { + $msg .= ': ' . $dsn_codes{'subject'}{$subject}; + } + if (exists $dsn_codes{'complete'}{$dsn}) { + $msg .= ': ' . $dsn_codes{'complete'}{$dsn}; + } + elsif (exists $dsn_codes{'detail'}{$detail}) { + $msg .= ': ' . $dsn_codes{'detail'}{$detail}; + } + + #print "get_dsn_msg: $msg\n" if ($msg); + return $dsn . ': ' . $msg; +} + +1; + +#MODULE: ../Logreporters/PolicySPF.pm +package Logreporters::PolicySPF; + +use 5.008; +use strict; +use re 'taint'; +use warnings; + +BEGIN { + use Exporter (); + use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION); + $VERSION = '1.000'; + @ISA = qw(Exporter); + @EXPORT = qw(&postfix_policy_spf); +} + +use subs @EXPORT; + +BEGIN { + import Logreporters::TreeData qw(%Totals %Counts $END_KEY); + import Logreporters::Utils; + import Logreporters::Reports qw(&inc_unmatched); +} + +# Handle postfix/policy_spf entries +# +# Mail::SPF::Result +# Pass the SPF record designates the host to be allowed to send accept +# Fail the SPF record has designated the host as NOT being allowed to send reject +# SoftFail the SPF record has designated the host as NOT being allowed to send but is in transition accept but mark +# Neutral the SPF record specifies explicitly that nothing can be said about validity accept +# None the domain does not have an SPF record or the SPF record does not evaluate to a result accept +# PermError a permanent error has occured (eg. badly formatted SPF record) unspecified +# TempError a transient error has occured accept or reject + +sub postfix_policy_spf($) { + my $line = shift; + + if ( + $line =~ /^Attribute: / or + # handler sender_policy_framework: is decisive. + $line =~ /^handler [^:]+/ or + # decided action=REJECT Please see http://www.openspf.org/why.html?sender=jrzjcez%40telecomitalia.it&ip=81.178.62.236&receiver=protegate1.zmi.at + $line =~ /^decided action=/ or + + # pypolicyd-spf-0.7.1 + # + # Read line: "request=smtpd_access_policy" + # Found the end of entry + # Config: {'Mail_From_reject': 'Fail', 'PermError_reject': 'False', 'HELO_reject': 'SPF_Not_Pass', 'defaultSeedOnly': 1, 'debugLevel': 4, 'skip_addresses': '127.0.0.0/8,::ffff:127.0.0.0//104,::1//128', 'TempError_Defer': 'False'} + # spfcheck: pyspf result: "['Pass', 'sender SPF authorized', 'helo']" + # ERROR: Could not match line "#helo pass and mfrom none" + # Traceback (most recent call last): + # File "/usr/local/bin/policyd-spf", line 405, in + # line = sys.stdin.readline() + # KeyboardInterrupt + $line =~ /^Read line: "/ or + $line =~ /^Found the end of entry$/ or + $line =~ /^Config: {/ or + $line =~ /^spfcheck: pyspf result/ or + $line =~ /^Starting$/ or + $line =~ /^Normal exit$/ or + $line =~ /^ERROR: Could not match line/ or + $line =~ /^Traceback / or + $line =~ /^KeyboardInterrupt/ or + $line =~ /^\s\s+/ + ) + { + #print "IGNORING...\n\tORIG: $::OrigLine\n"; + return + } + + # Keep policy-spf warnings in its section + if (my ($warn,$msg) = $line =~ /^warning: ([^:]+): (.*)$/) { + #TDspf warning: ignoring garbage: # No SPF + + $msg =~ s/^# ?//; + $Totals{'policyspf'}++; + $Counts{'policyspf'}{'*Warning'}{ucfirst $warn}{$msg}{$END_KEY}++ if ($Logreporters::TreeData::Collecting{'policyspf'}); + return; + } + + # pypolicyd-spf-0.7.1 + + # Fail; identity=helo; client-ip=192.168.0.1; helo=example.com; envelope-from=f@example.com; receiver=bogus2@example.net + # Fail; identity=helo; client-ip=192.168.0.2; helo=example.com; envelope-from=<>; receiver=bogus@example.net + # Neutral; identity=helo; client-ip=192.168.0.3; helo=example.com; envelope-from=f@example.com; receiver=bogus2@example.net + # None; identity=helo; client-ip=192.168.0.4; helo=example.com; envelope-from=f@example.com; receiver=bogus2@example.net + # None; identity=helo; client-ip=192.168.0.5; helo=example.com; envelope-from=f@example.com; receiver=bogus2@example.net + # None; identity=mailfrom; client-ip=192.168.0.1; helo=example.com; envelope-from=f@example.com; receiver=bogus2@example.net + # None; identity=mailfrom; client-ip=192.168.0.2; helo=example.com; envelope-from=f@example.com; receiver=bogus2@example.net + # Pass; identity=helo; client-ip=192.168.0.2; helo=example.com; envelope-from=<>; receiver=bogus@example.net + # Permerror; identity=helo; client-ip=192.168.0.4; helo=example.com; envelope-from=f@example.com; receiver=bogus2@example.net + # Softfail; identity=mailfrom; client-ip=192.168.0.6; helo=example.com; envelope-from=f@example.com; receiver=yahl@example.org + if ($line =~ /^(Pass|Fail|None|Neutral|Permerror|Softfail|Temperror); (.*)$/) { + my $result = $1; + my %params = $2 =~ /([-\w]+)=([^;]+)/g; + #$params{'s'} = '*unknown' unless $params{'s'}; + $Totals{'policyspf'}++; + if ($Logreporters::TreeData::Collecting{'policyspf'}) { + my ($id) = $params{'identity'}; + $id =~ s/mailfrom/envelope-from/; + + $Counts{'policyspf'}{'Policy Action'}{"SPF: $result"}{join(': ',$params{'identity'},$params{$id})}{$params{'client-ip'}}{$params{'receiver'}}++; + } + return; + } + elsif ($line =~ /^ERROR /) { + $line =~ s/^ERROR //; + $Totals{'warningsother'}++; return unless ($Logreporters::TreeData::Collecting{'warningsother'}); + $Counts{'warningsother'}{"$Logreporters::service_name: $line"}++; + return; + } + + # Strip QID if it exists, and trailing ": ", leaving just the message. + $line =~ s/^(?:$Logreporters::re_QID|): //; + + # other ignored + if ( + $line =~ /^SPF \S+ \(.+?\): .*$/ or + $line =~ /^Mail From/ or + $line =~ /^:HELO check failed/ or # log entry has no space after : + $line =~ /^testing:/ + ) + { + #TDspf testing: stripped sender=jrzjcez@telecomitalia.it, stripped rcpt=hengstberger@adv.at + # postfix-policyd-spf-perl-2.007 + #TDspf SPF pass (Mechanism 'ip4:10.0.0.2/22' matched): Envelope-from: foo@example.com + #TDspf SPF pass (Mechanism 'ip4:10.10.10.10' matched): Envelope-from: anyone@sample.net + #TDspf SPF pass (Mechanism 'ip4:10.10.10.10' matched): HELO/EHLO (Null Sender): mailout2.example.com + #TDspf SPF fail (Mechanism '-all' matched): HELO/EHLO: mailout1.example.com + #TDspf SPF none (No applicable sender policy available): Envelope-from: efrom@example.com + #TDspf SPF permerror (Included domain 'example.com' has no applicable sender policy): Envelope-from: efrom@example.com + #TDspf SPF permerror (Maximum DNS-interactive terms limit (10) exceeded): Envelope-from: efrom@example.com + #TDspf Mail From (sender) check failed - Mail::SPF->new(10.0.0.1, , test.DNSreport.com) failed: 'identity' option must not be empty + #TDspf HELO check failed - Mail::SPF->new(, , ) failed: Missing required 'identity' option + + #TDspf SPF not applicable to localhost connection - skipped check + + #print "IGNORING...\n\tLINE: $line\n\tORIG: \"$Logreporters::Reports::origline\"\n"; + return; + } + + my ($action, $domain, $ip, $message, $mechanism); + ($domain, $ip, $message, $mechanism) = ('*unknown', '*unknown', '', '*unavailable'); + #print "LINE: '$line'\n"; + + # postfix-policyd-spf-perl: http://www.openspf.org/Software + if ($line =~ /^Policy action=(.*)$/) { + $line = $1; + + #: Policy action=DUNNO + return if $line =~ /^DUNNO/; + # Policy action=PREPEND X-Comment: SPF not applicable to localhost connection - skipped check + return if $line =~ /^PREPEND X-Comment: SPF not applicable to localhost connection - skipped check$/; + + #print "LINE: '$line'\n"; + if ($line =~ /^DEFER_IF_PERMIT SPF-Result=\[?(.*?)\]?: (.*) of .*$/) { + my ($lookup,$message) = ($1,$2); + # Policy action=DEFER_IF_PERMIT SPF-Result=[10.0.0.1]: Time-out on DNS 'SPF' lookup of '[10.0.0.1]' + # Policy action=DEFER_IF_PERMIT SPF-Result=example.com: 'SERVFAIL' error on DNS 'SPF' lookup of 'example.com' + $message =~ s/^(.*?) on (DNS SPF lookup)$/$2: $1/; + $message =~ s/'//g; + $Totals{'policyspf'}++; + $Counts{'policyspf'}{'Policy Action'}{'defer_if_permit'}{$message}{$lookup}{$END_KEY}++ if ($Logreporters::TreeData::Collecting{'policyspf'}); + return; + } + + if ($line =~ /^550 Please see http:\/\/www\.openspf\.org\/Why\?(.*)$/) { + # Policy action=550 Please see http://www.openspf.org/Why?s=mfrom&id=from%40example.com&ip=10.0.0.1&r=example.net + # Policy action=550 Please see http://www.openspf.org/Why?s=helo;id=mailout03.example.com;ip=192.168.0.1;r=mx1.example.net + # Policy action=550 Please see http://www.openspf.org/Why?id=someone%40example.com&ip=10.0.0.1&receiver=vps.example.net + + my %params; + for (split /[&;]/, $1) { + my ($id,$val) = split /=/, $_; + $params{$id} = $val; + } + $params{'id'} =~ s/^.*%40//; + $params{'s'} = '*unknown' unless $params{'s'}; + #print "Please see...:\n\tMessage: $message\n\tIP: $ip\n\tDomain: $domain\n"; + $Totals{'policyspf'}++; + $Counts{'policyspf'}{'Policy Action'}{'550 reject'}{'See http://www.openspf.org/Why?...'}{$params{'s'}}{$params{'ip'}}{$params{'id'}}++ if ($Logreporters::TreeData::Collecting{'policyspf'}); + return; + } + + if ($line =~ /^[^:]+: (none|pass|fail|softfail|neutral|permerror|temperror) \((.+?)\) receiver=[^;]+(?:; (.*))?$/) { + # iehc is identity, envelope-from, helo, client-ip + my ($result,$message,$iehc,$subject) = ($1,$2,$3,undef); + my %params = (); + #TDspf Policy action=PREPEND Received-SPF: pass (bounces.example.com ... _spf.example.com: 10.0.0.1 is authorized to use 'from@bounces.example.com' in 'mfrom' identity (mechanism 'ip4:10.0.0.1/24' matched)) receiver=sample.net; identity=mfrom; envelope-from="from@bounces.example.com"; helo=out.example.com; client-ip=10.0.0.1 + + # Note: "identity=mailfrom" new in Mail::SPF version 2.006 Aug. 17 + #TDspf Policy action=PREPEND Received-SPF: pass (example.com: 10.0.0.1 is authorized to use 'from@example.com' in 'mfrom' identity (mechanism 'ip4:10.0.0.0/24' matched)) receiver=mx.example.com; identity=mailfrom; envelope-from="from@example.com"; helo=example.com; client-ip=10.0.0.1 + + #TDspf Policy action=PREPEND Received-SPF: none (example.com: No applicable sender policy available) receiver=sample.net; identity=mfrom; envelope-from="f@example.com"; helo=example.com; client-ip=10.0.0.1 + + #TDspf Policy action=PREPEND Received-SPF: neutral (example.com: Domain does not state whether sender is authorized to use 'f@example.com' in 'mfrom' identity (mechanism '?all' matched)) receiver=sample.net identity=mfrom; envelope-from="f@example.com"; helo="[10.0.0.1]"; client-ip=192.168.0.1 + + #TDspf Policy action=PREPEND Received-SPF: none (example.com: No applicable sender policy available) receiver=sample.net; identity=helo; helo=example.com; client-ip=192.168.0.1 + #TDspf Policy action=PREPEND Received-SPF: none (example.com: No applicable sender policy available) receiver=mx1.example + + #print "LINE: $iehc\n"; + if ($iehc) { + %params = $iehc =~ /([-\w]+)=([^;]+)/g; + + if (exists $params{'identity'}) { + $params{'identity'} =~ s/identity=//; + if ($params{'identity'} eq 'mfrom' or $params{'identity'} eq 'mailfrom') { + $params{'identity'} = 'mail from'; + } + $params{'identity'} = uc $params{'identity'}; + } + $params{'envelope-from'} =~ s/"//g if exists $params{'envelope-from'}; + #($helo = $params{'helo'}) =~ s/"//g if exists $params{'helo'}; + $ip = $params{'client-ip'} if exists $params{'client-ip'}; + } + + $message =~ s/^([^:]+): // and $subject = $1; + + if ($message =~ /^No applicable sender policy available$/) { + $message = 'No sender policy'; + } + elsif ($message =~ s/^(Junk encountered in mechanism) '(.*?)'/$1/) { + #TDspf Policy action=PREPEND Received-SPF: permerror (example.com: Junk encountered in mechanism 'a:10.0.0.1') receiver=example.net; identity=mfrom; envelope-from="ef@example.com"; helo=h; client-ip=10.0.0.2 + $ip = formathost ($ip, 'mech: ' . $2); + } + elsif ($message =~ s/^(Included domain) '(.*?)' (has no .*)$/$1 $3/) { + #TDspf Policy action=PREPEND Received-SPF: permerror (example.com: Included domain 's.example.net' has no applicable sender policy) receiver=x.sample.com; identity=mfrom; envelope-from="ef@example.com"; helo=example.net; client-ip=10.0.0.2 + $subject .= " (included: $2)"; + } + elsif ($message =~ /^Domain does not state whether sender is authorized to use '.*?' in '\S+' identity \(mechanism '(.+?)' matched\)$/) { + # Domain does not state whether sender is authorized to use 'returns@example.com' in 'mfrom' identity (mechanism '?all' matched)) + ($mechanism,$message) = ($1,'Domain does not state if sender authorized to use'); + } + elsif ($message =~ /^(\S+) is (not )?authorized( by default)? to use '.*?' in '\S+' identity(?:, however domain is not currently prepared for false failures)? \(mechanism '(.+?)' matched\)$/) { + # Sender is not authorized by default to use 'from@example.com' in 'mfrom' identity, however domain is not currently prepared for false failures (mechanism '~all' matched)) + # 192.168.1.10 is authorized by default to use 'from@example.com' in 'mfrom' identity (mechanism 'all' matched)) + $message = join (' ', + $1 eq 'Sender' ? 'Sender' : 'IP', # canonicalize IP address + $2 ? 'not authorized' : 'authorized', + $3 ? 'by default to use' : 'to use', + ); + $mechanism = $4; + } + elsif ($message =~ /^Maximum DNS-interactive terms limit \S+ exceeded$/) { + $message = 'Maximum DNS-interactive terms limit exceeded'; + } + elsif ($message =~ /^Invalid IPv4 prefix length encountered in (.*)$/) { + $subject .= " (invalid: $1)"; + $message = 'Invalid IPv4 prefix length encountered'; + } + + #print "Result: $result, Identity: $params{'identity'}, Mech: $mechanism, Subject: $subject, IP: $ip\n"; + $Totals{'policyspf'}++; + if ($Logreporters::TreeData::Collecting{'policyspf'}) { + $message = join (' ', $message, $params{'identity'}) if exists $params{'identity'}; + $Counts{'policyspf'}{'Policy Action'}{"SPF $result"}{$message}{'mech: ' .$mechanism}{$subject}{$ip}++ + } + return; + } + + inc_unmatched('postfix_policy_spf(2)'); + return; + } + +=pod + Mail::SPF::Query + libmail-spf-query-perl 1:1.999 + + XXX incomplete + + Some possible smtp_comment results: + pass "localhost is always allowed." + none "SPF", "domain of sender $query->{sender} does not designate mailers + unknown $explanation, "domain of sender $query->{sender} does not exist" + $query->{spf_error_explanation}, $query->is_looping + $query->{spf_error_explanation}, $directive_set->{hard_syntax_error} + $query->{spf_error_explanation}, "Missing SPF record at $query->{domain}" + error $query->{spf_error_explanation}, $query->{error} + + $result $explanation, $comment, $query->{directive_set}->{orig_txt} + + Possible header_comment results: + pass "$query->{spf_source} designates $ip as permitted sender" + fail "$query->{spf_source} does not designate $ip as permitted sender" + softfail "transitioning $query->{spf_source} does not designate $ip as permitted sender" + /^unknown / "encountered unrecognized mechanism during SPF processing of $query->{spf_source}" + unknown "error in processing during lookup of $query->{sender}" + neutral "$ip is neither permitted nor denied by domain of $query->{sender}" + error "encountered temporary error during SPF processing of $query->{spf_source}" + none "$query->{spf_source} does not designate permitted sender hosts" + "could not perform SPF query for $query->{spf_source}" ); +=cut + + #TDspf 39053DC: SPF none: smtp_comment=SPF: domain of sender user@example.com does not designate mailers, header_comment=sample.net: domain of user@example.com does not designate permitted sender hosts + #TDspf : SPF none: smtp_comment=SPF: domain of sender user@example.com does not designate mailers, header_comment=sample.net: domain of user@example.com does not designate permitted sender hosts + #TDspf : SPF pass: smtp_comment=Please see http://www.openspf.org/why.html?sender=user%40example.com&ip=10.0.0.1&receiver=sample.net: example.com MX mail.example.com A 10.0.0.1, header_comment=example.com: domain of user@example.com designates 10.0.0.1 as permitted sender + #TDspf : SPF fail: smtp_comment=Please see http://www.openspf.org/why.html?sender=user%40example.com&ip=10.0.0.1&receiver=sample.net, header_comment=sample.net: domain of user@example.com does not designate 10.0.0.1 as permitted sender + #TDspf : : SPF none: smtp_comment=SPF: domain of sender does not designate mailers, header_comment=mx1.example.com: domain of does not designate permitted sender hosts + + if (my ($result, $reply) = ($line =~ /^(SPF [^:]+): (.*)$/)) { + + #print "result: $result\n\treply: $reply\n\tORIG: \"$Logreporters::Reports::origline\"\n"; + + if ($reply =~ /^(?:smtp_comment=)(.*)$/) { + $reply = $1; + + # SPF none + if ($reply =~ /^SPF: domain of sender (?:(?:[^@]+@)?(\S+) )?does not designate mailers/) { + $domain = $1 ? $1 : '*unknown'; + #print "result: $result: domain: $domain\n"; + } + elsif ($reply =~ /^Please see http:\/\/[^\/]+\/why\.html\?sender=(?:.+%40)?([^&]+)&ip=([^&]+)/) { + ($domain,$ip) = ($1,$2); + #print "result: $result: domain: $domain, IP: $ip\n"; + } + + # SPF unknown + elsif ($reply =~ /^SPF record error: ([^,]+), .*: error in processing during lookup of (?:[^@]+\@)?(\S+)/) { + ($message, $domain) = ($1, $2); + #print "result: $result: domain: $domain, Problem: $message\n"; + } + elsif ($reply =~ /^SPF record error: ([^,]+), .*: encountered unrecognized mechanism during SPF processing of domain (?:[^@]+\@)?(\S+)/) { + ($message, $domain) = ($1,$2); + #print "result: \"$result\": domain: $domain, Problem: $message\n"; + $result = "SPF permerror" if ($result =~ /SPF unknown mx-all/); + } + else { + inc_unmatched('postfix_policy_spf(3)'); + return; + } + } + else { + inc_unmatched('postfix_policy_spf(4)'); + return; + } + + $Totals{'policyspf'}++; + if ($message) { + $Counts{'policyspf'}{'Policy Action'}{$result}{$domain}{$ip}{$message}{$END_KEY}++ if ($Logreporters::TreeData::Collecting{'policyspf'}); + } + else { + $Counts{'policyspf'}{'Policy Action'}{$result}{$domain}{$ip}{$END_KEY}++ if ($Logreporters::TreeData::Collecting{'policyspf'}); + } + return; + } + + + inc_unmatched('postfix_policy_spf(5)'); +} + +1; + +#MODULE: ../Logreporters/Postfwd.pm +package Logreporters::Postfwd; + +use 5.008; +use strict; +use re 'taint'; +use warnings; + +BEGIN { + use Exporter (); + use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION); + $VERSION = '1.000'; + @ISA = qw(Exporter); + @EXPORT = qw(&postfix_postfwd); +} + +use subs @EXPORT; + +BEGIN { + import Logreporters::TreeData qw(%Totals %Counts $END_KEY); + import Logreporters::Utils; + import Logreporters::Reports qw(&inc_unmatched); +} + +# postfwd: http://postfwd.org/ +# +# +sub postfix_postfwd($) { + my $line = shift; + + return if ( + #TDpfw [STATS] Counters: 213000 seconds uptime, 39 rules + #TDpfw [LOGS info]: compare rbl: "example.com[10.1.0.7]" -> "localrbl.local" + #TDpfw [DNSBL] object 10.0.0.1 listed on rbl:list.dnswl.org (answer: 127.0.15.0, time: 0s) + $line =~ /^\[STATS\] / or + $line =~ /^\[DNSBL\] / or + $line =~ /^\[LOGS info\]/ or + $line =~ /^Process Backgrounded/ or + $line =~ /^Setting [ug]id to/ or + $line =~ /^Binding to TCP port/ or + $line =~ /^terminating\.\.\./ or + $line =~ /^Setting status interval to \S+ seconds/ or + $line =~ /^postfwd .+ ready for input$/ or + $line =~ /postfwd .+ (?:starting|terminated)/ + ); + + my ($type,$rule,$id,$action,$host,$hostip,$recipient); + + if ($line =~ /^\[(RULES|CACHE)\] rule=(\d+), id=([^,]+), client=([^[]+)\[([^]]+)\], sender=.*?, recipient=<(.*?)>,.*? action=(.*)$/) { + #TDpfw [RULES] rule=0, id=OK_DNSWL, client=example.com[10.0.0.1], sender=, recipient=, helo=, proto=ESMTP, state=RCPT, delay=0s, hits=OK_DNSWL, action=DUNNO + #TDpfw [CACHE] rule=14, id=GREY_NODNS, client=unknown[192.168.0.1], sender=, recipient=, helo=, proto=ESMTP, state=RCPT, delay=0s, hits=SET_NODNS;EVAL_DNSBLS;EVAL_RHSBLS;GREY_NODNS, action=greylist + ($type,$rule,$id,$host,$hostip,$recipient,$action) = ($1,$2,$3,$4,$5,$6,$7); + $recipient = '*unknown' if (not defined $recipient); + $Counts{'postfwd'}{"Rule $rule"}{$id}{$action}{$type}{$recipient}{formathost($hostip,$host)}++ if ($Logreporters::TreeData::Collecting{'postfwd'}); + } + elsif (($line =~ /Can't connect to TCP port/) or + ($line =~ /Pid_file already exists for running process/) + ) + { + $line =~ s/^[-\d\/:]+ //; # strip leading date/time stamps 2009/07/18-20:09:49 + $Totals{'warningsother'}++; return unless ($Logreporters::TreeData::Collecting{'warningsother'}); + $Counts{'warningsother'}{"$Logreporters::service_name: $line"}++; + return; + } + + # ignoring [DNSBL] lines + #elsif ($line =~ /^\[DNSBL\] object (\S+) listed on (\S+) \(answer: ([^,]+), .*\)$/) { + # #TDpfw [DNSBL] object 10.0.0.60 listed on rbl:list.dnswl.org (answer: 127.0.15.0, time: 0s) + # ($type,$rbl) = split (/:/, $2); + # $Counts{'postfwd'}{"DNSBL: $type"}{$rbl}{$1}{$3}{''}++ if ($Logreporters::TreeData::Collecting{'postfwd'}); + #} + else { + inc_unmatched('postfwd'); + return; + } + + $Totals{'postfwd'}++; +} + +1; + +#MODULE: ../Logreporters/Postgrey.pm +package Logreporters::Postgrey; + +use 5.008; +use strict; +use re 'taint'; +use warnings; + +my (%pgDelays,%pgDelayso); + +BEGIN { + use Exporter (); + use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION); + $VERSION = '1.000'; + @ISA = qw(Exporter); + @EXPORT = qw(&postfix_postgrey &print_postgrey_reports); +} + +use subs @EXPORT; + +BEGIN { + import Logreporters::TreeData qw(%Totals %Counts $END_KEY); + import Logreporters::Utils; + import Logreporters::Config qw(%Opts); + import Logreporters::Reports qw(&inc_unmatched &print_percentiles_report2); +} + +# postgrey: http://postgrey.schweikert.ch/ +# +# Triplet: (client IP, envelope sender, envelope recipient address) +# +sub postfix_postgrey($) { + my $line = shift; + + return if ( + #TDpg cleaning up old logs... + #TDpg cleaning up old entries... + #TDpg cleaning clients database finished. before: 207, after: 207 + #TDpg cleaning main database finished. before: 3800, after: 2539 + #TDpg delayed 603 seconds: client=10.0.example.com, from=anyone@sample.net, to=joe@example.com + + #TDpg Setting uid to "504" + #TDpg Setting gid to "1002 1002" + #TDpg Process Backgrounded + #TDpg 2008/03/08-15:54:49 postgrey (type Net::Server::Multiplex) starting! pid(21961) + #TDpg Binding to TCP port 10023 on host 127.0.0.1 + #TDpg 2007/01/25-14:58:24 Server closing! + #TDpg Couldn't unlink "/var/run/postgrey.pid" [Permission denied] + #TDpg ignoring garbage: + #TDpg unrecognized request type: '' + #TDpg rm /var/spool/postfix/postgrey/log.0000000002 + #TDpg 2007/01/25-14:48:00 Pid_file already exists for running process (4775)... aborting at line 232 in file /usr/lib/perl5/vendor_perl/5.8.7/Net/Server.pm + + + $line =~ /^cleaning / or + $line =~ /^delayed / or + $line =~ /^cleaning / or + $line =~ /^Setting [ug]id/ or + $line =~ /^Process Backgrounded/ or + $line =~ /^Binding to / or + $line =~ /^Couldn't unlink / or + $line =~ /^ignoring garbage: / or + $line =~ /^unrecognized request type/ or + $line =~ /^rm / or + # unanchored last + $line =~ /Pid_file already exists/ or + $line =~ /postgrey .* starting!/ or + $line =~ /Server closing!/ + ); + + my ($action,$reason,$delay,$host,$ip,$sender,$recip); + + if ($line =~ /^(?:$Logreporters::re_QID: )?action=(.*?), reason=(.*?)(?:, delay=(\d+))?, client_name=(.*?), client_address=(.*?)(?:, sender=(.*?))?(?:, +recipient=(.*))?$/o) { + #TDpg action=greylist, reason=new, client_name=example.com, client_address=10.0.0.1, sender=from@example.com, recipient=to@sample.net + #TDpgQ action=greylist, reason=new, client_name=example.com, client_address=10.0.0.1, sender=from@example.com + #TDpgQ action=pass, reason=triplet found, client_name=example.com, client_address=10.0.0.1, sender=from@example.com, recipient=to@sample.net + #TDpg action=pass, reason=triplet found, client_name=example.com, client_address=10.0.0.1, sender=from@example.com, recipient=to@sample.net + #TDpg action=pass, reason=triplet found, client_name=example.com, client_address=10.0.0.1, recipient=to@sample.net + #TDpg action=pass, reason=triplet found, delay=99, client_name=example.com, client_address=10.0.0.1, recipient=to@sample.net + ($action,$reason,$delay,$host,$ip,$sender,$recip) = ($1,$2,$3,$4,$5,$6,$7); + $reason =~ s/^(early-retry) \(.* missing\)$/$1/; + $recip = '*unknown' if (not defined $recip); + $sender = '' if (not defined $sender); + + $Totals{'postgrey'}++; + if ($Logreporters::TreeData::Collecting{'postgrey'}) { + $Counts{'postgrey'}{"\u$action"}{"\u$reason"}{formathost($ip,$host)}{$recip}{$sender}++; + + if (defined $delay and $Logreporters::TreeData::Collecting{'postgrey_delays'}) { + $pgDelays{'1: Total'}{$delay}++; + + push @{$pgDelayso{'Postgrey'}}, $delay + } + } + } + elsif ($line =~ /^whitelisted: (.*?)(?:\[([^]]+)\])?$/) { + #TDpg: whitelisted: example.com[10.0.0.1] + $Totals{'postgrey'}++; + if ($Logreporters::TreeData::Collecting{'postgrey'}) { + $Counts{'postgrey'}{'Whitelisted'}{defined $2 ? formathost($2,$1) : $1}{$END_KEY}++; + } + } + elsif ($line =~ /^tarpit whitelisted: (.*?)(?:\[([^]]+)\])?$/) { + #TDpg: tarpit whitelisted: example.com[10.0.0.1] + $Totals{'postgrey'}++; + if ($Logreporters::TreeData::Collecting{'postgrey'}) { + $Counts{'postgrey'}{'Tarpit whitelisted'}{defined $2 ? formathost($2,$1) : $1}{$END_KEY}++; + } + } + else { + inc_unmatched('postgrey'); + } + + return; +} + +sub print_postgrey_reports() { + #print STDERR "pgDelays memory usage: ", commify(Devel::Size::total_size(\%pgDelays)), "\n"; + + if ($Opts{'postgrey_delays'}) { + my @table; + for (sort keys %pgDelays) { + # anon array ref: label, array ref of $Delay{key} + push @table, [ substr($_,3), $pgDelays{$_} ]; + } + if (@table) { + print_percentiles_report2(\@table, "Postgrey Delays Percentiles", $Opts{'postgrey_delays_percentiles'}); + } + } +} + +1; + +#MODULE: ../Logreporters/PolicydWeight.pm +package Logreporters::PolicydWeight; + +use 5.008; +use strict; +use re 'taint'; +use warnings; + +BEGIN { + use Exporter (); + use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION); + $VERSION = '1.000'; + @ISA = qw(Exporter); + @EXPORT = qw(&postfix_policydweight); +} + +use subs @EXPORT; + +BEGIN { + import Logreporters::Reports qw(&inc_unmatched); + import Logreporters::TreeData qw(%Totals %Counts); + import Logreporters::Utils; +} + +# Handle postfix/policydweight entries +# +sub postfix_policydweight($) { + my $line = shift; + my ($r1, $code, $reason, $reason2); + + if ( + $line =~ /^weighted check/ or + $line =~ /^policyd-weight .* started and daemonized/ or + $line =~ /^(cache|child|master): / or + $line =~ /^cache (?:spawned|killed)/ or + $line =~ /^child \d+ exited/ or + $line =~ /^Daemon terminated/ or + $line =~ /^Daemon terminated/ + ) + { + #print "$OrigLine\n"; + return; + } + + if ($line =~ s/^decided action=//) { + $line =~ s/; delay: \d+s$//; # ignore, eg.: "delay: 3s" + #print "....\n\tLINE: $line\n\tORIG: '$Logreporters::Reports::origline'\n"; + if (($code,$r1) = ($line =~ /^(\d+)\s+(.*)$/ )) { + my @problems = (); + for (split /; */, $r1) { + + if (/^Mail appeared to be SPAM or forged\. Ask your Mail\/DNS-Administrator to correct HELO and DNS MX settings or to get removed from DNSBLs/ ) { + push @problems, 'spam/forged: bad DNS/hit DNSRBLs'; + } + elsif (/^Your MTA is listed in too many DNSBLs/) { + push @problems, 'too many DNSBLs'; + } + elsif (/^temporarily blocked because of previous errors - retrying too fast\. penalty: \d+ seconds x \d+ retries\./) { + push @problems, 'temp blocked: retrying too fast'; + } + elsif (/^Please use DynDNS/) { + push @problems, 'use DynDNS'; + } + elsif (/^please relay via your ISP \([^)]+\)/) { + push @problems, 'use ISP\'s relay'; + } + elsif (/^in (.*)/) { + push @problems, $1; + } + elsif (m#^check http://rbls\.org/\?q=#) { + push @problems, 'see http://rbls.org'; + } + elsif (/^MTA helo: .* \(helo\/hostname mismatch\)/) { + push @problems, 'helo/hostname mismatch'; + } + elsif (/^No DNS entries for your MTA, HELO and Domain\. Contact YOUR administrator\s+/) { + push @problems, 'no DNS entries'; + } + else { + push @problems, $_; + } + } + + $reason = $code; $reason2 = join (', ', @problems); + } + elsif ($line =~ s/^DUNNO\s+//) { + #decided action=DUNNO multirecipient-mail - already accepted by previous query; delay: 0s + $reason = 'DUNNO'; $reason2 = $line; + } + elsif ($line =~ s/^check_greylist//) { + #decided action=check_greylist; delay: 16s + $reason = 'Check greylist'; $reason2 = $line; + } + elsif ($line =~ s/^PREPEND X-policyd-weight:\s+//) { + #decided action=PREPEND X-policyd-weight: using cached result; rate: -7.6; delay: 0s + if ($line =~ /(using cached result); rate:/) { + $reason = 'PREPEND X-policyd-weight: mail accepted'; $reason2 = "\u$1"; + } + else { + #decided action=PREPEND X-policyd-weight: NOT_IN_SBL_XBL_SPAMHAUS=-1.5 P0F_LINUX=0 , rate: -7.6; delay: 2s + $reason = 'PREPEND X-policyd-weight: mail accepted'; $reason2 = 'Varies'; + } + } + else { + return; + } + } + elsif ($line =~ /^err/) { + # coerrce policyd-weight err's into general warnings + $Totals{'startuperror'}++; + $Counts{'startuperror'}{'Service: policyd-weight'}{$line}++ if ($Logreporters::TreeData::Collecting{'startuperror'}); + return; + } + else { + inc_unmatched('policydweight'); + return; + } + + $Totals{'policydweight'}++; + $Counts{'policydweight'}{$reason}{$reason2}++ if ($Logreporters::TreeData::Collecting{'policydweight'}); +} + +1; + + +package Logreporters; + +BEGIN { + import Logreporters::Utils; + import Logreporters::Config; + import Logreporters::TreeData qw(%Totals %Counts %Collecting printTree buildTree $END_KEY); + import Logreporters::RegEx; + import Logreporters::Reports; + import Logreporters::RFC3463; + import Logreporters::PolicySPF; + import Logreporters::Postfwd; + import Logreporters::Postgrey; + import Logreporters::PolicydWeight; +} +use 5.008; +use strict; +use warnings; +no warnings "uninitialized"; +use re 'taint'; + +use File::Basename; +our $progname = fileparse($0); + +my @supplemental_reports = qw(delays postgrey_delays); + +# Default values for various options. These are used +# to reset default values after an option has been +# disabled (via undef'ing its value). This allows +# a report to be disabled via config file or --nodetail, +# but reenabled via subsequent command line option +my %Defaults = ( + detail => 10, # report level detail + max_report_width => 100, # maximum line width for report output + line_style => undef, # lines > max_report_width, 0=truncate,1=wrap,2=full + syslog_name => 'postfix', # service name (postconf(5), syslog_name) + sect_vars => 0, # show section vars in detail report hdrs + unknown => 1, # show 'unknown' in address/hostname pairs + ipaddr_width => 15, # width for printing ip addresses + long_queue_ids => 0, # enable long queue ids (2.9+) + delays => 1, # show message delivery delays report + delays_percentiles => '0 25 50 75 90 95 98 100', # percentiles shown in delays report + reject_reply_patterns => '5.. 4.. warn', # reject reply grouping patterns + postgrey_delays => 1, # show postgrey delays report + postgrey_delays_percentiles => '0 25 50 75 90 95 98 100', # percentiles shown in postgrey delays report +); + +my $usage_str = <<"END_USAGE"; +Usage: $progname [ ARGUMENTS ] [logfile ...] + ARGUMENTS can be one or more of options listed below. Later options + override earlier ones. Any argument may be abbreviated to an unambiguous + length. Input is read from the named logfile(s), or STDIN. + + --debug AREAS provide debug output for AREAS + --help print usage information + --version print program version + + --config_file FILE, -f FILE use alternate configuration file FILE + --ignore_services PATTERN ignore postfix/PATTERN services + --syslog_name PATTERN only consider log lines that match + syslog service name PATTERN + + --detail LEVEL print LEVEL levels of detail + (default: 10) + --nodetail set all detail levels to 0 + --[no]summary display the summary section + + --ipaddr_width WIDTH use WIDTH chars for IP addresses in + address/hostname pairs + --line_style wrap|full|truncate disposition of lines > max_report_width + (default: truncate) + --full same as --line_style=full + --truncate same as --line_style=truncate + --wrap same as --line_style=wrap + --max_report_width WIDTH limit report width to WIDTH chars + (default: 100) + --limit L=V, -l L=V set level limiter L with value V + --[no]long_queue_ids use long queue ids + --[no]unknown show the 'unknown' hostname in + formatted address/hostnames pairs + --[no]sect_vars [do not] show config file var/cmd line + option names in section titles + + --recipient_delimiter C split delivery addresses using + recipient delimiter char C + --reject_reply_patterns "R1 [R2 ...]" set reject reply patterns used in + to group rejects to R1, [R2 ...], + where patterns are [45][.0-9][.0-9] + or "Warn" (default: 5.. 4.. Warn) + Supplimental reports + --[no]delays [do not] show msg delays percentiles report + --delays_percentiles "P1 [P2 ...]" set delays report percentiles to + P1 [P2 ...] (range: 0...100) + --[no]postgrey_delays [do not] show postgrey delays percentiles + report + --postgrey_delays_percentiles "P1 [P2 ...]" + set postgrey delays report percentiles to + P1 [P2 ...] (range: 0...100) +END_USAGE + +my @RejectPats; # pattern list used to match against reject replys +my @RejectKeys; # 1-to-1 with RejectPats, but with 'x' replacing '.' (for report output) +my (%DeferredByQid, %SizeByQid, %AcceptedByQid, %Delays); + +# local prototypes +sub usage; +sub init_getopts_table; +sub init_defaults; +sub build_sect_table; +sub postfix_bounce; +sub postfix_cleanup; +sub postfix_panic; +sub postfix_fatal; +sub postfix_error; +sub postfix_warning; +sub postfix_script; +sub postfix_postsuper; +sub process_delivery_attempt; +sub cleanhostreply; +sub strip_ftph; +sub get_reject_key; +sub expand_bare_reject_limiters; +sub create_ignore_list; +sub in_ignore_list; +sub header_body_checks; +sub milter_common; + +# lines that match any RE in this list will be ignored. +# see create_ignore_list(); +my @ignore_list = (); + +# The Sections table drives Summary and Detail reports. For each entry in the +# table, if there is data avaialable, a line will be output in the Summary report. +# Additionally, a sub-section will be output in the Detail report if both the +# global --detail, and the section's limiter variable, are sufficiently high (a +# non-existent section limiter variable is considered to be sufficiently high). +# +my @Sections = (); + +# List of reject variants. See also: "Add reject variants" below, and conf file(s). +my @RejectClasses = qw( + rejectrelay rejecthelo rejectdata rejectunknownuser rejectrecip rejectsender + rejectclient rejectunknownclient rejectunknownreverseclient rejectunverifiedclient + rejectrbl rejectheader rejectbody rejectcontent rejectsize rejectmilter rejectproxy + rejectinsufficientspace rejectconfigerror rejectverify rejectetrn rejectlookupfailure +); + +# Dispatch table of the list of supported policy services +# XXX have add-ins register into the dispatch table +my @policy_services = ( + [ qr/^postfwd/, \&Logreporters::Postfwd::postfix_postfwd ], + [ qr/^postgrey/, \&Logreporters::Postgrey::postfix_postgrey ], + [ qr/^policyd?-spf/, \&Logreporters::PolicySPF::postfix_policy_spf ], + [ qr/^policyd-?weight/, \&Logreporters::PolicydWeight::postfix_policydweight ], +); + +# Initialize main running mode and basic opts +init_run_mode($config_file); + +# Configure the Getopts options table +init_getopts_table(); + +# Place configuration file/environment variables onto command line +init_cmdline(); + +# Initialize default values +init_defaults(); + +# Process command line arguments, 0=no_permute,no_pass_through +get_options(0); + +# Build the Section table, after reject_reply_patterns is final +build_sect_table(); + +# Expand bare rejects before generic processing +expand_bare_reject_limiters(); + +# Run through the list of Limiters, setting the limiters in %Opts. +process_limiters(@Sections); + +# Set collection for any enabled supplemental sections +foreach (@supplemental_reports) { + $Logreporters::TreeData::Collecting{$_} = (($Opts{'detail'} >= 5) && $Opts{$_}) ? 1 : 0; +} + +if (! defined $Opts{'line_style'}) { + # default line style to full if detail >= 11, or truncate otherwise + $Opts{'line_style'} = + ($Opts{'detail'} > 10) ? $line_styles{'full'} : $line_styles{'truncate'}; +} + +# Set the QID RE to capture either pre-2.9 short style or 2.9+ long style. +$re_QID = $Opts{'long_queue_ids'} ? $re_QID_l : $re_QID_s; + +# Create the list of REs used to match against log lines +create_ignore_list(); + +# Notes: +# +# - IN REs, always use /o flag or qr// at end of RE when RE uses interpolated vars +# - In REs, email addresses may be empty "<>" - capture using *, not + ( eg. from=<.*?> ) +# - See additional notes below, search for "Note:". +# - XXX indicates change, fix or thought required + + +# Main processing loop +# +LINE: while ( <> ) { + chomp; + s/\s+$//; + next unless length $_; + + $Logreporters::Reports::origline = $_; + + # Linux: Jul 1 20:08:06 mailhost postfix/smtpd[4379]: connect from unknown[10.0.0.1] + # FreeBSD: Jul 1 20:08:06 mailhost postfix/smtpd[4379]: connect from unknown[10.0.0.1] + # Aug 17 15:16:12 mailhost postfix/cleanup[14194]: [ID 197553 mail.info] EC2B339E5: message-id=<2616.EC2B339E5@example.com> + # Dec 25 05:20:28 mailhost policyd-spf[14194]: [ID 27553 mail.info] ... policyd-spf stuff ... + + next unless /^[A-Z][a-z]{2} [ \d]\d \d{2}:\d{2}:\d{2} (?:<[^>]+> )?(\S+) ($Opts{'syslog_name'}(?:\/([^:[]+))?)(?:\[\d+\])?: (?:\[ID \d+ \w+\.\w+\] )?(.*)$/o; + + our $service_name = $3; + my ($mailhost,$server_name,$p1) = ($1,$2,$4); + #print "mailhost: $mailhost, servername: $server_name, servicename: $service_name, p1: $p1\n"; + + $service_name = $server_name unless $service_name; + #print "service_name: $service_name\n"; + + # ignored postfix services... + next if $service_name eq 'postlog'; + next if $service_name =~ /^$Opts{'ignore_services'}$/o; + + # common log entries up front + if ($p1 =~ s/^connect from //) { + #TD25 connect from sample.net[10.0.0.1] + #TD connect from mail.example.com[2001:dead:beef::1] + #TD connect from localhost.localdomain[127.0.0.1] + #TD connect from unknown[unknown] + $Totals{'connectioninbound'}++; + next unless ($Collecting{'connectioninbound'}); + + my $host = $p1; my $hostip; + if (($host,$hostip) = ($host =~ /^([^[]+)\[([^]]+)\]/)) { + $host = formathost($hostip,$host); + } + $Counts{'connectioninbound'}{$host}++; + next; + } + + if ($p1 =~ /^disconnect from /) { + #TD25 disconnect from sample.net[10.0.0.1] + #TD disconnect from mail.example.com[2001:dead:beef::1] + $Totals{'disconnection'}++; + next; + } + + if ($p1 =~ s/^connect to //) { + next if ($p1 =~ /^subsystem /); + $Totals{'connecttofailure'}++; + next unless ($Collecting{'connecttofailure'}); + + my ($host,$hostip,$reason,$port) = ($p1 =~ /^([^[]+)\[([^]]+)\](?::\d+)?: (.*?)(?:\s+\(port (\d+)\))?$/); + # all "connect to" messages indicate a problem with the connection + #TDs connect to example.org[10.0.0.1]: Connection refused (port 25) + #TDs connect to mail.sample.com[10.0.0.1]: No route to host (port 25) + #TDs connect to sample.net[192.168.0.1]: read timeout (port 25) + #TDs connect to mail.example.com[10.0.0.1]: server dropped connection without sending the initial SMTP greeting (port 25) + #TDs connect to mail.example.com[192.168.0.1]: server dropped connection without sending the initial SMTP greeting (port 25) + #TDs connect to ipv6-1.example.com[2001:dead:beef::1]: Connection refused (port 25) + #TDs connect to ipv6-2.example.com[FEDC:BA98:7654:3210:FEDC:BA98:7654:3210]: Connection refused (port 25) + #TDs connect to ipv6-3.example.com[1080:0:0:0:8:800:200C:4171]: Connection refused (port 25) + #TDs connect to ipv6-4.example.com[3ffe:2a00:100:7031::1]: Connection refused (port 25) + #TDs connect to ipv6-5.example.com[1080::8:800:200C:417A]: Connection refused (port 25) + #TDs connect to ipv6-6.example.com[::192.9.5.5]: Connection refused (port 25) + #TDs connect to ipv6-7.example.com[::FFFF:129.144.52.38]: Connection refused (port 25) + #TDs connect to ipv6-8.example.com[2010:836B:4179::836B:4179]: Connection refused (port 25) + #TDs connect to mail.example.com[10.0.0.1]: server refused to talk to me: 452 try later (port 25) + + $host = join(' :', $host, $port) if ($port and $port ne '25'); + # Note: See ConnectToFailure below + if ($reason =~ /^server (refused to talk to me): (.*)$/) { + $Counts{'connecttofailure'}{ucfirst($1)}{formathost($hostip,$host)}{$2}++; + } else { + $Counts{'connecttofailure'}{ucfirst($reason)}{formathost($hostip,$host)}{''}++; + } + next; + } + +=pod +real 3m43.997s +user 3m39.038s +sys 0m3.005s +=pod + # Handle before panic, fatal, warning, so that service-specific code gets first crack + # XXX replace w/dispatch table for add-ins, so user's can add their own... + if ($service_name eq 'postfwd') { postfix_postfwd($p1); next; } + if ($service_name eq 'postgrey') { postfix_postgrey($p1); next; } + if ($service_name =~ /^policyd?-spf/) { postfix_policy_spf($p1); next; } # postfix/policy-spf + if ($service_name =~ /^policyd-?weight/) { postfix_policydweight($p1); next; } # postfix/policydweight + +=cut + # Handle policy service handlers before panic, fatal, warning, etc. + # messages so that service-specific code gets first crack. + # 5:25 + foreach (@policy_services) { + if ($service_name =~ $_->[0]) { + #print "Calling policy service helper: $service_name:('$p1')\n"; + &{$_->[1]}($p1); + next LINE; + } + }; +#=cut + + # ^warning: ... + # ^fatal: ... + # ^panic: ... + # ^error: ... + if ($p1 =~ /^warning: +(.*)$/) { postfix_warning($1); next; } + if ($p1 =~ /^fatal: +(.*)$/) { postfix_fatal($1); next; } + if ($p1 =~ /^panic: +(.*)$/) { postfix_panic($1); next; } + if ($p1 =~ /^error: +(.*)$/) { postfix_error($1); next; } + + # output by all services that use table lookups - process before specific messages + if ($p1 =~ /(?:lookup )?table (?:[^ ]+ )?has changed -- (?:restarting|exiting)$/) { + #TD table hash:/var/mailman/data/virtual-mailman(0,lock|fold_fix) has changed -- restarting + #TD table hash:/etc/postfix/helo_checks has changed -- restarting + $Totals{'tablechanged'}++; + next; + } + + # postfix/postscreen and postfix/verify services + if ($service_name eq 'postscreen' + or $service_name eq 'verify') { postfix_postscreen($p1); next; } # postfix/postscreen, postfix/verify + if ($service_name eq 'dnsblog') { postfix_dnsblog($p1); next; } # postfix/dnsblog + if ($service_name =~ /^cleanup/) { postfix_cleanup($p1); next; } # postfix/cleanup* + if ($service_name =~ /^bounce/) { postfix_bounce($p1); next; } # postfix/bounce* + if ($service_name eq 'postfix-script') { postfix_script($p1); next; } # postfix/postfix-script + if ($service_name eq 'postsuper') { postfix_postsuper($p1); next; } # postfix/postsuper + + # ignore tlsproxy for now + if ($service_name eq 'tlsproxy') { next; } # postfix/tlsproxy + + my ($helo, $relay, $from, $origto, $to, $domain, $status, + $type, $reason, $reason2, $filter, $site, $cmd, $qid, + $rej_type, $reject_name, $host, $hostip, $dsn, $reply, $fmthost, $bytes); + + $rej_type = undef; + + # ^$re_QID: ... + if ($p1 =~ s/^($re_QID): //o) { + $qid = $1; + + next if ($p1 =~ /^host \S*\[\S*\] said: 4\d\d/); # deferrals, picked up in "status=deferred" + + if ($p1 =~ /^removed\s*$/ ) { + # Note: See REMOVED elsewhere + # 52CBDC2E0F: removed + delete $SizeByQid{$qid} if (exists $SizeByQid{$qid}); + $Totals{'removedfromqueue'}++; + next; + } + + # coerce into general warning + if (($p1 =~ /^Cannot start TLS: handshake failure/) or + ($p1 =~ /^non-E?SMTP response from/)) { + postfix_warning($p1); + next; + } + + if ($p1 eq 'status=deferred (bounce failed)') { + #TDqQ status=deferred (bounce failed) + $Totals{'bouncefailed'}++; + next; + } + + # this test must preceed access checks below + #TDsQ replace: header From: "Postmaster" : From: "Postmaster" + if ($service_name eq 'smtp' and header_body_checks($p1)) { + #print "main: header_body_checks\n"; + next; + } + + # Postfix access actions + # REJECT optional text... + # DISCARD optional text... + # HOLD optional text... + # WARN optional text... + # FILTER transport:destination + # REDIRECT user@domain + # BCC user@domain (2.6 experimental branch) + # The following actions are indistinguishable in the logs + # 4NN text + # 5NN text + # DEFER_IF_REJECT optional text... + # DEFER_IF_PERMIT optional text... + # UCE restriction... + # The following actions are not logged + # PREPEND headername: headervalue + # DUNNO + # + # Reject actions based on remote client information: + # - one of host name, network address, envelope sender + # or + # - recipient address + + # Template of access controls. Rejects look like the first line, other access actions the second. + # ftph is envelope from, envelope to, proto and helo. + # QID: ACTION STAGE from host[hostip]: DSN trigger: explanation; ftph + # QID: ACTION STAGE from host[hostip]: trigger: explanation; ftph + + # $re_QID: reject: RCPT|MAIL|CONNECT|HELO|DATA from ... + # $re_QID: reject_warning: RCPT|MAIL|CONNECT|HELO|DATA from ... + if ($p1 =~ /^(reject(?:_warning)?|discard|filter|hold|redirect|warn|bcc|replace): /) { + my $action = $1; + $p1 = substr($p1, length($action) + 2); + + #print "action: \"$action\", p1: \"$p1\"\n"; + if ($p1 !~ /^(RCPT|MAIL|CONNECT|HELO|EHLO|DATA|VRFY|ETRN|END-OF-MESSAGE) from ([^[]+)\[([^]]+)\](?::\d+)?: (.*)$/) { + inc_unmatched('unexpected access'); + next; + } + my ($stage,$host,$hostip,$p1) = ($1,$2,$3,$4); #print "stage: \"$stage\", host: \"$host\", hostip: \"$hostip\", p1: \"$p1\"\n"; + my ($efrom,$eto,$proto,$helo) = strip_ftph($p1); #print "efrom: \"$efrom\", eto: \"$eto\", proto: \"$proto\", helo: \"$helo\"\n"; + #print "p1 now: \"$p1\"\n"; + +# QID: ACTION STAGE from host[hostip]: DSN trigger: explanation; ftph +#TDsdN reject_warning: VRFY from host[10.0.0.1]: 450 4.1.2 <<1F4@bs>>: Recipient address rejected: Domain not found; to=<<1F4@bs>> proto=SMTP helo= +#TDsdN reject: VRFY from host[10.0.0.1]: 550 5.1.1 <:>: Recipient address rejected: User unknown in local recipient table; to=<:> proto=SMTP helo=<10.0.0.1> +#TDsdN reject: VRFY from host[10.0.0.1]: 450 4.1.8 : Sender address rejected: Domain not found; from= to= proto=SMTP +#TDsdN reject: VRFY from host[10.0.0.1]: 554 5.7.1 Service unavailable; Client host [10.0.0.1] blocked using zen.spamhaus.org; http://www.spamhaus.org/query/bl?ip=10.0.0.1; to= proto=SMTP +#TDsdN reject: RCPT from host[10.0.0.1]: 450 4.1.2 : Recipient address rejected: User unknown in local recipient table; from=<> to= proto=SMTP helo= +#TDsdN reject: RCPT from host[10.0.0.1]: 550 : Recipient address rejected: User unknown in local recipient table; from=<> to= proto=SMTP helo= +#TDsdN reject_warning: RCPT from host[10.0.0.1]: 550 : Recipient address rejected: User unknown in local recipient table; from=<> to= proto=SMTP helo= +#TDsdN reject: RCPT from host[10.0.0.1]: 550 5.1.1 : Recipient address rejected: User unknown in virtual address table; from= to= proto=ESMTP helo= +#TDsdN reject: RCPT from host[10.0.0.1]: 450 4.1.1 : Recipient address rejected: User unknown in virtual mailbox table; from= to= proto=ESMTP helo= +#TDsdN reject: RCPT from host[10.0.0.1]: 550 5.5.0 : Recipient address rejected: User unknown; from= to= proto=ESMTP helo=<[10.0.0.1]> +#TDsdN reject: RCPT from host[10.0.0.1]: 450 : Recipient address rejected: Greylisted; from= to= proto=ESMTP helo= +#TDsdN reject: RCPT from host[10.0.0.1]: 454 4.7.1 : Recipient address rejected: Access denied; from= to= proto=SMTP helo= +#TDsdN reject_warning: RCPT from host[10.0.0.1]: 454 4.7.1 : Recipient address rejected: Access denied; from= to= proto=ESMTP helo= +#TDsdN reject: RCPT from host[10.0.0.1]: 450 4.1.2 : Recipient address rejected: Domain not found; from= to= proto=ESMTP helo= +#TDsdN reject: RCPT from host[10.0.0.1]: 554 : Recipient address rejected: Please see http://www.openspf.org/why.html?sender=from%40example.net&ip=10.0.0.1&receiver=example.net; from= to= proto=ESMTP helo= +#TDsdN reject: RCPT from host[10.0.0.1]: 550 : Recipient address rejected: undeliverable address: host example.net[192.168.0.1] said: 550 : User unknown in virtual alias table (in reply to RCPT TO command); from= to= proto=SMTP helo= +#TDsdN reject: RCPT from host[10.0.0.1]: 554 : Recipient address rejected: Please see http://spf.pobox.com/why.html?sender=user%40example.com&ip=10.0.0.1&receiver=mail; from= to= proto=ESMTP helo=<10.0.0.1> +#TDsdN reject: RCPT from host[10.0.0.1]: 554 : Relay access denied; from= to= proto=SMTP helo= +#TDsdN reject_warning: HELO from host[10.0.0.1]: 554 : Relay access denied; proto=SMTP helo= +#TDsdN reject: RCPT from host[10.0.0.1]: 450 4.1.8 : Sender address rejected: Domain not found; from= to= proto=ESMTP helo= +#TDsdN reject_warning: RCPT from host[10.0.0.1]: 450 4.1.8 : Sender address rejected: Domain not found; from= to= proto=ESMTP helo= +#TDsdN reject: RCPT from host[10.0.0.1]: 550 : Sender address rejected: undeliverable address: host example.net[10.0.0.1] said: 550 : User unknown in virtual alias table (in reply to RCPT TO command); from= to= proto=SMTP helo= +#TDsdN reject_warning: RCPT from host[10.0.0.1]: 554 : Client host rejected: Access denied; from= to= proto=SMTP helo= +#TDsdN reject: RCPT from host[10.0.0.1]: 554 : Client host rejected: Optional text; from= to= proto=SMTP helo= +#TDsdN reject: CONNECT from host[10.0.0.1]: 503 5.5.0 : Client host rejected: Improper use of SMTP command pipelining; proto=SMTP + +#TDsdN reject_warning: RCPT from unk[10.0.0.1]: 450 Client host rejected: cannot find your hostname, [10.0.0.1]; from= to= proto=ESMTP helo= +#TDsdN reject: RCPT from unk[10.0.0.1]: 450 Client host rejected: cannot find your hostname, [10.0.0.1]; from= to= proto=ESMTP helo= +#TDsdN reject: RCPT from unk[10.0.0.1]: 450 Client host rejected: cannot find your hostname, [10.0.0.1]; proto=ESMTP +#TDsdN reject: RCPT from unk[10.0.0.1]: 550 5.7.1 Client host rejected: cannot find your reverse hostname, [10.0.0.1] +#TDsdN reject: CONNECT from unk[unknown]: 421 4.7.1 Client host rejected: cannot find your reverse hostname, [unknown]; proto=SMTP + +#TDsdN reject: RCPT from host[10.0.0.1]: 554 5.7.1 Service unavailable; Client host [10.0.0.1] blocked using sbl-xbl.spamhaus.org; http://www.spamhaus.org/query/bl?ip=10.0.0.1; from= to= proto=ESMTP helo= +#TDsdN reject_warning: RCPT from host[10.0.0.1]: 554 5.7.1 Service unavailable; Client host [10.0.0.1] blocked using sbl-xbl.spamhaus.org; http://www.spamhaus.org/query/bl?ip=10.0.0.1; from= to= proto=ESMTP helo= +#TDsdN reject: RCPT from host[10.0.0.1]: 554 Service denied; Client host [10.0.0.1] blocked using bl.spamcop.net; Blocked - see http://www.spamcop.net/bl.shtml?83.164.27.124; from= to= proto=ESMTP helo= +#TDsdN reject: RCPT from host[10.0.0.1]: 454 4.7.1 : Helo command rejected: Access denied; from= to= proto=SMTP helo= +#TDsdN reject_warning: RCPT from host[10.0.0.1]: 454 4.7.1 : Helo command rejected: Access denied; from= to= proto=SMTP helo= +#TDsdN reject: EHLO from host[10.0.0.1]: 504 5.5.2 : Helo command rejected: need fully-qualified hostname; proto=SMTP helo= +#TDsdQ reject: DATA from host[10.0.0.1]: 550 5.5.3 : Data command rejected: Multi-recipient bounce; from=<> proto=ESMTP helo= +#TDsdN reject: ETRN from host[10.0.0.1]: 554 5.7.1 : Etrn command rejected: Access denied; proto=ESMTP helo= +#TDsdN reject: RCPT from host[10.0.0.1]: 452 Insufficient system storage; from= to= +#TDsdN reject_warning: RCPT from host[10.0.0.1]: 451 4.3.5 Server configuration error; from= to= proto=ESMTP helo= +#TDsdN reject: RCPT from host[10.0.0.1]: 450 Server configuration problem; from= to= proto=ESMTP helo= +#TDsdN reject: MAIL from host[10.0.0.1]: 552 Message size exceeds fixed limit; proto=ESMTP helo= +#TDsdN reject: RCPT from unk[10.0.0.1]: 554 5.7.1 : Unverified Client host rejected: Access denied; from= to= proto=SMTP helo= +#TDsdN reject: MAIL from host[10.0.0.1]: 451 4.3.0 : Temporary lookup failure; from= proto=ESMTP helo= + + # reject, reject_warning + if ($action =~ /^reject/) { + my ($recip); + + if ($p1 !~ /^($re_DSN) (.*)$/o) { + inc_unmatched('reject1'); + next; + } + ($dsn,$p1) = ($1,$2); #print "dsn: $dsn, p1: \"$p1\"\n"; + $fmthost = formathost($hostip,$host); + + # reject_warning override temp or perm reject types + $rej_type = ($action eq 'reject_warning' ? 'warn' : get_reject_key($dsn)); + #print "REJECT stage: '$rej_type'\n"; + + if ($Collecting{'byiprejects'} and substr($rej_type,0,1) eq '5') { + $Counts{'byiprejects'}{$fmthost}++; + } + + if ($stage eq 'VRFY') { + if ($p1 =~ /^(?:<(\S*)>: )?(.*);$/) { + my ($trigger,$reason) = ($1,$2); + $Totals{$reject_name = "${rej_type}rejectverify" }++; next unless ($Collecting{$reject_name}); + + if ($reason =~ /^Service unavailable; Client host \[[^]]+\] (blocked using [^;]*);/) { + $reason = join (' ', 'Client host blocked using', $1); + $trigger = ''; + } + $Counts{$reject_name}{$reason}{$fmthost}{ucfirst($trigger)}++; + } else { + inc_unmatched('vrfyfrom'); + } + next; + } + + # XXX there may be several semicolon-separated messages + # Recipient address rejected: Unknown users and via check_recipient_access + if ($p1 =~ /^<(.*)>: Recipient address rejected: ([^;]*);/) { + ($recip,$reason) = ($1,$2); + + my ($localpart,$domainpart); + # Unknown users; local mailbox, alias, virtual, relay user, unspecified + if ($recip eq '') { ($localpart, $domainpart) = ('<>', '*unspecified'); } + else { + ($localpart, $domainpart) = split (/@/, lc $recip); + ($localpart, $domainpart) = ($recip, '*unspecified') if ($domainpart eq ''); + } + + if ($reason =~ s/^User unknown *//) { + $Totals{$reject_name = "${rej_type}rejectunknownuser" }++; next unless ($Collecting{$reject_name}); + + my ($table) = ($reason =~ /^in ((?:\w+ )+table)/); + $table = 'Address table unavailable' if ($table eq ''); # when show_user_unknown_table_name=no + $Counts{$reject_name}{ucfirst($table)}{$domainpart}{$localpart}{$fmthost}++; + } else { + # check_recipient_access + $Totals{$reject_name = "${rej_type}rejectrecip" }++; next unless ($Collecting{$reject_name}); + + if ($reason =~ m{^Please see http://[^/]+/why\.html}) { + $reason = 'SPF reject'; + } + elsif ($reason =~ /^undeliverable address: host ([^[]+)\[([^]]+)\](?::\d+)? said:/) { + $reason = 'undeliverable address: remote host rejected recipient'; + } + $Counts{$reject_name}{ucfirst($reason)}{$domainpart}{$localpart}{$fmthost}++; + } + + } elsif ($p1 =~ /^<(.*?)>.* Relay access denied/) { + $Totals{$reject_name = "${rej_type}rejectrelay" }++; next unless ($Collecting{$reject_name}); + $Counts{$reject_name}{$fmthost}{$eto}++; + + } elsif ($p1 =~ /^<(.*)>: Sender address rejected: (.*);/) { + $Totals{$reject_name = "${rej_type}rejectsender" }++; next unless ($Collecting{$reject_name}); + ($from,$reason) = ($1,$2); + + if ($reason =~ /^undeliverable address: host ([^[]+)\[([^]]+)\](?::\d+)? said:/) { + $reason = 'undeliverable address: remote host rejected sender'; + } + $Counts{$reject_name}{ucfirst($reason)}{$fmthost}{$from ne '' ? $from : '<>'}++; + + } elsif ($p1 =~ /^(?:<.*>: )?Unverified Client host rejected: /) { + # check_reverse_client_hostname_access (postfix 2.6+) + $Totals{$reject_name = "${rej_type}rejectunverifiedclient" }++; next unless ($Collecting{$reject_name}); + $Counts{$reject_name}{$fmthost}{$helo}{$eto}{$efrom}++; + + } elsif ($p1 =~ s/^(?:<.*>: )?Client host rejected: //) { + # reject_unknown_client + # client IP->name mapping fails + # name->IP mapping fails + # name->IP mapping =! client IP + if ($p1 =~ /^cannot find your hostname/) { + $Totals{$reject_name = "${rej_type}rejectunknownclient" }++; next unless ($Collecting{$reject_name}); + $Counts{$reject_name}{$fmthost}{$helo}{$eto}{$efrom}++; + } + # reject_unknown_reverse_client_hostname (no PTR record for client's IP) + elsif ($p1 =~ /^cannot find your reverse hostname/) { + $Totals{$reject_name = "${rej_type}rejectunknownreverseclient" }++; next unless ($Collecting{$reject_name}); + $Counts{$reject_name}{$hostip}++ + } + else { + $Totals{$reject_name = "${rej_type}rejectclient" }++; next unless ($Collecting{$reject_name}); + $p1 =~ s/;$//; + $Counts{$reject_name}{ucfirst($p1)}{$fmthost}{$eto}{$efrom}++; + } + } elsif ($p1 =~ /^Service (?:temporarily )?(?:unavailable|denied)[^;]*; (?:(?:Unverified )?Client host |Sender address |Helo command )?\[[^ ]*\] blocked using ([^;]+);/) { + # Note: similar code below: search RejectRBL + + # postfix 2.1 + #TDsdN reject: RCPT from example.com[10.0.0.5]: 554 Service unavailable; Client host [10.0.0.5] blocked using bl.spamcop.net; Blocked - see http://www.spamcop.net/bl.shtml?10.0.0.5; from= to= proto=ESMTP helo= + # postfix 2.3+ + #TDsdN reject: RCPT from example.com[10.0.0.6]: 554 5.7.1 Service unavailable; Client host [10.0.0.6] blocked using bl.spamcop.net; Blocked - see http://www.spamcop.net/bl.shtml?10.0.0.6; from= to= proto=SMTP helo= + #TDsdN reject: RCPT from example.com[10.0.0.1]: 550 5.7.1 Service unavailable; Client host [10.0.0.1] blocked using Trend Micro RBL+. Please see http://www.mail-abuse.com/cgi-bin/lookup?ip_address=10.0.0.1; Mail from 10.0.0.1 blocked using Trend Micro Email Reputation database. Please see ; from= to= proto=SMTP helo=<10.0.0.1> + + $Totals{$reject_name = "${rej_type}rejectrbl" }++; next unless ($Collecting{$reject_name}); + ($site,$reason) = ($1 =~ /^(.+?)(?:$|(?:[.,] )(.*))/); + $reason =~ s/^reason: // if ($reason); + $Counts{$reject_name}{$site}{$fmthost}{$reason ? $reason : ''}++; + + } elsif ($p1 =~ /^<.*>: Helo command rejected: (.*);$/) { + $Totals{$reject_name = "${rej_type}rejecthelo" }++; next unless ($Collecting{$reject_name}); + $Counts{$reject_name}{ucfirst($1)}{$fmthost}{$helo}++; + + } elsif ($p1 =~ /^<.*>: Etrn command rejected: (.*);$/) { + $Totals{$reject_name = "${rej_type}rejectetrn" }++; next unless ($Collecting{$reject_name}); + $Counts{$reject_name}{ucfirst($1)}{$fmthost}{$helo}++; + + } elsif ($p1 =~ /^<.*>: Data command rejected: (.*);$/) { + $Totals{$reject_name = "${rej_type}rejectdata" }++; next unless ($Collecting{$reject_name}); + $Counts{$reject_name}{$1}{$fmthost}{$helo}++; + + } elsif ($p1 =~ /^Insufficient system storage;/) { + $Totals{'warninsufficientspace'}++; # force display in Warnings section also + $Totals{$reject_name = "${rej_type}rejectinsufficientspace" }++; next unless ($Collecting{$reject_name}); + $Counts{$reject_name}{$fmthost}{$eto}{$efrom}++; + + } elsif ($p1 =~ /^Server configuration (?:error|problem);/) { + $Totals{'warnconfigerror'}++; # force display in Warnings section also + $Totals{$reject_name = "${rej_type}rejectconfigerror" }++; next unless ($Collecting{$reject_name}); + $Counts{$reject_name}{$fmthost}{$eto}{$efrom}++; + + } elsif ($p1 =~ /^Message size exceeds fixed limit;$/) { + # Postfix responds with this message after a MAIL FROM:<...> SIZE=nnn command, where postfix consider's nnn excessive + # Note: similar code below: search RejectSize + # Note: reject_warning does not seem to occur + $Totals{$reject_name = "${rej_type}rejectsize" }++; next unless ($Collecting{$reject_name}); + $Counts{$reject_name}{$fmthost}{$eto}{$efrom}++; + + } elsif ($p1 =~ /^<(.*?)>: Temporary lookup failure;/) { + $Totals{$reject_name = "${rej_type}rejectlookupfailure" }++; next unless ($Collecting{$reject_name}); + $Counts{$reject_name}{$fmthost}{$eto}{$efrom}++; + + # This would capture all other rejects, but I think it might be more useful to add + # additional capture sections based on user reports of uncapture lines. + # + #} elsif ( ($reason) = ($p1 =~ /^([^;]+);/)) { + # $Totals{$rej_type . 'rejectother'}++; + # $Counts{$rej_type . 'rejectother'}{$reason}++; + } else { + inc_unmatched('rejectother'); + } + } + # end of $re_QID: reject: + +# QID: ACTION STAGE from host[hostip]: trigger: reason; ftph +# +#TDsdN warn: RCPT from host[10.0.0.1]: TEST access WARN action; from= to= proto=ESMTP helo= +#TDsdN warn: RCPT from host[10.0.0.1]: ; from= to= proto=ESMTP helo= +#TDsdN discard: RCPT from host[10.0.0.1]: : Sender address TEST DISCARD action; from= to= proto=ESMTP helo= +#TDsdN discard: RCPT from host[10.0.0.1]: : Client host TEST DISCARD action w/ip(client_checks); from= to= proto=ESMTP helo= +#TDsdN discard: RCPT from host[10.0.0.1]: : Unverified Client host triggers DISCARD action; from= to= proto=ESMTP helo=<10.0.0.1> +#TDsdN hold: RCPT from host[10.0.0.1]: : Recipient address triggers HOLD action; from= to= proto=SMTP helo=<10.0.0.1> +#TDsdN hold: RCPT from host[10.0.0.1]: : Helo command optional text...; from= to= proto=ESMTP helo= +#TDsdN hold: RCPT from host[10.0.0.1]: : Helo command triggers HOLD action; from= to= proto=ESMTP helo= +#TDsdN hold: DATA from host[10.0.0.1]: : Helo command triggers HOLD action; from= to= proto=ESMTP helo= +#TDsdN filter: RCPT from host[10.0.0.1]: <>: Sender address triggers FILTER filter:somefilter; from=<> to= proto=SMTP helo= +#TDsdN filter: RCPT from host[10.0.0.1]: : Recipient address triggers FILTER smtp-amavis:[127.0.0.1]:10024; from= to= proto=SMTP helo= +#TDsdN redirect: RCPT from host[10.0.0.1]: : Client host triggers REDIRECT root@localhost; from= to= proto=SMTP helo= +#TDsdN redirect: RCPT from host[10.0.0.1]: : Recipient address triggers REDIRECT root@localhost; from= to= proto=ESMTP helo= + +# BCC action (postfix 2.6+) +#TDsdN bcc: RCPT from host[10.0.0.1]: : Sender address triggers BCC root@localhost; from= to= proto=ESMTP helo= + + # $re_QID: discard, filter, hold, redirect, warn, bcc, replace ... + else { + my $trigger; + ($trigger,$reason) = ($p1 =~ /^(?:<(\S*)>: )?(.*);$/ ); + if ($trigger eq '') { $trigger = '*unavailable'; } + else { $trigger =~ s/^<(.+)>$/$1/; } + $reason = '*unavailable' if ($reason eq ''); + $fmthost = formathost ($hostip,$host); + #print "trigger: \"$trigger\", reason: \"$reason\"\n"; + + # reason -> subject text + # subject -> "Helo command" : smtpd_helo_restrictions + # subject -> "Client host" : smtpd_client_restrictions + # subject -> "Unverified Client host" : smtpd_client_restrictions + # subject -> "Client certificate" : smtpd_client_restrictions + # subject -> "Sender address" : smtpd_sender_restrictions + # subject -> "Recipient address" : smtpd_recipient_restrictions + + # subject -> "Data command" : smtpd_data_restrictions + # subject -> "End-of-data" : smtpd_end_of_data_restrictions + # subject -> "Etrn command" : smtpd_etrn_restrictions + + # text -> triggers action|triggers |optional text... + + my ($subject, $text) = + ($reason =~ /^((?:Recipient|Sender) address|(?:Unverified )?Client host|Client certificate|(?:Helo|Etrn|Data) command|End-of-data) (.+)$/o); + #printf "ACTION: '$action', SUBJECT: %-30s TEXT: \"$text\"\n", '"' . $subject . '"'; + + if ($action eq 'filter') { + $Totals{'filtered'}++; next unless ($Collecting{'filtered'}); + # See "Note: Counts" before changing $Counts below re: Filtered + $text =~ s/triggers FILTER //; + if ($subject eq 'Recipient address') { $Counts{'filtered'}{$text}{$subject}{$trigger}{$efrom}{$fmthost}++; } + elsif ($subject =~ /Client host$/) { $Counts{'filtered'}{$text}{$subject}{$fmthost}{$eto}{$efrom}++; } + else { $Counts{'filtered'}{$text}{$subject}{$trigger}{$eto}{$fmthost}++; } + } + elsif ($action eq 'redirect') { + $Totals{'redirected'}++; next unless ($Collecting{'redirected'}); + $text =~ s/triggers REDIRECT //; + # See "Note: Counts" before changing $Counts below re: Redirected + if ($subject eq 'Recipient address') { $Counts{'redirected'}{$text}{$subject}{$trigger}{$efrom}{$fmthost}++; } + elsif ($subject =~ /Client host$/) { $Counts{'redirected'}{$text}{$subject}{$fmthost}{$eto}{$efrom}++; } + else { $Counts{'redirected'}{$text}{$subject}{$trigger}{$eto}{$fmthost}++; } + } + # hold, discard, and warn allow "optional text" + elsif ($action eq 'hold') { + $Totals{'hold'}++; next unless ($Collecting{'hold'}); + # See "Note: Counts" before changing $Counts below re: Hold + $subject = $reason unless $text eq 'triggers HOLD action'; + if ($subject eq 'Recipient address') { $Counts{'hold'}{$subject}{$trigger}{$efrom}{$fmthost}++; } + elsif ($subject =~ /Client host$/) { $Counts{'hold'}{$subject}{$fmthost}{$eto}{$efrom}++; } + else { $Counts{'hold'}{$subject}{$trigger}{$eto}{$fmthost}++; } + } + elsif ($action eq 'discard') { + $Totals{'discarded'}++; next unless ($Collecting{'discarded'}); + # See "Note: Counts" before changing $Counts below re: Discarded + $subject = $reason unless $text eq 'triggers DISCARD action'; + if ($subject eq 'Recipient address') { $Counts{'discarded'}{$subject}{$trigger}{$efrom}{$fmthost}++; } + elsif ($subject =~ /Client host$/) { $Counts{'discarded'}{$subject}{$fmthost}{$eto}{$efrom}++; } + else { $Counts{'discarded'}{$subject}{$trigger}{$eto}{$fmthost}++; } + } + elsif ($action eq 'warn') { + $Totals{'warned'}++; next unless ($Collecting{'warned'}); + $Counts{'warned'}{$reason}{$fmthost}{$eto}{''}++; + # See "Note: Counts" before changing $Counts above... + } + elsif ($action eq 'bcc') { + $Totals{'bcced'}++; next unless ($Collecting{'bcced'}); + # See "Note: Counts" before changing $Counts below re: Filtered + $text =~ s/triggers BCC //o; + if ($subject eq 'Recipient address') { $Counts{'bcced'}{$text}{$subject}{$trigger}{$efrom}{$fmthost}++; } + elsif ($subject =~ /Client host$/) { $Counts{'bcced'}{$text}{$subject}{$fmthost}{$eto}{$efrom}++; } + else { $Counts{'bcced'}{$text}{$subject}{$trigger}{$eto}{$fmthost}++; } + } + else { + die "Unexpected ACTION: '$action'"; + } + } + } + + elsif ($p1 =~ s/^client=(([^ ]*)\[([^ ]*)\](?::(?:\d+|unknown))?)//) { + my ($hip,$host,$hostip) = ($1,$2,$3); + + # Increment accepted when the client connection is made and smtpd has a QID. + # Previously, accepted was being incorrectly incremented when the first qmgr + # "from=xxx, size=nnn ..." line was seen. This is erroneous when the smtpd + # client connection occurred outside the date range of the log being analyzed. + $AcceptedByQid{$qid} = $hip; + $Totals{'msgsaccepted'}++; + + #TDsdQ client=unknown[192.168.0.1] + #TDsdQ client=unknown[192.168.0.1]:unknown + #TDsdQ client=unknown[192.168.0.1]:10025 + #TDsd client=example.com[192.168.0.1], helo=example.com + #TDsdQ client=mail.example.com[2001:dead:beef::1] + + #TDsdQ client=localhost[127.0.0.1], sasl_sender=someone@example.com + #TDsdQ client=example.com[192.168.0.1], sasl_method=PLAIN, sasl_username=anyone@sample.net + #TDsdQ client=example.com[192.168.0.1], sasl_method=LOGIN, sasl_username=user@example.com, sasl_sender= + #TDsdQ client=unknown[10.0.0.1], sasl_sender=user@examine.com + next if ($p1 eq ''); + my ($method,$user,$sender) = ($p1 =~ /^(?:, sasl_method=([^,]+))?(?:, sasl_username=([^,]+))?(?:, sasl_sender=?)?$/); + + # sasl_sender occurs when AUTH verb is present in MAIL FROM, typically used for relaying + # the username (eg. sasl_username) of authenticated users. + if ($sender or $method or $user) { + $Totals{'saslauth'}++; next unless ($Collecting{'saslauth'}); + $method ||= '*unknown method'; + $user ||= '*unknown user'; + $Counts{'saslauth'}{$user . ($sender ? " ($sender)" : '')}{$method}{formathost($hostip,$host)}++; + } + } + + # ^$re_QID: ... (not access(5) action) + elsif ($p1 =~ /^from=<(.*?)>, size=(\d+), nrcpt=(\d+)/) { + my ($efrom,$bytes,$nrcpt) = ($1,$2,$3); + #TDsdQ from=, size=4051, nrcpt=1 (queue active) + #TDsdQ(12) from=, size=25302, nrcpt=2 (queue active) + #TDsdQ from=, size=5529, nrcpt=1 (queue active) + #TDsdQ from=, size=5335, nrcpt=1 (queue active) + + # Distinguish bytes accepted vs. bytes delivered due to multiple recips + + # Increment bytes accepted on the first qmgr "from=..." line... + next if (exists $SizeByQid{$qid}); + $SizeByQid{$qid} = $bytes; + # ...but only when the smtpd "client=..." line has been seen too. + # This under-counts when the smtpd "client=..." connection log entry and the + # qmgr "from=..." log entry span differnt periods (as fed to postfix-logwatch). + next if (! exists $AcceptedByQid{$qid}); + + $Totals{'bytesaccepted'} += $bytes; + + $Counts{'envelopesenders'}{$efrom ne '' ? $efrom : '<>'}++ if ($Collecting{'envelopesenders'}); + if ($Collecting{'envelopesenderdomains'}) { + my ($localpart, $domain); + if ($efrom eq '') { ($localpart, $domain) = ('<>', '*unknown'); } + else { ($localpart, $domain) = split (/@/, lc $efrom); } + + $Counts{'envelopesenderdomains'}{$domain ne '' ? $domain : '*unknown'}{$localpart}++; + } + delete $AcceptedByQid{$qid}; # prevent incrementing BytesAccepted again + } + + ### sent, forwarded, bounced, softbounce, deferred, (un)deliverable + elsif ($p1 =~ s/^to=<(.*?)>,(?: orig_to=<(.*?)>,)? relay=([^,]*).*, ($re_DDD), status=(\S+) //o) { + ($relay,$status) = ($3,$5); + + my ($to,$origto,$localpart,$domainpart,$dsn,$p1) = process_delivery_attempt ($1,$2,$4,$p1); + + #TD 552B6C20E: to=, relay=mail.example.net[10.0.0.1]:25, delay=1021, delays=1020/0.04/0.56/0.78, dsn=2.0.0, status=sent (250 Ok: queued as 6EAC4719EB) + #TD 552B6C20E: to=, relay=mail.example.net[10.0.0.1]:25, conn_use=2 delay=1021, delays=1020/0.04/0.56/0.78, dsn=2.0.0, status=sent (250 Ok: queued as 6EAC4719EB) + #TD DD925BBE2: to=, orig_to=, relay=mail.example.net[2001:dead:beef::1], delay=2, status=sent (250 Ok: queued as 5221227246) + + ### sent + if ($status eq 'sent') { + if ($p1 =~ /forwarded as /) { + $Totals{'bytesforwarded'} += $SizeByQid{$qid} if (exists $SizeByQid{$qid}); + $Totals{'forwarded'}++; next unless ($Collecting{'forwarded'}); + $Counts{'forwarded'}{$domainpart}{$localpart}{$origto}++; + } + else { + if ($service_name eq 'lmtp') { + $Totals{'bytessentlmtp'} += $SizeByQid{$qid} if (exists $SizeByQid{$qid}); + $Totals{'sentlmtp'}++; next unless ($Collecting{'sentlmtp'}); + $Counts{'sentlmtp'}{$domainpart}{$localpart}{$origto}++; + } + elsif ($service_name eq 'smtp') { + $Totals{'bytessentsmtp'} += $SizeByQid{$qid} if (exists $SizeByQid{$qid}); + $Totals{'sent'}++; next unless ($Collecting{'sent'}); + $Counts{'sent'}{$domainpart}{$localpart}{$origto}++; + } + # virtual, command, ... + else { + $Totals{'bytesdelivered'} += $SizeByQid{$qid} if (exists $SizeByQid{$qid}); + $Totals{'delivered'}++; next unless ($Collecting{'delivered'}); + $Counts{'delivered'}{$domainpart}{$localpart}{$origto}++; + } + } + } + + elsif ($status eq 'deferred') { + #TDsQ to=, relay=none, delay=27077, delays=27077/0/0.57/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=example.com type=MX: Host not found, try again) + #TDsQ to=, relay=none, delay=141602, status=deferred (connect to mx1.example.com[10.0.0.1]: Connection refused) + #TDsQ to=, relay=none, delay=141602, status=deferred (delivery temporarily suspended: connect to example.com[192.168.0.1]: Connection refused) + #TDsQ to=, relay=none, delay=306142, delays=306142/0.04/0.18/0, dsn=4.4.1, status=deferred (connect to example.com[10.0.0.1]: Connection refused) + #TDsQ to=, relay=example.org[10.0.0.1], delay=48779, status=deferred (lost connection with mail.example.org[10.0.0.1] while sending MAIL FROM) + #TDsQ to=, relay=sample.net, delay=26541, status=deferred (conversation with mail.example.com timed out while sending end of data -- message may be sent more than once) + #TDsQ to=, relay=sample.net[10.0.0.1]:25, delay=322, delays=0.04/0/322/0, dsn=4.4.2, status=deferred (conversation with example.com[10.0.0.01] timed out while receiving the initial server greeting) + #TDsQ to=, orig_to=, relay=none, delay=238024, status=deferred (delivery temporarily suspended: transport is unavailable) + + # XXX postfix reports dsn=5.0.0, host's reply may contain its own dsn's such as 511 and #5.1.1 + # XXX should these be used instead? + #TDsQ to=, relay=sample.net[10.0.0.1]:25, delay=5.7, delays=0.05/0.02/5.3/0.3, dsn=4.7.1, status=deferred (host sample.net[10.0.0.1] said: 450 4.7.1 : Recipient address rejected: Greylisted (in reply to RCPT TO command)) + #TDsQ to=, relay=example.com[10.0.0.1]:25, delay=79799, delays=79797/0.02/0.4/1.3, dsn=4.0.0, status=deferred (host example.com[10.0.0.1] said: 450 : User unknown in local recipient table (in reply to RCPT TO command)) + #TDsQ to=, relay=example.com[10.0.0.1]:25, delay=97, delays=0.03/0/87/10, dsn=4.0.0, status=deferred (host example.com[10.0.0.1] said: 450 : Recipient address rejected: undeliverable address: User unknown in virtual alias table (in reply to RCPT TO command)) + + ($reply,$fmthost) = cleanhostreply($p1,$relay,$to,$domainpart); + + $Totals{'deferred'}++ if ($DeferredByQid{$qid}++ == 0); + $Totals{'deferrals'}++; next unless ($Collecting{'deferrals'}); + $Counts{'deferrals'}{get_dsn_msg($dsn)}{$reply}{$domainpart}{$localpart}{$fmthost}++; + } + + ### bounced + elsif ($status eq 'bounced' or $status eq 'SOFTBOUNCE') { + # local agent + #TDlQ to=, relay=local, delay=2.5, delays=2.1/0.22/0/0.21, dsn=5.1.1, status=bounced (unknown user: "friend") + + # smtp agent + #TDsQ to=, orig_to=, relay=sample.net[10.0.0.1]:25, delay=22, delays=0.02/0.09/22/0.07, dsn=5.0.0, status=bounced (host sample.net[10.0.0.1] said: 551 invalid address (in reply to MAIL FROM command)) + + #TDsQ to=, relay=sample.net[10.0.0.1]:25, delay=11, delays=0.13/0.07/0.98/0.52, dsn=5.0.0, status=bounced (host sample.net[10.0.0.1] said: 550 MAILBOX NOT FOUND (in reply to RCPT TO command)) + #TDsQ to=, orig_to=, relay=sample.net[10.0.0.1]:25, delay=22, delays=0.02/0.09/22/0.07, dsn=5.0.0, status=bounced (host sample.net[10.0.0.1] said: 551 invalid address (in reply to MAIL FROM command)) + + + #TDsQ to=, relay=none, delay=0.57, delays=0.57/0/0/0, dsn=5.4.6, status=bounced (mail for sample.net loops back to myself) + #TDsQ to=<>, relay=none, delay=1, status=bounced (mail for sample.net loops back to myself) + #TDsQ to=, relay=none, delay=0, status=bounced (Host or domain name not found. Name service error for name=unknown.com type=A: Host not found) + # XXX verify these... + #TD EB0B8770: to=, orig_to=, relay=none, delay=1, status=bounced (User unknown in virtual alias table) + #TD EB0B8770: to=, orig_to=, relay=sample.net[192.168.0.1], delay=1.1, status=bounced (User unknown in relay recipient table) + #TD D8962E54: to=, relay=local, conn_use=2 delay=0.21, delays=0.05/0.02/0/0.14, dsn=4.1.1, status=SOFTBOUNCE (unknown user: "to") + #TD F031C832: to=, orig_to=, relay=local, delay=0.17, delays=0.13/0.01/0/0.03, dsn=5.1.1, status=bounced (unknown user: "to") + + #TD C76431E2: to=, relay=local, delay=2, status=SOFTBOUNCE (host sample.net[192.168.0.1] said: 450 : User unknown in local recipient table (in reply to RCPT TO command)) + #TD 04B0702E: to=, relay=example.com[10.0.0.1]:25, delay=12, delays=6.5/0.01/0.03/5.1, dsn=5.1.1, status=bounced (host example.com[10.0.0.1] said: 550 5.1.1 User unknown (in reply to RCPT TO command)) + #TD 9DAC8B2D: to=, relay=example.com[10.0.0.1]:25, delay=1.4, delays=0.04/0/0.27/1.1, dsn=5.0.0, status=bounced (host example.com[10.0.0.1] said: 511 sorry, no mailbox here by that name (#5.1.1 - chkuser) (in reply to RCPT TO command)) + #TD 79CB702D: to=, relay=example.com[10.0.0.1]:25, delay=0.3, delays=0.04/0/0.61/0.8, dsn=5.0.0, status=bounced (host example.com[10.0.0.1] said: 550 , Recipient unknown (in reply to RCPT TO command)) + #TD 88B7A079: to=, relay=example.com[10.0.0.1]:25, delay=45, delays=0.03/0/5.1/40, dsn=5.0.0, status=bounced (host example.com[10.0.0.1] said: 550-"The recipient cannot be verified. Please check all recipients of this 550 message to verify they are valid." (in reply to RCPT TO command)) + #TD 47B7B074: to=, relay=example.com[10.0.0.1]:25, delay=6.6, delays=6.5/0/0/0.11, dsn=5.1.1, status=bounced (host example.com[10.0.0.1] said: 550 5.1.1 User unknown; rejecting (in reply to RCPT TO command)) + #TDppQ to=, relay=dbmail-pipe, delay=0.15, delays=0.09/0.01/0/0.06, dsn=5.3.0, status=bounced (Command died with signal 11: "/usr/sbin/dbmail-smtp") + + # print "bounce message from " . $to . " msg : " . $relay . "\n"; + + # See same code elsewhere "Note: Bounce" + ### local bounce + # XXX local v. remote bounce seems iffy, relative + if ($relay =~ /^(?:none|local|virtual|127\.0\.0\.1|maildrop|avcheck)/) { + $Totals{'bouncelocal'}++; next unless ($Collecting{'bouncelocal'}); + $Counts{'bouncelocal'}{get_dsn_msg($dsn)}{$domainpart}{ucfirst($p1)}{$localpart}++; + } + else { + $Totals{'bounceremote'}++; next unless ($Collecting{'bounceremote'}); + ($reply,$fmthost) = cleanhostreply($p1,$relay,$to,$domainpart); + $Counts{'bounceremote'}{get_dsn_msg($dsn)}{$domainpart}{$localpart}{$fmthost}{$reply}++; + } + } + + + elsif ($status =~ 'undeliverable') { + #TDsQ to=, relay=sample.com[10.0.0.1], delay=0, dsn=5.0.0, status=undeliverable (host sample.com[10.0.0.1] refused to talk to me: 554 5.7.1 example.com Connection not authorized) + #TDsQ to=, relay=mx.example.com[10.0.0.1]:25, conn_use=2, delay=5.5, delays=0.03/0/0.21/5.3, dsn=5.0.0, status=undeliverable-but-not-cached (host mx.example.com[10.0.0.1] said: 550 RCPT TO: User unknown (in reply to RCPT TO command)) + #TDvQ to=, relay=virtual, delay=0.14, delays=0.06/0/0/0.08, dsn=5.1.1, status=undeliverable (unknown user: "u@example.com") + #TDlQ to=, relay=local, delay=0.02, delays=0.01/0/0/0, dsn=5.1.1, status=undeliverable-but-not-cached (unknown user: "to") + $Totals{'undeliverable'}++; next unless ($Collecting{'undeliverable'}); + if ($p1 =~ /^unknown user: ".+?"$/) { + $Counts{'undeliverable'}{get_dsn_msg($dsn)}{'Unknown user'}{$domainpart}{$localpart}{$origto ? $origto : ''}++; + } + else { + my ($reply,$fmthost) = cleanhostreply($p1,'',$to ne '' ? $to : '<>',$domainpart); + $Counts{'undeliverable'}{get_dsn_msg($dsn)}{$reply}{$domainpart}{$localpart}{$fmthost}++; + } + } + + elsif ($status eq 'deliverable') { + # address verification, sendmail -bv deliverable reports + #TDvQ to=, relay=virtual, delay=0.09, delays=0.03/0/0/0.06, dsn=2.0.0, status=deliverable (delivers to maildir) + $Totals{'deliverable'}++; next unless ($Collecting{'deliverable'}); + my $dsn = ($p1 =~ s/^($re_DSN) // ? $1 : '*unavailable'); + $Counts{'deliverable'}{$dsn}{$p1}{$origto ? "$to ($origto)" : $to}++; + } + + else { + # keep this as the last condition in this else clause + inc_unmatched('unknownstatus'); + } + } # end of sent, forwarded, bounced, softbounce, deferred, (un)deliverable + + # pickup + elsif ($p1 =~ /^(uid=\S* from=<.*?>)/) { + #TDpQ2 uid=0 from= + $AcceptedByQid{$qid} = $1; + $Totals{'msgsaccepted'}++; + } + + elsif ($p1 =~ /^from=<(.*?)>, status=expired, returned to sender$/) { + #TDqQ from=, status=expired, returned to sender + $Totals{'returnedtosender'}++; next unless ($Collecting{'returnedtosender'}); + $Counts{'returnedtosender'}{$1 ne '' ? $1 : '<>'}++; + } + + elsif ($p1 =~ s/^host ([^[]+)\[([^]]+)\](?::\d+)? refused to talk to me://) { + #TDsQ host mail.example.com[10.0.0.1] refused to talk to me: 553 Connections are being blocked due to previous incidents of abuse + #TDsQ host mail.example.com[10.0.0.1] refused to talk to me: 501 Connection from 192.168.2.1 (XY) rejected + # Note: See ConnectToFailure above + $Totals{'connecttofailure'}++; next unless ($Collecting{'connecttofailure'}); + $Counts{'connecttofailure'}{'Refused to talk to me'}{formathost($2,$1)}{$p1}++; + } + + elsif ($p1 =~ /^lost connection with ([^[]*)\[([^]]+)\](?::\d+)? (while .*)$/) { + # outbound smtp + #TDsQ lost connection with sample.net[10.0.0.1] while sending MAIL FROM + #TDsQ lost connection with sample.net[10.0.0.2] while receiving the initial server greeting + $Totals{'connectionlostoutbound'}++; next unless ($Collecting{'connectionlostoutbound'}); + $Counts{'connectionlostoutbound'}{ucfirst($3)}{formathost($2,$1)}++; + } + + elsif ($p1 =~ /^conversation with ([^[]*)\[([^]]+)\](?::\d+)? timed out (while .*)$/) { + #TDsQ conversation with sample.net[10.0.0.1] timed out while receiving the initial SMTP greeting + # Note: see TimeoutInbound below + $Totals{'timeoutinbound'}++; next unless ($Collecting{'timeoutinbound'}); + $Counts{'timeoutinbound'}{ucfirst($3)}{formathost($2,$1)}{''}++; + } + + elsif ($p1 =~ /^enabling PIX (\.) workaround for ([^[]+)\[([^]]+)\](?::\d+)?/ or + $p1 =~ /^enabling PIX workarounds: (.*) for ([^[]+)\[([^]]+)\](?::\d+)?/) { + #TDsQ enabling PIX . workaround for example.com[192.168.0.1] + #TDsQ enabling PIX . workaround for mail.sample.net[10.0.0.1]:25 + #TDsQ enabling PIX workarounds: disable_esmtp delay_dotcrlf for spam.example.org[10.0.0.1]:25 + $Totals{'pixworkaround'}++; next unless ($Collecting{'pixworkaround'}); + $Counts{'pixworkaround'}{$1}{formathost($3,$2)}++; + } + + # milter-reject, milter-hold, milter-discard + elsif ($p1 =~ s/^milter-//) { + milter_common($p1); + } + + elsif ($p1 =~ s/^SASL (\[CACHED\] )?authentication failed; //) { + #TDsQ SASL authentication failed; cannot authenticate to server smtp.example.com[10.0.0.1]: no mechanism available + #TDsQ SASL authentication failed; server example.com[10.0.0.1] said: 535 Error: authentication failed + #TDsQ SASL [CACHED] authentication failed; server example.com[10.0.0.1] said: 535 Error: authentication failed + # see saslauthfail elsewhere + + $Totals{'saslauthfail'}++; next unless ($Collecting{'saslauthfail'}); + my $cached = $1; + + if ($p1 =~ /^(authentication protocol loop with server): ([^[]+)\[([^]]+)\](?::\d+)?$/) { + ($reason,$host,$hostip,$reason2) = ($1,$2,$3,''); + } + elsif ($p1 =~ /^(cannot authenticate to server) ([^[]+)\[([^]]+)\](?::\d+)?: (.*)$/) { + ($reason,$host,$hostip,$reason2) = ($1,$2,$3,$4); + } + elsif ($p1 =~ /^server ([^[]+)\[([^]]+)\](?::\d+)? said: (.+)$/) { + ($reason,$host,$hostip,$reason2) = ('server ... said',$1,$2,$3); + } + else { + inc_unmatched('saslauthfail'); + next; + } + + $reason .= ': ' . $reason2 if $reason2; + $Counts{'saslauthfail'}{$cached . $reason}{formathost($hostip,$host)}++; + } + + else { + # keep this as the last condition in this else clause + inc_unmatched('unknownqid') if ! in_ignore_list ($p1); + } + } + # end of $re_QID section + + elsif ($p1 =~ /^(timeout|lost connection) (after [^ ]+)(?: \((?:approximately )?(\d+) bytes\))? from ([^[]*)\[([^]]+)\](?::\d+)?$/) { + my ($lort,$reason,$bytes,$host,$hostip) = ($1,$2,$3,$4,$5); + if ($lort eq 'timeout') { + # see also TimeoutInbound in $re_QID section + #TDsd timeout after RSET from example.com[192.168.0.1] + #TDsd timeout after DATA (6253 bytes) from example.com[10.0.0.1] + + $Totals{'timeoutinbound'}++; next unless ($Collecting{'timeoutinbound'}); + $Counts{'timeoutinbound'}{ucfirst($reason)}{formathost($hostip,$host)}{commify($bytes)}++; + } else { + #TDsd lost connection after CONNECT from mail.example.com[192.168.0.1] + # postfix 2.5:20071003 + #TDsd lost connection after DATA (494133 bytes) from localhost[127.0.0.1] + # postfix 2.6:20080621 + #TDsd lost connection after DATA (approximately 0 bytes) from example.com[10.0.0.1] + + $Totals{'connectionlostinbound'}++; next unless ($Collecting{'connectionlostinbound'}); + $Counts{'connectionlostinbound'}{ucfirst($reason)}{formathost($hostip,$host)}{commify($bytes)}++; + } + } + + elsif ($p1 =~ /^(reject(?:_warning)?): RCPT from ([^[]+)\[([^]]+)\](?::\d+)?: ($re_DSN) Service (?:temporarily )?(?:unavailable|denied)[^;]*; (?:(?:Unverified )?Client host |Sender address |Helo command )?\[[^ ]*\] blocked using ([^;]+);/o) { + my ($rej_type,$host,$hostip,$dsn,) = ($1,$2,$3,$4); + ($site,$reason) = ($5 =~ /^(.+?)(?:$|(?:[.,] )(.*))/); + $reason =~ s/^reason: // if ($reason); + $rej_type = ($rej_type =~ /_warning/ ? 'warn' : get_reject_key($dsn)); + #print "REJECT RBL NOQ: '$rej_type'\n"; + # Note: similar code above: search RejectRBL + + # This section required: postfix didn't always log QID (eg. postfix 1.1) + # Also, "reason:" was probably always present in this case, but I'm not certain + # postfix 1.1 + #TDsd reject_warning: RCPT from example.com[10.0.0.1]: 554 Service unavailable; [10.0.0.1] blocked using orbz.org, reason: Open relay. Please see http://orbz.org/?10.0.0.1; from= to= + #TDsd reject: RCPT from example.com[10.0.0.2]: 554 Service unavailable; [10.0.0.2] blocked using orbz.org, reason: Open relay. Please see http://orbz.org/?10.0.0.2; from= to= + #TDsd reject: RCPT from unknown[10.0.0.3]: 554 Service unavailable; [10.0.0.3] blocked using bl.spamcop.net, reason: Blocked - see http://www.spamcop.net/bl.shtml?10.0.0.3; from= to= + #TDsd reject: RCPT from example.com[10.0.0.4]: 554 Service unavailable; [10.0.0.4] blocked using sbl.spamhaus.org, reason: http://www.spamhaus.org/SBL/sbl.lasso?query=B12057; from= to= + + if ($Collecting{'byiprejects'} and substr($rej_type,0,1) eq '5') { + $fmthost = formathost($hostip,$host); + $Counts{'byiprejects'}{$fmthost}++; + } + + $Totals{$reject_name = "${rej_type}rejectrbl" }++; next unless ($Collecting{$reject_name}); + $Counts{$reject_name}{$site}{$fmthost ? $fmthost : formathost($hostip,$host)}{$reason ? $reason : ''}++; + } + + # proxy-reject, proxy-accept + elsif ($p1 =~ s/^proxy-(reject|accept): ([^:]+): //) { + # 2.7 + #TDsdN proxy-accept: END-OF-MESSAGE: 250 2.0.0 Ok: queued as 9BE3547AFE; from= to= proto=ESMTP helo= + #TDsdN proxy-reject: END-OF-MESSAGE: 554 5.7.0 Reject, id=11912-03 - INFECTED: Eicar-Test-Signature; from= to= proto=ESMTP helo= + #TDsdN proxy-reject: END-OF-MESSAGE: ; from= to= proto=SMTP helo= + + next if $1 eq 'accept'; #ignore accepts + + my ($stage) = ($2); + my ($efrom,$eto,$proto,$helo) = strip_ftph($p1); + #print "efrom: '$efrom', eto: '$eto', proto: '$proto', helo: '$helo'\n"; + #print "stage: '$stage', reply: '$p1'\n"; + + my ($dsn,$reject_name); + ($dsn,$reply) = ($1,$2) if $p1 =~ /^($re_DSN) (.*)$/o; + #print " dsn: '$dsn', reply: '$reply', key: ", get_reject_key($dsn), "\n"; + # DSN may not be present. Can occur, for example, when queue file size limit is reached, + # which is logged as a Warning. Ignore these, since they can't be add to any + # reject section (no SMTP reply code). + if (! defined $dsn) { + next; + } + + $Totals{$reject_name = get_reject_key($dsn) . 'rejectproxy' }++; next unless ($Collecting{$reject_name}); + $Counts{$reject_name}{$stage}{$reply}{$eto}++; + } + + ### smtpd_tls_loglevel >= 1 + # Server TLS messages + elsif (($status,$host,$hostip,$type) = ($p1 =~ /^(?:(Anonymous|Trusted|Untrusted) )?TLS connection established from ([^[]+)\[([^]]+)\](?::\d+)?: (.*)$/)) { + #TDsd TLS connection established from example.com[192.168.0.1]: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits) + # Postfix 2.5+: status: Untrusted or Trusted + #TDsd Untrusted TLS connection established from example.com[192.168.0.1]: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits) + #TDsd Anonymous TLS connection established from localhost[127.0.0.1]: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits) + + $Totals{'tlsserverconnect'}++; next unless ($Collecting{'tlsserverconnect'}); + $Counts{'tlsserverconnect'}{$status ? "$status: $type" : $type}{formathost($hostip,$host)}++; + } + + # Client TLS messages + elsif ( ($status,$host,$type) = ($p1 =~ /^(?:(Verified|Trusted|Untrusted|Anonymous) )?TLS connection established to ([^ ]*): (.*)$/o)) { + #TD TLS connection established to example.com: TLSv1 with cipher AES256-SHA (256/256 bits) + # Postfix 2.5+: peer verification status: Untrusted, Trusted or Verified when + # server's trust chain is valid and peername is matched + #TD Verified TLS connection established to 127.0.0.1[127.0.0.1]:26: TLSv1 with cipher DHE-DSS-AES256-SHA (256/256 bits) + + $Totals{'tlsclientconnect'}++; next unless ($Collecting{'tlsclientconnect'}); + $Counts{'tlsclientconnect'}{$status ? "$status: $type" : $type}{$host}++; + } + + # smtp_tls_note_starttls_offer=yes + elsif ($p1 =~ /^Host offered STARTTLS: \[(.*)\]$/o) { + #TD Host offered STARTTLS: [mail.example.com] + $Totals{'tlsoffered'}++; next unless ($Collecting{'tlsoffered'}); + $Counts{'tlsoffered'}{$1}++; + } + + ### smtpd_tls_loglevel >= 1 + elsif ($p1 =~ /^Unverified: (.*)/o) { + #TD Unverified: subject_CN=(www|smtp|mailhost).(example.com|sample.net), issuer=someuser + $Totals{'tlsunverified'}++; next unless ($Collecting{'tlsunverified'}); + $Counts{'tlsunverified'}{$1}++; + } + + # Note: no QID + elsif (($host,$hostip,$dsn,$from,$to) = ($p1 =~ /^reject: RCPT from ([^[]+)\[([^]]+)\](?::\d+)?: ([45]52) Message size exceeds fixed limit; from=<(.*?)> to=<(.*?)>/)) { + #TD reject: RCPT from size.example.com[192.168.0.1]: 452 Message size exceeds fixed limit; from= to= + #TD reject: RCPT from size.example.com[192.168.0.1]: 552 Message size exceeds fixed limit; from= to= proto=ESMTP helo= + # Note: similar code above: search RejectSize + # Note: reject_warning does not seem to occur + if ($Collecting{'byiprejects'} and substr($dsn,0,1) eq '5') { + $fmthost = formathost($hostip,$host); + $Counts{'byiprejects'}{$fmthost}++; + } + $Totals{$reject_name = get_reject_key($dsn) . 'rejectsize' }++; next unless ($Collecting{$reject_name}); + $Counts{$reject_name}{$fmthost ? $fmthost : formathost($hostip,$host)}{$to}{$from ne '' ? $from : '<>'}++; + } + + elsif ($p1 =~ /looking for plugins in (.*)$/) { + #TD looking for plugins in '/usr/lib/sasl2', failed to open directory, error: No such file or directory + $Totals{'warnconfigerror'}++; next unless ($Collecting{'warnconfigerror'}); + $Counts{'warnconfigerror'}{$1}++; + } + + # SMTP/ESMTP protocol violations + elsif ($p1 =~ /^(improper command pipelining) (after \S+) from ([^[]*)\[([^]]+)\](?::\d+)?/) { + # ProtocolViolation + #TDsd postfix/smtpd[24928]: improper command pipelining after RCPT from unknown[192.168.0.1] + my ($error,$stage,$host,$hostip) = ($1,$2,$3,$4); + $Totals{'smtpprotocolviolation'}++; next unless ($Collecting{'smtpprotocolviolation'}); + $Counts{'smtpprotocolviolation'}{ucfirst($error)}{ucfirst($stage)}{formathost($hostip,$host)}++; + } + + elsif ($p1 =~ /^(too many errors) (after [^ ]*)(?: \((?:approximately )?\d+ bytes\))? from ([^[]*)\[([^]]+)\](?::\d+)?$/) { + my ($error,$stage,$host,$hostip) = ($1,$2,$3,$4); + #TDsd too many errors after AUTH from sample.net[10.0.0.1] + #TDsd too many errors after DATA (0 bytes) from 1-0-0-10.example.com[10.0.0.1] + $Totals{'smtpprotocolviolation'}++; next unless ($Collecting{'smtpprotocolviolation'}); + $Counts{'smtpprotocolviolation'}{ucfirst($error)}{ucfirst($stage)}{formathost($hostip,$host)}++; + } + + # coerce these into general warnings + elsif ( $p1 =~ /^cannot load Certificate Authority data/ or + $p1 =~ /^SSL_connect error to /) + { + #TDsQ Cannot start TLS: handshake failure + #TDsd cannot load Certificate Authority data + #TDs SSL_connect error to mail.example.com: 0 + + postfix_warning($p1); + } + + else { + # add to the unmatched list if not on the ignore_list + inc_unmatched('final') if ! in_ignore_list ($p1); + } +} + +######################################## +# Final tabulations, and report printing + +for my $code (@RejectKeys) { + for my $type (@RejectClasses) { + $Totals{'totalrejects' . $code} += $Totals{$code . $type}; + } + + if ($code =~ /^5/o) { + $Totals{'totalrejects'} += $Totals{'totalrejects' . $code}; + } +} + +# XXX this was naive - the goal was to avoid recounting messages +# released from quarantine, but externally introduced messages may +# contain resent-message-id; trying to track only internally resent +# messages does not seem useful. +# make some corrections now, due to double counting +#$Totals{'msgsaccepted'} -= $Totals{'resent'} if ($Totals{'msgsaccepted'} >= $Totals{'resent'}); + +$Totals{'totalacceptplusreject'} = $Totals{'msgsaccepted'} + $Totals{'totalrejects'}; + +# Print the Summary report if any key has non-zero data. +# Note: must explicitely check for any non-zero data, +# as Totals always has some keys extant. +# +if ($Opts{'summary'}) { + for (keys %Totals) { + if ($Totals{$_}) { + print_summary_report (@Sections); + last; + } + } +} + +# Print the Detail report, if detail is sufficiently high +# +if ($Opts{'detail'} >= 5) { + #print STDERR "Counts memory usage: ", commify(Devel::Size::total_size(\%Counts)), "\n"; + #print STDERR "Delays memory usage: ", commify(Devel::Size::total_size(\%Delays)), "\n"; + print_detail_report(@Sections); + + if ($Opts{'delays'}) { + my @table; + for (sort keys %Delays) { + # anon array ref: label, array ref of $Delay{key} + push @table, [ substr($_,3), $Delays{$_} ]; + } + if (@table) { + print_percentiles_report2(\@table, "Delivery Delays Percentiles", $Opts{'delays_percentiles'}); + } + } + + print_postgrey_reports(); + +} + +# debug: show which ignore_list items are hit most +#my %IGNORED; +#for (sort { $IGNORED{$b} <=> $IGNORED{$a} } keys %IGNORED) { +# printf "%10d: KEY: %s\n", $IGNORED{$_}, $_; +#} + +# Finally, print any unmatched lines +# +print_unmatched_report(); + +# +# End of main +# +################################################## + +# Create the list of REs against which log lines are matched. +# Lines that match any of the patterns in this list are ignored. +# +# Note: This table is created at runtime, due to a Perl bug which +# I reported as perl bug #56202: +# +# http://rt.perl.org/rt3/Public/Bug/Display.html?id=56202 +# +sub create_ignore_list() { + # top 3 hitters up front + push @ignore_list, qr/^statistics:/; + push @ignore_list, qr/^setting up TLS connection (?:from|to)/; + push @ignore_list, qr/^Verified: /; + push @ignore_list, qr/^skipped, still being delivered/; + + # SSL info at/above mail.info level + push @ignore_list, qr/^read from [a-fA-F\d]{8}/; + push @ignore_list, qr/^write to [a-fA-F\d]{8}/; + push @ignore_list, qr/^[a-f\d]{4} [a-f\d]{2}/; + push @ignore_list, qr/^[a-f\d]{4} - ]+ /; + + push @ignore_list, qr/^premature end-of-input (?:on|from) .* socket while reading input attribute name$/; + push @ignore_list, qr/^certificate peer name verification failed/; + push @ignore_list, qr/^Peer certi?ficate could not be verified$/; # missing i was a postfix typo + push @ignore_list, qr/^Peer cert verify depth=/; + push @ignore_list, qr/^Peer verification:/; + push @ignore_list, qr/^Server certificate could not be verified/; + push @ignore_list, qr/^cannot load .SA certificate and key data/; + push @ignore_list, qr/^tlsmgr_cache_run_event/; + push @ignore_list, qr/^SSL_accept/; + push @ignore_list, qr/^SSL_connect:/; + push @ignore_list, qr/^connection (?:closed|established)/; + push @ignore_list, qr/^delete smtpd session/; + push @ignore_list, qr/^put smtpd session/; + push @ignore_list, qr/^save session/; + push @ignore_list, qr/^Reusing old/; + push @ignore_list, qr/^looking up session/; + push @ignore_list, qr/^lookup smtpd session/; + push @ignore_list, qr/^lookup \S+ type/; + push @ignore_list, qr/^xsasl_(?:cyrus|dovecot)_/; + push @ignore_list, qr/^watchdog_/; + push @ignore_list, qr/^read smtpd TLS/; + push @ignore_list, qr/^open smtpd TLS/; + push @ignore_list, qr/^write smtpd TLS/; + push @ignore_list, qr/^read smtp TLS cache entry/; + push @ignore_list, qr/^starting TLS engine$/; + push @ignore_list, qr/^initializing the server-side TLS/; + push @ignore_list, qr/^global TLS level: /; + push @ignore_list, qr/^auto_clnt_/; + push @ignore_list, qr/^generic_checks:/; + push @ignore_list, qr/^inet_addr_/; + push @ignore_list, qr/^mac_parse:/; + push @ignore_list, qr/^cert has expired/; + push @ignore_list, qr/^daemon started/; + push @ignore_list, qr/^master_notify:/; + push @ignore_list, qr/^rewrite_clnt:/; + push @ignore_list, qr/^rewrite stream/; + push @ignore_list, qr/^dict_/; + push @ignore_list, qr/^send attr /; + push @ignore_list, qr/^match_/; + push @ignore_list, qr/^input attribute /; + push @ignore_list, qr/^Run-time/; + push @ignore_list, qr/^Compiled against/; + push @ignore_list, qr/^private\//; + push @ignore_list, qr/^reject_unknown_/; # don't combine or shorten these reject_ patterns + push @ignore_list, qr/^reject_unauth_/; + push @ignore_list, qr/^reject_non_/; + push @ignore_list, qr/^permit_/; + push @ignore_list, qr/^idle timeout/; + push @ignore_list, qr/^get_dns_/; + push @ignore_list, qr/^dns_/; + push @ignore_list, qr/^chroot /; + push @ignore_list, qr/^process generation/; + push @ignore_list, qr/^fsspace:/; + push @ignore_list, qr/^master disconnect/; + push @ignore_list, qr/^resolve_clnt/; + push @ignore_list, qr/^ctable_/; + push @ignore_list, qr/^extract_addr/; + push @ignore_list, qr/^mynetworks:/; + push @ignore_list, qr/^name_mask:/; + #TDm reload -- version 2.6-20080814, configuration /etc/postfix + #TDm reload configuration /etc/postfix + push @ignore_list, qr/^reload (?:-- version \S+, )?configuration/; + push @ignore_list, qr/^terminating on signal 15$/; + push @ignore_list, qr/^verify error:num=/; + push @ignore_list, qr/^verify return:/; + push @ignore_list, qr/^nss_ldap: /; + push @ignore_list, qr/^discarding EHLO keywords: /; + push @ignore_list, qr/^sql auxprop plugin/; + push @ignore_list, qr/^sql plugin/; + push @ignore_list, qr/^sql_select/; + push @ignore_list, qr/^auxpropfunc error/; + push @ignore_list, qr/^commit transaction/; + push @ignore_list, qr/^begin transaction/; + push @ignore_list, qr/^maps_find: /; + push @ignore_list, qr/^check_access: /; + push @ignore_list, qr/^check_domain_access: /; + push @ignore_list, qr/^check_mail_access: /; + push @ignore_list, qr/^check_table_result: /; + push @ignore_list, qr/^mail_addr_find: /; + push @ignore_list, qr/^mail_addr_map: /; + push @ignore_list, qr/^mail_flow_put: /; + push @ignore_list, qr/^smtp_addr_one: /; + push @ignore_list, qr/^smtp_connect_addr: /; + push @ignore_list, qr/^smtp_connect_unix: trying: /; + push @ignore_list, qr/^smtp_find_self: /; + push @ignore_list, qr/^smtp_get: /; + push @ignore_list, qr/^smtp_fputs: /; + push @ignore_list, qr/^smtp_parse_destination: /; + push @ignore_list, qr/^smtp_sasl_passwd_lookup: /; + push @ignore_list, qr/^smtpd_check_/; + push @ignore_list, qr/^smtpd_chat_notify: /; + push @ignore_list, qr/^been_here: /; + push @ignore_list, qr/^set_eugid: /; + push @ignore_list, qr/^deliver_/; + push @ignore_list, qr/^flush_send_file: queue_id/; + push @ignore_list, qr/^milter_macro_lookup/; + push @ignore_list, qr/^milter8/; + push @ignore_list, qr/^skipping non-protocol event/; + push @ignore_list, qr/^reply: /; + push @ignore_list, qr/^event: /; + push @ignore_list, qr/^trying... /; + push @ignore_list, qr/ all milters$/; + push @ignore_list, qr/^vstream_/; + push @ignore_list, qr/^server features/; + push @ignore_list, qr/^skipping event/; + push @ignore_list, qr/^Using /; + push @ignore_list, qr/^rec_(?:put|get): /; + push @ignore_list, qr/^subject=/; + push @ignore_list, qr/^issuer=/; + push @ignore_list, qr/^pref /; # yes, multiple spaces + push @ignore_list, qr/^request: \d/; + push @ignore_list, qr/^done incoming queue scan$/; + push @ignore_list, qr/^qmgr_/; + push @ignore_list, qr/^trigger_server_accept_fifo: /; + push @ignore_list, qr/^proxymap stream/; + push @ignore_list, qr/^(?:start|end) sorted recipient list$/; + push @ignore_list, qr/^connecting to \S+ port /; + push @ignore_list, qr/^Write \d+ chars/; + push @ignore_list, qr/^Read \d+ chars/; + push @ignore_list, qr/^(?:lookup|delete) smtp session/; + push @ignore_list, qr/^delete smtp session/; + push @ignore_list, qr/^(?:reloaded|remove|looking for) session .* cache$/; + push @ignore_list, qr/^(?:begin|end) \S+ address list$/; + push @ignore_list, qr/^mapping DSN status/; + push @ignore_list, qr/^record [A-Z]/; + push @ignore_list, qr/^dir_/; + push @ignore_list, qr/^transport_event/; + push @ignore_list, qr/^read [A-Z](?: |$)/; + push @ignore_list, qr/^relay: /; + push @ignore_list, qr/^why: /; + push @ignore_list, qr/^fp: /; + push @ignore_list, qr/^path: /; + push @ignore_list, qr/^level: /; + push @ignore_list, qr/^recipient: /; + push @ignore_list, qr/^delivered: /; + push @ignore_list, qr/^queue_id: /; + push @ignore_list, qr/^queue_name: /; + push @ignore_list, qr/^user: /; + push @ignore_list, qr/^sender: /; + push @ignore_list, qr/^offset: /; + push @ignore_list, qr/^offset: /; + push @ignore_list, qr/^verify stream disconnect/; + push @ignore_list, qr/^event_request_timer: /; + push @ignore_list, qr/^smtp_sasl_authenticate: /; + push @ignore_list, qr/^flush_add: /; + push @ignore_list, qr/^disposing SASL state information/; + push @ignore_list, qr/^starting new SASL client/; + push @ignore_list, qr/^error: dict_ldap_connect: /; + push @ignore_list, qr/^error: to submit mail, use the Postfix sendmail command/; + push @ignore_list, qr/^local_deliver[:[]/; + push @ignore_list, qr/^_sasl_plugin_load /; + push @ignore_list, qr/^exp_type: /; + push @ignore_list, qr/^wakeup [\dA-Z]/; + push @ignore_list, qr/^defer (?:site|transport) /; + push @ignore_list, qr/^local: /; + push @ignore_list, qr/^exp_from: /; + push @ignore_list, qr/^extension: /; + push @ignore_list, qr/^owner: /; + push @ignore_list, qr/^unmatched: /; + push @ignore_list, qr/^domain: /; + push @ignore_list, qr/^initializing the client-side TLS engine/; + push @ignore_list, qr/^header_token: /; + push @ignore_list, qr/^(?:PUSH|POP) boundary/; + push @ignore_list, qr/^recipient limit \d+$/; + push @ignore_list, qr/^scan_dir_next: found/; + push @ignore_list, qr/^open (?:btree|incoming)/; + push @ignore_list, qr/^Renamed to match inode number/; + push @ignore_list, qr/^cleanup_[^:]+:/; + push @ignore_list, qr/^(?:before|after) input_transp_cleanup: /; + push @ignore_list, qr/^event_enable_read: /; + push @ignore_list, qr/^report recipient to all milters /; + push @ignore_list, qr/_action = defer_if_permit$/; + push @ignore_list, qr/^reject_invalid_hostname: /; + push @ignore_list, qr/^cfg_get_/; + push @ignore_list, qr/^sacl_check: /; + + # non-anchored + #push @ignore_list, qr/: Greylisted for /; + push @ignore_list, qr/certificate verification (?:depth|failed for)/; + push @ignore_list, qr/re-using session with untrusted certificate, look for details earlier in the log$/; + push @ignore_list, qr/socket: wanted attribute: /; + push @ignore_list, qr/ smtpd cache$/; + push @ignore_list, qr/ old session$/; + push @ignore_list, qr/fingerprint=/; + push @ignore_list, qr/TLS cipher list "/; +} + +# Evaluates a given line against the list of ignore patterns. +# +sub in_ignore_list($) { + my $line = shift; + + foreach (@ignore_list) { + #return 1 if $line =~ /$_/; + if ($line =~ /$_/) { + #$IGNORED{$_}++; + return 1; + } + } + + return 0; +} + +# Accepts common fields from a standard delivery attempt, processing then +# and returning modified values +# +sub process_delivery_attempt ($ $ $ $) { + my ($to,$origto,$DDD,$reason) = @_; + + $reason =~ s/\((.*)\)/$1/; # Makes capturing nested parens easier + # leave $to/$origto undefined, or strip < > chars if not null address (<>). + defined $to and $to = ($to eq '') ? '<>' : lc $to; + defined $origto and $origto = ($origto eq '') ? '<>' : lc $origto; + my ($localpart, $domainpart) = split ('@', $to); + ($localpart, $domainpart) = ($to, '*unspecified') if ($domainpart eq ''); + my ($dsn); + + # If recipient_delimiter is set, break localpart into user + extension + # and save localpart in origto if origto is empty + # + if ($Opts{'recipient_delimiter'} and $localpart =~ /\Q$Opts{'recipient_delimiter'}\E/o) { + + # special cases: never split mailer-daemon or double-bounce + # or owner- or -request if delim is "-" (dash). + unless ($localpart =~ /^(?:mailer-daemon|double-bounce)$/i or + ($Opts{'recipient_delimiter'} eq '-' and $localpart =~ /^owner-.|.-request$/i)) { + my ($user,$extension) = split (/\Q$Opts{'recipient_delimiter'}\E/, $localpart, 2); + $origto = $localpart if ($origto eq ''); + $localpart = $user; + } + } + + unless (($dsn) = ($DDD =~ /dsn=(\d\.\d+\.\d+)/o)) { + $dsn = ''; + } + + if ($Collecting{'delays'} and $DDD =~ m{delay=([\d.]+)(?:, delays=([\d.]+)/([\d.]+)/([\d.]+)/([\d.]+))?}) { + # Message delivery time stamps + # delays=a/b/c/d, where + # a = time before queue manager, including message transmission + # b = time in queue manager + # c = connection setup including DNS, HELO and TLS; + # d = message transmission time. + if (defined $2) { + $Delays{'1: Before qmgr'}{$2}++; + $Delays{'2: In qmgr'}{$3}++; + $Delays{'3: Conn setup'}{$4}++; + $Delays{'4: Transmission'}{$5}++; + } + $Delays{'5: Total'}{$1}++; + } + + return ($to,$origto,$localpart,$domainpart,$dsn,$reason); +} + +# Processes postfix/bounce messages +# +sub postfix_bounce($) { + my $line = shift; + my $type; + + $line =~ s/^(?:$re_QID): //o; + if ($line =~ /^(sender|postmaster) non-delivery notification/o) { + #TDbQ postmaster non-delivery notification: 7446BCD68 + #TDbQ sender non-delivery notification: 7446BCD68 + $type = 'Non-delivery'; + } + elsif ($line =~ /^(sender|postmaster) delivery status notification/o ) { + #TDbQ sender delivery status notification: 7446BCD68 + $type = 'Delivery'; + } + elsif ($line =~ /^(sender|postmaster) delay notification: /o) { + #TDbQ sender delay notification: AA61EC2F9A + $type = 'Delayed'; + } + else { + inc_unmatched('bounce') if ! in_ignore_list($line); + return; + } + + $Totals{'notificationsent'}++; return unless ($Collecting{'notificationsent'}); + $Counts{'notificationsent'}{$type}{$1}++; +} + +# Processes postfix/cleanup messages +# cleanup always has a QID +# +sub postfix_cleanup($) { + my $line = shift; + my ($qid,$reply,$fmthost,$reject_name); + + ($qid, $line) = ($1, $2) if ($line =~ /^($re_QID): (.*)$/o ); + + #TDcQ message-id= + return if ($line =~ /^message-id=/); + + # milter-reject, milter-hold, milter-discard + if ($line =~ s/^milter-//) { + milter_common($line); + return; + } + + ### cleanup bounced messages (always_bcc, recipient_bcc_maps, sender_bcc_maps) + # Note: Bounce + # See same code elsewhere "Note: Bounce" + #TDcQ to=, relay=none, delay=0.11, delays=0.11/0/0/0, dsn=5.7.1, status=bounced optional text... + #TDcQ to=, orig_to=, relay=none, delay=0.13, delays=0.13/0/0/0, dsn=5.7.1, status=bounced optional text... + if ($line =~ /^to=<(.*?)>,(?: orig_to=<(.*?)>,)? relay=([^,]*).*, ($re_DDD), status=([^ ]+) (.*)$/o) { + # ($to,$origto,$relay,$DDD,$status,$reason) = ($1,$2,$3,$4,$5,$6); + if ($5 ne 'bounced' and $5 ne 'SOFTBOUNCE') { + inc_unmatched('cleanupbounce'); + return; + } + + my ($to,$origto,$relay,$DDD,$reason) = ($1,$2,$3,$4,$6); + my ($localpart,$domainpart,$dsn); + ($to,$origto,$localpart,$domainpart,$dsn,$reason) = process_delivery_attempt ($to,$origto,$DDD,$reason); + + ### local bounce + # XXX local v. remote bounce seems iffy, relative + if ($relay =~ /^(?:none|local|virtual|maildrop|127\.0\.0\.1|avcheck)/) { + $Totals{'bouncelocal'}++; return unless ($Collecting{'bouncelocal'}); + $Counts{'bouncelocal'}{get_dsn_msg($dsn)}{$domainpart}{ucfirst($reason)}{$localpart}++; + } + ### remote bounce + else { + ($reply,$fmthost) = cleanhostreply($reason,$relay,$to ne '' ? $to : '<>',$domainpart); + $Totals{'bounceremote'}++; return unless ($Collecting{'bounceremote'}); + $Counts{'bounceremote'}{get_dsn_msg($dsn)}{$domainpart}{$localpart}{$fmthost}{$reply}++; + } + } + + # *header_checks and body_checks + elsif (header_body_checks($line)) { + #print "cleanup: header_body_checks\n"; + return; + } + + #TDcQ resent-message-id=4739073.1 + #TDcQ resent-message-id= + #TDcQ resent-message-id=? <120B11@samplepc> + elsif ( ($line =~ /^resent-message-id=?$/o )) { + $Totals{'resent'}++; + } + + #TDcN unable to dlopen .../sasl2/libplain.so.2: .../sasl2/libplain.so.2: failed to map segment from shared object: Operation not permitted + elsif ($line =~ /^unable to dlopen /) { + # strip extraneous doubling of library path + $line = "$1$2 $3" if ($line =~ /(unable to dlopen )([^:]+: )\2(.+)$/); + postfix_warning($line); + } + + else { + inc_unmatched('cleanup(last)') if ! in_ignore_list($line); + } +} + +=pod + header_body_checks + + Handle cleanup's header_checks and body_checks, and smtp's smtp_body_checks/smtp_*header_checks + + Possible actions that log are: + + REJECT optional text... + DISCARD optional text... + FILTER transport:destination + HOLD optional text... + REDIRECT user@domain + PREPEND text... + REPLACE text... + WARN optional text... + + DUNNO and IGNORE are not logged + +Returns: + 1: if line matched or handled + 0: otherwise +=cut + +sub header_body_checks($) +{ + my $line = shift; + + # bcc, discard, filter, hold, prepend, redirect, reject, replace, warning + return 0 if ($line !~ /^[bdfhprw]/) or # short circuit alternation when no match possible + ($line !~ /^(re(?:ject|direct|place)|filter|hold|discard|prepend|warning|bcc): (header|body|content) (.*)$/); + + my ($action,$part,$p3) = ($1,$2,$3); + + #print "header_body_checks: action: \"$action\", part: \"$part\", p3: \"$p3\"\n"; + + my ($trigger,$host,$eto,$p4,$fmthost,$reject_name); + # $re_QID: reject: body ... + # $re_QID: reject: header ... + # $re_QID: reject: content ... + + + if ($p3 =~ /^(.*) from ([^;]+); from=<.*?>(?: to=<(.*?)>)?(?: proto=\S*)?(?: helo=<.*?>)?(?:: (.*)|$)/) { + ($trigger,$host,$eto,$p4) = ($1,$2,$3,$4); + + # $action $part $trigger $host $eto $p4 + #TDcQ reject: body Subject: Cheap cialis from local; from=: optional text... + #TDcQ reject: body Quality replica watches!!! from hb.example.com[10.0.0.1]; from= to= proto=SMTP helo=: optional text... + #TDcQ reject: header To: from hb.example.com[10.0.0.1]; from= to= proto=ESMTP helo=: optional text... + # message_reject_characters (postfix >= 2.3) + #TDcQ reject: content Received: by example.com Postfix from example.com[10.0.0.1]; from= to= proto=ESMTP helo=.example.com>: 5.7.1 disallowed character + + #TDcQ filter: header To: to@example.com from hb.example.com[10.0.0.1]; from= to= proto=ESMTP helo=: transport:destination + #TDcQ hold: header Message-ID: from localhost[127.0.0.1]; from= to= proto=ESMTP helo=: optional text... + #TDcQ hold: header Subject: Hold Test from local; from= to=: optional text... + #TDcQ hold: header Received: by example.com...from x from local; from= + #TDcQ hold: header Received: from x.com (x.com[10.0.0.1])??by example.com (Postfix) with ESMTP id 630BF??for ; Thu, 20 Oct 2006 13:27: from example.com[10.0.0.1]; from= to= proto=ESMTP helo= + # hold: header Received: from [10.0.0.1] by example.com Thu, 9 Jan 2008 18:06:06 -0500 from sample.net[10.0.0.2]; from=<> to= proto=SMTP helo=: faked header + #TDcQ redirect: header From: "Attn Men" from hb.example.com[10.0.0.1]; from= to= proto=ESMTP helo=: user@domain + #TDcQ redirect: header From: "Superman" from hb.example.com[10.0.0.2]; from= to= proto=ESMTP helo=: user@domain + #TDcQ redirect: body Original drugs from hb.example.com[10.0.0.1]; from= to= proto=SMTP helo=: user@domain + #TDcQ discard: header Subject: **SPAM** Blah... from hb.example.com[10.0.0.1]; from= to= proto=ESMTP helo= + #TDcQ prepend: header Rubble: Mr. from localhost[127.0.0.1]; from= to= proto=ESMTP helo=: text... + #TDcQ replace: header Rubble: flintstone from localhost[127.0.0.1]; from= to= proto=ESMTP helo=: text... + #TDcQ warning: header Date: Tues, 99:34:67 from localhost[127.0.0.1]; from= to= proto=ESMTP helo=: optional text... + # BCC action (2.6 experimental branch) + #TDcQ bcc: header To: to@example.com from hb.example.com[10.0.0.1]; from= to= proto=ESMTP helo=: user@domain + + # Note: reject_warning does not seem to occur + } + + else { + # smtp_body_checks, smtp_header_checks, smtp_mime_header_checks, smtp_nested_header_checks (postfix >= 2.5) + #TDsQ replace: header Sender: : Sender: + + $trigger = $p3; $host = ''; $eto = ''; $p4 = $part eq 'body' ? 'smtp_body_checks' : 'smtp_*header_checks'; + + #inc_unmatched('header_body_checks'); + #return 1; + } + + #print " trigger: \"$trigger\", host: \"$host\", eto: \"$eto\", p4: \"$p4\"\n"; + $trigger =~ s/\s+/ /g; + $trigger = '*unknown reason' if ($trigger eq ''); + $eto = '*unknown' if ($eto eq ''); + + my ($trig,$trig_opt,$text); + if ($part eq 'header') { ($trig = $trigger) =~ s/^([^:]+:).*$/Header check "$1"/; } + elsif ($part eq 'body') { $trig = "Body check"; } + else { $trig = "Content check"; } # message_reject_characters (postfix >= 2.3) + + if ($p4 eq '') { $text = '*generic'; $trig_opt = $trig; } + else { $text = $p4; $trig_opt = "$trig ($p4)"; } + + if ($host eq 'local') { $fmthost = formathost('127.0.0.1', 'local'); } + elsif ($host =~ /([^[]+)\[([^]]+)\]/) { $fmthost = formathost($2,$1); } + else { $fmthost = '*unknown'; } + + # Note: Counts + # Ensure each $Counts{key} accumulator is consistently + # used with the same number of hash key levels throughout the code. + # For example, $Counts{'hold'} below has 4 keys; ensure that every + # other usage of $Counts{'hold'} also has 4 keys. Currently, it is + # OK to set the last key as '', but only the last. + + if ($action eq 'reject') { + $Counts{'byiprejects'}{$fmthost}++ if $Collecting{'byiprejects'}; + + # Note: no temporary or reject_warning + # Note: no reply code - force into a 5xx reject + # XXX this won't be seen if the user has no 5.. entry in reject_reply_patterns + $Totals{$reject_name = "5xxreject$part" }++; + $Counts{$reject_name}{$text}{$eto}{$fmthost}{$trigger}++ if $Collecting{$reject_name}; + } + elsif ( $action eq 'filter' ) { + $Totals{'filtered'}++; + $Counts{'filtered'}{$text}{$trig}{$trigger}{$eto}{$fmthost}++ if $Collecting{'filtered'}; + } + elsif ( $action eq 'hold' ) { + $Totals{'hold'}++; + $Counts{'hold'}{$trig_opt}{$fmthost}{$eto}{$trigger}++ if $Collecting{'hold'}; + } + elsif ( $action eq 'redirect' ) { + $Totals{'redirected'}++; + $Counts{'redirected'}{$trig}{$text}{$eto}{$fmthost}{$trigger}++ if $Collecting{'redirected'}; + } + elsif ( $action eq 'discard' ) { + $Totals{'discarded'}++; + $Counts{'discarded'}{$trig}{$fmthost}{$eto}{$trigger}++ if $Collecting{'discarded'}; + } + elsif ( $action eq 'prepend' ) { + $Totals{'prepended'}++; + $Counts{'prepended'}{"$trig ($text)"}{$fmthost}{$eto}{$trigger}++ if $Collecting{'prepended'}; + } + elsif ( $action eq 'replace' ) { + $Totals{'replaced'}++; + $Counts{'replaced'}{"$trig ($text)"}{$fmthost}{$eto}{$trigger}++ if $Collecting{'replaced'}; + } + elsif ( $action eq 'warning' ) { + $Totals{'warned'}++; + $Counts{'warned'}{$trig}{$fmthost}{$eto}{$trigger}++ if $Collecting{'warned'}; + } + elsif ( $action eq 'bcc' ) { + $Totals{'bcced'}++; + $Counts{'bcced'}{$text}{$trig}{$trigger}{$eto}{$fmthost}++ if $Collecting{'bcced'}; + } + else { + inc_unmatched('header_body_checks unexpected action'); + } + + return 1; +} + + +# Handle common milter actions: +# milter-reject, milter-hold, milter-discard +# which are created by both smtpd and cleanup +# +sub milter_common($) { + my $line = shift; + + #TDsdN milter-reject: MAIL from milterS.example.com[10.0.0.1]: 553 5.1.7 address incomplete; proto=ESMTP helo= + #TDsdN milter-reject: CONNECT from milterS.example.com[10.0.0.2]: 451 4.7.1 Service unavailable - try again later; proto=SMTP + #TDsdQ milter-reject: END-OF-MESSAGE from milterS.example.com[10.0.0.3]: 5.7.1 black listed URL host sample.com by ...uribl.com; from= to= proto=ESMTP helo= + #TDsdQ milter-hold: END-OF-MESSAGE from milterS.example.com[10.0.0.4]: milter triggers HOLD action; from= to= proto=ESMTP helo= + + #TDcQ milter-reject: END-OF-MESSAGE from milterC.example.com[10.0.0.1]: 5.7.1 Some problem; from= to= proto=SMTP helo= + #TDcQ milter-reject: CONNECT from milterC.example.com[10.0.0.2]: 5.7.1 Some problem; proto=SMTP + #TDcQ milter-hold: END-OF-MESSAGE from milterC.example.com[10.0.0.3]: milter triggers HOLD action; from= to= proto=ESMTP helo= + #TDcQ milter-discard: END-OF-MESSAGE from milterC.example.com[10.0.0.4]: milter triggers DISCARD action; from= to= proto=ESMTP helo= +# 84B82AC8B3: milter-reject: END-OF-MESSAGE from localhost[127.0.0.1]: 5.7.1 Blocked + + my ($efrom,$eto,$proto,$helo) = strip_ftph($line); + #print "efrom: '$efrom', eto: '$eto', proto: '$proto', helo: '$helo'\n"; + $line =~ s/;$//; + + if ($line =~ /^(reject|hold|discard): (\S+) from ([^[]+)\[([^]]+)\](?::\d+)?: (.*)$/) { + + my ($action,$stage,$host,$hostip,$reply) = ($1,$2,$3,$4,$5); + #print "action: '$action', stage: '$stage', host: '$host', hostip: '$hostip', reply: '$reply'\n"; + + if ($action eq 'reject') { + my ($dsn,$fmthost,$reject_name); + ($dsn,$reply) = ($1,$2) if $reply =~ /^($re_DSN) (.*)$/o; + #print " dsn: '$dsn', reply: '$reply'\n"; + + if ($Collecting{'byiprejects'} and substr($dsn,0,1) eq '5') { + $fmthost = formathost($hostip,$host); + $Counts{'byiprejects'}{$fmthost}++; + } + # Note: reject_warning does not seem to occur + # Note: See rejectmilter elsewhere + $Totals{$reject_name = get_reject_key($dsn) . 'rejectmilter' }++; return unless ($Collecting{$reject_name}); + $Counts{$reject_name}{$stage}{$fmthost ? $fmthost : formathost($hostip,$host)}{$reply}++; + } + # milter-hold + elsif ($action eq 'hold') { + $Totals{'hold'}++; return unless ($Collecting{'hold'}); + $Counts{'hold'}{'milter'}{$stage}{formathost($hostip,$host)}{$eto}++; + } + # milter-discard + else { # $action eq 'discard' + $Totals{'discarded'}++; return unless ($Collecting{'discarded'}); + $Counts{'discarded'}{'milter'}{$stage}{formathost($hostip,$host)}{$eto}++; + } + + } + else { + inc_unmatched('milter_common)'); + } +} + +sub postfix_dnsblog { + my $line = shift; + + #postfix/dnsblog[16943]: addr 192.168.0.1 listed by domain bl.spamcop.net as 127.0.0.2 + #postfix/dnsblog[78598]: addr 192.168.0.1 blocked by domain zen.spamhaus.org as 127.0.0.11 + if ($line =~ /^addr (\S+) (?:listed|blocked) by domain (\S+) as (\S+)$/) { + $Counts{'dnsblog'}{$2}{$1}{$3}++ if $Collecting{'dnsblog'}; + } + else { + inc_unmatched('dnsblog') if ! in_ignore_list($line); + return; + } +} + +sub postfix_postscreen { + my $line = shift; + + return if ( + $line =~ /^cache / or + $line =~ /discarding EHLO keywords: / or + $line =~ /: discard_mask / or + $line =~ /: sq=\d+ cq=\d+ event/ or + $line =~ /: replacing command "/ + ); + + + if (($line =~ /^(PREGREET) \d+ (?:after \S+)? from \[([^]]+)\](?::\d+)?/) or + # PREGREET 20 after 0.31 from [192.168.0.1]:12345: HELO 10.0.0.1?? + # HANGUP after 0.7 from [192.168.0.4]:12345 + ($line =~ /^(HANGUP) (?:after \S+)? from \[([^]]+)\](?::\d+)?/)) { + $Counts{'postscreen'}{lc $1}{$2}{$END_KEY}++ if $Collecting{'postscreen'}; + } + elsif ($line =~ /^(WHITELISTED|BLACKLISTED|PASS \S+) \[([^]]+)\](?::\d+)?$/) { + # PASS NEW [192.168.0.2]:12345 + # PASS OLD [192.168.0.3]:12345 + $Counts{'postscreen'}{lc $1}{$2}{$END_KEY}++ if $Collecting{'postscreen'}; + } + elsif ($line =~ /^DNSBL rank (\S+) for \[([^]]+)\](?::\d+)?$/) { + $Counts{'postscreen'}{'dnsbl'}{$2}{$1}++ if $Collecting{'postscreen'}; + } + + elsif ($line =~ /^(CONNECT|COMMAND (?:(?:TIME|COUNT|LENGTH) LIMIT|PIPELINING)|NON-SMTP COMMAND|BARE NEWLINE) from \[([^\]]+)\]:\d+/) { + # CONNECT from [192.168.1.1]:12345 + $Counts{'postscreen'}{lc($1)}{$2}{$END_KEY}++ if $Collecting{'postscreen'}; + } + elsif ($line =~ /^DISCONNECT \[([^\]]+)\]:\d+$/) { + # DISCONNECT [192.168.1.1]:12345 + $Counts{'postscreen'}{'disconnect'}{$1}{$END_KEY}++ if $Collecting{'postscreen'}; + } + + elsif ($line =~ /^NOQUEUE: reject: RCPT from \[([^]]+)\](?::\d+)?: ($re_DSN) ([^;]+)/o) { + #NOQUEUE: reject: RCPT from [192.168.0.1]:12345: 550 5.7.1 Service unavailable; client [192.168.0.1] blocked using b.barracudacentral.org; from=, to=, proto=SMTP, helo= + my ($ip,$dsn,$msg) = ($1,$2,$3); + + if ($dsn =~ /^([54])/) { + $Counts{'postscreen'}{$1 . 'xx reject'}{"$dsn $msg"}{$ip}++ if $Collecting{'postscreen'}; + } + else { + $Counts{'postscreen'}{'reject'}{"$dsn $msg"}{$ip}{$END_KEY}++ if $Collecting{'postscreen'}; + } + } + + elsif ($line =~ /^NOQUEUE: reject: CONNECT from \[([^]]+)\](?::\d+)?: too many connections/) { + # NOQUEUE: reject: CONNECT from [192.168.0.1]:7197: too many connections + $Counts{'postscreen'}{'reject'}{'Too many connections'}{$1}{$END_KEY}++ if $Collecting{'postscreen'}; + } + + elsif ($line =~ /^reject: connect from \[([^]]+)\](?::\d+)?: (.+)$/) { + # reject: connect from [192.168.0.1]:21225: all screening ports busy + $Counts{'postscreen'}{'reject'}{"\u$2"}{$1}{$END_KEY}++ if $Collecting{'postscreen'}; + } + + elsif ($line =~ /^(?:WHITELIST VETO) \[([^]]+)\](?::\d+)?$/) { + # WHITELIST VETO [192.168.0.8]:43579 + $Counts{'postscreen'}{'whitelist veto'}{$1}{$END_KEY}++ if $Collecting{'postscreen'}; + } + + elsif ($line =~ /^(entering|leaving) STRESS mode with (\d+) connections$/) { + # entering STRESS mode with 90 connections + $Counts{'postscreen'}{'stress mode: ' . $1}{$2}{$END_KEY}++ if $Collecting{'postscreen'}; + } + + elsif ($line =~ /^close database (\S+): No such file or directory/) { + # close database /var/lib/postfix/postscreen_cache.db: No such file or directory (possible Berkeley DB bug) + $Counts{'postscreen'}{'close database'}{$1}{$END_KEY}++ if $Collecting{'postscreen'}; + } + + else { + inc_unmatched('postscreen') if ! in_ignore_list($line); + return; + } + + $Totals{'postscreen'}++; +} + + +# Handles postfix/postsuper lines +# +sub postfix_postsuper($) { + my $line = shift; + + return if $line =~ /^Deleted: \d+ messages?$/; + + if ($line =~ /^Placed on hold: (\d+) messages?$/o) { + #TDps Placed on hold: 2 messages + # Note: See Hold elsewhere + $Totals{'hold'} += $1; return unless ($Collecting{'hold'}); + $Counts{'hold'}{'Postsuper'}{'localhost'}{"bulk hold: $1"}{''} += $1; + } + elsif ($line =~ /^Released from hold: (\d+) messages?$/o) { + #TDps Released from hold: 1 message + $Totals{'releasedfromhold'} += $1; + } + elsif ($line =~ /^Requeued: (\d+) messages?$/o) { + #TDps Requeued: 1 message + $Totals{'requeued'} += $1; + } + elsif (my($qid,$p2) = ($line =~ /($re_QID): (.*)$/)) { + # postsuper double reports the following 3 lines + return if ($p2 eq 'released from hold'); + return if ($p2 eq 'placed on hold'); + return if ($p2 eq 'requeued'); + + if ($p2 =~ /^removed\s*$/o) { + # Note: See REMOVED elsewhere + # 52CBDC2E0F: removed + delete $SizeByQid{$qid} if (exists $SizeByQid{$qid}); + $Totals{'removedfromqueue'}++; + } + elsif (! in_ignore_list ($p2)) { + inc_unmatched('postsuper2'); + } + } + elsif (! in_ignore_list ($line)) { + inc_unmatched('postsuper1'); + } +} + +# Handles postfix panic: lines +# +sub postfix_panic($) { + #TD panic: myfree: corrupt or unallocated memory block + $Totals{'panicerror'}++; return unless ($Collecting{'panicerror'}); + $Counts{'panicerror'}{ucfirst($1)}++; +} + +# Handles postfix fatal: lines +# +sub postfix_fatal($) { + my ($reason) = shift; + + if ($reason =~ /^\S*\(\d+\): Message file too big$/o) { + #TD fatal: root(0): Message file too big + $Totals{'fatalfiletoobig'}++; + + + # XXX its not clear this is at all useful - consider falling through to last case + } elsif ( $reason =~ /^config variable (\S*): (.*)$/o ) { + #TD fatal: config variable inet_interfaces: host not found: 10.0.0.1:2525 + #TD fatal: config variable inet_interfaces: host not found: all:2525 + $Totals{'fatalconfigerror'}++; return unless ($Collecting{'fatalconfigerror'}); + $Counts{'fatalconfigerror'}{ucfirst($reason)}++; + } + else { + #TD fatal: watchdog timeout + #TD fatal: bad boolean configuration: smtpd_use_tls = + + #TDvN fatal: update queue file active/4B709F060E: File too large + $reason =~ s/(^update queue file \w+\/)\w+:/$1*:/; + + $Totals{'fatalerror'}++; return unless ($Collecting{'fatalerror'}); + $Counts{'fatalerror'}{ucfirst($reason)}++; + } +} +# Handles postfix fatal: lines +# +sub postfix_error($) { + my ($reason) = shift; + # postfix/postfix-script[4271]: error: unknown command: 'rel' + + $Totals{'error'}++; return unless ($Collecting{'fatalerror'}); + $Counts{'error'}{ucfirst($reason)}++; +} + +# Handles postfix warning: lines +# and additional lines coerced into warnings +# +sub postfix_warning($) { + my ($warning) = shift; + + # Skip these + return if ($warning =~ /$re_QID: skipping further client input$/o); + return if ($warning =~ /^Mail system is down -- accessing queue directly$/o); + return if ($warning =~ /^SASL authentication failure: (?:Password verification failed|no secret in database)$/o); + return if ($warning =~ /^no MX host for .* has a valid A record$/o); + return if ($warning =~ /^uid=\d+: Broken pipe$/o); + + #TD warning: connect to 127.0.0.1:12525: Connection refused + #TD warning: problem talking to server 127.0.0.1:12525: Connection refused + #TD warning: valid_ipv4_hostaddr: invalid octet count: + + my ($domain,$to,$type,$site,$helo,$cmd); + my ($addr,$size,$hostip,$host,$port,$reason,$qid,$queue,$reason2,$process,$status,$service); + + if (($hostip,$host,$reason) = ($warning =~ /^(?:smtpd_peer_init: )?([^:]+): hostname ([^ ]+) verification failed: (.*)$/) or + ($hostip,$reason,$host) = ($warning =~ /^(?:smtpd_peer_init: )?([^:]+): (address not listed for hostname) (.*)$/) or + ($host,$reason,$hostip,$reason2) = ($warning =~ /^(?:smtpd_peer_init: )?hostname (\S+) (does not resolve to address) ([\d.]+)(: host not found, try again)?$/)) { + #TD warning: 10.0.0.1: hostname sample.com verification failed: Host not found + #TD warning: smtpd_peer_init: 192.168.0.1: hostname example.com verification failed: Name or service not known + #TD warning: 192.168.0.1: address not listed for hostname sample.net + # post 2.8 + #TD warning: hostname 281.example.net does not resolve to address 192.168.0.1: host not found, try again + #TD warning: hostname 281.example.net does not resolve to address 192.168.0.1 + + $reason .= $reason2 if $reason2; + $Totals{'hostnameverification'}++; return unless ($Collecting{'hostnameverification'}); + $Counts{'hostnameverification'}{ucfirst($reason)}{formathost($hostip,$host)}++; + + } elsif (($warning =~ /^$re_QID: queue file size limit exceeded$/o) or + ($warning =~ /^uid=\d+: File too large$/o)) { + $Totals{'warnfiletoobig'}++; + + } elsif ($warning =~ /^database (?:[^ ]*) is older than source file ([\w\/]+)$/o) { + #TD warning: database /etc/postfix/client_checks.db is older than source file /etc/postfix/client_checks + $Totals{'databasegeneration'}++; return unless ($Collecting{'databasegeneration'}); + $Counts{'databasegeneration'}{$1}++; + + } elsif (($reason,$qid,$reason2) = ($warning =~ /^(open active) ($re_QID): (.*)$/o) or + ($reason,$qid,$reason2) = ($warning =~ /^qmgr_active_corrupt: (save corrupt file queue active) id ($re_QID): (.*)$/o) or + ($qid,$reason,$reason2) = ($warning =~ /^($re_QID): (write queue file): (.*)$/o)) { + + #TD warning: open active BDB9B1309F7: No such file or directory + #TD warning: qmgr_active_corrupt: save corrupt file queue active id 4F4272F342: No such file or directory + #TD warning: E669DE52: write queue file: No such file or directory + + $Totals{'queuewriteerror'}++; return unless ($Collecting{'queuewriteerror'}); + $Counts{'queuewriteerror'}{"$reason: $reason2"}{$qid}++; + + } elsif (($qid,$reason) = ($warning =~ /^qmgr_active_done_3_generic: remove ($re_QID) from active: (.*)$/o)) { + #TD warning: qmgr_active_done_3_generic: remove AF0F223FC05 from active: No such file or directory + $Totals{'queuewriteerror'}++; return unless ($Collecting{'queuewriteerror'}); + $Counts{'queuewriteerror'}{"remove from active: $reason"}{$qid}++; + + } elsif (($queue,$qid) = ($warning =~ /^([^\/]*)\/($re_QID): Error writing message file$/o )) { + #TD warning: maildrop/C9E66ADF: Error writing message file + $Totals{'messagewriteerror'}++; return unless ($Collecting{'messagewriteerror'}); + $Counts{'messagewriteerror'}{$queue}{$qid}++; + + } elsif (($process,$status) = ($warning =~ /^process ([^ ]*) pid \d+ exit status (\d+)$/o)) { + #TD warning: process /usr/lib/postfix/smtp pid 9724 exit status 1 + $Totals{'processexit'}++; return unless ($Collecting{'processexit'}); + $Counts{'processexit'}{"Exit status $status"}{$process}++; + + } elsif ($warning =~ /^mailer loop: (.*)$/o) { + #TD warning: mailer loop: best MX host for example.com is local + $Totals{'mailerloop'}++; return unless ($Collecting{'mailerloop'}); + $Counts{'mailerloop'}{$1}++; + + } elsif ($warning =~ /^no (\S+) host for (\S+) has a valid address record$/) { + #TDs warning: no MX host for example.com has a valid address record + $Totals{'dnserror'}++; return unless ($Collecting{'dnserror'}); + $Counts{'dnserror'}{"No $1 host has a valid address record"}{$2}{$END_KEY}++; + + } elsif ($warning =~ /^(Unable to look up \S+ host) (.+)$/) { + #TDsd warning: Unable to look up MX host for example.com: Host not found + #TDsd warning: Unable to look up MX host mail.example.com for Sender address from@example.com: hostname nor servname provided, or not known + #TDsd warning: Unable to look up NS host ns1.example.logal for Sender address bounce@example.local: No address associated with hostname + $Totals{'dnserror'}++; return unless ($Collecting{'dnserror'}); + + my ($problem,$target,$reason) = ($1, split(/: /,$2)); + $reason =~ s/, try again//; + + if ($target =~ /^for (\S+)$/) { + $Counts{'dnserror'}{$problem}{ucfirst($reason)}{$1}{$END_KEY}++; + } + elsif ($target =~ /^(\S+)( for \S+ address) (\S+)$/) { + $Counts{'dnserror'}{$problem . lc($2)}{ucfirst($reason)}{$1}{$3}++; + } + + } elsif ($warning =~ /^((?:malformed|numeric) domain name in .+? of \S+ record) for (.*):(.*)?$/) { + my ($problem,$domain,$reason) = ($1,$2,$3); + #TDsd warning: malformed domain name in resource data of MX record for example.com: + #TDsd warning: malformed domain name in resource data of MX record for example.com: mail.example.com\\032 + #TDsd warning: numeric domain name in resource data of MX record for sample.com: 192.168.0.1 + $Totals{'dnserror'}++; return unless ($Collecting{'dnserror'}); + $Counts{'dnserror'}{ucfirst($problem)}{$domain}{$reason eq '' ? '*unknown' : $reason}{$END_KEY}++; + + } elsif ($warning =~ /^numeric hostname: ([\S]+)$/) { + #TD warning: numeric hostname: 192.168.0.1 + $Totals{'numerichostname'}++; return unless ($Collecting{'numerichostname'}); + $Counts{'numerichostname'}{$1}++; + + } elsif ( ($host,$hostip,$port,$type,$reason) = ($warning =~ /^([^[]+)\[([^]]+)\](?::(\d+))? (sent \w+ header instead of SMTP command): (.*)$/) or + ($type,$host,$hostip,$port,$reason) = ($warning =~ /^(non-E?SMTP command) from ([^[]+)\[([^]]+)\](?::(\d+))?: (.*)$/) or + ($type,$host,$hostip,$port,$reason) = ($warning =~ /^(?:$re_QID: )?(non-E?SMTP response) from ([^[]+)\[([^]]+)\](?::(\d+))?:(?: (.*))?$/o)) { + # ancient + #TDsd warning: example.com[192.168.0.1] sent message header instead of SMTP command: From: "Someone" <40245426501example.com> + # current + #TDsd warning: non-SMTP command from sample.net[10.0.0.1]: Received: from 192.168.0.1 (HELO bogus.sample.com) + #TDs warning: 6B01A8DEF: non-ESMTP response from mail.example.com[192.168.0.1]:25: + + $Totals{'smtpconversationerror'}++; return unless ($Collecting{'smtpconversationerror'}); + $host .= ' :' . $port if ($port and $port ne '25'); + $Counts{'smtpconversationerror'}{ucfirst($type)}{formathost($hostip,$host)}{$reason}++; + + } elsif ($warning =~ /^valid_hostname: (.*)$/o) { + #TD warning: valid_hostname: empty hostname + $Totals{'hostnamevalidationerror'}++; return unless ($Collecting{'hostnamevalidationerror'}); + $Counts{'hostnamevalidationerror'}{$1}++; + + } elsif (($host,$hostip,$type,$reason) = ($warning =~ /^([^[]+)\[([^]]+)\](?::\d+)?: SASL (.*) authentication failed(.*)$/)) { + #TDsd warning: unknown[10.0.0.1]: SASL LOGIN authentication failed: bad protocol / cancel + #TDsd warning: example.com[192.168.0.1]: SASL DIGEST-MD5 authentication failed + # see saslauthfail elsewhere + $Totals{'saslauthfail'}++; return unless ($Collecting{'saslauthfail'}); + if ($reason) { $reason = $type . $reason; } + else { $reason = $type; } + $Counts{'saslauthfail'}{$reason}{formathost($hostip,$host)}++; + + } elsif (($host,$reason) = ($warning =~ /^(\S+): RBL lookup error:.* Name service error for (?:name=)?\1(?: type=[^:]+)?: (.*)$/o)) { + #TD warning: 192.168.0.1.sbl.spamhaus.org: RBL lookup error: Host or domain name not found. Name service error for name=192.168.0.1.sbl.spamhaus.org type=A: Host not found, try again + + #TD warning: 10.0.0.1.relays.osirusoft.com: RBL lookup error: Name service error for 10.0.0.1.relays.osirusoft.com: Host not found, try again + $Totals{'rblerror'}++; return unless ($Collecting{'rblerror'}); + $Counts{'rblerror'}{$reason}{$host}++; + + } elsif ( + ($host,$hostip,$reason,$helo) = ($warning =~ /^host ([^[]+)\[([^]]+)\](?::\d+)? (greeted me with my own hostname) ([^ ]*)$/ ) or + ($host,$hostip,$reason,$helo) = ($warning =~ /^host ([^[]+)\[([^]]+)\](?::\d+)? (replied to HELO\/EHLO with my own hostname) ([^ ]*)$/ )) { + #TDs warning: host example.com[192.168.0.1] greeted me with my own hostname example.com + #TDs warning: host example.com[192.168.0.1] replied to HELO/EHLO with my own hostname example.com + $Totals{'heloerror'}++; return unless ($Collecting{'heloerror'}); + $Counts{'heloerror'}{ucfirst($reason)}{formathost($hostip,$host)}++; + + } elsif (($size,$host,$hostip) = ($warning =~ /^bad size limit "([^"]+)" in EHLO reply from ([^[]+)\[([^]]+)\](?::\d+)?$/ )) { + #TD warning: bad size limit "-679215104" in EHLO reply from example.com[192.168.0.1] + $Totals{'heloerror'}++; return unless ($Collecting{'heloerror'}); + $Counts{'heloerror'}{"Bad size limit in EHLO reply"}{formathost($hostip,$host)}{"$size"}++; + + } elsif ( ($host,$hostip,$cmd,$addr) = ($warning =~ /^Illegal address syntax from ([^[]+)\[([^]]+)\](?::\d+)? in ([^ ]*) command: (.*)/ )) { + #TD warning: Illegal address syntax from example.com[192.168.0.1] in MAIL command: user@sample.net + $addr =~ s/[<>]//g unless ($addr eq '<>'); + $Totals{'illegaladdrsyntax'}++; return unless ($Collecting{'illegaladdrsyntax'}); + $Counts{'illegaladdrsyntax'}{$cmd}{$addr}{formathost($hostip,$host)}++; + + } elsif ($warning =~ /^(timeout|premature end-of-input) on (.+) while reading (.*)$/o + or $warning =~ /^(malformed (?:base64|numerical)|unexpected end-of-input) from (.+) while reading (.*)$/o) { + + #TDs warning: premature end-of-input on private/anvil while reading input attribute name + #TDs warning: timeout on private/anvil while reading input attribute data + #TDs warning: unexpected end-of-input from 127.0.0.1:10025 socket while reading input attribute name + #TDs warning: malformed base64 data from %s while reading input attribute data: ... + #TDs warning: malformed numerical data from %s while reading input attribute data: ... + + $Totals{'attrerror'}++; return unless ($Collecting{'attrerror'}); + $Counts{'attrerror'}{$2}{$1}{$3}++; + + } elsif ($warning =~ /^(.*): (bad command startup -- throttling)/o) { + #TD warning: /usr/libexec/postfix/trivial-rewrite: bad command startup -- throttling + $Totals{'startuperror'}++; return unless ($Collecting{'startuperror'}); + $Counts{'startuperror'}{ucfirst($2)}{$1}++; + + } elsif ($warning =~ /(problem talking to service [^:]*): (.*)$/o) { + #TD warning: problem talking to service rewrite: Connection reset by peer + #TD warning: problem talking to service rewrite: Success + $Totals{'communicationerror'}++; return unless ($Collecting{'communicationerror'}); + $Counts{'communicationerror'}{ucfirst($1)}{$2}++; + + } elsif (my ($map,$key) = ($warning =~ /^$re_QID: ([^ ]*) map lookup problem for (.*)$/o)) { + #TD warning: 6F74F74431: virtual_alias_maps map lookup problem for root@example.com + $Totals{'mapproblem'}++; return unless ($Collecting{'mapproblem'}); + $Counts{'mapproblem'}{$map}{$key}++; + + } elsif (($map,$reason) = ($warning =~ /^pcre map ([^,]+), (.*)$/o)) { + #TD warning: pcre map /etc/postfix/body_checks, line 92: unknown regexp option "F": skipping this rule + $Totals{'mapproblem'}++; return unless ($Collecting{'mapproblem'}); + $Counts{'mapproblem'}{$map}{$reason}++; + + } elsif (($reason) = ($warning =~ /dict_ldap_lookup: (.*)$/o)) { + #TD warning: dict_ldap_lookup: Search error 80: Internal (implementation specific) error + $Totals{'ldaperror'}++; return unless ($Collecting{'ldaperror'}); + $Counts{'ldaperror'}{$reason}++; + + } elsif (($type,$size,$host,$hostip,$service) = ($warning =~ /^(.+) limit exceeded: (\d+) from ([^[]+)\[([^]]+)\](?::\d+)? for service (.*)/ )) { + #TDsd warning: Connection concurrency limit exceeded: 51 from example.com[192.168.0.1] for service smtp + #TDsd warning: Connection rate limit exceeded: 20 from mail.example.com[192.168.0.1] for service smtp + #TDsd warning: Connection rate limit exceeded: 30 from unknown[unknown] for service smtp + #TDsd warning: Recipient address rate limit exceeded: 21 from example.com[10.0.0.1] for service smtp + #TDsd warning: Message delivery request rate limit exceeded: 11 from example.com[10.0.0.1] for service smtp + #TDsd warning: New TLS session rate limit exceeded: 49 from example.com[10.0.0.1] for service smtp + $Totals{'anvil'}++; return unless ($Collecting{'anvil'}); + $Counts{'anvil'}{$service}{$type}{formathost($hostip,$host)}{$size}++; + + } elsif (my ($extname,$intname,$limit) = ($warning =~ /service "([^"]+)" \(([^)]+)\) has reached its process limit "([^"]+)":/o)) { + #TD warning: service "smtp" (25) has reached its process limit "50": new clients may experience noticeable delays + $Totals{'processlimit'}++; return unless ($Collecting{'processlimit'}); + $Counts{'processlimit'}{'See http://www.postfix.org/STRESS_README.html'}{"$extname ($intname)"}{$limit}++; + + } else { + #TDsd warning: No server certs available. TLS won't be enabled + #TDs warning: smtp_connect_addr: bind : Address already in use + + # These two messages follow ProcessLimit message above + #TDm warning: to avoid this condition, increase the process count in master.cf or reduce the service time per client + #TDm warning: see http://www.postfix.org/STRESS_README.html for examples of stress-dependent configuration settings + return if ($warning =~ /^to avoid this condition,/o); + return if ($warning =~ /^see http:\/\/www\.postfix\.org\/STRESS_README.html/o); + + #TDsd warning: 009314BD9E: read timeout on cleanup socket + $warning =~ s/^$re_QID: (read timeout on \S+ socket)/$1/; + + #TDsd warning: Read failed in network_biopair_interop with errno=0: num_read=0, want_read=11 + #TDs warning: Read failed in network_biopair_interop with errno=0: num_read=0, want_read=11 + $warning =~ s/^(Read failed in network_biopair_interop) with .*$/$1/; + +=cut + $warning =~ s/^(TLS library problem: )\d+:(error:.*)$/$1$2/; + $warning =~ s/^(network_biopair_interop: error reading) \d+ bytes(.*)$/$1$2/; + +1 TLS library problem: 10212:error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher... +1 TLS library problem: 10217:error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher... +1 network_biopair_interop: error reading 1102 bytes from the network: Connection reset by peer +1 network_biopair_interop: error reading 1120 bytes from the network: Connection reset by peer +=cut + + + $Totals{'warningsother'}++; return unless ($Collecting{'warningsother'}); + $Counts{'warningsother'}{$warning}++; + } +} + +# Handles postfix/postfix-script lines +# +sub postfix_script($) { + my $line = shift; + + return if ($line =~ /^the Postfix mail system is running: PID: /o); + + if ($line =~ /^starting the Postfix mail system/o) { + $Totals{'postfixstart'}++; + } + elsif ($line =~ /^stopping the Postfix mail system/o) { + $Totals{'postfixstop'}++; + } + elsif ($line =~ /^refreshing the Postfix mail system/o) { + $Totals{'postfixrefresh'}++; + } + elsif ($line =~ /^waiting for the Postfix mail system to terminate/o) { + $Totals{'postfixwaiting'}++; + } + elsif (! in_ignore_list ($line)) { + inc_unmatched('postfix_script'); + } +} + +# Clean up a server's reply, to give some uniformity to reports +# +sub cleanhostreply($ $ $ $) { + my ($hostreply,$relay,$recip,$domain) = @_; + + my $fmtdhost = ''; + my ($r1, $r2, $dsn, $msg, $host, $event); + + #print "RELAY: $relay, RECIP: $recip, DOMAIN: $domain\n"; + #print "HOSTREPLY: \"$hostreply\"\n"; + return ('Accepted', '*unknown') if $hostreply =~ /^25\d/o; + + # Host or domain name not found. Name service error for name=example.com type=MX: Host not found... + if ($hostreply =~ /^Host or domain name not found. Name service error for name=([^:]+): Host not found/o) { + return ('Host not found', $1); + } + + if (($host,$dsn,$r1) = ($hostreply =~ /host (\S+) said: ($re_DSN)[\- :]*"?(.*)"?$/o)) { + # Strip recipient address from host's reply - we already have it in $recip. + $r1 =~ s/[<(]?\Q$recip\E[>)]?\W*//ig; + + # Strip and capture "in reply to XYZ command" from host's reply + if ($r1 =~ s/\s*[(]?(?:in reply to (.*) command)[)]?//o) { + $r2 = ": $1"; + } + $r1 =~ s/^Recipient address rejected: //o; + # Canonicalize numerous forms of "recipient unknown" + if ( $r1 =~ /^user unknown/i + or $r1 =~ /^unknown user/i + or $r1 =~ /^unknown recipient address/i + or $r1 =~ /^invalid recipient/i + or $r1 =~ /^recipient unknown/i + or $r1 =~ /^sorry, no mailbox here by that name/i + or $r1 =~ /^User is unknown/ + or $r1 =~ /^User not known/ + or $r1 =~ /^MAILBOX NOT FOUND/ + or $r1 =~ /^Recipient Rejected: No account by that name here/ + or $r1 =~ /^Recipient does not exist here/ + or $r1 =~ /The email account that you tried to reach does not exist./ # Google's long mess + or $r1 =~ /(?:no such user|user unknown)/i + ) + { + #print "UNKNOWN RECIP: $r1\n"; + $r1 = 'Unknown recipient'; + } + elsif ($r1 =~ /greylisted/oi) { + #print "GREYLISTED RECIP: $r1\n"; + $r1 = 'Recipient greylisted'; + } + elsif ($r1 =~ /^Message temporarily deferred - (\d\.\d+\.\d+)\. Please refer to (.+)$/o) { + # Yahoo: 421 Message temporarily deferred - 4.16.51. Please refer to http://... (in reply to end of DATA command)) + $dsn = "$dsn $1"; $r1 = "see $2"; + } + elsif ($r1 =~ /^Resources temporarily not available - Please try again later \[#(\d\.\d+\.\d+)\]\.$/o) { + #Yahoo 451 Resources temporarily not available - Please try again later [#4.16.5]. + $dsn = "$dsn $1"; $r1 = "resources not available"; + } + elsif ($r1 =~ /^Message temporarily deferred - (\[\d+\])/o) { + # Yahoo: 451 Message temporarily deferred - [160] + $dsn = "$dsn $1"; $r1 = ''; + } + } + + elsif ($hostreply =~ /^connect to (\S+): (.*)$/o) { + #print "CONNECT: $hostreply\n"; + $host = $1; $r1 = $2; $r1 =~ s/server refused to talk to me/refused/; + } + + elsif ($hostreply =~ /^host (\S+) refused to talk to me: (.*)$/o) { + $host = $1; $msg = $2; + #print "HOSTREFUSED: $hostreply\n"; + #Yahoo: '421 Message from (10.0.0.1) temporarily deferred - 4.16.50. Please refer to http://... + if ($msg =~ /^(\d+) Message from \([^)]+\) temporarily deferred - (\d\.\d+\.\d+)\. Please refer to (.+)$/) { + $dsn = "$1 $2"; $msg = "see $3"; + } + #$r1 = join(': ', 'refused', $msg); + $r1 = $msg; + } + elsif ($hostreply =~ /^(delivery temporarily suspended): connect to (\S+): (.*)$/o) { + #print "DELIVERY SUSP: $hostreply\n"; + $host = $2; $r1 = join(': ', $1, $3); + } + elsif ($hostreply =~ /^(delivery temporarily suspended: conversation) with (\S+) (.*)$/o) { + # delivery temporarily suspended: conversation with example.com[10.0.0.1] timed out while receiving the initial server greeting) + #print "DELIVERY SUSP2: $hostreply\n"; + $host = $2; $r1 = join(' ', $1, $3); + } + elsif (($event,$host,$r1) = ($hostreply =~ /^(lost connection|conversation) with (\S+) (.*)$/o)) { + #print "LOST conv/conn: $hostreply\n"; + $r1 = join(' ',$event,$r1); + } + elsif ($hostreply =~ /^(.*: \S+maildrop: Unable to create a dot-lock) at .*$/o) { + #print "MAILDROP: $hostreply\n"; + $r1 = $1; + } + elsif ($hostreply =~ /^mail for (\S+) loops back to myself/o) { + #print "LOOP: $hostreply\n"; + $host = $1; $r1 = 'mailer loop'; + } + elsif ($hostreply =~ /^unable to find primary relay for (\S+)$/o) { + #print "NORELAY: $hostreply\n"; + $host = $1; $r1 = 'no relay found'; + } + elsif ($hostreply =~ /^message size \d+ exceeds size limit \d+ of server (\S+)\s*$/o) { + #print "TOOBIG: $hostreply\n"; + $host = $1; $r1 = 'message too big'; + } + else { + #print "UNMATCH: $hostreply\n"; + $r1 = $hostreply; + } + + #print "R1: $r1, R2: $r2\n"; + $r1 =~ s/for name=\Q$domain\E //ig; + + if ($host eq '') { + if ($relay =~ /([^[]+)\[([^]]+)\]/) { + $fmtdhost = formathost($2,$1); + } + else { + $fmtdhost = '*unknown'; + } + } + elsif ($host =~ /^([^[]+)\[([^]]+)\]/) { + $fmtdhost = formathost($2,$1); + } + else { + $fmtdhost = $host; + } + + return (($dsn ? "$dsn " : '' ) . "\u$r1$r2", $fmtdhost); +} + +# Strip and return from, to, proto, and helo information from a log line +# From is set to the empty envelope sender <> as necessary, and To is +# always lowercased. +# +# Note: modifies its input for efficiency +# +sub strip_ftph($) { + my ($helo, $proto, $to, $from); + #print "strip_ftph: '$_[0]\n"; + $helo = ($_[0] =~ s/\s+helo=<(.*?)>\s*$//) == 1 ? $1 : '*unavailable'; + $proto = ($_[0] =~ s/\s+proto=(\S+)\s*$//) == 1 ? $1 : '*unavailable'; + $to = ($_[0] =~ s/\s+to=<(.*?)>\s*$//) == 1 ? (lc($1) || '<>') : '*unavailable'; + $from = ($_[0] =~ s/\s+from=<(.*?)>\s*$//) == 1 ? ( $1 || '<>') : '*unavailable'; + + #print "helo: $helo, proto: $proto, to: $to, from: $from\n"; + #print "strip_ftph: final: '$_[0]'\n"; + return ($from,$to,$proto,$helo); +} + +# Initialize the Getopts option list. Requires the Section table to +# be built already. +# +sub init_getopts_table() { + print "init_getopts_table: enter\n" if $Opts{'debug'} & Logreporters::D_ARGS; + + init_getopts_table_common(@supplemental_reports); + + add_option ('recipient_delimiter=s'); + add_option ('delays!'); + add_option ('show_delays=i', sub { $Opts{'delays'} = $_[1]; 1; }); + add_option ('delays_percentiles=s'); + add_option ('reject_reply_patterns=s'); + add_option ('ignore_services=s'); + add_option ('postgrey_delays!'); + add_option ('postgrey_show_delays=i', sub { $Opts{'postgrey_delays'} = $_[1]; 1; }); + add_option ('postgrey_delays_percentiles=s'); + add_option ('unknown!', sub { $Opts{'unknown'} = $_[1]; 1; }); + add_option ('show_unknown=i', sub { $Opts{'unknown'} = $_[1]; 1; }); + add_option ('enable_long_queue_ids=i', sub { $Opts{'long_queue_ids'} = $_[1]; 1; }); + add_option ('long_queue_ids!'); + +=pod + # aliases and backwards compatibility + add_option ('msgsdeferred=s', \$Opts{'deferred'}); + add_option ('msgsdelivered=s', \$Opts{'delivered'}); + add_option ('msgssent=s', \$Opts{'sent'}); + add_option ('msgssentlmtp=s', \$Opts{'sentlmtp'}); + add_option ('msgsforwarded=s', \$Opts{'forwarded'}); + add_option ('msgsresent=s', \$Opts{'resent'}); + add_option ('warn=s', \$Opts{'warned'}); + add_option ('held=s', \$Opts{'hold'}); +=cut +} + +# Builds the entire @Section table used for data collection +# +# Each Section entry has as many as six fields: +# +# 1. Section array reference +# 2. Key to %Counts, %Totals accumulator hashes, and %Collecting hash +# 3. Output in Detail report? (must also a %Counts accumulator) +# 4. Numeric output format specifier for Summary report +# 5. Section title for Summary and Detail reports +# 6. A hash to a divisor used to calculate the percentage of a total for that key +# +# Use begin_section_group/end_section_group to create groupings around sections. +# +# Sections can be freely reordered if desired, but maintain proper group nesting. +# +# +# The reject* entries of this table are dynamic, in that they are built based +# upon the value of $Opts{'reject_reply_patterns'}, which can be specified by +# either command line or configuration file. This allows various flavors, of +# reject sections based on SMTP reply code (eg. 421 45x, 5xx, etc.). Instead +# of creating special sections for each reject variant, the primary key of each +# reject section could have been the SMTP reply code. However, this would +# require special-case processing to distinguish 4xx temporary rejects from 5xx +# permanent rejects in various Totals{'totalrejects*'} counts, and in the +# Totals{'totalrejects'} tally. +# +# Sections can be freely reordered if desired. +sub build_sect_table() { + if ($Opts{'debug'} & Logreporters::D_SECT) { + print "build_sect_table: enter\n"; + print "\treject patterns: $Opts{'reject_reply_patterns'}\n"; + } + my $S = \@Sections; + + # References to these are used in the Sections table below; we'll predeclare them. + $Totals{'totalrejects'} = 0; + $Totals{'totalrejectswarn'} = 0; + $Totals{'totalacceptplusreject'} = 0; + + # Configuration and critical errors appear first + + # SECTIONREF, NAME, DETAIL, FMT, TITLE, DIVISOR + begin_section_group ($S, 'warnings'); + add_section ($S, 'panicerror', 1, 'd', '*Panic: General panic'); + add_section ($S, 'fatalfiletoobig', 0, 'd', '*Fatal: Message file too big'); + add_section ($S, 'fatalconfigerror', 1, 'd', '*Fatal: Configuration error'); + add_section ($S, 'fatalerror', 1, 'd', '*Fatal: General fatal'); + add_section ($S, 'error', 1, 'd', '*Error: General error'); + add_section ($S, 'processlimit', 1, 'd', '*Warning: Process limit reached, clients may delay'); + add_section ($S, 'warnfiletoobig', 0, 'd', '*Warning: Queue file size limit exceeded'); + add_section ($S, 'warninsufficientspace', 0, 'd', '*Warning: Insufficient system storage error'); + add_section ($S, 'warnconfigerror', 1, 'd', '*Warning: Server configuration error'); + add_section ($S, 'queuewriteerror', 1, 'd', '*Warning: Error writing queue file'); + add_section ($S, 'messagewriteerror', 1, 'd', '*Warning: Error writing message file'); + add_section ($S, 'databasegeneration', 1, 'd', '*Warning: Database is older than source file'); + add_section ($S, 'mailerloop', 1, 'd', '*Warning: Mailer loop'); + add_section ($S, 'startuperror', 1, 'd', '*Warning: Startup error'); + add_section ($S, 'mapproblem', 1, 'd', '*Warning: Map lookup problem'); + add_section ($S, 'attrerror', 1, 'd', '*Warning: Error reading attribute data'); + add_section ($S, 'anvil', 1, 'd', '*Warning: Anvil limit reached'); + add_section ($S, 'processexit', 1, 'd', 'Process exited'); + add_section ($S, 'hold', 1, 'd', 'Placed on hold'); + add_section ($S, 'communicationerror', 1, 'd', 'Postfix communications error'); + add_section ($S, 'saslauthfail', 1, 'd', 'SASL authentication failed'); + add_section ($S, 'ldaperror', 1, 'd', 'LDAP error'); + add_section ($S, 'warningsother', 1, 'd', 'Miscellaneous warnings'); + add_section ($S, 'totalrejectswarn', 0, 'd', 'Reject warnings (warn_if_reject)'); + end_section_group ($S, 'warnings'); + + begin_section_group ($S, 'bytes', "\n"); + add_section ($S, 'bytesaccepted', 0, 'Z', 'Bytes accepted '); # Z means print scaled as in 1k, 1m, etc. + add_section ($S, 'bytessentsmtp', 0, 'Z', 'Bytes sent via SMTP'); + add_section ($S, 'bytessentlmtp', 0, 'Z', 'Bytes sent via LMTP'); + add_section ($S, 'bytesdelivered', 0, 'Z', 'Bytes delivered'); + add_section ($S, 'bytesforwarded', 0, 'Z', 'Bytes forwarded'); + end_section_group ($S, 'bytes', $sep1); + + begin_section_group ($S, 'acceptreject', "\n"); + begin_section_group ($S, 'acceptreject2', "\n"); + add_section ($S, 'msgsaccepted', 0, 'd', 'Accepted', \$Totals{'totalacceptplusreject'}); + add_section ($S, 'totalrejects', 0, 'd', 'Rejected', \$Totals{'totalacceptplusreject'}); + end_section_group ($S, 'acceptreject2', $sep2); + add_section ($S, 'totalacceptplusreject', 0, 'd', 'Total', \$Totals{'totalacceptplusreject'}); + end_section_group ($S, 'acceptreject', $sep1); + + # The various Reject sections are built dynamically based upon a list of reject reply keys, + # which are user-configured via $Opts{'reject_reply_patterns'} + @RejectPats = (); + foreach my $rejpat (split /[ ,]/, $Opts{'reject_reply_patterns'}) { + if ($rejpat !~ /^(warn|[45][\d.]{2})$/io) { + print STDERR usage "Invalid pattern \"$rejpat\" in reject_reply_patterns"; + exit (2); + } + if (grep (/\Q$rejpat\E/, @RejectPats) == 0) { + push @RejectPats, $rejpat + } + else { + print STDERR "Ignoring duplicate pattern \"$rejpat\" in reject_reply_patterns\n"; + } + } + @RejectKeys = @RejectPats; + for (@RejectKeys) { + s/\./x/g; + } + + print "\tRejectPat: \"@RejectPats\", RejectKeys: \"@RejectKeys\"\n" if $Opts{'debug'} & Logreporters::D_SECT; + + # Add reject variants + foreach my $key (@RejectKeys) { + $key = lc($key); + my $keyuc = ucfirst($key); + my $totalsref = \$Totals{'totalrejects' . $key}; + print "\t reject key: $key\n" if $Opts{'debug'} & Logreporters::D_SECT; + + begin_section_group ($S, 'rejects', "\n"); + begin_section_group ($S, 'rejects2', "\n"); + add_section ($S, $key . 'rejectrelay', 1, 'd', $keyuc . ' Reject relay denied', $totalsref); + add_section ($S, $key . 'rejecthelo', 1, 'd', $keyuc . ' Reject HELO/EHLO', $totalsref); + add_section ($S, $key . 'rejectdata', 1, 'd', $keyuc . ' Reject DATA', $totalsref); + add_section ($S, $key . 'rejectunknownuser', 1, 'd', $keyuc . ' Reject unknown user', $totalsref); + add_section ($S, $key . 'rejectrecip', 1, 'd', $keyuc . ' Reject recipient address', $totalsref); + add_section ($S, $key . 'rejectsender', 1, 'd', $keyuc . ' Reject sender address', $totalsref); + add_section ($S, $key . 'rejectclient', 1, 'd', $keyuc . ' Reject client host', $totalsref); + add_section ($S, $key . 'rejectunknownclient', 1, 'd', $keyuc . ' Reject unknown client host', $totalsref); + add_section ($S, $key . 'rejectunknownreverseclient', 1, 'd', $keyuc . ' Reject unknown reverse client host', $totalsref); + add_section ($S, $key . 'rejectunverifiedclient', 1, 'd', $keyuc . ' Reject unverified client host', $totalsref); + add_section ($S, $key . 'rejectrbl', 1, 'd', $keyuc . ' Reject RBL', $totalsref); + add_section ($S, $key . 'rejectheader', 1, 'd', $keyuc . ' Reject header', $totalsref); + add_section ($S, $key . 'rejectbody', 1, 'd', $keyuc . ' Reject body', $totalsref); + add_section ($S, $key . 'rejectcontent', 1, 'd', $keyuc . ' Reject content', $totalsref); + add_section ($S, $key . 'rejectsize', 1, 'd', $keyuc . ' Reject message size', $totalsref); + add_section ($S, $key . 'rejectmilter', 1, 'd', $keyuc . ' Reject milter', $totalsref); + add_section ($S, $key . 'rejectproxy', 1, 'd', $keyuc . ' Reject proxy', $totalsref); + add_section ($S, $key . 'rejectinsufficientspace', 1, 'd', $keyuc . ' Reject insufficient space', $totalsref); + add_section ($S, $key . 'rejectconfigerror', 1, 'd', $keyuc . ' Reject server config error', $totalsref); + add_section ($S, $key . 'rejectverify', 1, 'd', $keyuc . ' Reject VRFY', $totalsref); + add_section ($S, $key . 'rejectetrn', 1, 'd', $keyuc . ' Reject ETRN', $totalsref); + add_section ($S, $key . 'rejectlookupfailure', 1, 'd', $keyuc . ' Reject temporary lookup failure', $totalsref); + end_section_group ($S, 'rejects2', $sep2); + add_section ($S, 'totalrejects' . $key, 0, 'd', "Total $keyuc Rejects", $totalsref); + end_section_group ($S, 'rejects', $sep1); + + $Totals{'totalrejects' . $key} = 0; + } + + begin_section_group ($S, 'byiprejects', "\n"); + add_section ($S, 'byiprejects', 1, 'd', 'Reject by IP'); + end_section_group ($S, 'byiprejects'); + + begin_section_group ($S, 'general1', "\n"); + add_section ($S, 'connectioninbound', 1, 'd', 'Connections'); + add_section ($S, 'connectionlostinbound', 1, 'd', 'Connections lost (inbound)'); + add_section ($S, 'connectionlostoutbound', 1, 'd', 'Connections lost (outbound)'); + add_section ($S, 'disconnection', 0, 'd', 'Disconnections'); + add_section ($S, 'removedfromqueue', 0, 'd', 'Removed from queue'); + add_section ($S, 'delivered', 1, 'd', 'Delivered'); + add_section ($S, 'sent', 1, 'd', 'Sent via SMTP'); + add_section ($S, 'sentlmtp', 1, 'd', 'Sent via LMTP'); + add_section ($S, 'forwarded', 1, 'd', 'Forwarded'); + add_section ($S, 'resent', 0, 'd', 'Resent'); + add_section ($S, 'deferred', 1, 'd', 'Deferred'); + add_section ($S, 'deferrals', 1, 'd', 'Deferrals'); + add_section ($S, 'bouncelocal', 1, 'd', 'Bounced (local)'); + add_section ($S, 'bounceremote', 1, 'd', 'Bounced (remote)'); + add_section ($S, 'bouncefailed', 1, 'd', 'Bounce failure'); + add_section ($S, 'postscreen', 1, 'd', 'Postscreen'); + add_section ($S, 'dnsblog', 1, 'd', 'DNSBL log'); + + add_section ($S, 'envelopesenders', 1, 'd', 'Envelope senders'); + add_section ($S, 'envelopesenderdomains', 1, 'd', 'Envelope sender domains'); + + add_section ($S, 'bcced', 1, 'd', 'BCCed'); + add_section ($S, 'filtered', 1, 'd', 'Filtered'); + add_section ($S, 'redirected', 1, 'd', 'Redirected'); + add_section ($S, 'discarded', 1, 'd', 'Discarded'); + add_section ($S, 'prepended', 1, 'd', 'Prepended'); + add_section ($S, 'replaced', 1, 'd', 'Replaced'); + add_section ($S, 'warned', 1, 'd', 'Warned'); + + add_section ($S, 'requeued', 0, 'd', 'Requeued messages'); + add_section ($S, 'returnedtosender', 1, 'd', 'Expired and returned to sender'); + add_section ($S, 'notificationsent', 1, 'd', 'Notifications sent'); + + add_section ($S, 'policyspf', 1, 'd', 'Policy SPF'); + add_section ($S, 'policydweight', 1, 'd', 'Policyd-weight'); + add_section ($S, 'postfwd', 1, 'd', 'Postfwd'); + add_section ($S, 'postgrey', 1, 'd', 'Postgrey'); + end_section_group ($S, 'general1'); + + begin_section_group ($S, 'general2', "\n"); + add_section ($S, 'connecttofailure', 1, 'd', 'Connection failures (outbound)'); + add_section ($S, 'timeoutinbound', 1, 'd', 'Timeouts (inbound)'); + add_section ($S, 'heloerror', 1, 'd', 'HELO/EHLO conversations errors'); + add_section ($S, 'illegaladdrsyntax', 1, 'd', 'Illegal address syntax in SMTP command'); + add_section ($S, 'released', 0, 'd', 'Released from hold'); + add_section ($S, 'rblerror', 1, 'd', 'RBL lookup errors'); + add_section ($S, 'dnserror', 1, 'd', 'DNS lookup errors'); + add_section ($S, 'numerichostname', 1, 'd', 'Numeric hostname'); + add_section ($S, 'smtpconversationerror', 1, 'd', 'SMTP dialog errors'); + add_section ($S, 'hostnameverification', 1, 'd', 'Hostname verification errors (FCRDNS)'); + add_section ($S, 'hostnamevalidationerror', 1, 'd', 'Hostname validation errors'); + add_section ($S, 'smtpprotocolviolation', 1, 'd', 'SMTP protocol violations'); + add_section ($S, 'deliverable', 1, 'd', 'Deliverable (address verification)'); + add_section ($S, 'undeliverable', 1, 'd', 'Undeliverable (address verification)'); + add_section ($S, 'tablechanged', 0, 'd', 'Restarts due to lookup table change'); + add_section ($S, 'pixworkaround', 1, 'd', 'PIX workaround enabled'); + add_section ($S, 'tlsserverconnect', 1, 'd', 'TLS connections (server)'); + add_section ($S, 'tlsclientconnect', 1, 'd', 'TLS connections (client)'); + add_section ($S, 'saslauth', 1, 'd', 'SASL authenticated messages'); + add_section ($S, 'tlsunverified', 1, 'd', 'TLS certificate unverified'); + add_section ($S, 'tlsoffered', 1, 'd', 'Host offered TLS'); + end_section_group ($S, 'general2'); + + begin_section_group ($S, 'postfixstate', "\n"); + add_section ($S, 'postfixstart', 0, 'd', 'Postfix start'); + add_section ($S, 'postfixstop', 0, 'd', 'Postfix stop'); + add_section ($S, 'postfixrefresh', 0, 'd', 'Postfix refresh'); + add_section ($S, 'postfixwaiting', 0, 'd', 'Postfix waiting to terminate'); + end_section_group ($S, 'postfixstate'); + + + if ($Opts{'debug'} & Logreporters::D_SECT) { + print "\tSection table\n"; + printf "\t\t%s\n", (ref($_) eq 'HASH' ? $_->{NAME} : $_) foreach @Sections; + print "build_sect_table: exit\n" + } +} + +# XXX create array of defaults for detail <5, 5-9, >10 +sub init_defaults() { + map { $Opts{$_} = $Defaults{$_} unless exists $Opts{$_} } keys %Defaults; + if (! $Opts{'standalone'}) { + # LOGWATCH + # these take affect if no env present (eg. nothing in conf file) + # 0 to 4 nodelays + + if ($Opts{'detail'} < 5) { # detail 0 to 4, disable all supplimental reports + $Opts{'delays'} = 0; + $Opts{'postgrey_delays'} = 0; + } + } +} + + +# XXX ensure something is matched? +# XXX cache values so we don't have to substitute X for . each time +#match $dsn against list for best fit +sub get_reject_key($) { + my $reply = shift; + my $replyorig = $reply; + ($reply) = split / /, $reply; + for (my $i = 0; $i <= $#RejectPats; $i++) { + #print "TRYING: $RejectPats[$i]\n"; + # we'll allow extended DSNs to match (eg. 5.7.1 will match 5..) + if ($reply =~ /^$RejectPats[$i]/) { # no /o here, pattern varies + #print "MATCHED: orig: $replyorig, reply $reply matched pattern $RejectPats[$i], returning $RejectKeys[$i]\n"; + return $RejectKeys[$i]; + } + } + #print "NOT MATCHED: REPLY CODE: '$replyorig', '$reply'\n"; + return; +} + +# Replace bare reject limiters with specific reject limiters +# based on reject_reply_patterns +# +sub expand_bare_reject_limiters() +{ + # don't reorder the list of limiters. This breaks --nodetail followed by a + # bare reject such as --limit rejectrbl=10. Reordering is no longer necessary + # since process_limiters was instituted and using the special __none__ pseudo- + # limiter to indicate the position at which --nodefailt was found on the command + # line. + # my ($limiter, @reject_limiters, @non_reject_limiters); + my ($limiter, @new_list); + + # XXX check if limiter matches just one in rejectclasses + while ($limiter = shift @Limiters) { + if ($limiter =~ /^reject[^_]/) { + foreach my $reply_code (@RejectKeys) { + printf "bare_reject: \L$reply_code$limiter\n" if $Opts{'debug'} & Logreporters::D_VARS; + #push @reject_limiters, lc($reply_code) . $limiter; + push @new_list, lc($reply_code) . $limiter; + } + } + elsif ($limiter =~ /^(?:[45]\.\.|Warn)reject[^_]/) { + $limiter =~ s/^([45])\.\./$1xx/; + #push @reject_limiters, lc $limiter; + push @new_list, lc $limiter; + } + else { + #push @non_reject_limiters, $limiter; + push @new_list, $limiter; + } + } + #@Limiters = (@reject_limiters, @non_reject_limiters); + @Limiters = @new_list; +} + + +# Return a usage string, built from: +# arg1 + +# $usage_str + +# a string built from each usable entry in the @Sections table. +# reject patterns are special cased to minimize the number of +# command line options presented. +# +sub usage($) { + my $ret = ""; + $ret = "@_\n" if ($_[0]); + + $ret .= $usage_str; + my ($name, $desc, %reject_types); + foreach my $sect (get_usable_sectvars(@Sections, 0)) { + + if (my ($code,$rej) = ($sect->{NAME} =~ /^(...|warn)(reject.*)$/oi)) { + $rej = lc $rej; + next if (exists $reject_types{$rej}); + $reject_types{$rej}++; + $name = '[###]' . $rej; + $desc = '###' . substr($sect->{TITLE}, length($code)); + } + else { + $name = lc $sect->{NAME}; + $desc = $sect->{TITLE}; + } + $ret .= sprintf " --%-38s%s\n", "$name" . ' LEVEL', "$desc"; + } + $ret .= "\n"; + return $ret; +} + +1; + +# vi: shiftwidth=3 tabstop=3 syntax=perl et diff --git a/postfix-logwatch.1 b/postfix-logwatch.1 new file mode 100644 index 0000000..1e67246 --- /dev/null +++ b/postfix-logwatch.1 @@ -0,0 +1,890 @@ +.TH POSTFIX-LOGWATCH 1 +.ad +.fi +.SH NAME +postfix-logwatch +\- +A Postfix log parser and analysis utility +.SH "SYNOPSIS" +.na +.nf +.fi +\fBpostfix-logwatch\fR [\fIoptions\fR] [\fIlogfile ...\fR] +.SH DESCRIPTION +.ad +.fi +The \fBpostfix-logwatch\fR(1) utility is a Postfix MTA log parser +that produces summaries, details, and statistics regarding +the operation of Postfix. +.PP +This utility can be used as a +standalone program, or as a Logwatch filter module to produce +Postfix summary and detailed reports from within Logwatch. +.PP +\fBPostfix-logwatch\fR is able to produce +a wide range of reports with data grouped and sorted as much as possible +to reduce noise and highlight patterns. +Brief summary reports provide a +quick overview of general Postfix operations and message +delivery, calling out warnings that may require attention. +Detailed reports provide easy to scan, hierarchically-arranged +and organized information, with as much or little detail as +desired. +.PP +\fBPostfix-logwatch\fR outputs two principal sections: a \fBSummary\fR section +and a \fBDetailed\fR section. +For readability and quick scanning, all event or hit counts appear in the left column, +followed by brief description of the event type, and finally additional +statistics or count representations may appear in the rightmost column. + +The following segment from a sample Summary report illustrates: +.RS 4 +.nf + +****** Summary ******************************************** + + 81 *Warning: Connection rate limit reached (anvil) + 146 Warned + + 68.310M Bytes accepted 71,628,177 + 97.645M Bytes delivered 102,388,245 +======== ================================================ + + 3464 Accepted 41.44% + 4895 Rejected 58.56% +-------- ------------------------------------------------ + 8359 Total 100.00% +======== ================================================ + +.fi +.RE 0 +The report warns that anvil's connection rate was hit 81 times, +a Postfix access check WARN action was logged 146 times, and +a total of 68.310 megabytes (71,628,177 bytes) were accepted +into the Postfix system, delivering 97.645 megabytes of +data (due to multiple recipients). +The Accepted and Rejected lines show that Postfix accepted 3464 (41.44% of the total +messages) and rejected 4895 (the remaining 58.56%) of the 8359 +total messages (temporary rejects show up elsewhere). +.PP +There are dozens of sub-sections available in the \fBDetailed\fR report, each of +whose output can be controlled in various ways. +Each sub-section attempts to group and present the most meaningful data at superior levels, +while pushing less useful or \fInoisy\fR data towards inferior levels. +The goal is to provide as much benefit as possible from smart grouping of +data, to allow faster report scanning, pattern identification, and problem solving. +Data is always sorted in descending order by count, and then numerically by IP address +or alphabetically as appropriate. +.PP +The following MX errors segment from a sample \fBDetailed\fR report +illustrates the basic hierarchical level structure of \fBpostfix-logwatch\fR: +.RS 4 +.nf + +****** Detailed ******************************************* + + 261 MX errors -------------------------------------- + 261 Unable to look up MX host + 222 Host not found + 73 foolishspammer.local + 60 completely.bogus.domain.example + 11 friend.example.com + 39 No address associated with hostname + 23 dummymx.sample.net + 16 pushn.spam.sample.com + +.fi +.RE 0 +.PP +The \fBpostfix-logwatch\fR utility reads from STDIN or from the named Postfix +\fIlogfile\fR. +Multiple \fIlogfile\fR arguments may be specified, each processed +in order. +The user running \fBpostfix-logwatch\fR must have read permission on +each named log file. +.PP +.SS Options +The options listed below affect the operation of \fBpostfix-logwatch\fR. +Options specified later on the command line override earlier ones. +Any option may be abbreviated to an unambiguous length. + +.IP "\fB-f \fIconfig_file\fR" +.PD 0 +.IP "\fB--config_file \fIconfig_file\fR" +.PD +Use an alternate configuration file \fIconfig_file\fR instead of +the default. +This option may be used more than once. +Multiple configuration files will be processed in the order presented on the command line. +See \fBCONFIGURATION FILE\fR below. +.IP "\fB--debug \fIkeywords\fR" +Output debug information during the operation of \fBpostfix-logwatch\fR. +The parameter \fIkeywords\fR is one or more comma or space separated keywords. +To obtain the list of valid keywords, use --debug xxx where xxx is any invalid keyword. +.IP "\fB--[no]delays\fR" +Enables (disables) output of the message delays percentiles report. +The delays percentiles report shows percentiles for each of the 4 delivery latency times reported +by Postfix (available in version 2.3 and later) in the form \fBdelays=\fIa\fR/\fIb\fR/\fIc\fR/\fId\fR, where +\fIa\fR is the amount of time before the active queue (includes time for previous delivery attempts and time in the deferred queue), +\fIb\fR is the amount of time in the active queue up to delivery agent handoff, +\fIc\fR is the amount of time spent making connections (including DNS, HELO and TLS) and +\fId\fR is the amount of time spent delivering the message. +The total delay shown comes from the \fBdelay=\fR field in a message delivery log line. + +\fBNote:\fR This report may consume a large amount of memory; if you have no use for it, disable the delays report. + +.IP "\fB--delays_percentiles \fIp1 [p2 ...]\fR" +Specifies the percentiles to be used in the message delays percentiles report. +The percentiles \fIp1\fR, \fIp2\fR, \fI...\fR range from 0 to 100, inclusively. +The order of the list is not sorted - the report will output the percentiles +columns in the order you specify. +.IP "\fB--detail \fIlevel\fR" +Sets the maximum detail level for \fBpostfix-logwatch\fR to \fIlevel\fR. +This option is global, overriding any other output limiters described below. + +The \fBpostfix-logwatch\fR utility +produces a \fBSummary\fR section, a \fBDetailed\fR section, and +additional report sections. +With \fIlevel\fR less than 5, \fBpostfix-logwatch\fR will produce +only the \fBSummary\fR section. +At \fIlevel\fR 5 and above, the \fBDetailed\fR section, and any +additional report sections are candidates for output. +Each incremental increase in \fIlevel\fR generates one additional +hierarchical sub-level of output in the \fBDetailed\fR section of the report. +At \fIlevel\fR 10, all levels are output. +Lines that exceed the maximum report width (specified with +\fBmax_report_width\fR) will be cut. +Setting \fIlevel\fR to 11 will prevent lines in the report from being cut (see also \fB--line_style\fR). +.IP "\fB--help\fR" +Print usage information and a brief description about command line options. +.IP "\fB--ignore_service \fIpattern\fR" +Ignore log lines that contain the postfix service name \fBpostfix/\fIservice\fR. +The parameter \fIservice\fR is a regular expression. + +\fBNote:\fR if you use parenthesis in your regular expression, be sure they are cloistering +and not capturing: use \fB(?:\fIpattern\fB)\fR instead of \fB(\fIpattern\fB)\fR. +.IP "\fB--ipaddr_width \fIwidth\fR" +Specifies that IP addresses in address/hostname pairs should be printed +with a field width of \fIwidth\fR characters. +Increasing the default may be useful for systems using long IPv6 addresses. +.IP "\fB-l limiter=levelspec\fR" +.PD 0 +.IP "\fB--limit limiter=levelspec\fR" +.PD +Sets the level limiter \fIlimiter\fR with the specification \fIlevelspec\fR. +.IP "\fB--line_style \fIstyle\fR" +Specifies how to handle long report lines. +Three styles are available: \fBfull\fR, \fBtruncate\fR, and \fBwrap\fR. +Setting \fIstyle\fR to \fBfull\fR will prevent cutting lines to \fBmax_report_width\fR; +this is what occurs when \fBdetail\fR is 11 or higher. +When \fIstyle\fR is \fBtruncate\fR (the default), +long lines will be truncated according to \fBmax_report_width\fR. +Setting \fIstyle\fR to \fBwrap\fR will wrap lines longer than \fBmax_report_width\fR such that +left column hit counts are not obscured. +This option takes precedence over the line style implied by the \fBdetail\fR level. +The options \fB--full\fR, \fB--truncate\fR, and \fB--wrap\fR are synonyms. +.IP "\fB--[no]long_queue_ids\fR" +Enables (disables) interpretation of long queue IDs in Postfix (>= 2.9) logs. +.IP "\fB--nodetail\fR" +Disables the \fBDetailed\fR section of the report, and all supplemental reports. +This option provides a convenient mechanism to quickly disable all sections +under the \fBDetailed\fR report, where subsequent command line +options may re-enable one or more sections to create specific reports. +.IP "\fB--[no]summary\fR" +.IP "\fB--show_summary\fR" +Enables (disables) displaying of the the \fBSummary\fR section of the report. +The variable Posfix_Show_Summary in used in a configuration file. +.IP "\fB--recipient_delimiter \fIdelimiter\fR" +Split email delivery addresses using the recipient delimiter character \fIdelimiter\fR. +This should generally match +the \fBrecipient_delimiter\fR specified in the Postfix parameter +file \fBmain.cf\fR, or the default value indicated in +\fBpostconf -d recipient_delimiter\fR. +This is very useful for obtaining per-alias statistics +when a recipient delimeter is used for mail delivery. +.IP "\fB--reject_reply_patterns \fIr1 [r2 ...]\fR" +Specifies the list of reject reply patterns used to create reject groups. +Each entry in the list \fIr1 [r2 ...]\fR must be either a three character +regular expression reply code of the form [45][0-9.][0-9.], or the word "Warn". +The "." in the regular expression is a literal dot which matches any reject reply subcode; +this wildcarding allows creation of broad rejects groups. +List order is preserved, in that reject reports will be output in the same order as +the entries in the list. +Specific reject reply codes will take priority over wildcard patterns, regardless of +the list order. + +The default list is "5.. 4.. Warn", which creates three groups of rejects: +permanent rejects, temporary reject failures, and reject warnings (as in warn_if_reject). + +This feature allows, for example, distinguishing 421 transmission +channel closures from 45x errors (eg. 450 mailbox unavailable, 451 +local processing errors, 452 insufficient storage). +Such a grouping would be configured with the list: "421 4.. 5.. Warn". +See RFC 2821 for more information about reply codes. + +See also \fBCONFIGURATION FILE\fR regarding using \fBreject_reply_patterns\fR within a configuration file. +.IP "\fB--[no]sect_vars\fR" +.PD 0 +.IP "\fB--show_sect_vars \fIboolean\fR" +.PD +Enables (disables) supplementing each \fBDetailed\fR section title +with the name of that section's level limiter. +The name displayed is the command line option (or configuration +file variable) used to limit that section's output. +. +With the large number of level limiters available in \fBpostfix-logwatch\fR, +this a convenient mechanism for determining exactly which level limiter +affects a section. +.IP "\fB--syslog_name \fInamepat\fR" +Specifies the syslog service name that \fBpostfix-logwatch\fR uses +to match syslog lines. +Only log lines whose service name matches +the perl regular expression \fInamepat\fR will be used by +\fBpostfix-logwatch\fR; all non-matching lines are silently ignored. +This is useful when a pre-installed Postfix package uses a name +other than the default (\fBpostfix\fR), or when multiple Postfix +instances are in use and per-instance reporting is desired. + +The pattern \fInamepat\fR should match the \fBsyslog_name\fR configuration +parameter specified in the Postfix parameter file \fBmain.cf\fR, the +master control file \fBmaster.cf\fR, or the default value as indicated +by the output of \fBpostconf -d syslog_name\fR. + +\fBNote:\fR if you use parenthesis in your regular expression, be sure they are cloistering +and not capturing: use \fB(?:\fIpattern\fB)\fR instead of \fB(\fIpattern\fB)\fR. +.IP "\fB--[no]unknown\fR" +.PD 0 +.IP "\fB--show_unknown \fIboolean\fR" +.PD +Enables (disables) display of the postfix-generated name of 'unknown' in formated IP/hostname pairs in \fBDetailed\fR reports. +Default: enabled. +.IP "\fB--version\fR" +Print \fBpostfix-logwatch\fR version information. +.SS Level Limiters +.PP +The output of every section in the \fBDetailed\fR report is controlled by a level limiter. +The name of the level limiter variable will be output when the \fBsect_vars\fR option is set. +Level limiters are set either via command line in standalone mode with \fB--limit \fIlimiter\fB=\fIlevelspec\fR option, +or via configuration file variable \fB$postfix_\fIlimiter\fB=\fIlevelspec\fR. +Each limiter requires a \fIlevelspec\fR argument, which is described below in \fBLEVEL CONTROL\fR. + +The list of level limiters is shown below. + +There are several level limiters that control reject sub-sections (eg. \fBrejectbody\fR, \fBrejectsender\fR, etc.). +Because the list of reject variants is not known until runtime after \fBreject_reply_patterns\fR is seen, these reject limiters are shown below generically, +with the prefix \fB###\fR. +To use one of these reject limiters, substitute \fB###\fR with one of the reject reply codes in effect, +replacing each dot with an \fBx\fR character. +For example, using the default \fBreject_reply_patterns\fR list of "5.. 4.. Warn", three \fBrejectbody\fR variants are valid: +\fB--limit 5xxrejectbody\fR, \fB--limit 4xxrejectbody\fR and \fB--limit warnrejectbody\fR. +As a convenience, you may entirely eliminate the \fB###\fR prefix, and instead use the bare \fBreject\fIXXX\fR option, and +all reject level limiter variations will be auto-generated based on the \fBreject_reply_patterns\fR list. +For example, the command line segment: +.nf + + ... --reject_reply_patterns "421 5.." \\ + --limit rejectrbl="1:10:" + +.fi +would automatically become: +.nf + + ... --reject_reply_patterns "421 5.." \\ + --limit 421rejectrbl="1:10:" --limit 5xxrejectrbl="1:10:" + +.fi +See \fBreject_reply_patterns\fR above, and comments in the configuration file \fBpostfix-logwatch.conf\fR. + +.de TQ +. br +. ns +. TP \\$1 +.. + +[ THIS SECTION IS NOT YET COMPLETE ] + +.PD 0 +.IP "\fBAttrError" +Errors obtaining attribute data from service. +.IP "\fBBCCed" +Messages that triggered access, header_checks or body_checks BCC action. (postfix 2.6 experimental branch) +.IP "\fBBounceLocal" +.IP "\fBBounceRemote" +Local and remote bounces. +A bounce is considered a local bounce if the relay was one of none, local, virtual, +avcheck, maildrop or 127.0.0.1. +.IP "\fBByIpRejects" +Regrouping by client host IP address of all 5xx (permanent) reject variants. +.IP "\fBCommunicationError" +Postfix errors talking to one of its services. +.IP "\fBAnvil" +Anvil rate or concurrency limits. +.IP "\fBConnectionInbound" +Connections made to the \fBsmtpd\fR server. +.IP "\fBConnectionLostInbound" +Connections lost to the \fBsmtpd\fR server. +.IP "\fBConnectionLostOutbound" +Connections lost during \fBsmtp\fR communications with remote MTA. +.IP "\fBConnectToFailure" +Failures reported by \fBsmtp\fR when connecting to remote MTA. +.IP "\fBDatabaseGeneration" +Warnings noted when binary database map file requires \fBpostmap\fR update from newer source file. +.IP "\fBDeferrals" +.IP "\fBDeferred" +Message delivery deferrals. +A single \fBdeferred\fR message will have one or more \fBdeferrals\fR many times. +.IP "\fBDeliverable" +Address verification indicates recipient address is deliverable. +.IP "\fBDelivered" +Number of messages handed-off to a delivery agent such as local or virtual. +.IP "\fBDiscarded" +Messages that triggered access, header_checks or body_checks DISCARD action. +.IP "\fBDNSError" +Any one of several errors encounted during DNS lookups. +.IP "\fBEnvelopeSenderDomains" +List of sending domains. (2 levels: envelope sender domain, localpart) +.IP "\fBEnvelopeSenders" +List of envelope senders. (1 level: envelope sender) +.IP "\fBError" +Postfix general \fBerror\fR messages. +.IP "\fBFatalConfigError" +Fatal main.cf or master.cf configuration errors. +.IP "\fBFatalError" +Postfix general \fBfatal\fR messages. +.IP "\fBFiltered" +Messages that triggered access, header_checks or body_checks FILTER action. +.IP "\fBForwarded" +Messages forwarded by MDA for one address class to another (eg. local -> virtual). +.IP "\fBHeloError" +XXXXXXXXXXX +.IP "\fBHold" +Messages that were placed on hold by postsuper, or triggered by access, header_checks or body_checks HOLD action. +.IP "\fBHostnameValidationError" +Invalid hostname detected. +.IP "\fBHostnameVerification" +Lookup of hostname does not map back to the IP of the peer (ie. the remote system connecting to \fBsmtpd\fR). +Also known as forward-confirmed reverse DNS (FCRDNS). +When the reverse name has no DNS entry, the message "host not found, try again" is included; otherwise, it is not +(e.g. when the reverse has some IP address, but not the one Postfix expects). +.IP "\fBIllegalAddrSyntax" +Illegal syntax in an email address provided during the MAIL FROM or RCPT TO dialog. +.IP "\fBLdapError" +Any LDAP errors during LDAP lookup. +.IP "\fBMailerLoop" +An MX lookup for the best mailer to use to deliver mail would result in a sending to ourselves. +.IP "\fBMapProblem" +Problem with an access table map that needs correcting. +.IP "\fBMessageWriteError" +Postfix encountered an error when trying to create a message file somewhere in the spool directory. +.IP "\fBNumericHostname" +A hostname was found that was numeric, instead of alphabetic. +.IP "\fBPanicError" +Postfix general \fBpanic\fR messages. +.IP "\fBPixWorkaround" +Workarounds were enabled to avoid remote Cisco PIX SMTP "fixups". +.IP "\fBPolicydWeight" +Summarization of policyweight/policydweight results. +.IP "\fBPolicySpf" +Summarization of PolicySPF results. +.IP "\fBPostgrey" +Summarization of Postgrey results. +.IP "\fBPostscreen" +Summarization of 2.7's postscreen and verify services. +.IP "\fBDNSBLog" +Summarization of 2.7's dnsblog service. +.IP "\fBPrepended" +Messages that triggered header_checks or body_checks PREPEND action. +.IP "\fBProcessExit" +Postfix services that exited unexpectedly. +.IP "\fBProcessLimit" +A Postfix service has reached or exceeded the maximum number of processes allowed. +.IP "\fBQueueWriteError" +Problems writing a Postfix queue file. +.IP "\fBRblError" +Lookup errors for RBLs. +.IP "\fBRedirected" +Messages that triggered access, header_checks or body_checks REDIRECT action. +.IP "\fB###RejectBody" +Messages that triggered body_checks REJECT action. +.IP "\fB###RejectClient" +Messages rejected by client access controls (smtpd_client_restrictions). +.IP "\fB###RejectConfigError" +Message rejected due to server configuration errors. +.IP "\fB###RejectContent" +Messages rejected by message_reject_characters. +.IP "\fB###RejectData" +Messages rejected at DATA stage in SMTP conversation (smtpd_data_restrictions). +.IP "\fB###RejectEtrn" +Messages rejected at ETRN stage in SMTP conversation (smtpd_etrn_restrictions). +.IP "\fB###RejectHeader" +Messages that triggered header_checks REJECT action. +.IP "\fB###RejectHelo" +Messages rejected at HELO/EHLO stage in SMTP conversation (smtpd_helo_restrictions). +.IP "\fB###RejectInsufficientSpace" +Messages rejected due to insufficient storage space. +.IP "\fB###RejectLookupFailure" +Messages rejected due to temporary DNS lookup failures. +.IP "\fB###RejectMilter" +Milter rejects. No reject reply code is available for these rejects, but an extended 5.7.1 DSN is provided. +These rejects are forced into the generic 5xx rejects group. +If you redefine \fBreject_reply_patterns\fR such that it does not contain the pattern \fB5..\fR, milter rejects +will not be output. +.IP "\fB###RejectRbl" +Messages rejected by an RBL hit. +.IP "\fB###RejectRecip" +Messages rejected by recipient access controls (smtpd_recipient_restrictions). +.IP "\fB###RejectRelay" +Messages rejected by relay access controls. +.IP "\fB###RejectSender" +Messages rejected by sender access controls (smtpd_sender_restrictions). +.IP "\fB###RejectSize" +Messages rejected due to excessive message size. +.IP "\fB###RejectUnknownClient" +Messages rejected by unknown client access controls. +.IP "\fB###RejectUnknownReverseClient" +Messages rejected by unknown reverse client access controls. +.IP "\fB###RejectUnknownUser" +Messages rejected by unknown user access controls. +.IP "\fB###RejectUnverifiedClient" +Messages rejected by unverified client access controls. +.IP "\fB###RejectVerify" +Messages rejected dueo to address verification failures. +.IP "\fBReplaced" +Messages that triggered header_checks or body_checks REPLACE action. +.IP "\fBReturnedToSender" +Messages returned to sender due to exceeding queue lifetime (maximal_queue_lifetime). +.IP "\fBSaslAuth" +SASL authentication successes, includes SASL method, username, and sender when present. +.IP "\fBSaslAuthFail" +SASL authentication failures. +.IP "\fBSent" +Messages sent via the SMTP delivery agent. +.IP "\fBSentLmtp" +Messages sent via the LMTP delivery agent. +.IP "\fBSmtpConversationError" +Errors during the SMTP/ESMTP dialog. +.IP "\fBSmtpProtocolViolation" +Protocol violation during the SMTP/ESMTP dialog. +.IP "\fBStartupError" +Errors during Postfix server startup. +.IP "\fBTimeoutInbound" +Connections to \fBsmtpd\fR that timed out. +.IP "\fBTlsClientConnect" +TLS client connections. +.IP "\fBTlsOffered" +TLS communication offerred. +.IP "\fBTlsServerConnect" +TLS server connections. +.IP "\fBTlsUnverified" +Unverified TLS connections. +.IP "\fBUndeliverable" +Address verification indicates recipient address is undeliverable. +.IP "\fBWarn" +Messages that triggered access, header_checks or body_checks WARN action. +.IP "\fBWarnConfigError" +Warnings regarding Postfix configuration errors. +.IP "\fBWarningsOther" +Postfix general \fBwarning\fR messages. + +.PD +.SH LEVEL CONTROL +.ad +.fi +The \fBDetailed\fR section of the report consists of a number of sub-sections, +each of which is controlled both globally and independently. +Two settings influence the output provided in the \fBDetailed\fR report: +a global detail level (specified with \fB--detail\fR) which has final (big hammer) +output-limiting control over the \fBDetailed\fR section, +and sub-section specific detail settings (small hammer), which allow further limiting +of the output for a sub-section. +Each sub-section may be limited to a specific depth level, and each sub-level may be limited with top N or threshold limits. +The \fIlevelspec\fR argument to each of the level limiters listed above is used to accomplish this. + +It is probably best to continue explanation of sub-level limiting with the following well-known outline-style hierarchy, and +some basic examples: +.nf + + level 0 + level 1 + level 2 + level 3 + level 4 + level 4 + level 2 + level 3 + level 4 + level 4 + level 4 + level 3 + level 4 + level 3 + level 1 + level 2 + level 3 + level 4 +.fi +.PP +The simplest form of output limiting suppresses all output below a specified level. +For example, a \fIlevelspec\fR set to "2" shows only data in levels 0 through 2. +Think of this as collapsing each sub-level 2 item, thus hiding all inferior levels (3, 4, ...), +to yield: +.nf + + level 0 + level 1 + level 2 + level 2 + level 1 + level 2 +.fi +.PP +Sometimes the volume of output in a section is too great, and it is useful to suppress any data that does not exceed a certain threshold value. +Consider a dictionary spam attack, which produces very lengthy lists of hit-once recipient email or IP addresses. +Each sub-level in the hierarchy can be threshold-limited by setting the \fIlevelspec\fR appropriately. +Setting \fIlevelspec\fR to the value "2::5" will suppress any data at level 2 that does not exceed a hit count of 5. +.PP +Perhaps producing a top N list, such as top 10 senders, is desired. +A \fIlevelspec\fR of "3:10:" limits level 3 data to only the top 10 hits. +.PP +With those simple examples out of the way, a \fIlevelspec\fR is defined as a whitespace- or comma-separated list of one or more of the following: +.IP "\fIl\fR" +Specifies the maximum level to be output for this sub-section, with a range from 0 to 10. +if \fIl\fR is 0, no levels will be output, effectively disabling the sub-section +(level 0 data is already provided in the Summary report, so level 1 is considered the first useful level in the \fBDetailed\fR report). +Higher values will produce output up to and including the specified level. +.IP "\fIl\fB.\fIn\fR" +Same as above, with the addition that \fIn\fR limits this section's level 1 output to +the top \fIn\fR items. +The value for \fIn\fR can be any integer greater than 1. +(This form of limiting has less utility than the syntax shown below. It is provided for +backwards compatibility; users are encouraged to use the syntax below). +.IP "\fIl\fB:\fIn\fB:\fIt\fR" +This triplet specifies level \fIl\fR, top \fIn\fR, and minimum threshold \fIt\fR. +Each of the values are integers, with \fIl\fR being the level limiter as described above, \fIn\fR being +a top \fIn\fR limiter for the level \fIl\fR, and \fIt\fR being the threshold limiter for level \fIl\fR. +When both \fIn\fR and \fIt\fR are specified, \fIn\fR has priority, allowing top \fIn\fR lists (regardless of +threshold value). +If the value of \fIl\fR is omitted, the specified values for \fIn\fR and/or \fIt\fR are used for +all levels available in the sub-section. +This permits a simple form of wildcarding (eg. place minimum threshold limits on all levels). +However, specific limiters always override wildcard limiters. +The first form of level limiter may be included in \fIlevelspec\fR to restrict output, regardless of how many triplets are present. +.PP +All three forms of limiters are effective only when \fBpostfix-logwatch\fR's detail level is 5 +or greater (the \fBDetailed\fR section is not activated until detail is at least 5). +.PP +See the \fBEXAMPLES\fR section for usage scenarios. +.SH CONFIGURATION FILE +.ad +\fBPostfix-logwatch\fR can read configuration settings from a configuration file. +Essentially, any command line option can be placed into a configuration file, and +these settings are read upon startup. + +Because \fBpostfix-logwatch\fR can run either standalone or within Logwatch, +to minimize confusion, \fBpostfix-logwatch\fR inherits Logwatch's configuration +file syntax requirements and conventions. +These are: +.IP \(bu 4'. +White space lines are ignored. +.IP \(bu 4'. +Lines beginning with \fB#\fR are ignored +.IP \(bu 4'. +Settings are of the form: +.nf + + \fIoption\fB = \fIvalue\fR + +.fi +.IP \(bu 4'. +Spaces or tabs on either side of the \fB=\fR character are ignored. +.IP \(bu 4'. +Any \fIvalue\fR protected in double quotes will be case-preserved. +.IP \(bu 4'. +All other content is reduced to lowercase (non-preserving, case insensitive). +.IP \(bu 4'. +All \fBpostfix-logwatch\fR configuration settings must be prefixed with "\fB$postfix_\fR" or +\fBpostfix-logwatch\fR will ignore them. +.IP \(bu 4'. +When running under Logwatch, any values not prefixed with "\fB$postfix_\fR" are +consumed by Logwatch; it only passes to \fBpostfix-logwatch\fR (via environment variable) +settings it considers valid. +.IP \(bu 4'. +The values \fBTrue\fR and \fBYes\fR are converted to 1, and \fBFalse\fR and \fBNo\fR are converted to 0. +.IP \(bu 4'. +Order of settings is not preserved within a configuration file (since settings are passed +by Logwatch via environment variables, which have no defined order). +.PP +To include a command line option in a configuration file, +prefix the command line option name with the word "\fB$postfix_\fR". +The following configuration file setting and command line option are equivalent: +.nf + + \fB$postfix_Line_Style = Truncate\fR + + \fB--line_style Truncate\fR + +.fi +Level limiters are also prefixed with \fB$postfix_\fR, but on the command line are specified with the \fB--limit\fR option: +.nf + + \fB$postfix_Sent = 2\fR + + \fB--limit Sent=2\fR + +.fi + + +The order of command line options and configuration file processing occurs as follows: +1) The default configuration file is read if it exists and no \fB--config_file\fR was specified on a command line. +2) Configuration files are read and processed in the order found on the command line. +3) Command line options override any options already set either via command line or from any configuration file. + +Command line options are interpreted when they are seen on the command line, and later options will override previously set options. +The notable exception is with limiter variables, which are interpreted in the order found, but only after all other options have been processed. +This allows \fB--reject_reply_patterns\fR to determine the dynamic list of the various reject limiters. + +See also \fB--reject_reply_patterns\fR. +.SH "EXIT STATUS" +.na +.nf +.ad +.fi +The \fBpostfix-logwatch\fR utility exits with a status code of 0, unless an error +occurred, in which case a non-zero exit status is returned. +.SH "EXAMPLES" +.na +.nf +.ad +.fi +.SS Running Standalone +\fBNote:\fR \fBpostfix-logwatch\fR reads its log data from one or more named Postfix log files, or from STDIN. +For brevity, where required, the examples below use the word \fIfile\fR as the command line +argument meaning \fI/path/to/postfix.log\fR. +Obviously you will need to substitute \fIfile\fR with the appropriate path. +.nf +.PP +To run \fBpostfix-logwatch\fR in standalone mode, simply run: +.nf +.RS 4 +.PP +\fBpostfix-logwatch \fIfile\fR +.RE 0 +.nf +.PP +A complete list of options and basic usage is available via: +.nf +.RS 4 +.PP +\fBpostfix-logwatch --help\fR +.RE 0 +.nf +.PP +To print a summary only report of Postfix log data: +.nf +.RS 4 +.PP +\fBpostfix-logwatch --detail 1 \fIfile\fR +.RE 0 +.fi +.PP +To produce a summary report and a one-level detail report for May 25th: +.nf +.RS 4 +.PP +\fBgrep 'May 25' \fIfile\fB | postfix-logwatch --detail 5\fR +.RE 0 +.fi +.PP +To produce only a top 10 list of Sent email domains, the summary report and detailed reports +are first disabled. +Since commands line options are read and enabled left-to-right, +the Sent section is re-enabled to level 1 with a level 1 top 10 limiter: +.nf +.RS 4 +.PP +\fBpostfix-logwatch --nosummary --nodetail --limit sent='1 1:10:' \fIfile\fR +.RE 0 +.fi +.PP +The following command and its sample output shows a more complex level limiter example. +The command gives the top 3 Sent email addresses from the top 5 domains, +in addition, all level 3 items with a hit count of 2 or less are suppressed (in the Sent sub-section, +this happens to be email's Original To address). +Ellipses indicate top N or threshold-limited data: +.nf +.RS 4 +.PP +\fBpostfix-logwatch --nosummary --nodetail \\ + --limit sent '1:5: 2:3: 3::2' \fIfile\fR +.nf + +1762 Sent via SMTP ----------------------------------- + 352 example.com + 310 joe + 255 joe.bob@virtdomain.example.com + 7 info@virtdomain.example.com + 21 pooryoda3 + 11 hot93uh + ... + 244 sample.net + 97 buzz + 26 leroyjones + 14 sally + ... + 152 example.net + 40 jim_jameson + 23 sam_sampson + 19 paul_paulson + ... + 83 sample.us + 44 root + 39 jenny1 + 69 dom3.example.us + 10 kay + 7 ron + 6 mrsmith + ... + ... +.fi +.RE 0 +.fi +.PP +The next command uses both \fBreject_reply_patterns\fR and level limiters to see 421 RBL rejects, +threshold-limiting level 2 output to hits greater than 5 (level 2 in the Reject RBL sub-section +is the client's IP address / hostname pair). +This makes for a very nice RBL offenders list, shown in the sample output +(note the use of the unambiguous, abbreviated command line option reject_reply_pat): +.nf +.RS 4 +.PP +\fBpostfix-logwatch --reject_reply_pat '421 4.. 5.. Warn' \\ + --nosummary --nodetail --limit 421rejectrbl='2 2::5' \fIfile\fR +.nf + +300 421 Reject RBL --------------------------------------- +243 zen.spamhaus.org=127.0.0.2 +106 10.0.0.129 129.0.0.example.com + 41 192.168.10.70 hostx10.sample.net + 40 192.168.42.39 hostz42.sample.net + 15 10.1.1.152 dsl-10-1-1-152.example.us + 14 10.10.10.122 mail122.sample.com + 7 192.168.3.44 smalltime-spammer.example.com + ... + 48 zen.spamhaus.org=127.0.0.4 + 17 10.29.124.92 10-29-124-92.adsl-static.sample.us + ... + 8 zen.spamhaus.org=127.0.0.11 + ... + 1 zen.spamhaus.org=127.0.0.10 + ... +.fi +.RE 4 +.SS Running within Logwatch +\fBNote:\fR Logwatch versions prior to 7.3.6, unless configured otherwise, required the \fB--print\fR option to print to STDOUT instead of sending reports via email. +Since version 7.3.6, STDOUT is the default output destination, and the \fB--print\fR option has been replaced +by \fB--output stdout\fR. Check your configuration to determine where report output will be directed, and add the appropriate option to the commands below. +.PP +To print a summary report for today's Postfix log data: +.nf +.RS 4 +.PP +\fBlogwatch --service postfix --range today --detail 1\fR +.RE 0 +.nf +.PP +To print a report for today's Postfix log data, with one level +of detail in the \fBDetailed\fR section: +.nf +.RS 4 +.PP +\fBlogwatch --service postfix --range today --detail 5\fR +.RE 0 +.fi +.PP +To print a report for yesterday, with two levels of detail in the \fBDetailed\fR section: +.nf +.RS 4 +.PP +\fBlogwatch --service postfix --range yesterday --detail 6\fR +.RE 0 +.fi +.PP +To print a report from Dec 12th through Dec 14th, with four levels of detail in the \fBDetailed\fR section: +.nf +.RS 4 +.PP +\fBlogwatch --service postfix --range \\ + 'between 12/12 and 12/14' --detail 8\fR +.RE 0 +.PP +To print a report for today, with all levels of detail: +.nf +.RS 4 +.PP +\fBlogwatch --service postfix --range today --detail 10\fR +.RE 0 +.PP +Same as above, but leaves long lines uncut: +.nf +.RS 4 +.PP +\fBlogwatch --service postfix --range today --detail 11\fR +.RE 0 + +.SH "ENVIRONMENT" +.na +.nf +.ad +.fi +The \fBpostfix-logwatch\fR program uses the following (automatically set) environment +variables when running under Logwatch: +.IP \fBLOGWATCH_DETAIL_LEVEL\fR +This is the detail level specified with the Logwatch command line argument \fB--detail\fR +or the \fBDetail\fR setting in the ...conf/services/postfix.conf configuration file. +.IP \fBLOGWATCH_DEBUG\fR +This is the debug level specified with the Logwatch command line argument \fB--debug\fR. +.IP \fBpostfix_\fIxxx\fR +The Logwatch program passes all settings \fBpostfix_\fIxxx\fR in the configuration file ...conf/services/postfix.conf +to the \fBpostfix\fR filter (which is actually named .../scripts/services/postfix) via environment variable. +.SH "FILES" +.na +.nf +.SS Standalone mode +.IP "/usr/local/bin/postfix-logwatch" +The \fBpostfix-logwatch\fR program +.IP "/usr/local/etc/postfix-logwatch.conf" +The \fBpostfix-logwatch\fR configuration file in standalone mode +.SS Logwatch mode +.IP "/etc/logwatch/scripts/services/postfix" +The Logwatch \fBpostfix\fR filter +.IP "/etc/logwatch/conf/services/postfix.conf" +The Logwatch \fBpostfix\fR filter configuration file +.SH "SEE ALSO" +.na +.nf +logwatch(8), system log analyzer and reporter +.SH "README FILES" +.na +.ad +.nf +README, an overview of \fBpostfix-logwatch\fR +Changes, the version change list history +Bugs, a list of the current bugs or other inadequacies +Makefile, the rudimentary installer +LICENSE, the usage and redistribution licensing terms +.SH "LICENSE" +.na +.nf +.ad +Covered under the included MIT/X-Consortium License: +http://www.opensource.org/licenses/mit-license.php +.SH "AUTHOR(S)" +.na +.nf +Mike Cappella + +.fi +The original \fBpostfix\fR Logwatch filter was written by +Kenneth Porter, and has had many contributors over the years. +They are entirely not responsible for any errors, problems or failures since the current author's +hands have touched the source code. diff --git a/postfix-logwatch.1.html b/postfix-logwatch.1.html new file mode 100644 index 0000000..7a45ab4 --- /dev/null +++ b/postfix-logwatch.1.html @@ -0,0 +1,882 @@ + + + + Man page: postfix-logwatch(1) + 
+POSTFIX-LOGWATCH(1)         General Commands Manual        POSTFIX-LOGWATCH(1)
+
+
+
+NAME
+       postfix-logwatch - A Postfix log parser and analysis utility
+
+SYNOPSIS
+       postfix-logwatch [options] [logfile ...]
+
+DESCRIPTION
+       The  postfix-logwatch(1)  utility is a Postfix MTA log parser that pro-
+       duces summaries, details, and statistics  regarding  the  operation  of
+       Postfix.
+
+       This utility can be used as a standalone program, or as a Logwatch fil-
+       ter module to produce Postfix summary and detailed reports from  within
+       Logwatch.
+
+       Postfix-logwatch  is  able to produce a wide range of reports with data
+       grouped and sorted as much as possible to reduce  noise  and  highlight
+       patterns.   Brief  summary  reports provide a quick overview of general
+       Postfix operations and message delivery, calling out warnings that  may
+       require  attention.   Detailed reports provide easy to scan, hierarchi-
+       cally-arranged and organized information, with as much or little detail
+       as desired.
+
+       Postfix-logwatch  outputs two principal sections: a Summary section and
+       a Detailed section.  For readability and quick scanning, all  event  or
+       hit  counts appear in the left column, followed by brief description of
+       the event type, and finally additional statistics or count  representa-
+       tions may appear in the rightmost column.
+
+       The following segment from a sample Summary report illustrates:
+
+           ****** Summary ********************************************
+
+                 81   *Warning: Connection rate limit reached (anvil)
+                146   Warned
+
+             68.310M  Bytes accepted                        71,628,177
+             97.645M  Bytes delivered                      102,388,245
+           ========   ================================================
+
+               3464   Accepted                                  41.44%
+               4895   Rejected                                  58.56%
+           --------   ------------------------------------------------
+               8359   Total                                    100.00%
+           ========   ================================================
+
+       The report warns that anvil's connection rate was hit 81 times, a Post-
+       fix access check WARN action was logged  146  times,  and  a  total  of
+       68.310 megabytes (71,628,177 bytes) were accepted into the Postfix sys-
+       tem, delivering 97.645 megabytes of data (due to multiple  recipients).
+       The Accepted and Rejected lines show that Postfix accepted 3464 (41.44%
+       of the total messages) and rejected 4895 (the remaining 58.56%) of  the
+       8359 total messages (temporary rejects show up elsewhere).
+
+       There are dozens of sub-sections available in the Detailed report, each
+       of whose output can be controlled in various  ways.   Each  sub-section
+       attempts to group and present the most meaningful data at superior lev-
+       els, while pushing less useful or noisy data towards  inferior  levels.
+       The  goal is to provide as much benefit as possible from smart grouping
+       of data, to allow faster report scanning, pattern  identification,  and
+       problem  solving.   Data is always sorted in descending order by count,
+       and then numerically by IP address or alphabetically as appropriate.
+
+       The following MX errors segment from a sample  Detailed  report  illus-
+       trates the basic hierarchical level structure of postfix-logwatch:
+
+           ****** Detailed *******************************************
+
+                261   MX errors --------------------------------------
+                261      Unable to look up MX host
+                222         Host not found
+                 73            foolishspammer.local
+                 60            completely.bogus.domain.example
+                 11            friend.example.com
+                 39         No address associated with hostname
+                 23            dummymx.sample.net
+                 16            pushn.spam.sample.com
+
+
+       The postfix-logwatch utility reads from STDIN or from the named Postfix
+       logfile.  Multiple logfile arguments may be specified,  each  processed
+       in  order.  The user running postfix-logwatch must have read permission
+       on each named log file.
+
+   Options
+       The options listed below  affect  the  operation  of  postfix-logwatch.
+       Options specified later on the command line override earlier ones.  Any
+       option may be abbreviated to an unambiguous length.
+
+
+       -f config_file
+       --config_file config_file
+              Use an alternate configuration file config_file instead  of  the
+              default.  This option may be used more than once.  Multiple con-
+              figuration files will be processed in the order presented on the
+              command line.  See CONFIGURATION FILE below.
+
+       --debug keywords
+              Output  debug  information  during the operation of postfix-log-
+              watch.  The parameter keywords is one or  more  comma  or  space
+              separated  keywords.   To obtain the list of valid keywords, use
+              --debug xxx where xxx is any invalid keyword.
+
+       --[no]delays
+              Enables (disables) output  of  the  message  delays  percentiles
+              report.   The  delays  percentiles  report shows percentiles for
+              each of the 4 delivery latency times reported by Postfix (avail-
+              able in version 2.3 and later) in the form delays=a/b/c/d, where
+              a is the amount of time before the active queue  (includes  time
+              for  previous delivery attempts and time in the deferred queue),
+              b is the amount of time in the active queue up to delivery agent
+              handoff,  c  is  the  amount  of  time  spent making connections
+              (including DNS, HELO and TLS) and d is the amount of time  spent
+              delivering  the  message.   The total delay shown comes from the
+              delay= field in a message delivery log line.
+
+              Note: This report may consume a large amount of memory;  if  you
+              have no use for it, disable the delays report.
+
+
+       --delays_percentiles p1 [p2 ...]
+              Specifies  the percentiles to be used in the message delays per-
+              centiles report.  The percentiles p1, p2, ... range  from  0  to
+              100,  inclusively.   The  order  of the list is not sorted - the
+              report will output the percentiles  columns  in  the  order  you
+              specify.
+
+       --detail level
+              Sets  the  maximum  detail  level for postfix-logwatch to level.
+              This option is global,  overriding  any  other  output  limiters
+              described below.
+
+              The  postfix-logwatch  utility  produces  a  Summary  section, a
+              Detailed section, and additional report  sections.   With  level
+              less than 5, postfix-logwatch will produce only the Summary sec-
+              tion.  At level 5 and above, the Detailed section, and any addi-
+              tional  report  sections are candidates for output.  Each incre-
+              mental increase in level generates one  additional  hierarchical
+              sub-level  of  output in the Detailed section of the report.  At
+              level 10, all levels are output.  Lines that exceed the  maximum
+              report  width  (specified  with  max_report_width)  will be cut.
+              Setting level to 11 will prevent lines in the report from  being
+              cut (see also --line_style).
+
+       --help Print  usage  information  and a brief description about command
+              line options.
+
+       --ignore_service pattern
+              Ignore log lines that contain the  postfix  service  name  post-
+              fix/service.  The parameter service is a regular expression.
+
+              Note: if you use parenthesis in your regular expression, be sure
+              they are cloistering and not capturing: use  (?:pattern) instead
+              of (pattern).
+
+       --ipaddr_width width
+              Specifies  that IP addresses in address/hostname pairs should be
+              printed with a field width of width characters.  Increasing  the
+              default may be useful for systems using long IPv6 addresses.
+
+       -l limiter=levelspec
+       --limit limiter=levelspec
+              Sets the level limiter limiter with the specification levelspec.
+
+       --line_style style
+              Specifies  how  to  handle  long report lines.  Three styles are
+              available: full, truncate, and wrap.  Setting style to full will
+              prevent  cutting  lines to max_report_width; this is what occurs
+              when detail is 11  or  higher.   When  style  is  truncate  (the
+              default),   long   lines   will   be   truncated   according  to
+              max_report_width.  Setting style to wrap will wrap lines  longer
+              than  max_report_width  such that left column hit counts are not
+              obscured.  This option takes  precedence  over  the  line  style
+              implied  by  the  detail level.  The options --full, --truncate,
+              and --wrap are synonyms.
+
+       --[no]long_queue_ids
+              Enables (disables) interpretation of long queue IDs  in  Postfix
+              (>= 2.9) logs.
+
+       --nodetail
+              Disables  the Detailed section of the report, and all supplemen-
+              tal reports.  This option provides  a  convenient  mechanism  to
+              quickly  disable  all  sections under the Detailed report, where
+              subsequent command line options may re-enable one or  more  sec-
+              tions to create specific reports.
+
+       --[no]summary
+
+       --show_summary
+              Enables  (disables) displaying of the the Summary section of the
+              report.  The variable Posfix_Show_Summary in used in a  configu-
+              ration file.
+
+       --recipient_delimiter delimiter
+              Split  email  delivery  addresses  using the recipient delimiter
+              character delimiter.  This should generally  match  the  recipi-
+              ent_delimiter  specified  in the Postfix parameter file main.cf,
+              or the default value indicated in postconf  -d  recipient_delim-
+              iter.   This  is  very useful for obtaining per-alias statistics
+              when a recipient delimeter is used for mail delivery.
+
+       --reject_reply_patterns r1 [r2 ...]
+              Specifies the list of  reject  reply  patterns  used  to  create
+              reject  groups.   Each  entry  in  the  list r1 [r2 ...] must be
+              either a three character regular expression reply  code  of  the
+              form [45][0-9.][0-9.], or the word "Warn".  The "." in the regu-
+              lar expression is a literal dot which matches any  reject  reply
+              subcode;  this  wildcarding  allows  creation  of  broad rejects
+              groups.  List order is preserved, in that reject reports will be
+              output  in  the same order as the entries in the list.  Specific
+              reject reply codes will take priority  over  wildcard  patterns,
+              regardless of the list order.
+
+              The  default  list is "5.. 4.. Warn", which creates three groups
+              of rejects: permanent rejects, temporary  reject  failures,  and
+              reject warnings (as in warn_if_reject).
+
+              This  feature  allows, for example, distinguishing 421 transmis-
+              sion channel closures from 45x errors (eg. 450 mailbox  unavail-
+              able,  451  local  processing errors, 452 insufficient storage).
+              Such a grouping would be configured with the list: "421 4..  5..
+              Warn".  See RFC 2821 for more information about reply codes.
+
+              See  also  CONFIGURATION  FILE regarding using reject_reply_pat-
+              terns within a configuration file.
+
+       --[no]sect_vars
+       --show_sect_vars boolean
+              Enables (disables) supplementing  each  Detailed  section  title
+              with  the  name  of that section's level limiter.  The name dis-
+              played is the command line option (or configuration  file  vari-
+              able)  used to limit that section's output.  With the large num-
+              ber of level limiters available in postfix-logwatch, this a con-
+              venient  mechanism  for  determining exactly which level limiter
+              affects a section.
+
+       --syslog_name namepat
+              Specifies the syslog service name that postfix-logwatch uses  to
+              match  syslog  lines.  Only log lines whose service name matches
+              the perl regular expression namepat will be used by postfix-log-
+              watch;  all  non-matching  lines  are silently ignored.  This is
+              useful when a pre-installed Postfix package uses  a  name  other
+              than  the  default (postfix), or when multiple Postfix instances
+              are in use and per-instance reporting is desired.
+
+              The pattern namepat should match the  syslog_name  configuration
+              parameter  specified  in the Postfix parameter file main.cf, the
+              master control file master.cf, or the default value as indicated
+              by the output of postconf -d syslog_name.
+
+              Note: if you use parenthesis in your regular expression, be sure
+              they are cloistering and not capturing: use  (?:pattern) instead
+              of (pattern).
+
+       --[no]unknown
+       --show_unknown boolean
+              Enables  (disables)  display  of  the  postfix-generated name of
+              'unknown' in formated IP/hostname  pairs  in  Detailed  reports.
+              Default: enabled.
+
+       --version
+              Print postfix-logwatch version information.
+
+   Level Limiters
+       The  output  of every section in the Detailed report is controlled by a
+       level limiter.  The name of the level limiter variable will  be  output
+       when  the  sect_vars  option is set.  Level limiters are set either via
+       command line in standalone mode with --limit limiter=levelspec  option,
+       or  via  configuration  file variable $postfix_limiter=levelspec.  Each
+       limiter requires a levelspec argument,  which  is  described  below  in
+       LEVEL CONTROL.
+
+       The list of level limiters is shown below.
+
+       There  are several level limiters that control reject sub-sections (eg.
+       rejectbody, rejectsender, etc.).  Because the list of  reject  variants
+       is  not  known until runtime after reject_reply_patterns is seen, these
+       reject limiters are shown below generically, with the prefix  ###.   To
+       use one of these reject limiters, substitute ### with one of the reject
+       reply codes in effect, replacing each dot with  an  x  character.   For
+       example,  using  the  default  reject_reply_patterns  list  of "5.. 4..
+       Warn", three rejectbody  variants  are  valid:  --limit  5xxrejectbody,
+       --limit  4xxrejectbody  and  --limit warnrejectbody.  As a convenience,
+       you may entirely eliminate the ### prefix, and  instead  use  the  bare
+       rejectXXX option, and all reject level limiter variations will be auto-
+       generated based on the reject_reply_patterns list.   For  example,  the
+       command line segment:
+
+           ... --reject_reply_patterns "421 5.." \
+                   --limit rejectrbl="1:10:"
+
+       would automatically become:
+
+           ... --reject_reply_patterns "421 5.." \
+                   --limit 421rejectrbl="1:10:" --limit 5xxrejectrbl="1:10:"
+
+       See reject_reply_patterns above, and comments in the configuration file
+       postfix-logwatch.conf.
+
+
+       [ THIS SECTION IS NOT YET COMPLETE ]
+
+       AttrError
+              Errors obtaining attribute data from service.
+       BCCed  Messages that triggered access, header_checks or body_checks BCC
+              action. (postfix 2.6 experimental branch)
+       BounceLocal
+       BounceRemote
+              Local and remote bounces.  A bounce is considered a local bounce
+              if the relay was one of none, local, virtual, avcheck,  maildrop
+              or 127.0.0.1.
+       ByIpRejects
+              Regrouping  by  client  host  IP  address of all 5xx (permanent)
+              reject variants.
+       CommunicationError
+              Postfix errors talking to one of its services.
+       Anvil  Anvil rate or concurrency limits.
+       ConnectionInbound
+              Connections made to the smtpd server.
+       ConnectionLostInbound
+              Connections lost to the smtpd server.
+       ConnectionLostOutbound
+              Connections lost during smtp communications with remote MTA.
+       ConnectToFailure
+              Failures reported by smtp when connecting to remote MTA.
+       DatabaseGeneration
+              Warnings noted when binary database map  file  requires  postmap
+              update from newer source file.
+       Deferrals
+       Deferred
+              Message delivery deferrals.  A single deferred message will have
+              one or more deferrals many times.
+       Deliverable
+              Address verification indicates recipient address is deliverable.
+       Delivered
+              Number of messages handed-off to a delivery agent such as  local
+              or virtual.
+       Discarded
+              Messages  that  triggered  access,  header_checks or body_checks
+              DISCARD action.
+       DNSError
+              Any one of several errors encounted during DNS lookups.
+       EnvelopeSenderDomains
+              List of sending domains.  (2  levels:  envelope  sender  domain,
+              localpart)
+       EnvelopeSenders
+              List of envelope senders.  (1 level: envelope sender)
+       Error  Postfix general error messages.
+       FatalConfigError
+              Fatal main.cf or master.cf configuration errors.
+       FatalError
+              Postfix general fatal messages.
+       Filtered
+              Messages  that  triggered  access,  header_checks or body_checks
+              FILTER action.
+       Forwarded
+              Messages forwarded by MDA for one address class to another  (eg.
+              local -> virtual).
+       HeloError
+              XXXXXXXXXXX
+       Hold   Messages  that were placed on hold by postsuper, or triggered by
+              access, header_checks or body_checks HOLD action.
+       HostnameValidationError
+              Invalid hostname detected.
+       HostnameVerification
+              Lookup of hostname does not map back to the IP of the peer  (ie.
+              the  remote system connecting to smtpd).  Also known as forward-
+              confirmed reverse DNS (FCRDNS).  When the reverse  name  has  no
+              DNS  entry, the message "host not found, try again" is included;
+              otherwise, it is not (e.g. when the reverse has some IP address,
+              but not the one Postfix expects).
+       IllegalAddrSyntax
+              Illegal syntax in an email address provided during the MAIL FROM
+              or RCPT TO dialog.
+       LdapError
+              Any LDAP errors during LDAP lookup.
+       MailerLoop
+              An MX lookup for the best mailer to use to  deliver  mail  would
+              result in a sending to ourselves.
+       MapProblem
+              Problem with an access table map that needs correcting.
+       MessageWriteError
+              Postfix  encountered  an  error  when trying to create a message
+              file somewhere in the spool directory.
+       NumericHostname
+              A hostname was found that was numeric, instead of alphabetic.
+       PanicError
+              Postfix general panic messages.
+       PixWorkaround
+              Workarounds were enabled to avoid remote Cisco  PIX  SMTP  "fix-
+              ups".
+       PolicydWeight
+              Summarization of policyweight/policydweight results.
+       PolicySpf
+              Summarization of PolicySPF results.
+       Postgrey
+              Summarization of Postgrey results.
+       Postscreen
+              Summarization of 2.7's postscreen and verify services.
+       DNSBLog
+              Summarization of 2.7's dnsblog service.
+       Prepended
+              Messages  that  triggered  header_checks  or body_checks PREPEND
+              action.
+       ProcessExit
+              Postfix services that exited unexpectedly.
+       ProcessLimit
+              A Postfix service has reached or exceeded the maximum number  of
+              processes allowed.
+       QueueWriteError
+              Problems writing a Postfix queue file.
+       RblError
+              Lookup errors for RBLs.
+       Redirected
+              Messages that triggered access, header_checks or body_checks RE-
+              DIRECT action.
+       ###RejectBody
+              Messages that triggered body_checks REJECT action.
+       ###RejectClient
+              Messages     rejected     by     client     access      controls
+              (smtpd_client_restrictions).
+       ###RejectConfigError
+              Message rejected due to server configuration errors.
+       ###RejectContent
+              Messages rejected by message_reject_characters.
+       ###RejectData
+              Messages   rejected   at   DATA   stage   in  SMTP  conversation
+              (smtpd_data_restrictions).
+       ###RejectEtrn
+              Messages  rejected  at   ETRN   stage   in   SMTP   conversation
+              (smtpd_etrn_restrictions).
+       ###RejectHeader
+              Messages that triggered header_checks REJECT action.
+       ###RejectHelo
+              Messages  rejected  at  HELO/EHLO  stage  in  SMTP  conversation
+              (smtpd_helo_restrictions).
+       ###RejectInsufficientSpace
+              Messages rejected due to insufficient storage space.
+       ###RejectLookupFailure
+              Messages rejected due to temporary DNS lookup failures.
+       ###RejectMilter
+              Milter rejects.  No reject reply code  is  available  for  these
+              rejects,  but  an extended 5.7.1 DSN is provided.  These rejects
+              are forced into the generic 5xx rejects group.  If you  redefine
+              reject_reply_patterns  such that it does not contain the pattern
+              5.., milter rejects will not be output.
+       ###RejectRbl
+              Messages rejected by an RBL hit.
+       ###RejectRecip
+              Messages rejected by recipient  access  controls  (smtpd_recipi-
+              ent_restrictions).
+       ###RejectRelay
+              Messages rejected by relay access controls.
+       ###RejectSender
+              Messages      rejected     by     sender     access     controls
+              (smtpd_sender_restrictions).
+       ###RejectSize
+              Messages rejected due to excessive message size.
+       ###RejectUnknownClient
+              Messages rejected by unknown client access controls.
+       ###RejectUnknownReverseClient
+              Messages rejected by unknown reverse client access controls.
+       ###RejectUnknownUser
+              Messages rejected by unknown user access controls.
+       ###RejectUnverifiedClient
+              Messages rejected by unverified client access controls.
+       ###RejectVerify
+              Messages rejected dueo to address verification failures.
+       Replaced
+              Messages that triggered  header_checks  or  body_checks  REPLACE
+              action.
+       ReturnedToSender
+              Messages  returned  to  sender  due  to exceeding queue lifetime
+              (maximal_queue_lifetime).
+       SaslAuth
+              SASL authentication successes, includes SASL  method,  username,
+              and sender when present.
+       SaslAuthFail
+              SASL authentication failures.
+       Sent   Messages sent via the SMTP delivery agent.
+       SentLmtp
+              Messages sent via the LMTP delivery agent.
+       SmtpConversationError
+              Errors during the SMTP/ESMTP dialog.
+       SmtpProtocolViolation
+              Protocol violation during the SMTP/ESMTP dialog.
+       StartupError
+              Errors during Postfix server startup.
+       TimeoutInbound
+              Connections to smtpd that timed out.
+       TlsClientConnect
+              TLS client connections.
+       TlsOffered
+              TLS communication offerred.
+       TlsServerConnect
+              TLS server connections.
+       TlsUnverified
+              Unverified TLS connections.
+       Undeliverable
+              Address  verification  indicates recipient address is undeliver-
+              able.
+       Warn   Messages that triggered  access,  header_checks  or  body_checks
+              WARN action.
+       WarnConfigError
+              Warnings regarding Postfix configuration errors.
+       WarningsOther
+              Postfix general warning messages.
+
+
+LEVEL CONTROL
+       The  Detailed  section  of  the report consists of a number of sub-sec-
+       tions, each of which is controlled  both  globally  and  independently.
+       Two  settings  influence  the output provided in the Detailed report: a
+       global detail level (specified with --detail) which has final (big ham-
+       mer) output-limiting control over the Detailed section, and sub-section
+       specific detail settings (small hammer), which allow  further  limiting
+       of  the output for a sub-section.  Each sub-section may be limited to a
+       specific depth level, and each sub-level may be limited with top  N  or
+       threshold limits.  The levelspec argument to each of the level limiters
+       listed above is used to accomplish this.
+
+       It is probably best to continue explanation of sub-level limiting  with
+       the  following well-known outline-style hierarchy, and some basic exam-
+       ples:
+
+           level 0
+              level 1
+                 level 2
+                    level 3
+                       level 4
+                       level 4
+                 level 2
+                    level 3
+                       level 4
+                       level 4
+                       level 4
+                    level 3
+                       level 4
+                    level 3
+              level 1
+                 level 2
+                    level 3
+                       level 4
+
+       The simplest form of output limiting  suppresses  all  output  below  a
+       specified  level.   For example, a levelspec set to "2" shows only data
+       in levels 0 through 2.  Think of this as collapsing  each  sub-level  2
+       item, thus hiding all inferior levels (3, 4, ...), to yield:
+
+           level 0
+              level 1
+                 level 2
+                 level 2
+              level 1
+                 level 2
+
+       Sometimes  the  volume  of  output in a section is too great, and it is
+       useful to suppress any data that does not exceed  a  certain  threshold
+       value.   Consider a dictionary spam attack, which produces very lengthy
+       lists of hit-once recipient email or IP addresses.  Each  sub-level  in
+       the  hierarchy can be threshold-limited by setting the levelspec appro-
+       priately.  Setting levelspec to the value "2::5" will suppress any data
+       at level 2 that does not exceed a hit count of 5.
+
+       Perhaps  producing a top N list, such as top 10 senders, is desired.  A
+       levelspec of "3:10:" limits level 3 data to only the top 10 hits.
+
+       With those simple examples out of the way, a levelspec is defined as  a
+       whitespace- or comma-separated list of one or more of the following:
+
+       l      Specifies  the  maximum level to be output for this sub-section,
+              with a range from 0 to 10.  if l is 0, no levels will be output,
+              effectively  disabling  the sub-section (level 0 data is already
+              provided in the Summary report, so level  1  is  considered  the
+              first  useful level in the Detailed report).  Higher values will
+              produce output up to and including the specified level.
+
+       l.n    Same as above, with the addition that n  limits  this  section's
+              level  1  output to the top n items.  The value for n can be any
+              integer greater than 1.  (This form of limiting has less utility
+              than  the  syntax shown below. It is provided for backwards com-
+              patibility; users are encouraged to use the syntax below).
+
+       l:n:t  This triplet specifies level l, top n, and minimum threshold  t.
+              Each  of the values are integers, with l being the level limiter
+              as described above, n being a top n limiter for the level l, and
+              t  being  the  threshold limiter for level l.  When both n and t
+              are specified, n has priority, allowing top n lists  (regardless
+              of  threshold  value).  If the value of l is omitted, the speci-
+              fied values for n and/or t are used for all levels available  in
+              the sub-section.  This permits a simple form of wildcarding (eg.
+              place minimum threshold limits on all  levels).   However,  spe-
+              cific  limiters  always  override  wildcard limiters.  The first
+              form of level limiter may be included in levelspec  to  restrict
+              output, regardless of how many triplets are present.
+
+       All  three forms of limiters are effective only when postfix-logwatch's
+       detail level is 5 or greater (the Detailed  section  is  not  activated
+       until detail is at least 5).
+
+       See the EXAMPLES section for usage scenarios.
+
+CONFIGURATION FILE
+       Postfix-logwatch  can  read configuration settings from a configuration
+       file.  Essentially, any command line option can be placed into  a  con-
+       figuration file, and these settings are read upon startup.
+
+       Because  postfix-logwatch can run either standalone or within Logwatch,
+       to minimize confusion, postfix-logwatch inherits Logwatch's  configura-
+       tion file syntax requirements and conventions.  These are:
+
+       o   White space lines are ignored.
+
+       o   Lines beginning with # are ignored
+
+       o   Settings are of the form:
+
+                   option = value
+
+
+       o   Spaces or tabs on either side of the = character are ignored.
+
+       o   Any value protected in double quotes will be case-preserved.
+
+       o   All  other  content  is  reduced to lowercase (non-preserving, case
+           insensitive).
+
+       o   All postfix-logwatch configuration settings must be  prefixed  with
+           "$postfix_" or postfix-logwatch will ignore them.
+
+       o   When  running  under Logwatch, any values not prefixed with "$post-
+           fix_" are consumed by Logwatch; it only passes to  postfix-logwatch
+           (via environment variable) settings it considers valid.
+
+       o   The  values  True  and Yes are converted to 1, and False and No are
+           converted to 0.
+
+       o   Order of settings is not  preserved  within  a  configuration  file
+           (since  settings  are passed by Logwatch via environment variables,
+           which have no defined order).
+
+       To include a command line option in a configuration  file,  prefix  the
+       command line option name with the word "$postfix_".  The following con-
+       figuration file setting and command line option are equivalent:
+
+               $postfix_Line_Style = Truncate
+
+               --line_style Truncate
+
+       Level limiters are also prefixed with $postfix_,  but  on  the  command
+       line are specified with the --limit option:
+
+               $postfix_Sent = 2
+
+               --limit Sent=2
+
+
+
+       The  order  of  command  line options and configuration file processing
+       occurs as follows: 1) The default configuration  file  is  read  if  it
+       exists  and  no --config_file was specified on a command line.  2) Con-
+       figuration files are read and processed in the order found on the  com-
+       mand  line.   3)  Command line options override any options already set
+       either via command line or from any configuration file.
+
+       Command line options are interpreted when they are seen on the  command
+       line,  and  later  options  will  override previously set options.  The
+       notable exception is with limiter variables, which are  interpreted  in
+       the  order found, but only after all other options have been processed.
+       This allows --reject_reply_patterns to determine the  dynamic  list  of
+       the various reject limiters.
+
+       See also --reject_reply_patterns.
+
+EXIT STATUS
+       The  postfix-logwatch  utility exits with a status code of 0, unless an
+       error occurred, in which case a non-zero exit status is returned.
+
+EXAMPLES
+   Running Standalone
+       Note: postfix-logwatch reads its log data from one or more named  Post-
+       fix  log  files, or from STDIN.  For brevity, where required, the exam-
+       ples below use the word file  as  the  command  line  argument  meaning
+       /path/to/postfix.log.   Obviously you will need to substitute file with
+       the appropriate path.
+
+       To run postfix-logwatch in standalone mode, simply run:
+
+           postfix-logwatch file
+
+       A complete list of options and basic usage is available via:
+
+           postfix-logwatch --help
+
+       To print a summary only report of Postfix log data:
+
+           postfix-logwatch --detail 1 file
+
+       To produce a summary report and a one-level detail report for May 25th:
+
+           grep 'May 25' file | postfix-logwatch --detail 5
+
+       To produce only a top 10 list of Sent email domains, the summary report
+       and  detailed  reports are first disabled.  Since commands line options
+       are read and enabled left-to-right, the Sent section is  re-enabled  to
+       level 1 with a level 1 top 10 limiter:
+
+           postfix-logwatch --nosummary --nodetail --limit sent='1 1:10:' file
+
+       The  following command and its sample output shows a more complex level
+       limiter example.  The command gives the top 3 Sent email addresses from
+       the top 5 domains, in addition, all level 3 items with a hit count of 2
+       or less are suppressed (in the Sent sub-section,  this  happens  to  be
+       email's  Original  To  address).  Ellipses indicate top N or threshold-
+       limited data:
+
+           postfix-logwatch --nosummary --nodetail \
+                   --limit sent '1:5: 2:3: 3::2' file
+
+           1762   Sent via SMTP -----------------------------------
+            352      example.com
+            310         joe
+            255            joe.bob@virtdomain.example.com
+              7            info@virtdomain.example.com
+             21         pooryoda3
+             11         hot93uh
+                        ...
+            244      sample.net
+             97         buzz
+             26         leroyjones
+             14         sally
+                        ...
+            152      example.net
+             40         jim_jameson
+             23         sam_sampson
+             19         paul_paulson
+                        ...
+             83      sample.us
+             44         root
+             39         jenny1
+             69      dom3.example.us
+             10         kay
+              7         ron
+              6         mrsmith
+                        ...
+                     ...
+
+       The next command uses both reject_reply_patterns and level limiters  to
+       see  421 RBL rejects, threshold-limiting level 2 output to hits greater
+       than 5 (level 2 in the  Reject  RBL  sub-section  is  the  client's  IP
+       address  /  hostname  pair).   This makes for a very nice RBL offenders
+       list, shown in the sample output (note  the  use  of  the  unambiguous,
+       abbreviated command line option reject_reply_pat):
+
+           postfix-logwatch --reject_reply_pat '421 4.. 5.. Warn' \
+                   --nosummary --nodetail --limit 421rejectrbl='2 2::5' file
+
+           300   421 Reject RBL ---------------------------------------
+           243      zen.spamhaus.org=127.0.0.2
+           106         10.0.0.129       129.0.0.example.com
+            41         192.168.10.70    hostx10.sample.net
+            40         192.168.42.39    hostz42.sample.net
+            15         10.1.1.152       dsl-10-1-1-152.example.us
+            14         10.10.10.122     mail122.sample.com
+             7         192.168.3.44     smalltime-spammer.example.com
+                       ...
+            48      zen.spamhaus.org=127.0.0.4
+            17         10.29.124.92     10-29-124-92.adsl-static.sample.us
+                       ...
+             8      zen.spamhaus.org=127.0.0.11
+                       ...
+             1      zen.spamhaus.org=127.0.0.10
+                       ...
+
+   Running within Logwatch
+       Note:  Logwatch  versions  prior to 7.3.6, unless configured otherwise,
+       required the --print option to  print  to  STDOUT  instead  of  sending
+       reports  via  email.  Since version 7.3.6, STDOUT is the default output
+       destination, and the --print option has been replaced by --output  std-
+       out.  Check your configuration to determine where report output will be
+       directed, and add the appropriate option to the commands below.
+
+       To print a summary report for today's Postfix log data:
+
+           logwatch --service postfix --range today --detail 1
+
+       To print a report for today's Postfix log data, with one level
+       of detail in the Detailed section:
+
+           logwatch --service postfix --range today --detail 5
+
+       To print a report for yesterday, with  two  levels  of  detail  in  the
+       Detailed section:
+
+           logwatch --service postfix --range yesterday --detail 6
+
+       To  print  a report from Dec 12th through Dec 14th, with four levels of
+       detail in the Detailed section:
+
+           logwatch --service postfix --range \
+                   'between 12/12 and 12/14' --detail 8
+
+       To print a report for today, with all levels of detail:
+
+           logwatch --service postfix --range today --detail 10
+
+       Same as above, but leaves long lines uncut:
+
+           logwatch --service postfix --range today --detail 11
+
+
+ENVIRONMENT
+       The postfix-logwatch program uses  the  following  (automatically  set)
+       environment variables when running under Logwatch:
+
+       LOGWATCH_DETAIL_LEVEL
+              This  is  the  detail  level specified with the Logwatch command
+              line argument --detail or the Detail setting in the ...conf/ser-
+              vices/postfix.conf configuration file.
+
+       LOGWATCH_DEBUG
+              This is the debug level specified with the Logwatch command line
+              argument --debug.
+
+       postfix_xxx
+              The Logwatch program passes all settings postfix_xxx in the con-
+              figuration  file  ...conf/services/postfix.conf  to  the postfix
+              filter (which is  actually  named  .../scripts/services/postfix)
+              via environment variable.
+
+FILES
+   Standalone mode
+       /usr/local/bin/postfix-logwatch
+              The postfix-logwatch program
+
+       /usr/local/etc/postfix-logwatch.conf
+              The postfix-logwatch configuration file in standalone mode
+
+   Logwatch mode
+       /etc/logwatch/scripts/services/postfix
+              The Logwatch postfix filter
+
+       /etc/logwatch/conf/services/postfix.conf
+              The Logwatch postfix filter configuration file
+
+SEE ALSO
+       logwatch(8), system log analyzer and reporter
+
+README FILES
+       README, an overview of postfix-logwatch
+       Changes, the version change list history
+       Bugs, a list of the current bugs or other inadequacies
+       Makefile, the rudimentary installer
+       LICENSE, the usage and redistribution licensing terms
+
+LICENSE
+       Covered under the included MIT/X-Consortium License:
+       http://www.opensource.org/licenses/mit-license.php
+
+AUTHOR(S)
+       Mike Cappella
+
+       The original postfix Logwatch filter was written by Kenneth Porter, and
+       has had many contributors over the years.  They are entirely not
+       responsible for any errors, problems or failures since the current
+       author's hands have touched the source code.
+
+
+
+                                                           POSTFIX-LOGWATCH(1)
+
diff --git a/postfix-logwatch.conf b/postfix-logwatch.conf new file mode 100644 index 0000000..5b14639 --- /dev/null +++ b/postfix-logwatch.conf @@ -0,0 +1,331 @@ +# +# postfix.conf / postfix-logwatch.conf +# +# This is the postfix-logwatch configuration file. + +# Lines in this file are of the format: +# +# VAR = VALUE +# *VAR = VALUE +# $VAR = VALUE +# +# Whitespace surrounding the = assignment character is removed. Variable names +# and values are case insensitive. Double quotes can be used to preserve case and +# whitespace. +# +# Variables beginning with a * are used only by logwatch. +# Variables beginning with a $ are used only by the postfix-logwatch filter. +# Variables beginning with neither * nor $ are used only by logwatch, with the +# exception of the Detail variable which is passed via environment to the +# postfix-logwatch filter. +# +# Any of the equivalent boolean values below may be used where appropriate: +# +# 1, Yes, True, On +# 0, No, False, Off +# +# Lines that begin with a # are comment lines. Blank and whitespace lines +# are ignored. Whitespace at the beginning and end of a line is ignored. +# + +# Specifies the title used in the logwatch report +# +Title = "Postfix" + +# Specifies the logwatch logfile group +# +LogFile = maillog + +# Specifies the global, maximum detail level +# +#Detail = 10 + +# The *OnlyService selector is used solely by logwatch to select log lines +# to pass to the postfix-logwatch filter. And postfix-logwatch uses the +# $postfix_Syslog_Name variable for log line selection. +# +# When used in logwatch, both the *OnlyService and $postfix_Syslog_Name +# variables below should contain essentially the same REs so that lines passed +# by logwatch are also selected by postfix-logwatch. Note that *OnlyService +# also includes the / (eg. postfix/smtpd). +# +# If you change postfix's syslog_name for any postfix service, you will need to +# replace "postfix" below with an appropriate RE to capture the desired log entries. +# Do likewise for *OnlyService above when used under logwatch. For example, the +# settings: +# +# *OnlyService = "postfix\d?/[-a-zA-Z\d]*" +# $postfix_Syslog_Name = "postfix\d?" +# +# will capture postfix/smtpd, postfix2/virtual, ..., postfix9/cleanup +# +# Note: If you use parenthesis in your regular expression, be sure they +# are cloistering and not capturing: use (?:pattern) instead of (pattern). +# +# Performance Note: +# If you do not wish to analyze any or all of postgrey, postfwd, or policyd-spf +# consider simplifying $postfix_Syslog_Name to increase log scanning performance. The +# more complex the RE, the longer the scan time to select/reject a log line. The +# difference in scan times between the simple string 'postfix' and the more complex +# alternation RE that includes postfix, postgrey, postfwd and policyd-spf is about 40%. +# +# Includes: postfix/smtpd, etc, postfix/policy-spf +#*OnlyService = "postfix/[-\w]*" +#$postfix_Syslog_Name = "postfix" +# Includes: postfix/smtpd, etc, postfix/policy-spf, postgrey, postfwd, policyd-spf +*OnlyService = "(?:post(?:fix|grey|fwd)|policyd-spf)(?:/[-\w]*)?" +$postfix_Syslog_Name = "(?:post(?:fix|grey|fwd)|policyd-spf)" + +# Ignored postfix services +# +# Ignores postfix services postfix/SERVICE, where SERVICE is an RE +# pattern. The example below will ignore log lines whose syslog +# name is "postfix/myservice". +#$postfix_Ignore_Service = "myservice" + +# Specifies the maximum report width for Detail <= 10, +# or when postfix_Line_Style is not set to Truncate +# +$postfix_Max_Report_Width = 100 + +# Specifies how to handle line lengths greater than Max_Report_Width. +# Options are Truncate (default), Wrap, or Full. +# for Detail <= 10 +# +$postfix_Line_Style = Truncate + +# Set the variable below to the value set for "recipient_delimiter" +# in your postfix configuration, if you want your recipient email +# addresses split into their user + extension. +# +#$postfix_Recipient_Delimiter = "+" + +# Width of IP addresses for columnar output. Change to 40 for IPv6 addresses +#$postfix_ipaddr_width = 40 +$postfix_ipaddr_width = 15 + +# Switch to use Postfix 2.8 long queue IDs: +# Postfix option: enable_long_queue_ids +$postfix_Enable_Long_Queue_Ids = No + +# Show delays percentiles report. For command line, use --[no]delays, +# without an argument. +# +$postfix_Show_Delays = Yes + +# Show names of detail section variables/command line options in +# detail report titles. For command line, use --[no]sect_vars, +# without an argument. +# +$postfix_Show_Sect_Vars = No + +# Show the postfix-reported hostname of 'unknown' in formatted +# ip/hostname pairs. For command line, use --[no]unknown, +# without an argument. +# +$postfix_Show_Unknown = Yes + +# Show the summary section. For command line, use --[no]summary, +# without an argument. +$postfix_Show_Summary = Yes + +# Specifies the percentiles shown in the delivery delays report +# Valid values are from 0 to 100, inclusive. +$postfix_Delays_Percentiles = "0 25 50 75 90 95 98 100" + +# Specifies the list of reject sections that will be output in +# reports (eg. 5xx permanent or 4xx temporary failures). +# Each entry in the comma or whitespace separated list consists of 3 +# characters, where the first is either 4 or 5, and second and third +# are a digit or a dot "." match-anything character. Also allowed is +# the keyword "Warn" (which is used for postfix "warn_if_reject" rejects). +# In PCRE (perl regular expression) terms, any pattern that matches: +# +# ^([45][0-9.][0-9.]|Warn)$ +# +# is acceptable. +# +# Typical reject codes: +# +# 421 Service not available, closing transmission channel +# 450 Requested mail action not taken: mailbox unavailable +# 451 Requested action aborted: local error in processing +# 452 Requested action not taken: insufficient system storage +# +# 500 Syntax error, command unrecognized +# 501 Syntax error in parameters or arguments +# 502 Command not implemented +# 503 Bad sequence of commands +# 504 Command parameter not implemented +# 550 Requested action not taken: mailbox unavailable +# 551 User not local; please try +# 552 Requested mail action aborted: exceeded storage allocation +# 553 Requested action not taken: mailbox name not allowed +# 554 Transaction failed +# +# Specific codes take priority over wildcard patterns. The default list +# is: "5.. 4.. Warn". +# +# See also the various Reject... level limiters below +# +$postfix_Reject_Reply_Patterns = "5.. 4.. Warn" + +# Level Limiters +# +# The variables below control the maximum output level for a given +# category. A level of 1 indicates only one level of detailed output in +# the Detailed report section. The Summary section is only available +# at logwatch --Detail level >= 5. Increasing the Detail level +# by one adds one level of additional detail in the Summary section. +# +# For example, Detail 5 would output one additional level of detail, +# Detail 6 two levels, etc. all the way up to 10. Finally, Detail +# 11 yields uncropped lines of output. +# +# You can control the maximum number of level 1 lines by appending +# a period and a number. The value 2.10 would indicate 2 levels +# of detail, but only 10 level-1 lines. For example, setting +# $postfix_Sent = 1.20 yields a top 20 list of Messages Sent. +# +# A more useful form of limiting uses triplets in the form l:n:t. +# This triplet specifies level l, top n, and minimum threshold t. +# Each of the values are integers, with l being the level limiter +# as described above, n being a top n limiter for the level l, and +# t being the threshold limiter for level l. When both n and t +# are specified, n has priority, allowing top n lists (regardless +# of threshold value). If the value of l is omitted, the speci- +# fied values for n and/or t are used for all levels available in +# the sub-section. This permits a simple form of wildcarding (eg. +# place minimum threshold limits on all levels). However, spe- +# cific limiters always override wildcard limiters. The first +# form of level limiter may be included in levelspec to restrict +# output, regardless of how many triplets are present. + +$postfix_Sent = 1 +$postfix_SentLmtp = 1 +$postfix_Delivered = 1 +$postfix_Forwarded = 1 +$postfix_ConnectionLostInbound = 1 +$postfix_TimeoutInbound = 1 +$postfix_ConnectToFailure = 2 + +# Disabled by default to reduce noise and consume less memory. +# Enable at will +$postfix_EnvelopeSenders = 0 +$postfix_EnvelopeSenderDomains = 0 +$postfix_ConnectionInbound = 0 +# Reject by IP report +$postfix_ByIpRejects = 0 + +$postfix_PanicError = 10 +$postfix_FatalError = 10 +$postfix_Error = 10 +# warnings +$postfix_Anvil = 3 +$postfix_AttrError = 10 +$postfix_CommunicationError = 10 +$postfix_DatabaseGeneration = 10 +$postfix_DNSError = 10 +$postfix_HeloError = 10 +$postfix_HostnameValidationError = 10 +$postfix_HostnameVerification = 10 +$postfix_IllegalAddrSyntax = 10 +$postfix_LdapError = 10 +$postfix_MailerLoop = 10 +$postfix_MapProblem = 10 +$postfix_MessageWriteError = 10 +$postfix_NumericHostname = 10 +$postfix_ProcessExit = 10 +$postfix_ProcessLimit = 10 +$postfix_QueueWriteError = 10 +$postfix_RBLError = 10 +$postfix_SaslAuthFail = 10 +$postfix_SmtpConversationError = 10 +$postfix_StartupError = 10 +$postfix_WarningsOther = 10 + +# Common access control actions +$postfix_Bcced = 10 +$postfix_Discarded = 10 +$postfix_Filtered = 10 +$postfix_Hold = 10 +$postfix_Prepended = 10 +$postfix_Redirected = 10 +$postfix_Replaced = 10 +$postfix_Warned = 10 +# DUNNO action not logged +# IGNORE action not logged +# REJECT actions are below + +# Rejects +# The following are generic reject types, which are automatically +# expanded into each reject variant, based on the reply patterns +# listed in Reject_Reply_Patterns. By default, each item in the +# list below becomes 4xxReject..., 5xxReject..., and WarnReject... +$postfix_RejectBody = 10 +$postfix_RejectClient = 10 +$postfix_RejectConfigError = 10 +$postfix_RejectContent = 10 +$postfix_RejectData = 10 +$postfix_RejectEtrn = 10 +$postfix_RejectHeader = 10 +$postfix_RejectHelo = 10 +$postfix_RejectInsufficientSpace = 10 +$postfix_RejectLookupFailure = 10 +$postfix_RejectMilter = 10 +$postfix_RejectProxy = 10 +$postfix_RejectRBL = 10 +$postfix_RejectRecip = 10 +$postfix_RejectRelay = 10 +$postfix_RejectSender = 10 +$postfix_RejectSize = 10 +$postfix_RejectUnknownClient = 10 +$postfix_RejectUnknownReverseClient = 10 +$postfix_RejectUnknownUser = 10 +$postfix_RejectUnverifiedClient = 3 +$postfix_RejectVerify = 10 + +# For more precise control, you can comment out any of the reject +# types above and specify each variant manually, but the list must +# be consistent with the values specified in Reject_Reply_Patterns. +# +# For example, you could comment out $postfix_RejectHelo above, and +# instead uncomment the three RejectHelo variants, allowing you to +# specify different level limiters to each variant: +# +# Permanent 5xx variant +# $postfix_5xxRejectHelo = 1 +# Temporary 4xx variant +# $postfix_4xxRejectHelo = 2 +# Warn_if_reject variant +# $postfix_WarnRejectHelo = 2 +# + +$postfix_Deferred = 10 +$postfix_Deferrals = 10 +$postfix_BounceLocal = 10 +$postfix_BounceRemote = 10 + +$postfix_Discarded = 10 +$postfix_ReturnedToSender = 10 +$postfix_NotificationSent = 10 +$postfix_ConnectionLostOutbound = 10 + +$postfix_Deliverable = 10 +$postfix_Undeliverable = 10 +$postfix_PixWorkaround = 10 +$postfix_SaslAuth = 10 +$postfix_TlsServerConnect = 10 +$postfix_TlsClientConnect = 10 +$postfix_TlsUnverified = 10 +$postfix_TlsOffered = 10 +$postfix_SMTPProtocolViolation = 10 + +$postfix_Postscreen = 1 +$postfix_DNSBLog = 1 + +$postfix_PolicySPF = 10 +$postfix_PolicydWeight = 10 +$postfix_Postgrey = 10 + +# vi: shiftwidth=3 tabstop=3 et diff --git a/postfix-logwatch.conf-topn b/postfix-logwatch.conf-topn new file mode 100644 index 0000000..6d6b5ed --- /dev/null +++ b/postfix-logwatch.conf-topn @@ -0,0 +1,331 @@ +# +# postfix.conf / postfix-logwatch.conf +# +# This is the postfix-logwatch configuration file. + +# Lines in this file are of the format: +# +# VAR = VALUE +# *VAR = VALUE +# $VAR = VALUE +# +# Whitespace surrounding the = assignment character is removed. Variable names +# and values are case insensitive. Double quotes can be used to preserve case and +# whitespace. +# +# Variables beginning with a * are used only by logwatch. +# Variables beginning with a $ are used only by the postfix-logwatch filter. +# Variables beginning with neither * nor $ are used only by logwatch, with the +# exception of the Detail variable which is passed via environment to the +# postfix-logwatch filter. +# +# Any of the equivalent boolean values below may be used where appropriate: +# +# 1, Yes, True, On +# 0, No, False, Off +# +# Lines that begin with a # are comment lines. Blank and whitespace lines +# are ignored. Whitespace at the beginning and end of a line is ignored. +# + +# Specifies the title used in the logwatch report +# +Title = "Postfix" + +# Specifies the logwatch logfile group +# +LogFile = maillog + +# Specifies the global, maximum detail level +# +#Detail = 10 + +# The *OnlyService selector is used solely by logwatch to select log lines +# to pass to the postfix-logwatch filter. And postfix-logwatch uses the +# $postfix_Syslog_Name variable for log line selection. +# +# When used in logwatch, both the *OnlyService and $postfix_Syslog_Name +# variables below should contain essentially the same REs so that lines passed +# by logwatch are also selected by postfix-logwatch. Note that *OnlyService +# also includes the / (eg. postfix/smtpd). +# +# If you change postfix's syslog_name for any postfix service, you will need to +# replace "postfix" below with an appropriate RE to capture the desired log entries. +# Do likewise for *OnlyService above when used under logwatch. For example, the +# settings: +# +# *OnlyService = "postfix\d?/[-a-zA-Z\d]*" +# $postfix_Syslog_Name = "postfix\d?" +# +# will capture postfix/smtpd, postfix2/virtual, ..., postfix9/cleanup +# +# Note: If you use parenthesis in your regular expression, be sure they +# are cloistering and not capturing: use (?:pattern) instead of (pattern). +# +# Performance Note: +# If you do not wish to analyze any or all of postgrey, postfwd, or policyd-spf +# consider simplifying $postfix_Syslog_Name to increase log scanning performance. The +# more complex the RE, the longer the scan time to select/reject a log line. The +# difference in scan times between the simple string 'postfix' and the more complex +# alternation RE that includes postfix, postgrey, postfwd and policyd-spf is about 40%. +# +# Includes: postfix/smtpd, etc, postfix/policy-spf +#*OnlyService = "postfix/[-\w]*" +#$postfix_Syslog_Name = "postfix" +# Includes: postfix/smtpd, etc, postfix/policy-spf, postgrey, postfwd, policyd-spf +*OnlyService = "(?:post(?:fix|grey|fwd)|policyd-spf)(?:/[-\w]*)?" +$postfix_Syslog_Name = "(?:post(?:fix|grey|fwd)|policyd-spf)" + +# Ignored postfix services +# +# Ignores postfix services postfix/SERVICE, where SERVICE is an RE +# pattern. The example below will ignore log lines whose syslog +# name is "postfix/myservice". +#$postfix_Ignore_Service = "myservice" + +# Specifies the maximum report width for Detail <= 10, +# or when postfix_Line_Style is not set to Truncate +# +$postfix_Max_Report_Width = 100 + +# Specifies how to handle line lengths greater than Max_Report_Width. +# Options are Truncate (default), Wrap, or Full. +# for Detail <= 10 +# +$postfix_Line_Style = Truncate + +# Set the variable below to the value set for "recipient_delimiter" +# in your postfix configuration, if you want your recipient email +# addresses split into their user + extension. +# +#$postfix_Recipient_Delimiter = "+" + +# Width of IP addresses for columnar output. Change to 40 for IPv6 addresses +#$postfix_ipaddr_width = 40 +$postfix_ipaddr_width = 15 + +# Switch to use Postfix 2.8 long queue IDs: +# Postfix option: enable_long_queue_ids +$postfix_Enable_Long_Queue_Ids = No + +# Show delays percentiles report. For command line, use --[no]delays, +# without an argument. +# +$postfix_Show_Delays = Yes + +# Show names of detail section variables/command line options in +# detail report titles. For command line, use --[no]sect_vars, +# without an argument. +# +$postfix_Show_Sect_Vars = No + +# Show the postfix-reported hostname of 'unknown' in formatted +# ip/hostname pairs. For command line, use --[no]unknown, +# without an argument. +# +$postfix_Show_Unknown = Yes + +# Show the summary section. For command line, use --[no]summary, +# without an argument. +$postfix_Show_Summary = Yes + +# Specifies the percentiles shown in the delivery delays report +# Valid values are from 0 to 100, inclusive. +$postfix_Delays_Percentiles = "0 25 50 75 90 95 98 100" + +# Specifies the list of reject sections that will be output in +# reports (eg. 5xx permanent or 4xx temporary failures). +# Each entry in the comma or whitespace separated list consists of 3 +# characters, where the first is either 4 or 5, and second and third +# are a digit or a dot "." match-anything character. Also allowed is +# the keyword "Warn" (which is used for postfix "warn_if_reject" rejects). +# In PCRE (perl regular expression) terms, any pattern that matches: +# +# ^([45][0-9.][0-9.]|Warn)$ +# +# is acceptable. +# +# Typical reject codes: +# +# 421 Service not available, closing transmission channel +# 450 Requested mail action not taken: mailbox unavailable +# 451 Requested action aborted: local error in processing +# 452 Requested action not taken: insufficient system storage +# +# 500 Syntax error, command unrecognized +# 501 Syntax error in parameters or arguments +# 502 Command not implemented +# 503 Bad sequence of commands +# 504 Command parameter not implemented +# 550 Requested action not taken: mailbox unavailable +# 551 User not local; please try +# 552 Requested mail action aborted: exceeded storage allocation +# 553 Requested action not taken: mailbox name not allowed +# 554 Transaction failed +# +# Specific codes take priority over wildcard patterns. The default list +# is: "5.. 4.. Warn". +# +# See also the various Reject... level limiters below +# +$postfix_Reject_Reply_Patterns = "5.. 4.. Warn" + +# Level Limiters +# +# The variables below control the maximum output level for a given +# category. A level of 1 indicates only one level of detailed output in +# the Detailed report section. The Summary section is only available +# at logwatch --Detail level >= 5. Increasing the Detail level +# by one adds one level of additional detail in the Summary section. +# +# For example, Detail 5 would output one additional level of detail, +# Detail 6 two levels, etc. all the way up to 10. Finally, Detail +# 11 yields uncropped lines of output. +# +# You can control the maximum number of level 1 lines by appending +# a period and a number. The value 2.10 would indicate 2 levels +# of detail, but only 10 level-1 lines. For example, setting +# $postfix_Sent = 1.20 yields a top 20 list of Messages Sent. +# +# A more useful form of limiting uses triplets in the form l:n:t. +# This triplet specifies level l, top n, and minimum threshold t. +# Each of the values are integers, with l being the level limiter +# as described above, n being a top n limiter for the level l, and +# t being the threshold limiter for level l. When both n and t +# are specified, n has priority, allowing top n lists (regardless +# of threshold value). If the value of l is omitted, the speci- +# fied values for n and/or t are used for all levels available in +# the sub-section. This permits a simple form of wildcarding (eg. +# place minimum threshold limits on all levels). However, spe- +# cific limiters always override wildcard limiters. The first +# form of level limiter may be included in levelspec to restrict +# output, regardless of how many triplets are present. + +$postfix_Sent = "1:10:1 2::1" +$postfix_SentLmtp = "1:10:1 2::1" +$postfix_Delivered = "1:10:1" +$postfix_Forwarded = "1 1:10:1" +$postfix_ConnectionLostInbound = 1 +$postfix_TimeoutInbound = "2 :10:1" +$postfix_ConnectToFailure = 2 + +$postfix_EnvelopeSenders = "2 1:10:1" +$postfix_EnvelopeSenderDomains = "1 1:20:1" +$postfix_ConnectionInbound = "1 1:20:1" + +# Reject by IP report +$postfix_ByIpRejects = 0 + +$postfix_PanicError = 10 +$postfix_FatalError = 10 +$postfix_Error = 10 +# warnings +$postfix_Anvil = 2 +$postfix_AttrError = 10 +$postfix_CommunicationError = 10 +$postfix_DatabaseGeneration = 10 +$postfix_DNSError = 3 +$postfix_HeloError = 10 +$postfix_HostnameValidationError = 10 +$postfix_HostnameVerification = "2::1" +$postfix_IllegalAddrSyntax = 10 +$postfix_LdapError = 10 +$postfix_MailerLoop = 10 +$postfix_MapProblem = 10 +$postfix_MessageWriteError = 10 +$postfix_NumericHostname = 10 +$postfix_ProcessExit = 10 +$postfix_ProcessLimit = 10 +$postfix_QueueWriteError = 10 +$postfix_RBLError = 10 +$postfix_SaslAuthFail = 10 +$postfix_SmtpConversationError = 10 +$postfix_StartupError = 10 +$postfix_WarningsOther = 10 + +# Common access control actions +$postfix_Bcced = 10 +$postfix_Discarded = 10 +$postfix_Filtered = 10 +$postfix_Hold = 10 +$postfix_Prepended = 10 +$postfix_Redirected = 10 +$postfix_Replaced = 10 +$postfix_Warned = 10 +# DUNNO action not logged +# IGNORE action not logged +# REJECT actions are below + +# Rejects +# The following are generic reject types, which are automatically +# expanded into each reject variant, based on the reply patterns +# listed in Reject_Reply_Patterns. By default, each item in the +# list below becomes 4xxReject..., 5xxReject..., and WarnReject... +$postfix_RejectBody = "2 :10:1" +$postfix_RejectClient = "2 :10:1" +$postfix_RejectConfigError = "2 :10:1" +$postfix_RejectContent = "2 :10:1" +$postfix_RejectData = "1 :10:1" +$postfix_RejectEtrn = "2 :10:1" +$postfix_RejectHeader = "2 :10:1" +$postfix_RejectHelo = "2 :10:1" +$postfix_RejectInsufficientSpace = "2 :10:1" +$postfix_RejectLookupFailure = "2 :10:1" +$postfix_RejectMilter = "2 :10:1" +$postfix_RejectProxy = "2 :10:1" +$postfix_RejectRBL = "2 :10:1" +$postfix_RejectRecip = "2 :10:1" +$postfix_RejectRelay = "1 :10:1" +$postfix_RejectSender = "2 :10:1" +$postfix_RejectSize = "2 :10:1" +$postfix_RejectUnknownClient = "2 :10:1" +$postfix_RejectUnknownReverseClient = "2 :10:1" +$postfix_RejectUnknownUser = "2 :10:1" +$postfix_RejectUnverifiedClient = "2 :10:1" +$postfix_RejectVerify = "2 :10:1" + +# For more precise control, you can comment out any of the reject +# types above and specify each variant manually, but the list must +# be consistent with the values specified in Reject_Reply_Patterns. +# +# For example, you could comment out $postfix_RejectHelo above, and +# instead uncomment the three RejectHelo variants, allowing you to +# specify different level limiters to each variant: +# +# Permanent 5xx variant +# $postfix_5xxRejectHelo = 1 +# Temporary 4xx variant +# $postfix_4xxRejectHelo = 2 +# Warn_if_reject variant +# $postfix_WarnRejectHelo = 2 +# + +$postfix_Deferred = 10 +$postfix_Deferrals = 10 +$postfix_BounceLocal = 10 +$postfix_BounceRemote = "4 ::1" + +$postfix_Discarded = 10 +$postfix_ReturnedToSender = 10 +$postfix_NotificationSent = 10 +$postfix_ConnectionLostOutbound = 10 + +$postfix_Deliverable = 10 +$postfix_Undeliverable = 10 +$postfix_PixWorkaround = 10 +$postfix_SaslAuth = 10 +$postfix_TlsServerConnect = 10 +$postfix_TlsClientConnect = 10 +$postfix_TlsUnverified = 10 +$postfix_TlsOffered = 10 +$postfix_SmtpProtocolViolation = "2 :10:1" + +$postfix_Postscreen = 1 +$postfix_DNSBLog = 1 + +$postfix_PolicySPF = "3::10 4::1" +$postfix_PolicydWeight = 10 +$postfix_Postgrey = "4 3::1" + + +# vi: shiftwidth=3 tabstop=3 et