]>
Commit | Line | Data |
---|---|---|
ae52237f | 1 | Note for Users Upgrading to SpamAssassin 4.0.0 |
e04a3a9b SI |
2 | ---------------------------------------------- |
3 | ||
ae52237f SI |
4 | Apache SpamAssassin 4.0.0 represents years of work by the project with |
5 | numerous improvements, new rule types, and internal native handling | |
6 | of messages in international languages. We highly recommend looking | |
7 | through this file and all of the .pre files to evaluate your | |
8 | configuration thoroughly. Plugins have been added, removed, and | |
9 | improved throughout. | |
10 | ||
11 | - All rules, functions, command line options and modules that contain | |
12 | "whitelist" or "blacklist" have been renamed to contain more | |
13 | racially neutral "welcomelist" and "blocklist" terms. This allows | |
14 | acronyms like WL and BL to remain the same. Previous options will | |
15 | continue work at least until version 4.1.0 is released. If you have | |
16 | local settings including scores or meta rules referring to old rule | |
17 | names, these should be changed and "enable_compat | |
18 | welcomelist_blocklist" added in init.pre. See: | |
19 | https://wiki.apache.org/spamassassin/WelcomelistBlocklist (Bug 7826) | |
20 | ||
21 | - Meta rules no longer use priority values, they are evaluated | |
22 | dynamically when the rules they depend on are finished. (Bug 7735) | |
23 | ||
24 | - API: New $pms->rule_ready() function. Any asynchronous eval-function | |
25 | must now return undef (instead of 0 or 1), if rule result is not | |
26 | ready when exiting the function. $pms->rule_ready($rulename) or | |
27 | $pms->got_hit(...) must be called when the result has arrived. If | |
28 | these are not used, it can break depending meta rule evaluation. | |
e04a3a9b | 29 | |
ae52237f SI |
30 | - Setting normalize_charset is now enabled by default. Note that rules |
31 | should not expect specific non-UTF8 or UTF8 encoding in | |
32 | body. Matching is done against the raw data which may vary depending | |
33 | on normalize_charset setting and whether decoding to UTF8 was | |
34 | successful. See: | |
35 | https://wiki.apache.org/spamassassin/WritingRulesAdvanced | |
e04a3a9b | 36 | |
ae52237f SI |
37 | - DKIM plugin has added support for ARC signature verification |
38 | ||
39 | - The DecodeShortURL plugin has been added and decodes URIs from URL | |
40 | shorteners that may be used to evade scanning | |
e04a3a9b | 41 | |
ae52237f SI |
42 | - Strings can now be captured from rules and later reused using the |
43 | special %{TAGNAME} syntax | |
44 | ||
45 | - The Bayes stopwords, or noise words, are now configurable in order | |
46 | to optimize Bayes usage for non-English languages. Stopwords for 16 | |
47 | foreign languages have been included. See 60_bayes_stopwords.cf in | |
48 | the rules files. See Mail::SpamAssassin::Plugin::Bayes and the | |
49 | bayes_stopword_languages option if you wish to use a different | |
50 | stopword list. This is highly recommended if you are using Bayes and | |
51 | you are processing messages in languages other than English. | |
52 | ||
53 | - The OLEVBMacro plugin has been improved to identify more macros | |
54 | while also extracting uris from the attachments for automatic | |
55 | inclusion in RBL lookups | |
56 | ||
57 | - Internationalized domain name (IDN) support has been added and | |
58 | requires Net::LibIDN2 or Net::LibIDN module with a new | |
59 | Util::idn_to_ascii() function. (Bug 7215) | |
60 | ||
61 | - Improved internal header address (From/To/Cc) parser, now also | |
62 | handles multiple addresses and includes optional support for | |
63 | external Email::Address::XS parser, which can handle nested comments | |
64 | and other oddities. | |
65 | ||
66 | - Header :addr :name modifiers now return all addresses. Options of | |
67 | :first :last select only first (topmost) or last header to process | |
68 | when there are multiple headers with the same name. :addr and :name | |
69 | may still return multiple values from a single header. | |
70 | ||
71 | - API: $pms->get() can and should now be called in list | |
72 | context. Scalar context continues to return multiple values newline | |
73 | separated, but this should be considered deprecated. | |
74 | ||
75 | - New ExtractText plugin that extracts text from documents or images | |
76 | to feed the data into SpamAssassin for standard processing with | |
77 | existing rules, URIs extracted from documents will fall into normal | |
78 | RBL lookups. | |
79 | ||
80 | - New "nolog" tflag added to hide info coming from rules in | |
81 | SpamAssassin reports | |
82 | ||
83 | - All log output (stderr, file, syslog) is now escaped properly for \r | |
84 | \n \t \\, control chars, DEL, and UTF-8 sequences presented as | |
85 | \x{XX}. Whitespace is not normalized anymore like in versions prior | |
86 | to 4.0.0. | |
87 | ||
88 | - API: Logger::add() has new optional 'escape' parameter. New | |
89 | Logger::escape_str() function. | |
90 | ||
91 | - API: New $pms->add_uri_detail_list() function. Also new | |
92 | uri_detail_list types: unlinked, schemeless | |
93 | ||
94 | - Util::split_domain, trim_domain, and is_domain_valid functions have | |
95 | a new optional argument ($is_ascii) | |
96 | ||
97 | - Header names support new :host :domain :ip :revip modifiers | |
98 | ||
99 | - AskDNS: tag HEADER(hdrname) supported to query any header content | |
100 | similarly to header rules | |
101 | ||
102 | - The HashCash module and support has been removed completely, as it | |
103 | has been long since deprecated | |
104 | ||
105 | - URILocalBL: uri_block_cc/uri_block_cont now support negation (Bug | |
106 | 7528) | |
107 | ||
108 | - URILocalBL: IPv6 lookups for hosts is now support, if provided by | |
109 | your database | |
110 | ||
111 | - DNS and other asynchronous lookups such as Pyzor and DCC are now | |
112 | only launched when priority -100 is reached. This allows short | |
113 | circuiting at a lower priority without sending unneeded DNS queries | |
114 | and starting process forms. (Bug 5930) | |
115 | ||
116 | - API: New plugin method callback method check_dnsbl added to launch | |
117 | network lookups at priority -100 and check_post_dnsbl to harvest own | |
118 | network lookups | |
119 | ||
120 | - API: New plugin callback method check_cleanup for cleaning up | |
121 | things... | |
122 | ||
123 | - FreeMail: new options freemail_import_welcomelist_auth and | |
124 | freemail_import_def_welcomelist_auth added (Bug 6451) | |
125 | ||
126 | - New internal Mail::SpamAssassin::GeoDB module that provides a | |
127 | unified interface to modules MaxMind::DB::Reader (GeoIP2), Geo::IP, | |
128 | IP::Country::DB_File, and IP::Country::Fast. | |
129 | ||
130 | This is utilized by RelayCountry and URILocalBL with settings | |
131 | geodb_module, geodb_options, and geodb_search_path. | |
132 | ||
133 | Deprecated settings still work such as country_db_type, | |
134 | country_db_path, uri_country_db_path, and uri_country_db_isp_path | |
135 | but will print a warning to migrate to geodb_module/options. | |
136 | ||
137 | - Razor2 razor_fork option added to create separate Razor2 processes | |
138 | and read in the results later asynchronously, increasing throughput, | |
139 | and automatically adjusting rule priorities to -100. | |
140 | ||
141 | - DCC checks are now done asynchronously if using dccifd, improving | |
142 | throughput. With dccifd, rule priorities are automatically adjusted | |
143 | to -100. Commercial reputation rules can be ignored with the option | |
144 | "use_dcc_rep 0" to save a few CPU cycles. | |
145 | ||
146 | - Pyzor pyzor_fork option added to create separate Pyzor processes and | |
147 | read in the results later asynchronously, increasing throughput, and | |
148 | automatically adjusting rule priorities to -100. Renamed pyzor_max | |
149 | setting to pyzor_count_min. Added pyzor_welcomelist_min and | |
150 | pyzor_welcomelist_factor setting. Also try to improve false | |
151 | positives by ignoring "empty body" messages. | |
152 | ||
153 | - API: deprecated $pms->register_async_rule_start() and | |
154 | $pms->register_async_rule_finish() calls though left in for | |
155 | backwards compatibility. Plugins should only use | |
156 | $pms->bgsend_and_start_lookup(), which handles required things | |
157 | Automatically. Direct calls to bgsend or start_lookup should not be | |
158 | used. $pms->bgsend_and_start_lookup() should always contain | |
159 | $ent->{rulename} for correct meta dependency handling. Deprecated | |
160 | start_lookup, get_lookup, lookup_ns, harvest_until_rule_completes, | |
161 | and is_rule_complete. | |
e04a3a9b | 162 | |
ae52237f SI |
163 | - SPF: Mail::SPF is now the only supported perl module and |
164 | Mail::SPF::Query is deprecated along with the settings | |
165 | do_not_use_mail_spf, and do_not_use_mail_spf_query. SPF lookups are | |
166 | not done asynchronously so using an MTA filter such as pypolicyd-spf | |
167 | or spf-engine can generate Received-SPF for SpamAssassin to parse. | |
e04a3a9b | 168 | |
ae52237f SI |
169 | - "ALL" pseudo-header now returns decoded headers, so it's usage is |
170 | consistent with single header matching. Using the :raw option mimics | |
171 | the previous behavior of with undecoded and folded headers. | |
37ef5775 | 172 | |
ae52237f | 173 | - New dns_block_rule option handles blocked DNSBLs (Bug 6728) |
37ef5775 | 174 | |
ae52237f SI |
175 | - ASN: Support GeoDB for ASN lookups (asn_use_geodb, asn_prefer_geodb, |
176 | asn_use_dns). | |
37ef5775 | 177 | |
ae52237f SI |
178 | - ASN: Default sa-update ruleset doesn't make ASN lookups or add |
179 | headers anymore. Configure desired methods, asn_use_geodb or | |
180 | asn_use_dns, and add_header clauses manually as described in the | |
181 | plugin documentation. Usage of asn_use_geodb without DNS is | |
182 | recommended unless ASNCIDR is needed. Do not use rules that check | |
183 | metadata X-ASN header! Only the new eval function check_asn() | |
184 | described in plugin manual works reliably. | |
37ef5775 | 185 | |
ae52237f SI |
186 | - sa-update: New --score-multiplier, --score-limit, and --forcemirror |
187 | options added. | |
188 | #1 forcemirror: forces sa-update to use a specific mirror server, | |
189 | #2 score-multiplier: adjust all scores from update channel by a | |
190 | given multiplier to quickly level set scores to match your | |
191 | preferred threshold | |
192 | #3 score-limit adjusts all scores from update channel over a | |
193 | specified limit to a new limit | |
37ef5775 | 194 | |
ae52237f SI |
195 | - New dns_options "nov4" and "nov6" added. IMPORTANT:; You must set |
196 | nov6 if your DNS resolver is filtering IPv6 AAAA replies. | |
37ef5775 | 197 | |
ae52237f SI |
198 | - API: Added Message::get_pristine_body_digest(), |
199 | Message::get_msgid(), and Message::generate_msgid() | |
200 | functions. Removed deprecated private Plugin::Bayes::get_msgid() | |
201 | function. | |
202 | ||
203 | - Bayes and TxRep seen Message-ID tracking hashing method changed. No | |
204 | actions are required. If re-learning some old messages, they might | |
205 | be learned twice but old IDs should expire automatically. | |
206 | ||
207 | - report_charset defaults now to UTF-8. | |
208 | ||
209 | - Meta rules inherit net tflag setting from dependencies (Bug 7735) | |
210 | ||
211 | - BodyEval: Added plaintext_body_sig_ratio eval rules for the first | |
212 | text/plain MIME part's body and signature length ratio. | |
213 | ||
214 | - API: Now supports multiple calls of $pms->test_log() for | |
215 | rules. Added $pms->check_cleanup() to finalize tags, reports, | |
216 | etc. Deprecated internal $pms->{test_log_msgs}, renamed to | |
217 | $pms->{test_logs}. Deprecated $pms->clear_test_state() as it is not | |
218 | needed anymore. $pms->test_log() now accepts $rulename as second | |
219 | argument. | |
220 | ||
221 | - URIDNSBL: urirhsbl/urirhssub rules support "notrim" tflag to force | |
222 | querying the full hostname instead of just the domain. This works | |
223 | best if the specific uribl supports this mode. (Bug 7835) | |
224 | ||
225 | - Removed deprecated --auth-ident and --ident-timeout options from | |
226 | spamd | |
227 | ||
228 | - MIMEHeader: support matching ALL header, tflags range, and tflags | |
229 | concat | |
230 | ||
231 | - Autolearn: add new tflags autolearn_header/autolearn_body. These can | |
232 | force a rule to count as header or body points accordingly. (Bug | |
233 | 7907) | |
234 | ||
235 | - SSL client certificate support for spamc/spamd is now easier. New | |
236 | spamc options --ssl-cert, --ssl-key, --ssl-ca-file, and | |
237 | --ssl-ca-path. New spamd options --ssl-verify, --ssl-ca-file, and | |
238 | --ssl-ca-path (Bug 7267) | |
239 | ||
240 | - ArchiveIterator now automatically uncompressed all gzip, bzip2, xz, | |
241 | lz4, lzip, and lzo-compressed files (Bug 7598). These apply to | |
242 | spamassassin and sa-learn commands also. | |
243 | ||
244 | - New DMARC policy check plugin. | |
245 | ||
246 | - New project maintained DecodeShortURLs plugin which may not be | |
247 | directly compatible with rules from other third party plugins. See | |
248 | The plugin documentation for configuration and rule format. | |
249 | ||
250 | - Installing module Net::CIDR::Lite allows the use of dash-separated | |
251 | IP range format (e.g. 192.168.1.1-192.168.255.255) for NetSet tables | |
252 | including internal_networks, trusted_networks, msa_networks, and | |
253 | uri_local_cidr. | |
254 | ||
255 | - The HashBL plugin in 342.pre is now enabled by default. | |
37ef5775 | 256 | |
ae52237f | 257 | - HeaderEval check_for_unique_subject_id() function is deprecated. |
37ef5775 SI |
258 | |
259 | (end of UPGRADE) |