]>
Commit | Line | Data |
---|---|---|
4710c53d | 1 | #! /usr/bin/env python\r |
2 | \r | |
3 | """fixdiv - tool to fix division operators.\r | |
4 | \r | |
5 | To use this tool, first run `python -Qwarnall yourscript.py 2>warnings'.\r | |
6 | This runs the script `yourscript.py' while writing warning messages\r | |
7 | about all uses of the classic division operator to the file\r | |
8 | `warnings'. The warnings look like this:\r | |
9 | \r | |
10 | <file>:<line>: DeprecationWarning: classic <type> division\r | |
11 | \r | |
12 | The warnings are written to stderr, so you must use `2>' for the I/O\r | |
13 | redirect. I know of no way to redirect stderr on Windows in a DOS\r | |
14 | box, so you will have to modify the script to set sys.stderr to some\r | |
15 | kind of log file if you want to do this on Windows.\r | |
16 | \r | |
17 | The warnings are not limited to the script; modules imported by the\r | |
18 | script may also trigger warnings. In fact a useful technique is to\r | |
19 | write a test script specifically intended to exercise all code in a\r | |
20 | particular module or set of modules.\r | |
21 | \r | |
22 | Then run `python fixdiv.py warnings'. This first reads the warnings,\r | |
23 | looking for classic division warnings, and sorts them by file name and\r | |
24 | line number. Then, for each file that received at least one warning,\r | |
25 | it parses the file and tries to match the warnings up to the division\r | |
26 | operators found in the source code. If it is successful, it writes\r | |
27 | its findings to stdout, preceded by a line of dashes and a line of the\r | |
28 | form:\r | |
29 | \r | |
30 | Index: <file>\r | |
31 | \r | |
32 | If the only findings found are suggestions to change a / operator into\r | |
33 | a // operator, the output is acceptable input for the Unix 'patch'\r | |
34 | program.\r | |
35 | \r | |
36 | Here are the possible messages on stdout (N stands for a line number):\r | |
37 | \r | |
38 | - A plain-diff-style change ('NcN', a line marked by '<', a line\r | |
39 | containing '---', and a line marked by '>'):\r | |
40 | \r | |
41 | A / operator was found that should be changed to //. This is the\r | |
42 | recommendation when only int and/or long arguments were seen.\r | |
43 | \r | |
44 | - 'True division / operator at line N' and a line marked by '=':\r | |
45 | \r | |
46 | A / operator was found that can remain unchanged. This is the\r | |
47 | recommendation when only float and/or complex arguments were seen.\r | |
48 | \r | |
49 | - 'Ambiguous / operator (..., ...) at line N', line marked by '?':\r | |
50 | \r | |
51 | A / operator was found for which int or long as well as float or\r | |
52 | complex arguments were seen. This is highly unlikely; if it occurs,\r | |
53 | you may have to restructure the code to keep the classic semantics,\r | |
54 | or maybe you don't care about the classic semantics.\r | |
55 | \r | |
56 | - 'No conclusive evidence on line N', line marked by '*':\r | |
57 | \r | |
58 | A / operator was found for which no warnings were seen. This could\r | |
59 | be code that was never executed, or code that was only executed\r | |
60 | with user-defined objects as arguments. You will have to\r | |
61 | investigate further. Note that // can be overloaded separately from\r | |
62 | /, using __floordiv__. True division can also be separately\r | |
63 | overloaded, using __truediv__. Classic division should be the same\r | |
64 | as either of those. (XXX should I add a warning for division on\r | |
65 | user-defined objects, to disambiguate this case from code that was\r | |
66 | never executed?)\r | |
67 | \r | |
68 | - 'Phantom ... warnings for line N', line marked by '*':\r | |
69 | \r | |
70 | A warning was seen for a line not containing a / operator. The most\r | |
71 | likely cause is a warning about code executed by 'exec' or eval()\r | |
72 | (see note below), or an indirect invocation of the / operator, for\r | |
73 | example via the div() function in the operator module. It could\r | |
74 | also be caused by a change to the file between the time the test\r | |
75 | script was run to collect warnings and the time fixdiv was run.\r | |
76 | \r | |
77 | - 'More than one / operator in line N'; or\r | |
78 | 'More than one / operator per statement in lines N-N':\r | |
79 | \r | |
80 | The scanner found more than one / operator on a single line, or in a\r | |
81 | statement split across multiple lines. Because the warnings\r | |
82 | framework doesn't (and can't) show the offset within the line, and\r | |
83 | the code generator doesn't always give the correct line number for\r | |
84 | operations in a multi-line statement, we can't be sure whether all\r | |
85 | operators in the statement were executed. To be on the safe side,\r | |
86 | by default a warning is issued about this case. In practice, these\r | |
87 | cases are usually safe, and the -m option suppresses these warning.\r | |
88 | \r | |
89 | - 'Can't find the / operator in line N', line marked by '*':\r | |
90 | \r | |
91 | This really shouldn't happen. It means that the tokenize module\r | |
92 | reported a '/' operator but the line it returns didn't contain a '/'\r | |
93 | character at the indicated position.\r | |
94 | \r | |
95 | - 'Bad warning for line N: XYZ', line marked by '*':\r | |
96 | \r | |
97 | This really shouldn't happen. It means that a 'classic XYZ\r | |
98 | division' warning was read with XYZ being something other than\r | |
99 | 'int', 'long', 'float', or 'complex'.\r | |
100 | \r | |
101 | Notes:\r | |
102 | \r | |
103 | - The augmented assignment operator /= is handled the same way as the\r | |
104 | / operator.\r | |
105 | \r | |
106 | - This tool never looks at the // operator; no warnings are ever\r | |
107 | generated for use of this operator.\r | |
108 | \r | |
109 | - This tool never looks at the / operator when a future division\r | |
110 | statement is in effect; no warnings are generated in this case, and\r | |
111 | because the tool only looks at files for which at least one classic\r | |
112 | division warning was seen, it will never look at files containing a\r | |
113 | future division statement.\r | |
114 | \r | |
115 | - Warnings may be issued for code not read from a file, but executed\r | |
116 | using an exec statement or the eval() function. These may have\r | |
117 | <string> in the filename position, in which case the fixdiv script\r | |
118 | will attempt and fail to open a file named '<string>' and issue a\r | |
119 | warning about this failure; or these may be reported as 'Phantom'\r | |
120 | warnings (see above). You're on your own to deal with these. You\r | |
121 | could make all recommended changes and add a future division\r | |
122 | statement to all affected files, and then re-run the test script; it\r | |
123 | should not issue any warnings. If there are any, and you have a\r | |
124 | hard time tracking down where they are generated, you can use the\r | |
125 | -Werror option to force an error instead of a first warning,\r | |
126 | generating a traceback.\r | |
127 | \r | |
128 | - The tool should be run from the same directory as that from which\r | |
129 | the original script was run, otherwise it won't be able to open\r | |
130 | files given by relative pathnames.\r | |
131 | """\r | |
132 | \r | |
133 | import sys\r | |
134 | import getopt\r | |
135 | import re\r | |
136 | import tokenize\r | |
137 | \r | |
138 | multi_ok = 0\r | |
139 | \r | |
140 | def main():\r | |
141 | try:\r | |
142 | opts, args = getopt.getopt(sys.argv[1:], "hm")\r | |
143 | except getopt.error, msg:\r | |
144 | usage(msg)\r | |
145 | return 2\r | |
146 | for o, a in opts:\r | |
147 | if o == "-h":\r | |
148 | print __doc__\r | |
149 | return\r | |
150 | if o == "-m":\r | |
151 | global multi_ok\r | |
152 | multi_ok = 1\r | |
153 | if not args:\r | |
154 | usage("at least one file argument is required")\r | |
155 | return 2\r | |
156 | if args[1:]:\r | |
157 | sys.stderr.write("%s: extra file arguments ignored\n", sys.argv[0])\r | |
158 | warnings = readwarnings(args[0])\r | |
159 | if warnings is None:\r | |
160 | return 1\r | |
161 | files = warnings.keys()\r | |
162 | if not files:\r | |
163 | print "No classic division warnings read from", args[0]\r | |
164 | return\r | |
165 | files.sort()\r | |
166 | exit = None\r | |
167 | for filename in files:\r | |
168 | x = process(filename, warnings[filename])\r | |
169 | exit = exit or x\r | |
170 | return exit\r | |
171 | \r | |
172 | def usage(msg):\r | |
173 | sys.stderr.write("%s: %s\n" % (sys.argv[0], msg))\r | |
174 | sys.stderr.write("Usage: %s [-m] warnings\n" % sys.argv[0])\r | |
175 | sys.stderr.write("Try `%s -h' for more information.\n" % sys.argv[0])\r | |
176 | \r | |
177 | PATTERN = ("^(.+?):(\d+): DeprecationWarning: "\r | |
178 | "classic (int|long|float|complex) division$")\r | |
179 | \r | |
180 | def readwarnings(warningsfile):\r | |
181 | prog = re.compile(PATTERN)\r | |
182 | try:\r | |
183 | f = open(warningsfile)\r | |
184 | except IOError, msg:\r | |
185 | sys.stderr.write("can't open: %s\n" % msg)\r | |
186 | return\r | |
187 | warnings = {}\r | |
188 | while 1:\r | |
189 | line = f.readline()\r | |
190 | if not line:\r | |
191 | break\r | |
192 | m = prog.match(line)\r | |
193 | if not m:\r | |
194 | if line.find("division") >= 0:\r | |
195 | sys.stderr.write("Warning: ignored input " + line)\r | |
196 | continue\r | |
197 | filename, lineno, what = m.groups()\r | |
198 | list = warnings.get(filename)\r | |
199 | if list is None:\r | |
200 | warnings[filename] = list = []\r | |
201 | list.append((int(lineno), intern(what)))\r | |
202 | f.close()\r | |
203 | return warnings\r | |
204 | \r | |
205 | def process(filename, list):\r | |
206 | print "-"*70\r | |
207 | assert list # if this fails, readwarnings() is broken\r | |
208 | try:\r | |
209 | fp = open(filename)\r | |
210 | except IOError, msg:\r | |
211 | sys.stderr.write("can't open: %s\n" % msg)\r | |
212 | return 1\r | |
213 | print "Index:", filename\r | |
214 | f = FileContext(fp)\r | |
215 | list.sort()\r | |
216 | index = 0 # list[:index] has been processed, list[index:] is still to do\r | |
217 | g = tokenize.generate_tokens(f.readline)\r | |
218 | while 1:\r | |
219 | startlineno, endlineno, slashes = lineinfo = scanline(g)\r | |
220 | if startlineno is None:\r | |
221 | break\r | |
222 | assert startlineno <= endlineno is not None\r | |
223 | orphans = []\r | |
224 | while index < len(list) and list[index][0] < startlineno:\r | |
225 | orphans.append(list[index])\r | |
226 | index += 1\r | |
227 | if orphans:\r | |
228 | reportphantomwarnings(orphans, f)\r | |
229 | warnings = []\r | |
230 | while index < len(list) and list[index][0] <= endlineno:\r | |
231 | warnings.append(list[index])\r | |
232 | index += 1\r | |
233 | if not slashes and not warnings:\r | |
234 | pass\r | |
235 | elif slashes and not warnings:\r | |
236 | report(slashes, "No conclusive evidence")\r | |
237 | elif warnings and not slashes:\r | |
238 | reportphantomwarnings(warnings, f)\r | |
239 | else:\r | |
240 | if len(slashes) > 1:\r | |
241 | if not multi_ok:\r | |
242 | rows = []\r | |
243 | lastrow = None\r | |
244 | for (row, col), line in slashes:\r | |
245 | if row == lastrow:\r | |
246 | continue\r | |
247 | rows.append(row)\r | |
248 | lastrow = row\r | |
249 | assert rows\r | |
250 | if len(rows) == 1:\r | |
251 | print "*** More than one / operator in line", rows[0]\r | |
252 | else:\r | |
253 | print "*** More than one / operator per statement",\r | |
254 | print "in lines %d-%d" % (rows[0], rows[-1])\r | |
255 | intlong = []\r | |
256 | floatcomplex = []\r | |
257 | bad = []\r | |
258 | for lineno, what in warnings:\r | |
259 | if what in ("int", "long"):\r | |
260 | intlong.append(what)\r | |
261 | elif what in ("float", "complex"):\r | |
262 | floatcomplex.append(what)\r | |
263 | else:\r | |
264 | bad.append(what)\r | |
265 | lastrow = None\r | |
266 | for (row, col), line in slashes:\r | |
267 | if row == lastrow:\r | |
268 | continue\r | |
269 | lastrow = row\r | |
270 | line = chop(line)\r | |
271 | if line[col:col+1] != "/":\r | |
272 | print "*** Can't find the / operator in line %d:" % row\r | |
273 | print "*", line\r | |
274 | continue\r | |
275 | if bad:\r | |
276 | print "*** Bad warning for line %d:" % row, bad\r | |
277 | print "*", line\r | |
278 | elif intlong and not floatcomplex:\r | |
279 | print "%dc%d" % (row, row)\r | |
280 | print "<", line\r | |
281 | print "---"\r | |
282 | print ">", line[:col] + "/" + line[col:]\r | |
283 | elif floatcomplex and not intlong:\r | |
284 | print "True division / operator at line %d:" % row\r | |
285 | print "=", line\r | |
286 | elif intlong and floatcomplex:\r | |
287 | print "*** Ambiguous / operator (%s, %s) at line %d:" % (\r | |
288 | "|".join(intlong), "|".join(floatcomplex), row)\r | |
289 | print "?", line\r | |
290 | fp.close()\r | |
291 | \r | |
292 | def reportphantomwarnings(warnings, f):\r | |
293 | blocks = []\r | |
294 | lastrow = None\r | |
295 | lastblock = None\r | |
296 | for row, what in warnings:\r | |
297 | if row != lastrow:\r | |
298 | lastblock = [row]\r | |
299 | blocks.append(lastblock)\r | |
300 | lastblock.append(what)\r | |
301 | for block in blocks:\r | |
302 | row = block[0]\r | |
303 | whats = "/".join(block[1:])\r | |
304 | print "*** Phantom %s warnings for line %d:" % (whats, row)\r | |
305 | f.report(row, mark="*")\r | |
306 | \r | |
307 | def report(slashes, message):\r | |
308 | lastrow = None\r | |
309 | for (row, col), line in slashes:\r | |
310 | if row != lastrow:\r | |
311 | print "*** %s on line %d:" % (message, row)\r | |
312 | print "*", chop(line)\r | |
313 | lastrow = row\r | |
314 | \r | |
315 | class FileContext:\r | |
316 | def __init__(self, fp, window=5, lineno=1):\r | |
317 | self.fp = fp\r | |
318 | self.window = 5\r | |
319 | self.lineno = 1\r | |
320 | self.eoflookahead = 0\r | |
321 | self.lookahead = []\r | |
322 | self.buffer = []\r | |
323 | def fill(self):\r | |
324 | while len(self.lookahead) < self.window and not self.eoflookahead:\r | |
325 | line = self.fp.readline()\r | |
326 | if not line:\r | |
327 | self.eoflookahead = 1\r | |
328 | break\r | |
329 | self.lookahead.append(line)\r | |
330 | def readline(self):\r | |
331 | self.fill()\r | |
332 | if not self.lookahead:\r | |
333 | return ""\r | |
334 | line = self.lookahead.pop(0)\r | |
335 | self.buffer.append(line)\r | |
336 | self.lineno += 1\r | |
337 | return line\r | |
338 | def truncate(self):\r | |
339 | del self.buffer[-window:]\r | |
340 | def __getitem__(self, index):\r | |
341 | self.fill()\r | |
342 | bufstart = self.lineno - len(self.buffer)\r | |
343 | lookend = self.lineno + len(self.lookahead)\r | |
344 | if bufstart <= index < self.lineno:\r | |
345 | return self.buffer[index - bufstart]\r | |
346 | if self.lineno <= index < lookend:\r | |
347 | return self.lookahead[index - self.lineno]\r | |
348 | raise KeyError\r | |
349 | def report(self, first, last=None, mark="*"):\r | |
350 | if last is None:\r | |
351 | last = first\r | |
352 | for i in range(first, last+1):\r | |
353 | try:\r | |
354 | line = self[first]\r | |
355 | except KeyError:\r | |
356 | line = "<missing line>"\r | |
357 | print mark, chop(line)\r | |
358 | \r | |
359 | def scanline(g):\r | |
360 | slashes = []\r | |
361 | startlineno = None\r | |
362 | endlineno = None\r | |
363 | for type, token, start, end, line in g:\r | |
364 | endlineno = end[0]\r | |
365 | if startlineno is None:\r | |
366 | startlineno = endlineno\r | |
367 | if token in ("/", "/="):\r | |
368 | slashes.append((start, line))\r | |
369 | if type == tokenize.NEWLINE:\r | |
370 | break\r | |
371 | return startlineno, endlineno, slashes\r | |
372 | \r | |
373 | def chop(line):\r | |
374 | if line.endswith("\n"):\r | |
375 | return line[:-1]\r | |
376 | else:\r | |
377 | return line\r | |
378 | \r | |
379 | if __name__ == "__main__":\r | |
380 | sys.exit(main())\r |