\[\m@sdZddlmZdZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z mZmZddlmZdd lmZmZdd lmZdd lmZd d dZGdddeZddZddZddZddZej ddfkrpddl!Z!e!j"Z#ddZ$ddZ%ne"Z#eZ$eZ%ddZ&Gd d!d!eZ'Gd"d#d#e(Z)Gd$d%d%eZ*Gd&d'd'e)Z+dS)(zRefactoring framework. Used as a main program, this can refactor any number of files and/or recursively descend down directories. Imported as a module, this provides infrastructure to write your own refactoring tool. )with_statementz#Guido van Rossum N)chain)drivertokenizetoken) find_root)pytreepygram) btm_utils) btm_matcherTcCst|ggdg}tjj|j}g}xnttj|D]W}|jdrI|jdrI|r|dd}n|j |ddqIqIW|S)zEReturn a sorted list of all available fix names in the given package.*fix_z.pyN) __import__ospathdirname__file__sortedlistdir startswithendswithappend)Z fixer_pkgZ remove_prefixZpkgZ fixer_dirZ fix_namesnamer&/usr/lib/python3.4/lib2to3/refactor.pyget_all_fix_names!src@seZdZdS) _EveryNodeN)__name__ __module__ __qualname__rrrrr .s r cCst|tjtjfrC|jdkr3tnt|jgSt|tjrt|jrkt |jStnt|tj rt}x5|jD]*}x!|D]}|j t |qWqW|St d|dS)zf Accepts a pytree Pattern Node and returns a set of the pattern types which will match first. Nz$Oh no! I don't understand pattern %s) isinstancer Z NodePatternZ LeafPatterntyper setZNegatedPatternZcontent_get_head_typesZWildcardPatternupdate Exception)Zpatrpxrrrr'2s      r'c Cstjt}g}x|D]}|jryt|j}Wntk r_|j|YqXxU|D]}||j|qgWq|jdk r||jj|q|j|qWx:tt j j j t j j D]}||j|qWt|S)z^ Accepts a list of fixers and returns a dictionary of head node type --> fixer list. N) collections defaultdictlistpatternr'r rZ _accept_typerr python_grammarZ symbol2numbervaluestokensextenddict)Z fixer_listZ head_nodesZeveryfixerZheadsZ node_typerrr_get_headnode_dictNs"    r7cs fddtdDS)zN Return the fully qualified names for fixers in the package pkg_name. csg|]}d|qS).r).0fix_name)pkg_namerr ks z+get_fixers_from_package..F)r)r;r)r;rget_fixers_from_packagegsr=cCs|S)Nr)objrrr _identitynsr?rcCs|jddS)Nz  )replace)inputrrr_from_system_newlinesusrCcCs*tjdkr"|jdtjS|SdS)Nr@)rlineseprA)rBrrr_to_system_newlineswsrEc sd}tjtj|jfdd}ttjtjtj f}t }ydx]|\}}||krq`q`|tj kr|rPnd}q`|tj kr|dkr|\}}|tj ks|dkrPn|\}}|tj ks|dkrPn|\}}|tj krV|dkrV|\}}nxa|tj kr|j||\}}|tj ks|d krPn|\}}qYWq`Pq`Wntk rYnXt|S) NFcst}|d|dfS)Nrr)next)tok)genrradvances z(_detect_future_features..advanceTfrom __future__import(,)rgenerate_tokensioStringIOreadline frozensetrNEWLINENLCOMMENTr&STRINGNAMEOPadd StopIteration)sourceZhave_docstringrIignorefeaturesZtpvaluer)rHr_detect_future_featuressD     r`c@seZdZdZdS) FixerErrorzA fixer could not be loaded.N)r!r"r#__doc__rrrrras rac@skeZdZidd6dd6ZdZdZddddZd d Zd d Zd dZ ddZ ddZ ddddZ ddddZ ddZddddZddZdddZdd Zd!d"Zdddd#d$Zdd%d&Zd'Zd(Zd)d*Zd+d,Zd-d.Zd/d0Zd1d2Zd3d4ZdS)5RefactoringToolFprint_functionwrite_unchanged_filesZFixrNcCs||_|pg|_|jj|_|dk rI|jj|n|jdretj|_n tj |_|jj d|_ g|_ t jd|_g|_d|_tj|jdtjd|j|_|j\|_|_g|_tj|_g|_g|_x}t|j|jD]f}|j rT|jj!|q2||jkrv|jj"|q2||jkr2|jj"|q2q2Wt#|j|_$t#|j|_%dS)zInitializer. Args: fixer_names: a list of fixers to import options: an dict with configuration. explicit: a list of fixers to run even if they are explicit. NrdrercFconvertlogger)&fixersexplicit_default_optionscopyoptionsr(r !python_grammar_no_print_statementgrammarr1getreerrorsloggingZ getLoggerrg fixer_logwroterZDriverr rf get_fixers pre_order post_orderfilesbmZ BottomMatcherBMZ bmi_pre_orderZbmi_post_orderrZ BM_compatibleZ add_fixerrr7bmi_pre_order_headsbmi_post_order_heads)selfZ fixer_namesrlrir6rrr__init__s<            zRefactoringTool.__init__c Csg}g}x|jD]}t|iidg}|jddd}|j|jrx|t|jd}n|jd}|jdjdd|D}yt ||}Wn(t k rt d ||fYnX||j |j } | jr:|jd k r:||jkr:|jd |qn|jd || jd kri|j| q| jdkr|j| qt d| jqWtjd} |jd| |jd| ||fS)aInspects the options to load the requested patterns and handlers. Returns: (pre_order, post_order), where pre_order is the list of fixers that want a pre-order AST traversal, and post_order is the list that want post-order traversal. r r8rN_cSsg|]}|jqSr)title)r9r+rrrr<s z.RefactoringTool.get_fixers..zCan't find %s.%sTzSkipping implicit fixer: %szAdding transformation: %sZpreZpostzIllegal fixer order: %rZ run_orderkey)rhrrsplitr FILE_PREFIXlensplit CLASS_PREFIXjoingetattrAttributeErrorrarlrrri log_message log_debugZorderroperator attrgettersort) r|Zpre_order_fixersZpost_order_fixersZ fix_mod_pathmodr:partsZ class_nameZ fix_classr6Zkey_funcrrrrts8# zRefactoringTool.get_fixerscOsdS)zCalled when an error occurs.Nr)r|msgargskwdsrrr log_errorszRefactoringTool.log_errorcGs'|r||}n|jj|dS)zHook to log a message.N)rginfo)r|rrrrrrs zRefactoringTool.log_messagecGs'|r||}n|jj|dS)N)rgdebug)r|rrrrrrs zRefactoringTool.log_debugcCsdS)zTCalled with the old version, new version, and filename of a refactored file.Nr)r|old_textnew_textfilenameequalrrr print_output!szRefactoringTool.print_outputcCsPxI|D]A}tjj|r5|j|||q|j|||qWdS)z)Refactor a list of files and directories.N)rrisdir refactor_dir refactor_file)r|itemswrite doctests_onlyZ dir_or_filerrrrefactor&s zRefactoringTool.refactorc Cstjd}xtj|D]\}}}|jd||j|jxe|D]]}|jd rWtjj|d|krWtjj||} |j | ||qWqWWdd|D|dd.N) rextsepwalkrrrrsplitextrr) r|Zdir_namerrZpy_extdirpathZdirnames filenamesrfullnamerrrr/s    zRefactoringTool.refactor_dircCsyt|d}Wn<tk rQ}z|jd||dSWYdd}~XnXztj|jd}Wd|jXt|dd|}t|j |fSWdQXdS)zG Do our best to decode a Python source file correctly. rbzCan't open %s: %sNrr*encoding)NN) openOSErrorrrdetect_encodingrRclose_open_with_encodingrCread)r|rferrrrrr_read_python_sourceCs z#RefactoringTool._read_python_sourcecCs|j|\}}|dkr%dS|d7}|r|jd||j||}|jsl||kr|j|||||q|jd|ni|j||}|js|r|jr|jt|dd |d|d|n|jd|dS) zRefactors a file.Nr@zRefactoring doctests in %szNo doctest changes in %srrrzNo changes in %sr)rrrefactor_docstringreprocessed_filerefactor_string was_changedstr)r|rrrrBroutputtreerrrrSs  zRefactoringTool.refactor_filecCst|}d|kr*tj|j_nzby|jj|}WnEtk r}z%|jd||jj |dSWYdd}~XnXWd|j|j_X||_ |j d||j |||S)aFRefactor a given input string. Args: data: a string holding the code to be refactored. name: a human-readable name for use in error/log messages. Returns: An AST corresponding to the refactored input stream; None if there were errors during the parse. rdzCan't parse %s: %s: %sNzRefactoring %s) r`r rmrrnZ parse_stringr)r __class__r!future_featuresr refactor_tree)r|datarr^rrrrrrjs    zRefactoringTool.refactor_stringcCstjj}|ro|jd|j|d}|jsI||kr_|j|d|q|jdnS|j|d}|js|r|jr|jt |d|n |jddS)NzRefactoring doctests in stdinzzNo doctest changes in stdinzNo changes in stdin) sysstdinrrrrerrrr)r|rrBrrrrrrefactor_stdins zRefactoringTool.refactor_stdinc Csx-t|j|jD]}|j||qW|j|j|j|j|j|j|jj|j }xt |j rdx|jj D]}||kr||r||j dtjjdd|jr||j dtjjnx\t||D]G}|||kr9||j|nyt|Wntk r^wYnX|jr}||jkr}qn|j|}|r|j||}|dk rV|j|x9|jD]+}|jsg|_n|jj|qW|jj|j }x?|D]4} | |kr7g|| >> z... c Csg}d}d}d}d}x4|jddD] }|d7}|jj|jr|dk r|j|j||||n|}|g}|j|j} |d| }q1|dk r |j||js|||jjdkr |j |q1|dk r8|j|j||||nd}d}|j |q1W|dk r|j|j||||ndj |S)aRefactors a docstring, looking for doctests. This returns a modified version of the input string. It looks for doctests, which start with a ">>>" prompt, and may be continued with "..." prompts, as long as the "..." is indented the same as the ">>>". (Unfortunately we can't use the doctest module's parser, since, like most parsers, it is not geared towards preserving the original source.) NrkeependsTrr@r) splitlineslstriprPS1r4refactor_doctestfindPS2rstriprr) r|rBrresultblockZ block_linenoindentlinenolineirrrr(s:       z"RefactoringTool.refactor_docstringc syj||}Wntk r}zjjjtjrpx*|D]}jd|jdqJWnjd|||j j ||SWYdd}~XnXj ||rt |j dd}|d|d||dd} }| dg|dkst| |d jds@|d d7>>" (possibly indented), while the remaining lines start with "..." (identically indented). z Source: %sr@z+Can't parse docstring in %s line %s: %s: %sNrTrrcs!g|]}j|qSr)r)r9r)rr|rrr<ms z4RefactoringTool.refactor_doctest..rr) parse_blockr)rgZ isEnabledForrqDEBUGrrrrr!rrrAssertionErrorrrpop) r|rrrrrrrrZclippedr)rr|rrSs&   )#&z RefactoringTool.refactor_doctestcCs|jrd}nd}|js4|jd|n1|jd|x|jD]}|j|qNW|jr|jdx!|jD]}|j|qWn|jrt|jdkr|jdn|jdt|jx0|jD]"\}}}|j|||qWndS) Nwerez need to bezNo files %s modified.zFiles that %s modified:z$Warnings/messages while refactoring:rzThere was 1 error:zThere were %d errors:)rsrwrrrrpr)r|rfilemessagerrrrrr summarizeps$      zRefactoringTool.summarizecCs1|jj|j|||}t|_|S)zParses a block into a tree. This is necessary to get correct line number / offset information in the parser diagnostics and embedded into the parse tree. )rZ parse_tokens wrap_toksrSr)r|rrrrrrrrs! zRefactoringTool.parse_blockc cstj|j||j}xe|D]]\}}\}}\} } } ||d7}| |d7} ||||f| | f| fVq%WdS)z;Wraps a tokenize stream to systematically modify start/end.rN)rrO gen_lines__next__) r|rrrr3r%r_Zline0Zcol0Zline1Zcol1Z line_textrrrrs (zRefactoringTool.wrap_toksccs||j}||j}|}xo|D]g}|j|rT|t|dVn4||jdkrrdVntd||f|}q'WxdVqdS)zGenerates lines as expected by tokenize from a list of lines. This strips the first len(indent + self.PS1) characters off each line. Nr@zline=%r, prefix=%rr)rrrrrr)r|rrZprefix1Zprefix2prefixrrrrrs    zRefactoringTool.gen_lines)r!r"r#rjrrr}rtrrrrrrrrrrrrrrrrrrrrrrrrrrrcs:   4 (        O  +   rcc@seZdZdS)MultiprocessingUnsupportedN)r!r"r#rrrrrs rcsaeZdZfddZdddfddZfddZfd d ZS) MultiprocessRefactoringToolcs/tt|j||d|_d|_dS)N)superrr}queue output_lock)r|rkwargs)rrrr}s z$MultiprocessRefactoringTool.__init__Frcs[|dkr(ttj|||SyddlWntk rRtYnXjdk rqtdnj_j _ fddt |D}z;x|D]}|j qWttj|||Wdjj x$t |D]}jjdq Wx'|D]}|jr*|j q*q*Wd_XdS)Nrrz already doing multiple processescs%g|]}jdjqS)target)ZProcess_child)r9r)multiprocessingr|rrr<s z8MultiprocessRefactoringTool.refactor..)rrrr ImportErrorrr RuntimeErrorZ JoinableQueueZLockrrangestartrputZis_alive)r|rrrZ num_processesZ processesr+r)r)rr|rrs2         z$MultiprocessRefactoringTool.refactorc so|jj}xY|dk rj|\}}ztt|j||Wd|jjX|jj}qWdS)N)rrorrrZ task_done)r|Ztaskrr)rrrrs z"MultiprocessRefactoringTool._childcsE|jdk r(|jj||fntt|j||SdS)N)rrrrr)r|rr)rrrrsz)MultiprocessRefactoringTool.refactor_file)r!r"r#r}rrrrr)rrrs  r),rbrKr __author__rrrqrr-rP itertoolsrZpgen2rrrZ fixer_utilrrr r r Zbur rxrr)r r'r7r=r? version_infocodecsrrrCrEr`raobjectrcrrrrrr sF               (