ó ½ƒWc@s¤dZdZdZdZdZdZdZddlZdd lm Z ddl Z ddl Z e j d d krddl jZddljZdd lmZdd lmZddlmZddlmZddlmZddlmZddl m!Z"ddl#j$Z%n”ddl&Zddl'Zdd l(mZdd l(mZddl(mZddl(mZddl(mZddl(mZddl(m!Z"ddl)Z%ddl*m+Z+ddl,m-Z-ddl.m/Z/ej0e1ƒZ2defd„ƒYZ3de4fd„ƒYZ5ddd„Z7ddd„Z8ddd„Z9ddd„Z:dd„Z;d„Z<d e=fd!„ƒYZ>d"„Z?e1d#kr ej@ƒe?ƒndS($s`Utilities using NDG HTTPS Client, including a main module that can be used to fetch from a URL. s R B Wilkinsons09/12/11s2(C) 2011 Science and Technology Facilities Councils-BSD - see LICENSE file in top-level directorysPhilip.Kershaw@stfc.ac.uks$Id$iÿÿÿÿN(t OptionParserii(tRequest(t HTTPHandler(tHTTPCookieProcessor(tHTTPBasicAuthHandler(tHTTPPasswordMgrWithDefaultRealm(t ProxyHandler(t HTTPError(t build_opener(tHTTPSContextHandler(tssl_context_utiltAccumulatingHTTPCookieProcessorcBseZdZd„ZeZRS(stCookie processor that adds new cookies (instead of replacing the existing ones as HTTPCookieProcessor does) cCs¯d}t|jƒ|ji|j|jƒ}|jj|ƒ|j|ƒ}|r«|j|ƒr˜|j|ƒ}dj ||gƒ}|j ||ƒq«|j ||ƒn|S(s¹Processes cookies for a HTTP request. @param request: request to process @type request: urllib2.Request @return: request @rtype: urllib2.Request tCookies; ( tRequest_t get_full_urltdatatorigin_req_hostt unverifiablet cookiejartadd_cookie_headert get_headert has_headertjointadd_unredirected_header(tselftrequesttCOOKIE_HEADER_NAMEt tmp_requestt new_cookiest old_cookiestmerged_cookies((s9/usr/lib/python2.7/dist-packages/ndg/httpsclient/utils.pyt http_request3s   (t__name__t __module__t__doc__Rt https_request(((s9/usr/lib/python2.7/dist-packages/ndg/httpsclient/utils.pyR /s t URLFetchErrorcBseZdZRS(sError fetching content from URL(R R!R"(((s9/usr/lib/python2.7/dist-packages/ndg/httpsclient/utils.pyR$QscCsct||d|d|ƒ\}}}|rS|tjkrS|jƒ}|jƒ|St|ƒ‚dS(sáReturns data retrieved from a URL. @param url: URL to attempt to open @type url: basestring @param config: SSL context configuration @type config: Configuration @return data retrieved from URL or None RthandlersN(topen_urlt http_client_tOKtreadtcloseR$(turltconfigRR%t return_codetreturn_messagetresponset return_data((s9/usr/lib/python2.7/dist-packages/ndg/httpsclient/utils.pytfetch_from_urlUs  c Csˆt||d|d|ƒ\}}}|tjkrr|jƒ}|jƒt|dƒ} | j|ƒ| jƒn|||tjkfS(s¨Writes data retrieved from a URL to a file. @param url: URL to attempt to open @type url: basestring @param config: SSL context configuration @type config: Configuration @param output_file: output file @type output_file: basestring @return: tuple ( returned HTTP status code or 0 if an error occurred returned message boolean indicating whether access was successful) RR%tw(R&R'R(R)R*topentwrite( R+R,t output_fileRR%R-R.R/R0toutfile((s9/usr/lib/python2.7/dist-packages/ndg/httpsclient/utils.pytfetch_from_url_to_filefs     cCsMt||d|d|ƒ\}}}|r=|tjkr=|St|ƒ‚dS(sœReturns data retrieved from a URL. @param url: URL to attempt to open @type url: basestring @param config: SSL context configuration @type config: Configuration @param data: HTTP POST data @type data: str @param handlers: list of custom urllib2 handlers to add to the request @type handlers: iterable @return: data retrieved from URL or None @rtype: file derived type RR%N(R&R'R(R$(R+R,RR%R-R.R/((s9/usr/lib/python2.7/dist-packages/ndg/httpsclient/utils.pytfetch_stream_from_urls c Csâ|jrdnd}|jr*|j}n tjƒ}t|ƒ}|sQg}n|j|ƒ|jr¡td|ƒ}t|jd|ƒ}|j ||gƒn|j rùt t ƒƒ} | j ddd|d|j dd|j dƒ|j| ƒnt||jƒs.|jtiƒƒtjdƒn6|jrd|jt|jƒƒtjd |jƒntd |j|Œ} |j} | dkr—i} nt||| ƒ} d} d }d}yj| j| ƒ}|j}|j} tjtjƒr$x0t|ƒD]\}}tjd ||ƒqþWnWn­tk r~}|j} d |j}tjtjƒrÕtjd|j|jƒqÕnWtk rÔ}d |j ƒ}tjtjƒrÕddl!}tj|j"ƒƒqÕnX| ||fS(sÞAttempts to open a connection to a specified URL. @param url: URL to attempt to open @param config: SSL context configuration @type config: Configuration @param data: HTTP POST data @type data: str @param handlers: list of custom urllib2 handlers to add to the request @type handlers: iterable @return: tuple ( returned HTTP status code or 0 if an error occurred returned message or error description response object) iit debugleveltrealmturitusertpasswdsNot using proxysConfiguring proxies: %st ssl_contextts %s : %ss Error: %ss%s %siÿÿÿÿN(#tdebugtcookiet cookiejar_t CookieJarR tappendt HTTPHandler_R R>textendthttp_basicauthtHTTPBasicAuthHandler_t HTTPPasswordMgrWithDefaultRealm_t add_passwordtNonet_should_use_proxytno_proxyt ProxyHandler_tlogtproxiesRtheadersR R3tmsgtcodet isEnabledFortloggingtDEBUGt enumeratet HTTPError_t Exceptiont__str__t tracebackt format_exc(R+R,RR%R9tcjtcookie_handlert http_handlert https_handlert auth_handlertopenerRQRR-R.R/tindexRAtexcR[((s9/usr/lib/python2.7/dist-packages/ndg/httpsclient/utils.pyR&”sf                    cCs‰|dkr$tjjddƒ}n|}tjt|ƒƒ}xCg|jdƒD]}|jƒ^qRD]}|j |krht SqhWt S(sØDetermines whether a proxy should be used to open a connection to the specified URL, based on the value of the no_proxy environment variable. @param url: URL @type url: basestring or urllib2.Request RMR?t,N( RKtostenvirontgett urlparse_turlparset_url_as_stringtsplittstripthostnametFalsetTrue(R+RMtno_proxy_effectiveturlObjthtnp((s9/usr/lib/python2.7/dist-packages/ndg/httpsclient/utils.pyRLós /cCsFt|tƒr|jƒSt|tƒr,|Stdttfƒ‚dS(sÍReturns the URL string from a URL value that is either a string or urllib2.Request.. @param url: URL @type url: basestring or urllib2.Request @return: URL string @rtype: basestring sExpected type %r or %rN(t isinstanceR Rtstrt TypeError(R+((s9/usr/lib/python2.7/dist-packages/ndg/httpsclient/utils.pyRks  t ConfigurationcBs)eZdZedddddd„ZRS(sConnection configuration. cCsC||_||_||_||_||_||_||_dS(sÔ @param ssl_context: SSL context to use with this configuration @type ssl_context: OpenSSL.SSL.Context @param debug: if True, output debugging information @type debug: bool @param proxies: proxies to use for @type proxies: dict with basestring keys and values @param no_proxy: hosts for which a proxy should not be used @type no_proxy: basestring @param cookie: cookies to set for request @type cookie: cookielib.CookieJar (python 3 - http.cookiejar) @param http_basicauth: http authentication, or None @type http_basicauth: tuple of (username,password) @param headers: http headers @type headers: dict N(R>R@RPRMRARGRQ(RR>R@RPRMRARGRQ((s9/usr/lib/python2.7/dist-packages/ndg/httpsclient/utils.pyt__init__s      N(R R!R"RoRKRy(((s9/usr/lib/python2.7/dist-packages/ndg/httpsclient/utils.pyRxs c CsÒtddƒ}|jddddddd tjjd ƒd d ƒ|jd dddddd d8d dƒ|jddddddd d8d dƒ|jddddddd td dƒ|jddddddd d8d dƒ|jd d!dd"ddd d8d d#ƒ|jd$d%dddd&d td d'ƒ|jd(d)dd*dd+d d8d d,ƒ|jd-dd.dd/dd0d d1ƒ|jƒ\}}t|ƒd2kr´|j d3ƒn|d4}|j ràt j ƒj t jƒn|jr tjj|jƒr |j}nd8}|jr:tjj|jƒr:|j}nd8}|jrjtjj|jƒrj|j}nd8}|j }|jrÀtjj|jƒrÀt|jƒ}|jƒ} |jƒnd8} |jrç|jjd5d2ƒ} nd8} i} |jrAxB|jD]4} | jd5d2ƒ\} }|jƒ| | jƒR,R-R.((s9/usr/lib/python2.7/dist-packages/ndg/httpsclient/utils.pytmain4sš                  t__main__(AR"t __author__t__date__t __copyright__t __license__t __contact__t __revision__RUtoptparseRRftsyst version_infothttp.cookiejarRRBt http.clienttclientR'turllib.requestRR RRERtHTTPCookieProcessor_RRHRRIRRNt urllib.errorRRXt urllib.parsetparseRit cookielibthttplibturllib2Rjt$ndg.httpsclient.urllib2_build_openerRtndg.httpsclient.httpsR tndg.httpsclientR RR ROR RYR$RKR1R7R8R&RLRktobjectRxRœt basicConfig(((s9/usr/lib/python2.7/dist-packages/ndg/httpsclient/utils.pyts`      "_   e