U
e5dÐ$ ã @ s\ d Z ddlZddlZddlZdgZe dd¡ZG dd„ dƒZG dd„ dƒZ G d d
„ d
ƒZ
dS ) a% robotparser.py
Copyright (C) 2000 Bastian Kleineidam
You can choose between two licenses when using this package:
1) GNU GPLv2
2) PSF license for Python 2.2
The robots.txt Exclusion Protocol is implemented as specified in
http://www.robotstxt.org/norobots-rfc.txt
é NÚRobotFileParserÚ RequestRatezrequests secondsc @ sr e Zd ZdZd dd„Zdd„ Zdd„ Zd d
„ Zd d „ Zd
d„ Z dd„ Z
dd„ Z dd„ Z dd„ Z
dd„ Zdd„ ZdS ) r zs This class provides a set of methods to read, parse and answer
questions about a single robots.txt file.
Ú c C s2 g | _ g | _d | _d| _d| _| |¡ d| _d S )NFr )ÚentriesÚsitemapsÚ
default_entryÚ disallow_allÚ allow_allÚset_urlÚ last_checked©ÚselfÚurl© r ú*/usr/lib64/python3.8/urllib/robotparser.pyÚ__init__ s
zRobotFileParser.__init__c C s | j S )z·Returns the time the robots.txt file was last fetched.
This is useful for long-running web spiders that need to
check for new robots.txt files periodically.
)r ©r
r r r Úmtime% s zRobotFileParser.mtimec C s ddl }| ¡ | _dS )zYSets the time the robots.txt file was last fetched to the
current time.
r N)Útimer )r
r r r r Úmodified. s zRobotFileParser.modifiedc C s& || _ tj |¡dd… \| _| _dS )z,Sets the URL referring to a robots.txt file.é é N)r ÚurllibÚparseÚurlparseZhostÚpathr r r r r
6 s zRobotFileParser.set_urlc
C s† zt j | j¡}W nR t jjk
rd } z0|jdkr:d| _n|jdkrT|jdk rTd| _W 5 d}~X Y n X | ¡ }|
| d¡ ¡ ¡ dS )z4Reads the robots.txt URL and feeds it to the parser.)i‘ i“ Ti iô Nzutf-8)
r ZrequestZurlopenr ÚerrorZ HTTPErrorÚcoder r Úreadr ÚdecodeÚ
splitlines)r
ÚfÚerrÚrawr r r r ; s
zRobotFileParser.readc C s, d|j kr | jd kr(|| _n | j |¡ d S ©NÚ*)Ú
useragentsr r Úappend)r
Úentryr r r Ú
_add_entryH s
zRobotFileParser._add_entryc C sP d}t ƒ }| ¡ |D ] }|sP|dkr4t ƒ }d}n |dkrP| |¡ t ƒ }d}| d¡}|dkrn|d|… }| ¡ }|s|q| dd¡}t|ƒdkr|d ¡ ¡ |d< tj
|d ¡ ¡|d< |d dkrú|dkrä| |¡ t ƒ }|j |d ¡ d}q|d dkr.|dkr6|j
t|d d ƒ¡ d}q|d d
krb|dkr6|j
t|d d ƒ¡ d}q|d d kr |dkr6|d ¡ ¡ ršt|d ƒ|_d}q|d d
kr|dkr6|d d¡}t|ƒdkr|d ¡ ¡ r|d ¡ ¡ rtt|d ƒt|d ƒƒ|_d}q|d dkr| j |d ¡ q|dkrL| |¡ dS )z”Parse the input lines from a robots.txt file.
We allow that a user-agent: line is not preceded by
one or more blank lines.
r r é ú#Nú:z
user-agentZdisallowFZallowTz crawl-delayz request-rateú/Zsitemap)ÚEntryr r) ÚfindÚstripÚsplitÚlenÚlowerr r Úunquoter& r' Ú rulelinesÚRuleLineÚisdigitÚintÚdelayr Úreq_rater )r
ÚlinesÚstater( ÚlineÚiZnumbersr r r r Q sj
ÿ
zRobotFileParser.parsec C s | j r
dS | jrdS | js dS tj tj |¡¡}tj dd|j|j |j
|j f¡}tj |¡}|sfd}| j
D ] }| |¡rl| |¡ S ql| jrœ| j |¡S dS )z=using the parsed robots.txt decide if useragent can fetch urlFTr r- )r r r r r r r4 Ú
urlunparser ZparamsZqueryZfragmentÚquoter Ú
applies_toÚ allowancer )r
Ú useragentr Z
parsed_urlr( r r r Ú can_fetchš s* ÿ
zRobotFileParser.can_fetchc C s> | ¡ s d S | jD ]}| |¡r|j S q| jr:| jjS d S ©N)r r rA r9 r ©r
rC r( r r r Ú crawl_delay· s
zRobotFileParser.crawl_delayc C s> | ¡ s d S | jD ]}| |¡r|j S q| jr:| jjS d S rE )r r rA r: r rF r r r Ú request_rateÁ s
z RobotFileParser.request_ratec C s | j s
d S | j S rE )r r r r r Ú site_mapsË s zRobotFileParser.site_mapsc C s, | j }| jd k r || jg }d tt|ƒ¡S )Nz
)r r ÚjoinÚmapÚstr)r
r r r r Ú__str__Ð s
zRobotFileParser.__str__N)r )Ú__name__Ú
__module__Ú __qualname__Ú__doc__r r r r
r r) r rD rG rH rI rM r r r r r s
I
c @ s( e Zd ZdZdd„ Zdd„ Zdd„ ZdS ) r6 zoA rule line is a single "Allow:" (allowance==True) or "Disallow:"
(allowance==False) followed by a path.c C s< |dkr|sd}t j t j |¡¡}t j |¡| _|| _d S )Nr T)r r r? r r@ r rB )r
r rB r r r r Ú s
zRuleLine.__init__c C s | j dkp| | j ¡S r$ )r Ú
startswith)r
Úfilenamer r r rA â s zRuleLine.applies_toc C s | j r
dndd | j S )NZAllowZDisallowz: )rB r r r r r rM å s zRuleLine.__str__N)rN rO rP rQ r rA rM r r r r r6 × s r6 c @ s0 e Zd ZdZdd„ Zdd„ Zdd„ Zdd „ Zd
S ) r. z?An entry has one or more user-agents and zero or more rulelinesc C s g | _ g | _d | _d | _d S rE )r& r5 r9 r: r r r r r ë s zEntry.__init__c C s‚ g }| j D ]}| d|› ¡ q
| jd k r<| d| j› ¡ | jd k rf| j}| d|j› d|j› ¡ | tt| j ƒ¡ d
|¡S )Nz User-agent: z
Crawl-delay: zRequest-rate: r- Ú
) r& r' r9 r: ZrequestsZsecondsÚextendrK rL r5 rJ )r
ZretÚagentZrater r r rM ñ s
z
Entry.__str__c C sF | d¡d ¡ }| jD ](}|dkr* dS | ¡ }||kr dS qdS )z2check if this entry applies to the specified agentr- r r% TF)r1 r3 r& )r
rC rV r r r rA ý s
zEntry.applies_toc C s$ | j D ]}| |¡r|j S qdS )zZPreconditions:
- our agent applies to this entry
- filename is URL decodedT)r5 rA rB )r
rS r= r r r rB
s
zEntry.allowanceN)rN rO rP rQ r rM rA rB r r r r r. é s
r. ) rQ Ú collectionsZ urllib.parser Zurllib.requestÚ__all__Ú
namedtupler r r6 r. r r r r Ú<module> s B
All system for education purposes only. For more tools: Telegram @jackleet