xjU\dZddlmZmZmZmZmZmZmZm Z ddl m Z ddl mZddlZddlmZddlmZddlZddlmZddlmZdd lmZmZdd lmZdd lmZddl Z e jBd Z"e"jGe jHGd deZ%GddeZ&GddZ'y)z; Helper functions to communicate with replication servers. ) NamedTupleOptionalAnyIteratorcastMappingTupleDictN)URLError)contextmanager)ceil) HTTPAdapter)Retry)MergeInputReader BaseHandler)io)versionpyosmiumc<eZdZUdZeed< ejed<y) OsmosisStatez6 Represents a state file of a replication server. sequence timestampN)__name__ __module__ __qualname____doc__int__annotations__dtdatetimeI/build-osm/venv/lib/python3.12/site-packages/osmium/replication/server.pyrrsM5{{?r"rc4eZdZUdZeed< eed< eed<y)DownloadResultz Downloaded change. idreadernewestN)rrrrrrrr!r"r#r%r%'s  GG > K6r"r%ceZdZdZd*dededdfdZd+dZd,dZd ed ed eddfd Z d ededdfdZ dede jfdZ de jdefdZ d-dedeedeedeefdZ d.dededeedededeedeefdZ d/dedededeededeeeefdeedeedeeeeffdZ d0d ej4d!ed"edeefd#Zd1d$eed%edeefd&Zd$edefd'Zd$eedefd(Zd$edefd)Z y)2ReplicationServerac Represents a connection to a server that publishes replication data. Replication change files allow to keep local OSM data up-to-date without downloading the full dataset again. ReplicationServer may be used as a context manager. In this case, it internally keeps a connection to the server making downloads faster. url diff_typereturnNc||_||_tdd|_d|_t dddhgd |_y) a_ Set up the connection to a replication server. `url` contains the base URL of the replication service. This is the directory that contains the state file with the current state. If the replication service serves something other than osc.gz files, set the `diff_type` to the given file suffix. <T)timeoutstreamNg?GET)iiiiii)totalbackoff_factorallowed_methodsstatus_forcelist)baseurlr,dictextra_request_paramssessionrretry)selfr+r,s r#__init__zReplicationServer.__init__;s@ "48D4Q!37 3,JL r"c`|j"|jjd|_yy)z> Close any open connection to the replication server. N)r;closer=s r#r@zReplicationServer.closeKs* << # LL   DL $r"ctj|_|jjdt |j |jjdt |j |SNzhttp://) max_retrieszhttps://)requestsSessionr;mountrr<rAs r# __enter__zReplicationServer.__enter__RsQ'')  9kdjj&IJ :{tzz'JK r"exc_type exc_value tracebackc$|jy)N)r@)r=rIrJrKs r#__exit__zReplicationServer.__exit__Xs  r"keyvaluec"||j|<y)a Set a parameter which will be handed to the requests library when calling `requests.get()`. This may be used to set custom headers, timeouts and similar parameters. See the `requests documentation `_ for possible parameters. Per default, a timeout of 60 sec is set and streaming download enabled. N)r:)r=rNrOs r#set_request_parameterz'ReplicationServer.set_request_parameter[s*/!!#&r"cZddtji}tj||S)Nz User-Agentz pyosmium/)headers)rpyosmium_release urlrequestRequest)r=r+rSs r# make_requestzReplicationServer.make_requestes-9W-E-E,F!GH!!#w77r"cdjvr jn._get_url_with_sessionvsq!!#w i)LM j+$***MN%'++c&6&6&8GJG $##sBA2B BBB) r:r9 header_itemsr;rZr[r rrEResponse)r=r+kvr^r]s`` @r#open_urlzReplicationServer.open_urlis 11 122Jd778J696F6F6H$I6HdaQT6H$IJy ! << ##4<<##C$4$4$6E*E E  x0A0A'B   %&&%Js Cstart_idmax_sizeend_idc^|j}|||jkDry|}d}||dz}n|d}t}||dkDr|||kr||jkr |j|}t|dk(r||k(rynw|j||j} |tjd |n|| z}tjd ||dz |d z }||dkDr|||kr||jkrt|d z ||jS#tj $r} ||k(rJ| j >| j jdzdk(r"tjd| j | tjd| tjd d }Yd} ~ ;d} ~ wwxYw)a Create a MergeInputReader and download diffs starting with sequence id `start_id` into it. `end_id` optionally gives the highest sequence number to download. `max_size` restricts the number of diffs that are downloaded by size. If neither `end_id` nor `max_size` are given, then download default to stop after 1MB. The download stops as soon as 1. a diff cannot be downloaded or 2. the end_id (inclusive) is reached or 3. the unpacked data in memory exceeds `max_size` kB or, when no `end_id` and `max_size` are given, 1024kB. If some data was downloaded, returns a namedtuple with three fields: `id` contains the sequence id of the last downloaded diff, `reader` contains the MergeInputReader with the data and `newest` is a sequence id of the most recent diff available. Returns None if there was no new data was available. If there is an error during the download, then the function will simply return the already downloaded data. If the reported error is a client error (HTTP 4xx) and happens during the download of the first diff, then a ::request.HTTPError:: is raised: this condition is likely to be permanent and the caller should not simply retry without investigating the cause. NiirdzPermanent server error: %szError during diff download: %sz Bailing out.zDownloaded change %d.z:Downloaded change %d. (%d kB available in download buffer))get_state_inforrget_diff_blockrERequestExceptionresponse status_codeLOGerrorlen add_bufferr,debugr%) r=rdrerfr( current_id left_sizerddiffdataex diff_sizes r# collect_diffszReplicationServer.collect_diffss<$$& >X7 #'   4I ^#I   IM^zV';&//1 ..z:8}!z) h?I  1:>Y&  V$i$&68 !OJ? IM^zV';&//1>j1nb&//BB9,, z)KK3[[44s:a?II:BKKHH :B? .) sDF,%A|dz } | 7| |jkDr(|j| } | dz} | | |jkDr(| >|dz} | 7| |jkr(|j| } | dz } | | |jkr(| |jS| j|kr| }n| }|jdz|jk\r |jS)a Get the sequence number of the replication file that contains the given timestamp. The search algorithm is optimised for replication servers that publish updates in regular intervals. For servers with irregular change file publication dates 'balanced_search` should be set to true so that a standard binary search for the sequence will be used. The default is good for all known OSM replication services. When `limit_by_oldest_available` is set, then the function will return None when the server replication does not start at 0 and the given timestamp is older than the oldest available timestamp on the server. Some replication servers do not keep the full history and this flag avoids accidentally trying to download older data. The downside is that the function will never return the oldest available sequence ID when the flag is set. NrzTrying with Id %srk)rlrrrqrur total_secondsr )r=rrrupperlowerloweridnewid base_splitidts_intseq_intgoalsplitsplitids r#timestamp_to_sequencez'ReplicationServer.timestamp_to_sequenceEs(##% =  '5>>Q+>>> !m II)7 3''0E U__ %A>>Q&>>A%7#<4P%..P}Wu~~5:;G# >>)'m2"ENNU^^$Cq#HI //EOO;JJL..5>>9!EOO3BBD$~~TG^f5L0MM 5>>1#(>>A#5L'' 5E}&*m%..(@ //8EqLGm%..(@}&*m%..(@ //8EqLGm%..(@}~~%*~~!U^^3~~%Gr"seqretriesct|dzD]o}d}d} |j|j|j|5}t |dr!|j |j }n|}|D]}|jdj}d|vr|d|jd}n|j}|sN|jdd} t| dk7r dddy| dd k(rt| d}| dd k(s tjj| dd }|j#tj$j& }ddd|_|ct3||cSy#t $rY-wxYw#1swY2xYw#t(t*f$r*} t,j/d t1| Yd} ~ yd} ~ wwxYw)af Downloads and returns the state information for the given sequence. If the download is successful, a namedtuple with `sequence` and `timestamp` is returned, otherwise the function returns `None`. `retries` sets the number of times the download is retried when pyosmium detects a truncated state file. rkN iter_lineszutf-8#r=rsequenceNumberrz%Y-%m-%dT%H\:%M\:%SZ)tzinfoz"Loading state info failed with: %s)rr)rangercrW get_state_urlhasattrraise_for_statusrdecodestripindexrrsrrr strptime ValueErrorreplacetimezoneutcr IOErrorrqrurr) r=rr_tsnext_seqrolineslinekverrs r#rlz ReplicationServer.get_state_infosw{#ABH ]]4#4#4T5G5G5L#MNRZx6 113 ( 3 3 5 ( %#{{7399;$;#'$**S/#:D#'::~("6#XDDK$N(2!*$)!*/ON6g&  >CI sf/FA?F!F(F2F#F6,F"F F FF FF FG.GGcF|j|j|j|5}t|dr3|j t t |jcdddSt t |jcdddS#1swYyxYw)z Downloads the diff with the given sequence number and returns it as a byte sequence. Throws an :code:`requests.HTTPError` if the file cannot be downloaded. contentN) rcrW get_diff_urlrrrrrread)r=rresps r#rmz ReplicationServer.get_diff_blockst ]]4,,T->->s-CD EtY'%%'C. F ETYY[)F E Es6B0BB cb||jdzSd|j|dz |dzdz |dzfzS)z Returns the URL of the state.txt files for a given sequence id. If seq is `None` the URL for the latest state info is returned, i.e. the state file in the root directory of the replication service. z /state.txtz%s/%03i/%03i/%03i.state.txt@B)r8r=rs r#rzReplicationServer.get_state_urlsH ;<<,. ., cGmcGmt-CS4ZPQ Qr"cVd|j|dz |dzdz |dz|jfzS)zE Returns the URL to the diff file for the given sequence id. z%s/%03i/%03i/%03i.%srr)r8r,rs r#rzReplicationServer.get_diff_urls<& g g 5sTz   r")zosc.gz)r-N)r-r*)NN)NrjTN)NTNNN)FF)Nr)!rrrrrr>r@rHrrMrQrUrVrWrcrrr%r|rboolrrr rrr rrrlrmrrr!r"r#r*r*2soLCLCLtL   //S/T/88 (:(:8'J..'3'.FJ.2MCcMCXc]MC&smMC7?7OMC`/348,0);)#)&sm))-1)%SM)6>c])X7;;?IM7;48 I(#I(I(sI(&.smI(48I(,4GCH4E+FI((0} I( %-SM I( >FeCQTHo=V I(X7<@EZ&r{{Z&/3Z&9=Z&JRSV-Z&x.(3-..XVbMc.` *# *# * Q# Q3 Q   r"r*)(rtypingrrrrrrr r urllib.requestr\rU urllib.errorr r r contextlibr mathr rErequests.adaptersr urllib3.utilrosmiumrrrrrlogging getLoggerrq addHandler NullHandlerrr%r*r!r"r#rsRRR#!%)0g #"w""$%@:@7Z7@ @ r"