wh2 ddlmZmZmZddlmZddlmZmZddl Z ddl m Z ddl mZddlmZmZmZddlmZmZdd lmZmZddlmZdd lmZddl mZdd lmZmZdd lmZddlZddlZddlZdd lm Z ddl!m"Z"ddl#Z#ddl$m%Z%e"ejLdZ'e%jPZ(e%jRZ)e%jTZ*e%jVZ+e%jXZ,e#jZe(e)e*e+dZ.de/de0defdZ1dede/de/defdZ2ejLdZ3ejLdZ4ejLdZ5ejLdZ6ejLdZ7ejLd Z8d!edefd"Z9 d'dede/d#e/d$e/d%e/f d&Z:y)()Depends UploadFile HTTPException)Session)funccaseN) Datasource)get_current_user)process_csv_datacreate_datasource save_csv_file)StoreBranch)rr)get_db)ExternalDatasourceCreateInternalDatasourceCreate)create_external_data_source)urlparse) load_dotenv)settingsCSV_WEBHOOK_URLT)app_idkeysecretclusterssl datasource_id file_pathdbcBKt|||td{S7w)zP Controller for processing CSV file and sending reviews to the webhook. N)r rrrrs =/var/www/html/hubwallet-dev/src/apps/datasource/controller.pyprocess_csv_reviewsr#-s""mYP PP Ps filestore_id branch_idc .K|jtjtj|k(tj|jk(j }|st ddddd|jtjtj|k(tj|k(tj|jk(j }|st dddddt|} tj|}hd} | jt|jst dddd d t)||||} t+t-| j.t'||d{} d} t0j3| d| j.ddddddddd| j.t'| j4|| ddS#tjj $rt dd tjj"$rt dd t$$r} t d d t'| d} ~ wwxYw7#t$$r"} t d dd dt'| dd} ~ wwxYww)zB Controller for handling CSV file uploads and processing. Fz!Unauthorized access to this storestatuscodemessage status_codedetail"Unauthorized access to this branch>datenameratingcommentzFCSV format is incorrect. Required columns: name, comment, rating, datezCSV file is empty or invalid.zCSV file could not be parsed.zError reading CSV: Nr!zError processing CSV: import-updateprogress-updaterfetchingFetching recordsr processedtotalpercentr* status_msg event_namedataTz0CSV uploaded and processing started successfully)r snapshot_idfile_urlprocessing_resultr*r+r,rB)queryrfilterr%user_idfirstrrr&r pdread_csvissubsetsetcolumnserrorsEmptyDataError ParserError Exceptionstrr r#intds_id pusher_clienttriggerrD)r$r%r&ruserstorebranchrdfrequired_columnsedatasource_entryresult channel_names r"upload_csv_filerc4s HHUO " "5>>X#=u}}PTP\P\?\ ] c c eE #S=`a  XXf  $ $I%8#$,,& eg  #S=ab  d#IT [[ #@((RZZ9"' ##k :")Xy)L  *.445)n  #L$-33 ,   E-33/;;<!!'   O 99 # #U4STT 99 U4STT T6I#a&4RSST  #S?UVYZ[V\U]=^_   scDJAG)J#-I'I%I'AJ)AI"II""J%I'' J0J  JJBRIGHT_DATA_TOKENWEBHOOK_STATIC_TOKENWEBHOOK_BASE_URLBRIGHT_DATASET_IDGOOGLE_DATASET_IDBRIGHT_DATA_API_URLrBc t|jjdvr t}d}nt fddDrt }d}t t|j}|jxsd}|jdsad}|j|j}td |jtd |||_ntd d d d ddtdd} td} td} || dtdd| dd} dt|jig} t!j"t$| | | }|j'tdtd|j(tt+j,|j+d|j+}t5|t6rd$|vrtd%tdd dd&d|d$}t9||||}d'}t:j=|d(|j>d)d)d)d*d+d,-d.d/d0|j>|j@|jd1d2S#t j.j0$r0}td|tdd dd t|dd!}~wt j.j2$rtd"tdd dd#dwxYw)3zd Controller function to handle adding external data sources and triggering Bright Data API. yelpc3&K|]}|v yw)N).0keywordurl_strs r" z+add_external_data_source..s JGW  Js)googlemapszg.corrz google.comzwww.google.com)netloczOriginal Google URL: zCorrected Google URL: r5FzUnsupported data source URLr)r-zBearer zapplication/json) Authorizationz Content-Typez/webhookz/notifyjsontrue) dataset_idendpoint auth_headerformatuncompressed_webhooknotifyinclude_errorsurl)headersparamsrwz% ===== Bright Data API Response =====z Status Code: )indentz&Error making Bright Data API request: r6z Bright Data API request failed: Nz,Error: Bright Data API returned invalid JSONz%Bright Data API returned invalid JSONrDz,Error: Invalid response from Bright Data APIz%Invalid response from Bright Data APIr7r8rr9r:r;r@TrCz'External data source added successfully)rrDrrG)!rUrlowerrganyrhrhostnameendswith_replacegeturlprintrrdrfrerequestspostriraise_for_statusr.rwdumps exceptionsRequestExceptionJSONDecodeError isinstancedictrrXrYrWrD)rBrrZselected_dataset_id source_type parsed_urlrcorrected_hostname corrected_urlr webhook_url notify_urlr data_payloadresponsebright_data_responsereq_errrDr`rbrps @r"add_external_data_sourcers$((m!!#G/ J/I J J/ c$((m, &&,"  .!1 &//7I/JQQSM )$((4 5 *=/: ;$DH#S=Z[  ##4"56*G &&h/K$%W-J* !5 67 & FCM*+L ==!4gf[gh!!# 67 h22345 djj34'}} *D 1]J^5^ <=#S=de  '}5K34T;O"L$-33 ,   <-33+77#''  S    / /  6wi@A#S?_`cdk`l_m=no     . .  <=#S=de   s6BIK 8+J##=K rJpage page_sizec4|jtjtj|k(tj|k(j }|st ddddd|j}ttjdk(dftjdk(d ftjd k(d fd }|jtjtj|k(jtjj|}|||j} nO|j} |j!|dz |zj#|j} | |zdz |z} | sddddS| D cgc]i} | j$| j|| j&| j(| j| j*| j,| j.| jd k} } ddd| d}| |   ||d|d<|Scc} w)aHRetrieve datasource records for a given branch_id that belongs to the logged-in user. - If pagination parameters (`page` & `page_size`) are provided, return paginated results. - Otherwise, return the full list. - Sorting Order: Status (`completed` > `pending` > `failed`), then `created_at` (latest first). r(Fr0r)r- completedpendingfailedr)else_iz No records found for this branch) r%r& branch_namerlast_fetch_datelast_fetch_statusrWrrD created_atTrCz)Datasource records retrieved successfullyrG) total_records total_pages current_pager pagination)rHrrIr&rJrKrrrr rorder_byrdescallcountoffsetlimitr%rrrWrrD)rr&rJrrr\r status_orderdatasource_querydatasource_entriestotal_datasourcesrentryformatted_datars r"fetch_datasource_by_branchrs<XXf  $ $V%5%5%BFNNV]D] ^ d d fF #S=ab  $$K  % % 4a8  % % 2A6  % % 115 Lxx +22 )h""$ |y(-113-224.44dQh)5KLRRS\]aac)94q8YF  9  (( &99$44!&!8!8[[ ,, ,,** N"> H I1.& " "  OAsA.H)NN);fastapirrrsqlalchemy.ormr sqlalchemyrrpandasrLsrc.apps.datasource.modelsr src.apps.auth.controllerr src.apps.datasource.servicesr r r src.apps.stores.modelsrr src.utils.dbrsrc.apps.datasource.schemasrrrrosrw urllib.parserdotenvrpushersrc.utils.settingsrgetenvrrrrr batch_sizePusherrXrVrUr#rcrdrerfrgrhrirrrmr"rs66"!15ZZ0*"5ZD ! ' "))-. ll           QSQSQgQY YcYcYwYBBII12 ryy!78299/0BII12BII12bii 56z "zzH ````  `  `r