Hi all,
I am written a Python 3.6 ETL process where I am searching and then translating the 42000 products from a csv file.
The whole ETL process is taking more than 10 hours to complete.
I tried to wrap my code in multiprocessing block like below-
It seems I have not put the multiprocessing block in a right way.
So can anyone please see it and tell me what I am doing wrong here?
I am written a Python 3.6 ETL process where I am searching and then translating the 42000 products from a csv file.
The whole ETL process is taking more than 10 hours to complete.
I tried to wrap my code in multiprocessing block like below-
def process_product_data(product_df, language_code): logger.info("Processing of product data is started...") translated_product_combind = [] file_path = "../output/output_" + language_code + ".csv" with Pool(4) as p: logger.info("multiprocessing is started...") translated_product = p.map(fetch_product_details, product_df) translated_product_combind.append(translated_product) logger.info("multiprocessing is done in {} seconds".format(time.time() - start_time)) generate_csv(file_path,language_code, translated_product_combind) logger.info("Processing of product data is completed in {} seconds".format(time.time() - start_time)) return translated_product_combind
def fetch_product_details(product,language_code): check_flg = False title = product.Title.replace('%', '') logger.info('Product Title is: {}'.format(title)) desc = product.Description logger.info('Product Description is: {}'.format(desc)) product_url = search_product(language_code, title) logger.info("Product Url is: {}".format(product_url)) if product_url != 'Not Found': translated_product, check_flg = translate_data_web(product_url) if (check_flg == False): translated_product = translate_data(language_code, title, desc) else: translated_product = translate_data(language_code, title, desc) return translated_productIssue is when I run the script it got stuck in this line and there is no any error in logs-
multiprocessing is started...After this my next function(translated_product_combind()) is not called.
It seems I have not put the multiprocessing block in a right way.
So can anyone please see it and tell me what I am doing wrong here?