Upload Files to Azure Storage Container - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: Upload Files to Azure Storage Container (/thread-41321.html) |
Upload Files to Azure Storage Container - phillyfa - Dec-21-2023 Hello, I found the script online that uploads multiple JPG files. I changed it so I can upload multiple BLOB files, i.e.) csv, xls, etc. However, I am receiving the following error message when I run the script: The file names are lowercase and alphanumeric: bcp-notes-batch-0.csvHere is the script: # Replace with blob container MY_IMAGE_CONTAINER = "source_system" # Replace with the local folder which contains the files for upload LOCAL_FILE_PATH = 'I:/bcp' class AzureBlobFileUploader: def __init__(self): print("Initializing AzureBlobFileUploader") # Initialize the connection to Azure storage account self.blob_service_client = BlobServiceClient.from_connection_string(MY_CONNECTION_STRING) def upload_all_files_in_folder(self): all_file_names = [f for f in os.listdir(LOCAL_FILE_PATH) if os.path.isfile(os.path.join(LOCAL_FILE_PATH, f))] result = self.run(all_file_names) print(result) def run(self, all_file_names): with ThreadPool(processes=int(10)) as pool: return pool.map(self.upload_file, all_file_names) def upload_file(self, file_name): blob_client = self.blob_service_client.get_blob_client(container=MY_IMAGE_CONTAINER, blob=file_name) upload_file_path = os.path.join(LOCAL_FILE_PATH, file_name) content_settings = ContentSettings(content_type='application/octet-stream') print(f"Uploading file - {file_name}") with open(upload_file_path, "rb") as data: blob_client.upload_blob(data, overwrite=True, content_settings=content_settings) return file_name # Initialize class and upload files azure_blob_file_uploader = AzureBlobFileUploader() azure_blob_file_uploader.upload_all_files_in_folder() RE: Upload Files to Azure Storage Container - micseydel - Dec-21-2023 I would start with replacing the list comprehension on line 14 with a regular loop. Catch the exception, and when it happens, print out the filename with repr() in case it has weird things that aren't obvious. RE: Upload Files to Azure Storage Container - phillyfa - Dec-21-2023 These are the only two files in the directory. All characters in the name are valid:
RE: Upload Files to Azure Storage Container - deanhystad - Dec-21-2023 According to azure documentation: Quote:A path name is composed of one or more path name components (directory or file name) separated by the forward-slash (/) character. All path name components other than the last path name component denote directories. The last path name component denotes a directory or a file.Is the problem that os.path.join(LOCAL_FILE_PATH, file_name) uses '\'? RE: Upload Files to Azure Storage Container - phillyfa - Dec-21-2023 (Dec-21-2023, 06:50 PM)deanhystad Wrote: According to azure documentation: Still no luck: class AzureBlobFileUploader: def __init__(self): print("Initializing AzureBlobFileUploader") self.blob_service_client = BlobServiceClient.from_connection_string(MY_CONNECTION_STRING) def upload_all_files_in_folder(self): all_file_names = [f for f in os.listdir(LOCAL_FILE_PATH) if os.path.isfile(os.path.join(LOCAL_FILE_PATH, f))] result = self.run(all_file_names) print(result) def run(self, all_file_names): with ThreadPool(processes=int(10)) as pool: return pool.map(self.upload_file, all_file_names) def upload_file(self, file_name): try: print(repr(file_name)) # Print the repr() of the file name blob_client = self.blob_service_client.get_blob_client(container=MY_IMAGE_CONTAINER, blob=file_name) upload_file_path = os.path.join(LOCAL_FILE_PATH, file_name).replace('\\', '/') # Replace backslashes print(f"Uploading file - {file_name}") with open(upload_file_path, "rb") as data: blob_client.upload_blob(data, overwrite=True) return file_name except Exception as e: print(f"Exception uploading file - {repr(file_name)}: {e}") def get_containers(self): try: containers = self.blob_service_client.list_containers() print([container.name for container in containers]) except Exception as e: print(e.message) # Initialize class and upload files azure_blob_file_uploader = AzureBlobFileUploader() azure_blob_file_uploader.get_containers() azure_blob_file_uploader.upload_all_files_in_folder()
RE: Upload Files to Azure Storage Container - deanhystad - Dec-21-2023 I looked at the problem wrong. The \ has nothing to do with the issue because that is a local filename, not the name of the blob resource. I think our problem is the ".". From the azure docs Quote:Azure Storage resource names must follow certain rules, such as: RE: Upload Files to Azure Storage Container - Pedroski55 - Dec-22-2023 I don't know what azure storage is. If you need to change file names to suit, that can be done. I believe rsync will work under MS Windoze, but I have never tried. For syncing files I don't think rsync can be beat. Just use it on the command line, or make a bash script to copy from multiple folders. You must have write permission on the remote machine. For example, to send all files from /home/pedro/jpgs to another machine, you will be asked for your user, in this case pedro's, password. If the remote machine doesn't have the directory /home/pedro/jpgs, it will be created, as will any and all sub directories. rsync will only send (or receive) files which have changed since the last time you used rsync. This one-liner sends files over my LAN to another laptop, but I could put the address of my little cloud server. I just need to know the ip4 address of the other machine: Quote:rsync -av -e "ssh" --progress /home/pedro/jpgs pedro@192.168.1.8:/home/pedro/ You can even use rsync to send files to a usb stick! I believe -e "ssh" is redundant now, as all transfers use ssh, but I leave it there anyway! |