Binary File Read - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: Binary File Read (/thread-29405.html) |
Binary File Read - Aussie - Sep-01-2020 Hi, I have tried a number of examples to try and get the code below to step through correctly. The header needs to be stepped through once and the sample code at the end to run through until there are no 4 bytes left. At the moment the while does the whole lot over and over. The header is 20 bytes long and consists of values made up of different byte sizes. byte, int16 and int32. Hope you can help. Thanks, import time import struct # Header = 20 words # 32bit integer - Size # 1 byte - Variation # 1 byte - Object # Object Variation # 30 1 32-bit integer samples # 30 2 16-bit integer samples # 30 3 32-bit integer samples # 30 4 16-bit integer samples # 30 5 32-bit floating point samples # 16bit unsigned integer - Point index, analog point index being sampled # 32bit integer - Start Time EPOCH # 32bit integer - Sample Rate, the interval (in seconds) between samples of the analog point # 32bit integer - Number of Samples, the number of sample records in this file # Size = 20 + (Number of Samples) * 4 - 16bit integer # Size = 20 + (Number of Samples) * 8 - 32bit integer f = open("C:\Python\XXXXXXXXXXX", "rb") byte = f.read(4) while byte: #Step through once only #Header Section #Size 32 bit int, should be 484 byte = f.read(4) Size = struct.unpack('>i', byte)[0] # As integere print('The size is ') print(Size) #Variation 1 byte, should be 1 to 5 byte = f.read(1) Variation = int.from_bytes(byte, byteorder='big', signed=False) print('The variation is ') print(Variation) #Object 1 byte, should be 30 byte = f.read(1) Object = int.from_bytes(byte, byteorder='big', signed=False) print('The object is ') print(Object) #Point Index 16 bit int byte = f.read(2) PointIndex = int.from_bytes(byte, byteorder='big', signed=False) print('The point index is ') print(PointIndex) #StartTime 32 bit int byte = f.read(4) temp = struct.unpack('>i', byte)[0] # As integer bin(struct.unpack('>i', byte)[0]) # As binary StartTime = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(struct.unpack('>i', byte)[0])) # as time print('The start time is ') print(StartTime) #Sample Rate 32 bit int byte = f.read(4) SampleRate = int.from_bytes(byte, byteorder='big', signed=False) print('The sample rate is ') print(SampleRate) #Number of Samples byte = f.read(4) NoOfSamples = int.from_bytes(byte, byteorder='big', signed=False) print('The number of samples is ') print(NoOfSamples) #End of header section #Do until end of file, until only 4 more bytes left #Sample(s) byte = f.read(4) Sample = struct.unpack('>i', byte)[0] # As integer print(Sample) f.close() RE: Binary File Read - bowlofred - Sep-01-2020 Please put your code in python tags. We can't tell the program flow without them. Can you describe more what is happening or show the output? Right now I don't know what you mean by "the while does the whole lot over and over". RE: Binary File Read - Aussie - Sep-01-2020 Hi, I have put the code in Python tags, edited previous post. The output is below. The file has a header of 20 bytes. Broke down in various byte sizes. One byte Two bytes and four bytes The header provides setup data. The rest of the file is then sample values. Thanks, 66 The object is 220 The point index is 2220 The start time is 1970-01-17 08:17:36 The sample rate is 1121716396 The number of samples is 1441792 1121749822 The size is 1507328 The variation is 66 The object is 219 The point index is 45472 ........... The start time is 2004-07-15 17:11:17 The sample rate is 7012352 The number of samples is 1090932632 7208960 The size is 1090642939 The variation is 0 The object is 111 The point index is 0 The start time is 2004-07-12 14:54:27 The sample rate is 7471104 The number of samples is 1089652637 7536640 The size is 1090687508 The variation is 0 The object is 0 The point index is 0 Traceback (most recent call last): File "C:\Python\DNP3FileRead.py", line 55, in <module> temp = struct.unpack('>i', byte)[0] # As integer struct.error: unpack requires a buffer of 4 bytes >>> RE: Binary File Read - deanhystad - Sep-01-2020 I think this is your header (in C). unsigned int size; // 4 bytes unsigned char verification; // 1 byte unsigned char object; // 1 byte unsigned short pointindex; // 2 bytes unsigned int starttime; // 4 bytes unsigned int samplerate; // 4 bytes unsigned int samplecount; // 4 bytes // 20 bytes totalIf this is correct, and if the header starts at the very top of the file, then your program has already messed up reading the file by the end of this: f = open("C:\Python\XXXXXXXXXXX", "rb") byte = f.read(4)Why did you read 4 bytes? Isn't that supposed to be Size? I did my best to mimic what I think you are trying to do. writedata is used to make a file that I can read. readdata is what you are trying to do. Maybe there should be a readheader function and a readsamples function. But I just wanted to show you that this kind of code can be really compact and it should't be confusing at all. import struct import time def writedata(filename, data): '''Write data to a file with a header''' count = len(data) stime = int(time.time()) with open(filename, "wb") as file: buffer = struct.pack('=IBBHiII', 484, 1, 30, 0, stime, 10, count) file.write(buffer) buffer = struct.pack(f'={count}i', *data) file.write(buffer) def readdata(filename): '''Read data from a file. Data count is in the header''' with open(filename, "rb") as file: buffer = file.read(20) size, vtype, x, index, stime, srate, count = struct.unpack('=IBBHIII', buffer) print(size, x, index, stime, srate, count) buffer = file.read(4 * count) values = struct.unpack(f'={count}I', buffer) return values writedata('junk.bin', [1, 1, 2, 3, 5, 8, 13]) print(readdata('junk.bin')) I am assuming that the data in the file is in native format. If the file is a different format you can use '<' or '>' to fix any endian issues. If you are generating the file on your computer you should use '=' or '@'.I did not use the vtype or variation argument. It would be easy to check the type and unpack 16 bit (h or H) or 32 bit (i or I or maybe f?) sample values. RE: Binary File Read - Aussie - Sep-01-2020 Hi, I changed it a little bit def readdata(filename): '''Read data from a file. Data count is in the header''' with open(filename, "rb") as file: buffer = file.read(20) size, variation, objectType, pointIndex, startTime, sampleRate, count = struct.unpack('>IBBHiII', buffer) print(size, variation, objectType, startTime, sampleRate, count) buffer = file.read(4 * count) values = struct.unpack(f'={count}I', buffer) return values print(readdata('XXXXXXXXXXXXXXXX')) RE: Binary File Read - Aussie - Sep-03-2020 Hi, further to this I want to print to a csv file. The code below does create a file and prints to it. Though I want the "value" part of the writerow to an actual sample value. And each sample value to have its own row. Kind of step through. Would I use an array or list. Thanks, import struct import time import csv # Get Data out of Binary File def readdata(filename): '''Read data from a file. Data count is in the header''' with open(filename, "rb") as file: buffer = file.read(20) size, variation, objectType, pointIndex, startTime, sampleRate, count = struct.unpack('>IBBHiII', buffer) print(size, variation, objectType, startTime, sampleRate, count) buffer = file.read(4 * count) values = struct.unpack(f'={count}I', buffer) return values print(readdata('XXXXXXXXXXX')) # Create and Write to CSV File with open('XXXXXXXXXXX.csv', 'w', newline='') as file: writer = csv.writer(file) writer.writerow(["Blank1", "DateTime", "Blank2", "Value"]) RE: Binary File Read - deanhystad - Sep-03-2020 New question, start a new topic. |