model.fit and model.predict errors

model.fit and model.predict errors - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: model.fit and model.predict errors (/thread-41069.html)

model.fit and model.predict errors - hatflyer - Nov-06-2023

Having 2 issues. First, when I have test images not 150 x 150, it crashes, even though I have the line: img.resize((150,150)) before the line that causes the crash, "result = model.predict(img)"

Error message is- ValueError: Input 0 of layer "sequential" is incompatible with the layer: expected shape=(None, 150, 150, 3), found shape=(None, 499, 381, 3)

Second crash occurs during the execution of this line, "model.fit". It sometimes crashes part way through the epochs. It displayed too many lines of error messages to post here, so here are the first and last messages. I can post more of them if needed -

tensorflow/core/framework/op_kernel.cc:1816] UNKNOWN: UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x00000203FDBB87C0>

last line error message - PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x00000203FDBB87C0>
[[{{node PyFunc}}]]
[[IteratorGetNext]] [Op:__inference_train_function_1608]

Any idea why? I have plenty of RAM and SSD.

Thanks.

from keras.models import load_model, Sequential
from keras.layers import Conv2D,Flatten,Activation,Dense,MaxPooling2D,Dropout
from tensorflow.keras import optimizers
from tensorflow.keras.preprocessing.image import ImageDataGenerator, img_to_array
from PIL import Image
import numpy as np
import os
from sklearn.metrics import accuracy_score
from keras.preprocessing.image import ImageDataGenerator

trainRescale = ImageDataGenerator(rescale=1./255)
img_width = 150
img_height = 150

traindata = trainRescale.flow_from_directory('dog-vs-cat-classification/train', target_size = (150,150), batch_size = 32,  class_mode = 'binary')

model=Sequential()

model.add(Conv2D(32,(3,3),activation='relu',input_shape=(img_width,img_height,3)))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(64,(3,3),activation='relu'))         
model.add(MaxPooling2D(pool_size=(2,2)))                                                                                                         
model.add(Conv2D(64,(3,3),activation='relu')) 
model.add(MaxPooling2D(pool_size=(2,2))) 
model.add(Flatten())
model.add(Dense(64,activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1,activation='sigmoid'))

model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])

model.fit(traindata, steps_per_epoch = 8, epochs = 20) 

model.save_weights('model_weights.h5')
model.save('model_keras.h5')

testImage = os.listdir('dog-vs-cat-classification/test/')

for image in testImage:
    img = Image.open('dog-vs-cat-classification/test/' + image)
    img.resize((150,150))
    print("test image", testImage)
    img = img_to_array(img)
    img = np.expand_dims(img, axis=0)

    print("pre-predict img", img)
    result = model.predict(img)

    if result[0][0] >=0.5:
        prediction = 'dog'
    else:
        prediction = 'cat'    

print("image", image, "is a: ", prediction)

RE: model.fit and model.predict errors - Larz60+ - Nov-07-2023

Please post the actual error traceback, unaltered and complete as it contains very valuable debugging information
Use bbcode error tags (or circle X icon above)

RE: model.fit and model.predict errors - hatflyer - Nov-07-2023

Here is the first 1:

Error: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras 
format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(
Traceback (most recent call last):
  File "c:\Users\Jim\Documents\Pyro\renderbucket 1 for post .py", line 42, in <module>
    result = model.predict(img)
             ^^^^^^^^^^^^^^^^^^
  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\Users\JIMSCH~1\AppData\Local\Temp\__autograph_generated_file9q7ecm7e.py", line 15, in tf__predict_function
    retval_ = ag__.converted_call(ag__.ld(step_function), (ag__.ld(self), ag__.ld(iterator)), None, fscope)
    ^^^^^
ValueError: in user code:

    File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\engine\training.py", line 2341, in predict_function  *
        return step_function(self, iterator)
    File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\engine\training.py", line 2327, in step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\engine\training.py", line 2315, in run_step  **
        outputs = model.predict_step(data)
    File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\engine\training.py", line 2283, in predict_step
        return self(x, training=False)
    File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 70, in error_handler
        raise e.with_traceback(filtered_tb) from None
    File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\engine\input_spec.py", line 298, in assert_input_compatibility
        raise ValueError(

    ValueError: Input 0 of layer "sequential" is incompatible with the layer: expected shape=(None, 150, 150, 3), found shape=(None, 375, 500, 3)

Error:[b]Second [/b]1:

8/8 [==============================] - 2s 237ms/step - loss: 0.6899 - accuracy: 0.5977
Epoch 14/70
4/8 [==============>...............] - ETA: 0s - loss: 0.6612 - accuracy: 0.6484Traceback (most recent call last):
  File "c:\Users\Jim\Documents\Pyro\renderbucket 1 for post .py", line 32, in <module>
    model.fit(traindata, steps_per_epoch = 8, epochs = 70)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\engine\training.py", line 1742, in fit
    tmp_logs = self.train_function(iterator)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\util\traceback_utils.py", line 150, in error_handler
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\polymorphic_function\polymorphic_function.py", line 825, in __call__
    result = self._call(*args, **kwds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\polymorphic_function\polymorphic_function.py", line 857, in _call
    return self._no_variable_creation_fn(*args, **kwds)  # pylint: disable=not-callable
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\polymorphic_function\tracing_compiler.py", line 148, in __call__
    return concrete_function._call_flat(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\polymorphic_function\monomorphic_function.py", line 1349, in _call_flat
    return self._build_call_outputs(self._inference_function(*args))
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\polymorphic_function\atomic_function.py", line 196, in __call__
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
n\eager\context.py", line 1457, in call_function
    outputs = execute.execute(
              ^^^^^^^^^^^^^^^^
  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\execute.py", line 53, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
2023-11-07 15:52:38.626419: W tensorflow/core/kernels/data/generator_dataset_op.cc:108] Error occurred when finalizing GeneratorDataset iterator: FAILED_PRECONDITION: Python interpreter state is not initialized. The process may be terminated.
         [[{{node PyFunc}}]]
PS C:\Users\Jim\Documents\Pyro> cd "c:/Users/Jim/Documents/Pyro"
PS C:\Users\Jim\Documents\Pyro> & "C:/Users/Jim/AppData/Local/Microsoft/WindowsApps/python3.11.exe" "c:/Users/Jim/Documents/Pyro/renderbucket 1 for post .py"
Found 25000 images belonging to 2 classes.
2023-11-07 15:52:44.982197: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE SSE2 SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Epoch 1/70
8/8 [==============================] - 3s 233ms/step - loss: 0.8414 - accuracy: 0.4961
Epoch 2/70
8/8 [==============================] - 2s 233ms/step - loss: 0.6929 - accuracy: 0.5117
Epoch 3/70
8/8 [==============================] - 2s 235ms/step - loss: 0.6940 - accuracy: 0.4805
Epoch 4/70
8/8 [==============================] - 2s 232ms/step - loss: 0.6894 - accuracy: 0.5234
Epoch 5/70
8/8 [==============================] - 2s 232ms/step - loss: 0.7046 - accuracy: 0.5156
Epoch 6/70
8/8 [==============================] - 2s 234ms/step - loss: 0.6930 - accuracy: 0.5391
Epoch 7/70
8/8 [==============================] - 2s 233ms/step - loss: 0.6906 - accuracy: 0.5508
Epoch 8/70
8/8 [==============================] - 2s 229ms/step - loss: 0.6969 - accuracy: 0.4688
Epoch 9/70
8/8 [==============================] - 2s 233ms/step - loss: 0.6930 - accuracy: 0.4922
Epoch 10/70
8/8 [==============================] - 2s 231ms/step - loss: 0.6908 - accuracy: 0.5938
Epoch 11/70
2/8 [======>.......................] - ETA: 0s - loss: 0.6806 - accuracy: 0.52502023-11-07 15:53:05.604467: W tensorflow/core/framework/op_kernel.cc:1816] UNKNOWN: UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x000001B6142F9210>
Traceback (most recent call last):

  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\ops\script_ops.py", line 268, in __call__
    ret = func(*args)
          ^^^^^^^^^^^

  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\autograph\impl\api.py", line 643, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^

  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\data\ops\from_generator_op.py", line 198, in generator_py_func
    values = next(generator_state.get_iterator(iterator_id))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\engine\data_adapter.py", line 917, in wrapped_generator
    for data in generator_fn():

  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\engine\data_adapter.py", line 1064, in generator_fn
    yield x[i]
          ~^^^

  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\preprocessing\image.py", line 116, in __getitem__
    return self._get_batches_of_transformed_samples(index_array)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\preprocessing\image.py", line 370, in _get_batches_of_transformed_samples
    img = image_utils.load_img(
          ^^^^^^^^^^^^^^^^^^^^^

  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\image_utils.py", line 423, in load_img
    img = pil_image.open(io.BytesIO(f.read()))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\PIL\Image.py", line 3280, in open
    raise UnidentifiedImageError(msg)

PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x000001B6142F9210>


4/8 [==============>...............] - ETA: 0s - loss: 0.6799 - accuracy: 0.5481Traceback (most recent call last):
  File "c:\Users\Jim\Documents\Pyro\renderbucket 1 for post .py", line 32, in <module>
    model.fit(traindata, steps_per_epoch = 8, epochs = 70)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\execute.py", line 53, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tensorflow.python.framework.errors_impl.UnknownError: Graph execution error:

UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x000001B6142F9210>
Traceback (most recent call last):

  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\ops\script_ops.py", line 268, in __call__
    ret = func(*args)
          ^^^^^^^^^^^

  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\autograph\impl\api.py", line 643, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^

  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\data\ops\from_generator_op.py", line 198, in generator_py_func
    values = next(generator_state.get_iterator(iterator_id))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\Users\Jiim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\engine\data_adapter.py", line 917, in wrapped_generator
    for data in generator_fn():

  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\engine\data_adapter.py", line 1064, in generator_fn
    yield x[i]
          ~^^^

  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\preprocessing\image.py", line 116, in __getitem__
    return self._get_batches_of_transformed_samples(index_array)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\preprocessing\image.py", line 370, in _get_batches_of_transformed_samples
    img = image_utils.load_img(
          ^^^^^^^^^^^^^^^^^^^^^

  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\image_utils.py", line 423, in load_img
    img = pil_image.open(io.BytesIO(f.read()))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\Users\Jim\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\PIL\Image.py", line 3280, in open
    raise UnidentifiedImageError(msg)

PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x000001B6142F9210>


         [[{{node PyFunc}}]]
         [[IteratorGetNext]] [Op:__inference_train_function_1608]

RE: model.fit and model.predict errors - hatflyer - Nov-08-2023

Sorry, I think I made a mistake posting. Hopefully the info needed is there.

Is how I used Resize an issue?

RE: model.fit and model.predict errors - popejose - Nov-08-2023

expected shape=(None, 150, 150, 3), found shape=(None, 499, 381, 3)

What it is telling us is that the resize is not happening correctly.

This line
img.resize((150,150))

needs to be
img = img.resize((150,150)

RE: model.fit and model.predict errors - hatflyer - Nov-08-2023

Thanks. Any thought on the other issue, why it crashes on arbitrary epochs?

These are the relevant lines of code it seems:

traindata = trainRescale.flow_from_directory('dog-vs-cat-classification/train', target_size = (150,150), batch_size = 32, class_mode = 'binary')

model.fit(traindata, steps_per_epoch = 8, epochs = 70)

My training data is in this folder:

train/dogs , and train/cats.

Is that an issue, that dogs and cats are in separate folders within train? They have the same numbers, so they are separated. I also have some data where all cats and dogs are in 1 folder, in that case with them already labelled as cats and dogs.

RE: model.fit and model.predict errors - hatflyer - Nov-10-2023

Looks like the data I have was corrupted. Now it makes it to the end, but the conclusion is not correct, even tho there is 97% accuracy.

Is there something in my code that can explain this?

Thanks.