Cython, Pandas, and Chained Assignment

[sawtooth500]

So have a script called backtest.pyx. Script works fine with no errors/warnings.

Now I compliled the entire script using cython and I have the pyd file backtest.cp312-win_amd64.pyd

Next I have a script called test.py where the code in there is just:

import backtest

So I run test.py, and the cython compiled version of my script runs. FYI I did not make any cython specific optimizations to my backtest.pyx script, I'm just compiling the script as-is for nowsince this is the start of my cython journey.

When the compiled version of the script runs, I get a bunch of warnings:

Error:
C:\Users\thpfs\documents\python\cython\test.py:1: FutureWarning: ChainedAssignmentError: behaviour will change in pandas 3.0!
You are setting values through chained assignment. Currently this works in certain cases, but when using Copy-on-Write (which will become the default behaviour in pandas 3.0) this will never work to update the original DataFrame or Series, because the intermediate object on which we are setting values will behave as a copy.
A typical example is when you are setting values in a column of a DataFrame, like:

df["col"][row_indexer] = value

Use `df.loc[row_indexer, "col"] = values` instead, to perform the assignment in a single step and ensure this keeps updating the original `df`.

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

So I want to fix the chained assignments, so in my original, uncompiled script backtest.pyx I add

pd.set_option('mode.chained_assignment', 'raise')

This script still runs fine, no warnings/errors raised. I've scanned my code, I can't find any chained assingments. But cython compiled still throws warnings for chained assignments...

Now I suppose I could just try and silence those warnings. But I'd rather fix the core issue. I do need to do something because I need to run my script in parallel using joblib - right now the cython compiled version just gets hung when I'm using the joblib code, I'm guessing that may be because of the warnings thrown. All of the above examples was me using a for loop to run a function in series instead of using joblib to run in parallel.

What is the solution?

[sawtooth500]

So with some further sleuthing I found this to be the offending code block:

print('start time conversion')
    input()
    #Insert the time only value into backtestdf
    backtestdf['entry_time_only'] = pd.to_datetime(backtestdf['entry_time']).dt.time   
    print('end time conversion')
    input()    
    

That line throws me:

Error:
C:\Users\thpfs\documents\python\cython\test.py:4: FutureWarning: ChainedAssignmentError: behaviour will change in pandas 3.0!
You are setting values through chained assignment. Currently this works in certain cases, but when using Copy-on-Write (which will become the default behaviour in pandas 3.0) this will never work to update the original DataFrame or Series, because the intermediate object on which we are setting values will behave as a copy.
A typical example is when you are setting values in a column of a DataFrame, like:

df["col"][row_indexer] = value

Use `df.loc[row_indexer, "col"] = values` instead, to perform the assignment in a single step and ensure this keeps updating the original `df`.

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

  import backtest
end time conversion

But that is not chained assignment, or is it? Perhaps I don't understand chained assignment all that well if it is... still very strange how this is no problem in the regular python script by the cython compiled script has an issue with this.

[sawtooth500]

temptimedf = pd.DataFrame()
    print('problem start')
    temptimedf['entry_time_only'] = pd.to_datetime(backtestdf['entry_time']).dt.time
    print('problem end')
    backtestdf['entry_time_only'] = temptimedf['entry_time_only']
    print('now problem')

I tried the above to narrow the problem further....

backtestdf['entry_time_only'] = temptimedf['entry_time_only']

That's the line the cython has a problem with. I still have no idea as to the root cause...

[deanhystad]

Hard to tell when you just offer a peek into the code. How is temptimeddf created?

[sawtooth500]

backtestdf['entry_time_only'] = pd.to_datetime(backtestdf['entry_time']).dt.time

That was the original offensive line of code that triggered the chained assignments warnings.

    temptimedf = pd.DataFrame()
    print('problem start')
    temptimedf['entry_time_only'] = pd.to_datetime(backtestdf['entry_time']).dt.time
    print('problem end')
    backtestdf['entry_time_only'] = temptimedf['entry_time_only']
    print('now problem')

Here you see an attempt that I made to circumvent the issue. In this attempt,
backtestdf['entry_time_only'] = temptimedf['entry_time_only']
is the offending line.

So in backtestdf I have a col called 'entry_time' and this col has datetime objects in YYYY-MM-DD HH:MM:SS format. I need to append a new col to backtest DF that just has the time portion, HH:MM:SS so a time object.

Cython does not like this for some reason.