Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Joblib worker error
#2
Output:
b_index: 265 Finished backtesting 2024-01-24 @ 05-03-2024 01:32:15 PM - Execution time (HH:MM:SS.xx) : 00:00:08.69 b_index: 266 Finished backtesting 2024-01-25 @ 05-03-2024 01:32:16 PM - Execution time (HH:MM:SS.xx) : 00:00:08.86 b_index: 267 Finished backtesting 2024-01-26 @ 05-03-2024 01:32:17 PM - Execution time (HH:MM:SS.xx) : 00:00:08.69 b_index: 268 Finished backtesting 2024-01-29 @ 05-03-2024 01:32:17 PM - Execution time (HH:MM:SS.xx) : 00:00:08.65 b_index: 269 Finished backtesting 2024-01-30 @ 05-03-2024 01:32:18 PM - Execution time (HH:MM:SS.xx) : 00:00:08.47 b_index: 270 Finished backtesting 2024-01-31 @ 05-03-2024 01:32:19 PM - Execution time (HH:MM:SS.xx) : 00:00:08.49 b_index: 271 Finished backtesting 2024-02-01 @ 05-03-2024 01:32:20 PM - Execution time (HH:MM:SS.xx) : 00:00:08.53 b_index: 272 Finished backtesting 2024-02-02 @ 05-03-2024 01:32:21 PM - Execution time (HH:MM:SS.xx) : 00:00:08.81 b_index: 273 Finished backtesting 2024-02-05 @ 05-03-2024 01:32:21 PM - Execution time (HH:MM:SS.xx) : 00:00:08.62 b_index: 274 Finished backtesting 2024-02-06 @ 05-03-2024 01:32:22 PM - Execution time (HH:MM:SS.xx) : 00:00:08.73 b_index: 275 Finished backtesting 2024-02-07 @ 05-03-2024 01:32:23 PM - Execution time (HH:MM:SS.xx) : 00:00:08.41 b_index: 276 Finished backtesting 2024-02-08 @ 05-03-2024 01:32:24 PM - Execution time (HH:MM:SS.xx) : 00:00:08.65 b_index: 277 Finished backtesting 2024-02-09 @ 05-03-2024 01:32:24 PM - Execution time (HH:MM:SS.xx) : 00:00:08.47 b_index: 278 Finished backtesting 2024-02-12 @ 05-03-2024 01:32:25 PM - Execution time (HH:MM:SS.xx) : 00:00:08.31 b_index: 279 Finished backtesting 2024-02-13 @ 05-03-2024 01:32:26 PM - Execution time (HH:MM:SS.xx) : 00:00:08.42 b_index: 280 Finished backtesting 2024-02-14 @ 05-03-2024 01:32:26 PM - Execution time (HH:MM:SS.xx) : 00:00:08.26 /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/joblib/externals/loky/process_executor.py:752: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak. warnings.warn( b_index: 281 Finished backtesting 2024-02-15 @ 05-03-2024 01:32:27 PM - Execution time (HH:MM:SS.xx) : 00:00:08.55 b_index: 282 Finished backtesting 2024-02-16 @ 05-03-2024 01:32:28 PM - Execution time (HH:MM:SS.xx) : 00:00:08.50 b_index: 283 Finished backtesting 2024-02-20 @ 05-03-2024 01:32:29 PM - Execution time (HH:MM:SS.xx) : 00:00:08.59 b_index: 284 Finished backtesting 2024-02-21 @ 05-03-2024 01:32:30 PM - Execution time (HH:MM:SS.xx) : 00:00:08.61 b_index: 285 Finished backtesting 2024-02-22 @ 05-03-2024 01:32:31 PM - Execution time (HH:MM:SS.xx) : 00:00:08.51 b_index: 286 Finished backtesting 2024-02-23 @ 05-03-2024 01:32:32 PM - Execution time (HH:MM:SS.xx) : 00:00:08.94 b_index: 287 Finished backtesting 2024-02-26 @ 05-03-2024 01:32:33 PM - Execution time (HH:MM:SS.xx) : 00:00:08.99 b_index: 288 Finished backtesting 2024-02-27 @ 05-03-2024 01:32:33 PM - Execution time (HH:MM:SS.xx) : 00:00:08.96 b_index: 289 Finished backtesting 2024-02-28 @ 05-03-2024 01:32:34 PM - Execution time (HH:MM:SS.xx) : 00:00:08.83 b_index: 290 Finished backtesting 2024-02-29 @ 05-03-2024 01:32:35 PM - Execution time (HH:MM:SS.xx) : 00:00:08.77 b_index: 291 Finished backtesting 2024-03-01 @ 05-03-2024 01:32:35 PM - Execution time (HH:MM:SS.xx) : 00:00:08.75 b_index: 292 Finished backtesting 2024-03-04 @ 05-03-2024 01:32:36 PM - Execution time (HH:MM:SS.xx) : 00:00:08.88 b_index: 293 Finished backtesting 2024-03-05 @ 05-03-2024 01:32:38 PM - Execution time (HH:MM:SS.xx) : 00:00:08.89
So above is another run of joblib. I inserted a variable b_index to be shown in the output - as a reminder my joblib calls function do_backtest() which returns a dataframe - and b_index is inserted into one of the columns of the dataframe. So I thought that if a joblib worker fails, then it does not re-execute and you lose the data - but I guess I was wrong? Every b_index value in the vicinity is executed, and in the resultant dataframe b_index 280 has all the required values. I know that joblib sometimes does not execute all the items in order, my machine has 16 threads and I checked all the b_indexes from 260-300 just to be safe in my resultant dataframe and all the data is there... so even though I got the message, it processed all the data...

But I still don't know what triggers this message, or does it always still process and give me all the data or sometimes not...
Reply


Messages In This Thread
Joblib worker error - by sawtooth500 - May-03-2024, 05:30 PM
RE: Joblib worker error - by sawtooth500 - May-03-2024, 06:44 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  About command 'joblib.load' middlestudent 4 1,045 Sep-25-2023, 05:56 AM
Last Post: middlestudent
  Suspending worker threads olderguy 1 2,091 Apr-21-2021, 10:28 AM
Last Post: Larz60+
  Receive data from Redis RQ worker process freak14 0 1,962 Jul-15-2019, 12:39 PM
Last Post: freak14
  Sending a custom signal from a worker mhc 4 4,908 Jul-03-2018, 04:05 PM
Last Post: mhc
  Good way to have a worker queue accessible by multiple process? cheater 2 2,574 Dec-21-2017, 09:30 PM
Last Post: wavic
  Worker thread? micko 1 3,112 Feb-27-2017, 09:52 AM
Last Post: Ofnuts

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020