Python Forum
FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries
#5
See the below image. So that's an example of the edge case that I have highlighted. So as you see the price for AMD at this point is in time is hovering around $181.80 - then you see these three prints around $180.80 - so they are late prints that need to be removed.

I actually do just use a shift to compare columns, and that works great where the late print is a singular print. For each stock, I define a certain value that if the price moves greater than X in a single tick, that's a possible late print. So in this case let's use a greater than $0.18 movement in a single tick is a red flag.

I need to check:

1. If the abnormal movement is positive or negative - because if the initial movement is positive, for example, the corresponding movement after the abnormality should be negative. If we have a positive flag followed by a positive flag... that could indicate a very aggressive, but legit, price move so we don't want to filter out that. In the screenshot I provided the initial movement is negative, so the return was positive.

2. The price of the late prints could be the same, or it could be different.

3. If I start using shift, well by how many rows do I shift? This is an example of 3 ticks of late prints, but the actual count is unknown. I've seen as many as 13 late ticks in a row, but that doesn't mean that's 13 is an upper limit, that's just what I have seen.

It's easy to create a filter that filters out just a specific case, but I need to filter out all cases while keeping the false positives as low as possible.

[Image: AMD-late-prints.png]
Reply


Messages In This Thread
RE: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries - by sawtooth500 - Apr-22-2024, 01:33 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries i sawtooth500 3 2,241 Mar-22-2024, 03:08 AM
Last Post: deanhystad
  String concatenation in SQL update statement hammer 3 1,631 Feb-24-2022, 08:00 PM
Last Post: hammer
  f string concatenation problem growSeb 3 2,350 Jun-28-2021, 05:00 AM
Last Post: buran
  Concatenation ?? ridgerunnersjw 1 1,789 Sep-26-2020, 07:29 PM
Last Post: deanhystad
  FutureWarning: pandas.util.testing is deprecated buunaanaa 3 5,219 May-17-2020, 07:43 AM
Last Post: snippsat
  Combining two strings together (not concatenation) DreamingInsanity 6 3,251 Mar-29-2019, 04:32 PM
Last Post: DreamingInsanity
  Handling null or empty entries from Entry Widget KevinBrown 1 2,377 Mar-17-2019, 04:22 PM
Last Post: perfringo
  append elements into the empty dataframe jazzy 0 2,164 Sep-26-2018, 07:26 AM
Last Post: jazzy
  Regarding concatenation of a list to a string Kedar 2 22,874 Aug-19-2018, 12:57 PM
Last Post: ichabod801

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020