set.difference of two list gives empty result

[wardancer84]

hi,

i want to get the difference from two list of strings (using sets) but i always get an empty result. funny enough this worked until today and stopped working overnight without me changing something.

the function:

def delete_dead_ips():
    # delete ip's that exists in netbox but not in aix registry
    aix_reg_list = []
    for i in net_info:
        address,mask,broadcast,dns_name = i.split('/')
        mask_cidr= IPv4Network(address + "/" + mask, strict=False).prefixlen
        addr_recalc = address + "/" + str(mask_cidr)
        aix_reg_list.append(addr_recalc)
    print('items in aix registry list: "%s"' % len(aix_reg_list))
    print('items in aix_ips (netbox) list: "%s"' % len(aix_ips))
    #print("\n".join(map(str, aix_reg_list)))
    #print("\n".join(map(str, aix_ips)))
    dead_ips = list(set(aix_ips).difference(set(aix_reg_list)))
    print('items in aix registry but not in netbox "%s"' % len(dead_ips))
    if dead_ips and aix_reg_list:
        logging.info('ips in netbox but not in aix registry removed: "%s"' % ",".join(dead_ips))
        for x in dead_ips:
            all_ips.remove(x)
            aix_ips.remove(x)
            try:
                bla = nb.ipam.ip_addresses.get(q=x)
                result = nb.ipam.ip_addresses.delete([bla])
            except pynetbox.core.query.RequestError as e:
                logging.error(str(e.error))

result:

Output:
root@nimvie: /root/bin # ./netbox_import.py
2022-06-14 12:56:47,148 [INFO] netbox 3.2 connected
items in aix registry list: "1708"
items in aix_ips (netbox) list: "1710"
items in aix registry but not in netbox "0" -> this is clearly a lie

format of the lists (bog standard ip adresses):

['10.95.0.1/22', '10.95.0.1/22', '10.95.0.1/22', '10.95.0.1/22', '10.95.0.1/22', '10.95.0.1/22', '10.95.0.1/22', '10.95.0.1/22', '10.95.0.1/22', '10.95.0.1/22', '10.95.0.13/22', '10.95.0.119/22', '10.95.0.149/22', '10.95.0.211/22', '10.95.1.14/22', '10.95.1.15/22', '10.95.1.16/22', '10.95.1.18/22', '10.95.1.20/22', '10.95.1.21/22', '10.95.1.25/22', '10.95.1.48/22', '10.95.1.149/22', '10.111.160.6/24', '10.111.160.10/24']

any hints welcome, as i dont see a obvious reason for this....

wbr

chris

[deanhystad]

Why is it clearly a lie? If all items in aix_reg_list are also in aix_ips, the difference would be an empty set. a.difference(b) returns all items in a that are not in b. It does not include items in b that are not in a.
Maybe there are duplicates. is len(set(aix_ips)) == len(aix_ips)?
If I was trying to understand the problem I might do something like this:

for a in aix_ips:
    if a not in aix_reg_list:
        print(a)

[wardancer84]

this list contains 2 more items

items in aix_ips (netbox) list: "1710"

than this list

items in aix registry list: "1708"

so this 2 items should show up in the difference results IMHO

no sure if this is true, but i read somewhere that "sets" do not work well with lists that contain duplicate items which is absolutely the case here.

i tried list comprehensions instead of sets but this all gives empty results also

dead_ips = [item for item in aix_reg_list if item not in aix_ips]
dead_ips = [item for item in aix_ips if item not in aix_reg_list]

[deanhystad]

Looks like you have duplicates in aix_ips.

[wardancer84]

this diff method can handle duplicates...works well at first sight

from collections import Counter
...
dead_ips = list((Counter(aix_ips) - Counter(aix_reg_list)).elements()) # get list of whats in the first list but not in the second

mission completed