Don't run code in the TIMEOUT event (check that we don't do that)
I think the following logic is fundamentally broken:
for request_event in request_iterator: request_entry = RequestEntry(request_event) # Never Run time critical mechanism inside timeout # mechanism because timeout mechanism only comes # when no other event is happening. It means under # heavy load there would not be a timeout event. if request_entry.type == "TIMEOUT": # Detect hosts that are dead and set their status # to "DEAD", and their VMs' status to "KILLED" dead_hosts = dead_host_detection() if dead_hosts: logger.debug("Dead hosts: %s", dead_hosts) dead_host_mitigation(dead_hosts)
Just run the code that should ALWAYS be executed prior to the if statement.