As you suspect and know, there's no reason why it would take an arbitrary 5 days to give you back your own money. I've never deposited money in the bank and been told I'd have to wait 5 days to get it back out again, for instance.
Tell the agency that your conditions of returning the apartment are that the deposit is paid at the moment of or on the same day that the inspection is passed. What their specific policies are aren't really your concern. Hopefully on your contract these 5 days aren't mentioned, if they are, I'd suggest you get filming each and every corner of the apartment as evidence - though that won't really help all that much, it's something.