I have driven here for close go 15 years. It's a little funky, but not as bad as you have been advised. The route to Ica is pretty decent, and even at night you should not have a problem. The problem is other traffic, not the thieves. That said, there are other places in Peru where I would not drive at night, due to possibility of crime, but the Panamericana should be just fine.
Busses here are comfortable and inexpensive, by the way. It's another good option. Once you get to where you are going you can cab, also cheap.
But if you want to drive, and you know hw to use Google maps, and have a spirit of adventure, go for it!