When taking a taxi (wherever you are: Miraflores, San Isidro, Lima etc...) always set the fare before you get in. If you try to give him whatever you think, they can rightfully do something to you and get people to do something to you.
Check to see if they text or make a call .... If they do and you don't feel comfortable you can just ask him to stop...you forgot something. period.
As for taxi drivers robbing you. Yes, it happens. It used to happen much more and it usually happens on weekends and if you are drunk. In Trujillo, it was happening much more in the past 2-3 years.
Always look at what route he takes and you can tell him whatever route to take albeit when there is traffic, they will zig zag thru side streets. The problem is, you need to know where you are going and in what direction. For example, Surquillo (also called Chicago Chico) has its sections that are dangerous and it is not too far from Miraflores. El Callao has sections that are very dangerous.....
You have to see what the taxi driver does and what route he takes.
I'm surprised that taxi crimes hasn't happened much more to the expats in living in Peru. Americans are considered easy targets because they are not familiar with the modalities of robbery and/or the areas that are bad and good. Also keep in mind that many police officers are also in on crimes.
Things I do when in Taxis in Lima (stuff I've been told by taxi drivers, family, and friends who live in Peru):
1. Never talk on the phone with the phone facing the windows. Phone should always be facing the inside of car. Its easy to snatch a phone while it is close to the window.
2. All packages go on the floor and try to make them not visible.
3. Do not count money (especially when you first come to Peru and have all twenty dollar bills etc....) while in the car.... put different amounts of money in your pockets. For example, 10/20 soles in back pocket....hundreds of soles in the front, etc... That way, you know that you will pay for 12 soles ride from your back pocket and not count huge amounts of money.
4. if you have a tablet or smart phone or camera....put it in a green/black plastic bag that the small stores give you (cover it in a shirt or something).... As others suggested, dress down but people can tell when you are a foreigner (cologne, type of gym shoes, tan type, etc...)
I know so many people and family who have been robbed of everything when they go to or leave the airport. It is always better to pay a driver for the day(15 soles/hour) or take a service that you call via cell phone.
Also, keep in mind that the taxi drivers get robbed a lot and the types of scams are unheard of in the US. For example, a couple will pick up a taxi from Jockey Plaza all dressed up and carrying bags.... they get robbed at gun point going to Bren~a or Lince.
You see the taxis in Lima have metal cages to prevent people stabbing the driver or choking them.... You see taxis with the mirrors on chains and the metal frame locks on the front and back tail and head lights. These car accessories are stolen while a taxi is in traffic and the taxi driver can't do much other than reverse and go forward....if the taxi driver gets out he will get robbed.
Couples with a baby robbing taxi drivers. Two women robbing taxi driver.
The problem is that there are many guns (now as compared to 5 years ago) in Peru.
Crime is rampant in Peru, but has gotten better .... you just have to be on the lookout and be on alert.
Remember, you pay a fare and you can tell the taxi driver whatever you want (what route to take, what are you saying on the phone, etc...). I personally rent a taxi guy for 15 soles/hour or try to find an older gentleman or someone who looks peaceful and honest.
I haven't been robbed in Peru so far, but I am cautious .... I've been everywhere I probably should not have been such as Gamarra, Polvos Azules, Las Malvinas, Callao, San Martin, etc.... at all times of the day. It is always best to be with a local or in a group of more people and don't think because you may be 6 ft, 220 lbs and athletic/young that you can't get robbed.
Be safe and take precautions......you will be fine.