I have one customer that we setup nameservers for, but after 3 days I did a traceroute, and it shows dns error, reverse dns not configured...

Does anyone know how to fix this for the reverse dns to resolve to the IP

Hard to say with what you have posted, but I would say to start simple. Did you setup a PTR record for the IP address running that name server?