My mysql server is attacked heavily to one site on my server and make whole server load extremely high up to 120 and memory usage up to 90%
My server is dual opteron 244 with 4GB ram.
Any idea to prevent Mysql attack?
Are they attacking the mysql server directly or using one of the insecure php scripts you have in your server?
If it is the former case, make sure that you don't have wild card entries under the hostname allowed for the database. If it is through some insecure php scripts, update the scripts, tighten php by disabling dangerous functions and install mod_security. These will add extra layers of security.
If you do not know how to do it, hire some good admin to do it.
for example "SELECT * FROM Customers WHERE Name=" + variable, is the format in which you write your business logic. and if somebody passes variable as "'xyz'; delete from Customers" your entire query becomes
"SELECT * FROM Customers WHERE Name='xyz'; delete from Customers"
which is the way how someone can use search module or use any of your module to inject more dangerous sql and harm your server. Make sure phpbb forum is sql injection free, or if its not then disable the search till the time you resolve this issue.