After a bit of testing I have resolved the issue. It appears that if you are a warrior and a reaver there is a passive ability called Blood Frenzy. It gives you a buff of 5% damage for every 10%HP loss. Since the infinite HP button set you HP at 5000 and you maximum is below that, about 800 in my case. I'm assume that since your hp is over it causes the formula to go the opposite way and penalize your damage. There are two other abilities that give the same bonus Devour and Dragon-Rage. That are on use talents that didn't seem to cause the same issue after I redistributed talent point and left out Blood Frenzy. If in the future the same problem occurs i will just drop those ones as well.
I figure it worth posting my findings in case others have the same issues.
