Online advertising platforms thrive due to the customizable audiences they offer advertisers. However, recent studies show that ad gets shown in a discriminatory manner with respect to socially salient characteristics such as gender and race crossing ethical and legal boundaries. This can occur either directly, or inadvertently via spillover effects. Towards mitigating this, we propose a constrained optimization framework for revenue optimal single-item ad auctions that allows one to control of the distribution across relevant characteristics of each ad's audience. Building on Myerson's classic work, we first characterize what such optimal auction mechanism look like for a large class of fairness constraints. Finding the parameters of this optimal auction, however, turns out to be a non-convex problem. We show how this non-convex problem can be reformulated as a more structured problem with no saddle points or local-maxima; allowing us to develop a gradient-descent-based algorithm to solve it. Our empirical results on the A1 Yahoo! dataset demonstrate that our algorithm can obtain uniform coverage across different user attributes for each advertiser at a minor loss to the revenue of the platform, and a small change in the total number of advertisements each advertiser shows on the platform.