How is affirmative action regulated?

I know that both Kennedy and Johnson came up with executive orders that stated federal contractors must make an extra effort to employ and treat minorities. How is this regulated? Is it only if a person feels that they have been discriminated against that a company can get in trouble?