I had to hide a panel pane for certain users on my latest project. Babybrukt is a site where people can create their own ads. Buy, sell or give away used baby "stuff". The advertiser needs to know how many times his or her ad has been viewed. I created a view that gives me the data and went to the panels dashboard.

Panels visibility rules is the place you want to go. Create a new rule and give it these settings.

Visibility rule: user:Compare - First user = node author - second user = Logged in user
Grant access if user context are: Equal.

I spent almost 2 hours on this, even talked to a guy from the NL. We thought it had to be solved by php (grin). Check out the screenshot for the settings.