I was reading an article by Jean Tabaka on the 11 Ways Agile Adoptions Fail and her number 5 reason "Product Owner is Consistently Unavailable" hit a cord. In it she states
An unavailable product owner perpetuates the wait time and creates waste
Having experienced this in the past, I can't agree more.
At Ephox our Product Manager plays the role of client in our iterations. As part of his role he spends a certain amount of time travelling. In the past we tried to continue development while he was away. What we found was that, even though he was available via email and phone, his lack of availability within the office while a feature was development caused delays and in some cases wasted work.
The way we've dealt with this, especially with an increase in Product Managers recently, was to have agreement that, as the client, if you were not available your development release would stop at the end of the iteration. The team would then work on stories in the next iteration for the other product manager or spend time on process improvements within the Engineering team.