When it comes to the concept of Executive Privilege, we tend to think of more modern Presidents who are perhaps in trouble (Nixon/Clinton) or just in a tiff with Congress (Obama/Bush). In fact, the whole concept goes back much further, when the first real populist President refused to hand over to Congress a letter he himself had written to explain to his Cabinet why he wanted to withdraw US funds from the 2nd Bank of the United States.
The issue was so divisive, and Jacksons refusal so infuriating to the Senate Leader Henry Clay, that it caused the formation of the Whig Party and led to the idea that a Presidential Election is a referendum on a given policy.
It also made a clear reminder to the US Senate that it was not “Accusers, witnesses, counsel and judges” of a President. And it established the idea that the President has a responsibility to choose advisors and cabinet members that he wants to carry out his policies, not Congress.