"I get it," Clint assures him, straightening from his lean. "Nat and I have disappeared to safe houses to talk away from people a few times."

Even if Fury had read him the riot act every time, he definitely knows the need to not have authority around - even if Clint and Natasha have no authority over Matt, Clint gets that he feels like the Avengers are trying to, right now.