Do you think the number one reason Clint lies awake at night is because he was once compelled by Loki to tell him everything and so there is an angry, powerful, Norse Godish Alien out there, with a personal vendetta against the Earth and the Avengers 

And he knows about his children.