be not to know

Từ điển Oxford Advanced Learner 8th

be not to ˈknow idiom
to have no way of realizing or being aware that you have done sth wrong
•‘I'm sorry, I called when you were in bed.’ ‘Don't worry— you weren't to know.’
Main entry: ↑know_idiom