First, not all apologies begin with I’m sorry. In fact, some of the most graceful and moving apologies have ended with those words.
Second, and more significantly for you, not all utterances including “I’m sorry” are in fact apologies. For example, if you were to get in trouble with a teacher yet feel that you had done nothing, saying “I’m sorry you think that I…” only feels like an apology because it includes those sometimes-deceptive words. It is, in fact, an accusation.
Mildly amused and annoyed,