I don't think that's quite what their implication is? I mean - my interpretation of that was more like... like you'd say "Look, I may have said you could borrow the cat but nothing was said about you using it as a chimney brush."

When you definitely did say they could borrow the cat, the 'may' is just a figure of speech rather than throwing genuine doubt on whether you said it.