You can always point them to the Olympics Sucks website ...
http://www.olympicssuck.com/
From the minimal response in the Guestbook there, it appears most folks like (or at least don't hate the Olympics too much) ... rather I put the site up in large part due to the IOC's (and related organizations) over-handed response to people using the word "olympic" anywhere in domains, etc...
While their Olympic mark is very strong, they aren't the only ones with marks for olympic and similar.
There are numerous registered marks with 'olympic' in them, including some with solely that word. And some of those marks appear to be relatively recent - referring to the first use date.
End of tirade ... from a business aspect, likely the best course of action is what you did - deleting the domains and move on; every so often they do bulk cease and decist letters followed with bulk UDRPs, and sometimes worse.
Yep, some registrars offer a 5 day delete window for registrants - you're lucky you were able to get out of that mess without paying anything; even better, it's like those registrations never even happened - they won't come back to haunt you later on like 'expired' domains sometimes can.
Ron