If the games have names following format:
QM: Username1 vs Username2, you can turn them off by going to the QuickMatch page (in the MultiPlayer section), and turning off the box "invite me to multi day games".
Alternatively, you might have joined a tournament, which will be indicated by this icon:
If its not either of those, you will have to either block the players inviting you to their games, or if its a group of them doing it, simply ask them to stop doing it and hope they respect it (to save you the effort from having to block all of them).