Keys-Shabbat

Question

When I come to shul on Friday afternoon I put my car keys in my tallit bag. Is this a problem as on Shabbat morning I might touch the keys when I am reaching inside to get the tallit. I could put the tallit ooutside the tallit bag, but inside the plastic bag, but is this a problem since if I move the tallit bag I am also moving the keys?