You can change a WhatsApp setting, so only an added contact can add you to a group. 

• Open up WhatsApp
• Go to the 3 Dots (top right) 
• Select Settings
• Select Privacy
• Scroll down to Groups
• Change setting to 'My Contacts' 

Please be aware that the default for WhatsApp is 'Everyone'.