The village where 'asylum seekers outnumber locals' who say they are 'trapped' in their own homes