The future of a popular village fair is in doubt after a community group was told they have to move all of their stuff.