Sorry! Currently there is no way to remove people from crew.
Only selling the building and doing a new one.
How do I keep track of which friends I am a crew member?
You have to make a manual list. The game does not have anything for you to know, except a heart when you visit the neighbour. However this is also bugged and you MAY NOT see the heart, the best thing to do is to mouse over all Castle Buildings and if 'tend here!' appears then that's one you are crew too.