Currently, users aren't able to remove or block followers but this is a feature we are working on.
In the meantime, you can make your account private which will hide it from the leaderboard. You can do this by going to your profile and clicking Edit profile.
Then turn on the toggle next to Private profile.
If you have any other questions, please get in touch using the button below!