They likely can't buy shelter with a fiver or whatever you decide to give them either. The truth is that charities, food banks, and churches are much better equipped to supply the homeless with what they need because it isn't a one-size-fits-all solution, which is why I donate to those organizations instead of trying to hand out money.
Let's also be real: not wanting to enable addictions is not about policing morality. It's about harm prevention. The drugs and alcohol they may buy with your money is likely to do more harm than good in a very practical sense and has nothing to do with my personal beliefs.
At the end of the day, your money won't help them a majority of the time. Offering food or supplies gives them the opportunity to tell me what they need short term rather than me guessing or leaving things up to chance. Long term solutions are provided by other organizations, and your money is better spent there.
OK, what do you think they can do with the $50 that they manage to accumulate that will seriously benefit them?