First, try reading about restorative justice.
Meanwhile, particularly in the US slavery was outlawed EXCEPT for prisoners. It was explicitly allowed. And when laws were passed to specifically target PoC (the war on drugs, as a recent example, but i'm sure the laws go back much further) you get to enslave many PoC again without the same outcry.
And to your previous point about B&E, I already addressed the fact that if people are threatened it is violent. As per my definition of violence above.
But to you specific problem with drug dealing. Selling thing s that people want would be allowed in a free market. It is a service. Poor consumer protections, pushing drugs, taking advantage of addictive personalities or otherwise at risk groups is a problem generally with free markets, and just more stark in relation to drugs.
The easiest way to actually disempower drug dealers is to take away their main source of income. Regulation (incl consumer protections/quality of product) and taxation (vat and possibly an additional sin tax) would allow policing to be focused specifically on the most dangerous/coercive/violent aspects of the trade - while allowing law abiding citizens to provide competition to disrupt the current dealers income.
It will not be perfect by any means, but alcohol and tabacco are already in the imperfect area and we mange them using other economic means to minimise the harm (like bans on advertising/sin taxes).
But this whole area is a separate conversation to prison abolition.
Where in I advocate for removing (expensive and harmful) prison sentences and replacing them with (cheaper and proven effective) education/training and restorative policies.