There are lots of ideas about this, but generally the restorative/transformative justice frameworks are the things prison abolitionists tend to focus on. I'm not well versed on the research surrounding these things, I'm sure there are limitations and criticisms, but I've always thought they advocated a reasonable start for thinking about dismantling carceral systems. Mariame Kaba, an author and activist who writes a lot about this, has a good reading list here:

http://www.usprisonculture.com/blog/essential-pic-reading-li...