When we do logical deduction, we have facts and rules. We can apply rules to facts to get more facts. More specifically, formal logic is built from situations where we have some fact P, some rule P => Q, and some conclusion Q.
In a category, we have objects and arrows. We can imagine arrows as mapping one object to another, or sending one object to another. We might have objects P and Q, and an arrow f : P -> Q relating them.
The connection continues. We can take multiple rules in logic and apply them sequentially, building a proof tree; we can take multiple arrows in a category and compose them sequentially, building a path. In some logics, there is an idea of ex falso quodlibet, or from the false fact, anything can be proven; similarly, in some categories, there are initial objects, which come with arrows from the initial object to every other object.
Just like how logical formalism eventually removed the possibility that logic is non-mathematical, category theory removes the possibility that logic is unstructured. Logic is actually incredibly highly structured, and those structures happen to coincide with structures in other parts of maths and physics.