Image filtering _is_ signal processing, two-dimensional signal processing to be precise.
Traditionally, a college would take you through linear algebra -> differential equations -> signals and systems, to approach this subject.
I found it easier to go through the reverse: start at signals-and-systems (to see what you have to learn), then work your way back down to linear algebra, and then work your way back up to signals and systems.
---------------
From a "signals and systems" point of view, your image filtering functions are 99% going to just be a "kernel" applied to an image.
https://en.wikipedia.org/wiki/Kernel_(image_processing)
IMO, its easier to start with a 1-dimensional version, where you perform kernels upon sound and/or RADAR signals rather than 2-dimensional images.
https://en.wikipedia.org/wiki/Convolution#Visual_explanation
You can see that the 1-dimensional version of the convolution applied between (data x kernel) is extremely simple and "obvious" to think about, given this GIF: https://upload.wikimedia.org/wikipedia/commons/6/6a/Convolut...
Where blue-box is the original signal, and red-box is the convolution-kernel, and the black-line is the output of blue convolve with red.
From there, you generalize the 1-dimensional convolution, into a 2-dimensional convolution. To do so, you need to study linear algebra and matricies. But now that you're "focused" upon the convolution idea, as well as the idea of a "kernel", everything should be "more obvious" to you as you go through your studies.
You can see that a "Matrix", in your specific field of study, represents a kernel to a discrete system. The image you want to manipulate is a 2-dimensional signal. A "matrix" is many different things to many different mathematicians / engineers. "Focusing" upon your particular application is key to learning as quickly as possible. (You can generalize later after you've mastered your particular field).
Still, the study of signals / systems is a very generalized and large field. Mechanical engineers study this, because it turns out that an "impulse" that is "convoluted" with a "kernel" is descriptive of how a speed-bump affects your car's suspension system (!!!!). (EDIT: A youtube video demonstrating the same math for earthquakes vs buildings: https://www.youtube.com/watch?v=f1U4SAgy60c)
So studying signals-and-systems is still a very abstract goal of yours. It sounds like you need to focus upon the image-processing portions of signals-and-systems.
---------
IMO, you'll find that there's probably very little linear algebra you actually need to learn for your particular path.