"But they are focused on deep neural networks" - No. They're really not. If anything, they're focused on making matrix operations faster and faster; where the natural result is deeper and deeper networks. But, why not submit a pull request to a highly mature library and add these features so that you can have maximum impact instead of something fairly basic.
Usually each library has it's own architecture. This architecture based on some default classes that implements core functionality for models. I already have a chance to check source code for the popular neural network libraries and I found that their architecture requires some basic classes that are related to the Backpropagation algorithm and it's variation. In neural networks there are a lot of algorithms that really hard to compare and generalize. To make your idea possible I see two ways for that. First one is to change architecture for the main models classes in popular library. In practice for the popular libraries it's a huge amount of work. Even if you found possibility to generalize core architecture with other network that you want implement, probably, you will broke a big part of library that works before. The second one is build a new folder inside of existing library that contains localized code that don't have relation to the core part. How it differs from the decision of a single repository for a different library? In practice its a good way to separate something without relation, because further attempts to combine all in a big library will lead to breakage and bugs that are difficult to debug and fix.
Different strokes for different folks, man. Help contribute to the research/dev community by helping affect the research/dev community where it matters.