Refugees are by law not allowed to work until they have a semi-permanent residence permit of any kind ("Duldung" would be sufficient in some, but not all cases). So yeah, they're all lazy and at fault here.
Generally, a major problem is that practically no education that refugees may have is recognized in Germany. So even if the person is a doctor, baker or whatever, even if they speak proper German, they're not permitted to work in the job that they're educated for - so even if there's a lack of qualified workers on one side and qualified workers on the other side, you'd still need to go through at least large chunks of the education. My wife who has a full teacher education from a neighboring european country (ECTS credits and all) still had to go back to university before being allowed to teach in Germany. There's also language barriers (obviously)
It's also the case that in 2018 43% of the refugees had a job - that's certainly less than everyone would have wished for, but it's not as if all refugees are out of work.
Since you cited the case of having to import foreign workers for field/harvest work: That's something that the farmers prefer. The imported workers are unorganized, often speak no German and have little to no knowledge of their rights. They're kept in containers under the farmers control. They're easy to exploit in ways that people with even the least bit of local support network are not. They'd not take natural Germans either if they applied.