You may be experiencing a BufferBloat problem.

Try running this tool before, during and after you have problems: https://devina.io/speed-test