Worst time for me is first thing in the morning and for me that's between 4am and 6am., so I can't see a server load problem from visitors, although spam would do it from O/S around that time.

The site has grown and it becomes a delicate balance between bandwidth and cost effectiveness. Too little or too much of one, always effects the other. It does sound like the site needs a little more investment in bandwidth perhaps [if that's the problem].