I'll have a go.
I went to school with someone who became a (not very famous) trot driver. Now I'm just a bit over jockey size & he was almost twice as big as me. He told me that although he was probably a bit big to get anywhere in the game, light people were also not suitable as the gigs (which are very light!) tend to bounce around, putting the horse off stride, if the driver does not weigh a certain amount. Does that make sense?