There are a number of variables at play when it comes to the speed of a sledge, and weight is certainly an important one. Assuming there is a reasonable amount of snow on the ground, heavier people tend to go faster as they can cut through the snow on top to make contact with the fast, slippery ice underneath. The size that comes with additional weight is likely to slow you down, as it will make you less aerodynamic, but this will only have a pronounced effect at high speeds in very icy conditions.
Interestingly, during luge sledding, where such conditions exist, heavier sliders go faster at the top of the track and slimmer, more aerodynamic sliders go faster at the bottom of the track. This is due to improved aerodynamics, according to Mark Hatton, who represents the luge sledding Olympic GB team.