Set the temperature of the oven to 350°F (175°C).
If you haven't already, peel and boil the sweet potatoes until tender. Drain and mash them until smooth. You can also bake them until turn soft and then scoop out the flesh.
In a large mixing bowl, mix the mashed sweet potatoes, milk, brown sugar, granulated sugar, melted butter, eggs, vanilla extract, cinnamon, nutmeg, salt, and ginger (if using). Mix until smooth and throughly combined.
Pour the sweet potato mixture into a greased 9x9-inch baking dish or a similar-sized casserole dish.
If needed, sprinkle chopped pecans or walnuts on top of the pudding for added texture, crunch and flavor.
Bake in the oven until the pudding is set and the top turns slightly golden, it may take around 45-50 minutes.
Let the pudding cool for a few minutes before serving. You can enjoy it warm or at room temperature.
Try Serving with whipped cream, vanilla ice cream, or a drizzle of maple syrup for extra sweetness.