Using a stand mixer (or hand mixer + large bowl), whip together butter, powdered sugar, light brown sugar, cinnamon, and nutmeg on medium high speed until fluffy, about 5-7 minutes. Remove mixer blades and scrape along the side of the bowl to ensure all ingredients are combined.
Add rum extract and vanilla ice cream to mixing bowl; whip until mixture is thick and clumps are mostly gone, about another 8-10 minutes. Remove mixer blades and scrape along the side of the bowl to ensure all ingredients are combined.
Transfer butter rum mixture to a large sealable container and freeze until firm, about 2-4 hours (mixture will not turn fully solid).
When ready to serve, bring a few cups of water to a boil. Scoop out the rum butter mixture and add it to a glass, then pour boiling water on top. Stir until rum butter is mostly melted. To keep the correct ratio, it should be 1 part rum butter mixture to 2 parts boiling water (ex: 1/2 cup rum butter and 1 cup boiling water). Serve drinks with cinnamon sticks for stirring and added flavor (optional).