Preheat oven to 325°. Butter and flour a 10-inch tube pan.
Sprinkle almonds in bottom of prepared pan.
In a large bowl, whisk together flour, baking powder, and 3 cups granulated sugar. Add butter, buttermilk, eggs, and vanilla paste, and beat with a mixer at low speed until combined, about 1 minute. Scrape sides of bowl. Beat at medium speed for 2 minutes; fold in plums. Pour batter into prepared pan.
Bake until a wooden pick inserted near center comes out clean, 90 to 95 minutes. Let cool in pan for 20 minutes. Remove from pan, and let cool completely, top side up, on a wire rack.
Meanwhile, in a small saucepan, bring reserved plum peels, ¼ cup water, and remaining ¼ cup granulated sugar to a boil over medium heat, stirring constantly, until sugar is dissolved. Remove from heat, and let cool completely. Strain liquid into a small bowl, discarding peel.
In a small bowl, whisk together confectioners’ sugar and ¼ cup plum liquid until smooth. Spoon glaze onto cooled cake. Garnish with plums and mint, if desired.
Recipe by Paula Deen Magazine at https://pauladeenmagazine.com/plum-pound-cake-recipe/