Preheat your oven to 400°F. Prepare a baking sheet by lining it with a silicone mat or parchment paper.
In a large bowl, mix the all-purpose flour, whole wheat flour, brown sugar, baking powder, cinnamon, nutmeg, allspice, and salt. Blend the cold butter into the dry ingredients until it forms coarse crumbs.
In another bowl, whisk together the milk, vanilla extract, and egg. Pour this mixture into the dry ingredients and gently stir until combined. Fold in the chopped apples.
Drop the batter onto the prepared baking sheet using 1/4-cup scoops. For smaller fritters, use a heaping tablespoon.
Bake for 11-13 minutes, or until the bottoms are golden brown and the centers are set. For smaller fritters, begin checking for doneness at 8 minutes. Remove from the oven and transfer the fritters to a wire rack.
Switch your oven to broil. While it preheats, prepare the glaze by whisking the powdered sugar, milk, and vanilla extract until smooth.
Brush a thin layer of glaze over the fritters. Place them under the broiler for 2-4 minutes, until the glaze bubbles and the tops turn lightly golden.
Carefully remove the fritters and brush the remaining glaze over them while they are still warm. Allow the glaze to set, then enjoy!