Preheat your oven to 400°F. Line a baking sheet with a silicone baking mat or parchment paper.
In a large bowl, whisk together the all-purpose flour, white whole wheat flour, brown sugar, baking powder, cinnamon, nutmeg, allspice, and salt. Add the cold butter and blend it into the mixture until it resembles coarse crumbs.
In another bowl, whisk together the milk, egg, and vanilla extract. Pour this mixture into the dry ingredients and stir until just combined. Gently fold in the chopped apple.
Using a 1/4 cup measuring cup, drop portions of batter onto the prepared baking sheet. For smaller fritters, use a heaping tablespoon of batter.
Bake for 11 to 13 minutes until the bottoms are golden brown and the centers are set. For smaller fritters, start checking at 8 minutes. Remove from the oven and switch to broil mode.
While the oven heats for broiling, prepare the glaze by whisking together powdered sugar, milk, and vanilla extract until smooth. Brush a layer of glaze over each fritter.
Place the fritters under the broiler for 2 to 4 minutes, until the tops are golden and the glaze starts to bubble.
Remove the baking sheet from the oven and transfer the fritters to a wire rack. Brush the remaining glaze over the fritters while they are still warm.
Allow the glaze to set before serving. These apple fritters are best enjoyed on the same day they are made.