It's always worth giving it a good shake
Using the back of the broom gets the really stubborn bits off