Does anyone else have trouble with sharpening eyebrow pencils?

Every time I sharpen my eyebrow pencil it crumbles up and breaks off. Maybe it's just the brand I have "Rimmel London", or is this common?