How did we get to the point where we are using icon fonts? We used to use images, then moved on to sprite sheets to save HTTP requests. Then we had the problem of high resolution screens at different densities.
An icon font is one small request, it's completely scalable (vector), it is accessible (using the private use area of unicode), easily styled with CSS, supported in IE4+.
However: the process of generating them is fiddly, it has no intrinsic meaning, they are monochrome.
We can address the meaning issue by using ligatures. This allows us to change a whole word into a single character. Currenlty only Open Type and support is limited. Apple have created png's that are embedded into a font - still raster though. BBC have used a combination of glyphs that can be overlayed in different colours. Microsoft have created a font that has more layers that can be different colours.
There are off the shelf icon fonts that have a good set of icons (but sometimes very silly descriptions) so why make your own? Not always the right size, you might need a small version. The style may not fit with your design. There may be too many icons - ones you don't need. Github need icons that nobody else will need.
Creating your own: start with an audit of then icons already in use. Decide on size, style and other high level requirements. (Concider an odd numbered grid.) You might introduce more detail in larger versions of an icon.
- Metaphor - iconic (phone) easily understood, symbolic (sync) learned meaning. Follow conventions, don't be esoteric for the sake of it. The Noun project collects these so you can see if conventions exist for a word. Don't forget local cultural conventions (check with client).
- Style - there a number of different ways to represent an icon but don't get so fancy that it's not recognisable. The metaphor is just as important as the style. Keep them as simple as possible but not so simple that they can be easily confused. Context helps.
- Grid - balance, keep the visual weight the same - use borders to adjust. Use negative space to add depth.
There are a number of SVG creation tools. Decide on icon size and work on them all in the same canvass so you can see how they work together. Expand paths, strokes to outlines, unify to eliminate stray paths, reduce anchor points. We how have a folder of SVGs which need to be turned into a font - there are a number of tools that can automate this. If you create the icons at a large size then when they are scaled down then minor irregularities won't be as noticeable.
Alternatives to icon fonts. Have all your icons in a SVG file and use some form of pre-processor to build the CSS. Can use canvass but it doesn't scale very well.