will it make much of a difference to just keep it px instead of scaling it with em?

px is an absolute unit of measurement (like in, pt, or cm) that also happens to be 1/96 of an in unit (more on why later). Because it is an absolute measurement, it may be used any time you want to define something to be a particular size, rather than being proportional to something else like the size of the browser window or the font size.

Like all the other absolute units, px units don't scale according to the width of the browser window. Thus, if your entire page design uses absolute units such as px rather than %, it won't adapt to the width of the browser. This is not inherently good or bad, just a choice that the designer needs to make between adhering to an exact size and being inflexible versus stretching but in the process not adhering to an exact size. It would be typical for a site to have a mix of fixed-size and flexible-sized objects.

Fixed size elements often need to be incorporated into the page - such as advertising banners, logos or icons. This ensures you almost always need at least some px-based measurements in a design. Images, for example, will (by default) be scaled such that each pixel is 1*px* in size, so if you are designing around an image you'll need px units. It is also very useful for precise font sizing, and for border widths, where due to rounding it makes the most sense to use px units for the majority of screens.

All absolute measurements are rigidly related to each other; that is, 1in is always 96px, just as 1in is always 72pt. (Note that 1in is almost never actually a physical inch when talking about screen-based media). All absolute measurements assume a nominal screen resolution of 96ppi and a nominal viewing distance of a desktop monitor, and on such a screen one px will be equal to one physical pixel on the screen and one in will be equal to 96 physical pixels. On screens that differ significantly in either pixel density or viewing distance, or if the user has zoomed the page using the browser's zoom function, px will no longer necessarily relate to physical pixels.

em is not an absolute unit - it is a unit that is relative to the currently chosen font size. Unless you have overridden font style by setting your font size with an absolute unit (such as px or pt), this will be affected by the choice of fonts in the user's browser or OS if they have made one, so it does not make sense to use em as a general unit of length except where you specifically want it to scale as the font size scales.

Use em when you specifically want the size of something to depend on the current font size.