High speed working memory, such as RAM, ROM or, in the old days, core would not normally be referred to as peripherals. The more modern term "device" is also more general in that it is used for things such as a pseudo-tty, a RAM drive, or a network adaptor.
Some argue that, since the advent of the personal computer, the motherboard, hard disk, keyboard, mouse, and monitor are all parts of the base system, and only use the term "peripheral" for optional additional components.