If an alphabet has case, all or nearly all letters have both a majuscule and minuscule form. Both forms in each pair are considered to be the same letter: they have the same name, same pronunciation, and will be treated identically when sorting in alphabetical order. Languages have capitalisation rules to determine whether majuscules or minuscules are to be used in a given context.
An example of a letter without both forms is the German ß (ess-tsett), which exists only in minuscule. When capitalized it becomes two letters, "SS". This is because ß was originally a ligature of the two letters "ſs", both of which become "S" when capitalized. It evolved into a letter in its own right. (ß is occasionally referred to as a ligature of "sz", which recalls the way this consonant was pronounced in some medieval German dialects. The original pronunciation and the spelling "sz" are preserved in Hungarian.)
The distinction between hiragana and katakana in Japanese is similar to, but not the same as, case. The hiragana and katakana for the same sound are not considered two forms of the same letter. If a word is written with hiragana, it is not normally considered correct to write it with katakana, and vice versa.
The term case derives from early printers' jargon. The individual type blocks used in hand typesetting are stored in wooden or metal cases, sorted by letter. For an alphabet that uses majuscules and minuscules, typesetters need two cases at hand, one for each form. Historically, these were placed one above another on a rack on the typesetter's desk. The upper case contained majuscules, the lower case had the minuscules.