Each episode of Cold Case begins with a flashback scene informing viewers of the year in which it takes place. A set of characters are revealed in a seemingly mundane situation. The viewer is then shown the corpse of the victim as he or she was found. It is always one of the people introduced in the previous, seemingly innocent, scene.

The show then flashes to the present day. The detectives of the homicide division of the Philadelphia police department are told there is reason for further investigation of an old case gone cold, for one of several reasons:

New evidence has come to light;
The victim’s physical remains have been discovered;
A witness has decided to come forward, etc.
All or any of these cause the detectives to give the cold case a new look and begin researching the victim and interviewing their friends, acquaintances, and family.

The friends, family, co-workers, etc. who were introduced in the flashback at the beginning of the episode have aged (in one case, a six-year-old witness to the crime is interviewed by the detectives when she is 94). Gradually, the detectives gather enough evidence to determine the killer, who most of the time is then arrested.

At the end of each episode, the detectives mark the case solved, while an apparition of the murdered person looks on.

The problems in the detectives' personal lives are also featured, though the main emphasis of every story is on the victim and the search for the killer. Most notably, Detective Lilly Rush grew up in a severely dysfunctional, poverty-stricken home with an alcoholic mother.

Cold Case is also notable for double-casting: it will cast a young actor for the flashback sequences and an older actor for the shots in the present, and cut back and forth between the two, to show how the character has aged. The episode "One Night" managed to triple-cast one character, showing him in the present day, when he committed his murder, and when he was a teenager. If the murder takes place within the recent past, i.e. one to five years earlier, the same actors are often used with subtle changes in appearance and with a few exceptions (i.e: infants to adolescents). If the murder takes place in the distant past, i.e. 15 years plus, different actors with a few exceptions are used (e.g. adult persons who are not likely to age significantly other than gray hairs and/or weight gain/loss).

In 1998, a similar Canadian series called Cold Squad debuted, several years before Cold Case. Fans of Cold Squad accuse the American series of copying the basic premise and characters of the Canadian version, and at one point the creators of Cold Squad launched legal action against the makers of Cold Case. Both shows air in Canada (and on the same network, CTV). Waking the Dead is a similar British TV series about a "Cold Case Squad", which first aired in 2000. Another BBC series, New Tricks, also features cold cases being reinvestigated by a team of retired detectives lead by a woman. The series was first shown on TV in 2003 and stars Amanda Redman, Dennis Waterman, James Bolam, and Alun Armstrong.

The show's theme song is an excerpt from "Nara" by E.S. Posthumus with an introduction by series composer Michael A. Levine. Besides Levine's original music, each episode makes extensive use of era-appropriate music for flashbacks to the year in question. In an episode of a murder in 1994 (Detention), the episode featured references to Kurt Cobain and added grunge songs from the same era, a genre of music that Cobain had popularized. The fifth season premiere, also dealing with a 1994 murder, would later feature songs by Nirvana, which Cobain fronted. Six different episodes exclusively used the music of one particular musican or band. These include Johnny Cash, John Mellencamp , Bruce Springsteen, Tim McGraw, U2, and Bob Dylan, respectively. Also, there has been an episode featuring all Rocky Horror Picture Show songs, as well as an episode with all Cabaret songs.

One episode featured music from an era different than the one featured in the flashbacks. The episode "Blood on the Tracks", about former hippies turned yuppies, took place in 1981 but featured Bob Dylan's music from the 60s.

The episode that was taken the farthest back in Cold Case was the story of a suffragette who was murdered in 1919. This episode also features popular music of that year.

Rarely is the same song ever heard twice on the show. The sole exception being the fourth season episode Forever Blue which replayed The Monkees song "Daydream Believer" first heard in the third season episode Debut. This was due to the fact that "Daydream Believer" was a last-minute replacement as the producers had originally intended to use "Happy Together" by The Turtles in Forever Blue. The song "Landslide" has been used at the end of two episodes, though not the same version: A cover by The Smashing Pumpkins was used in the episode "Detention" while the original version recorded by Fleetwood Mac was used at the end of the episode "Fireflies".

In the 2006 season, the music aired on the show during some episodes has been sponsored by XM Satellite Radio, where music from the various years covered can be heard on XM's decade channels.