It was the former prison for the city of Lost Heaven. It is unknown when it was closed. The old prison is used extensively during the mission Election Campaign as the location in which Tommy Angelo assassinates a city council member.
The only way in is a sewer manhole. Once inside, you'll notice that all the jail cells had no toilets; they were squatters with a marked shoe-print where the prisoner was to stand while relieving himself. There was a hole in the center that was only a few inches across, so remnants of feces are seen near all of them. Moreover, that same squatter space was also a shower - to cleanse the prisoner, as well as wash away the excrement.
While you lurk your way through, you'll run into a bunch of very hostile hobos who took to having shelter in this abandoned building. They often have chairs and fires in metal trashcans, as well as baseball bats and firearms.
Like many other buildings in the game, The Lost Heaven County Jail is inspired by a real-world building; in this case it is a scaled-down version of a disused prison in city of Uherské Hradiště, Czech Republic.