A fairly early work of I. M. Pei (built in 1964), this was part of the massive redevelopment of the Lower Hill that cleared out all the poor people and replaced their houses, stores, clubs, bars, synagogues, churches, and schools with a modernist wasteland. It was originally called Washington Plaza, and it was meant to be an International Style city-in-a-tower, with everything you would need on the premises and no reason ever to go out into the grubby outdoors. For most of its life, it was gleaming white; the muddy brown came in with the new name.
Correction: Father Pitt had originally mistyped the date as “1864,” which in geological time is not much of a difference, but in stylistic time is almost enough for the universe to have been destroyed and created again. Much gratitude to “sandisk” for the correction (see the comment below).