New Yorkers may think of slavery as a Southern institution, but it thrived here in colonial times. And in “The Manor: Three Centuries at a Slave Plantation on Long Island” (Farrar, Straus and Giroux, $28), the landscape historian Mac Griswold tells the long story of a surviving slaveholders’ estate, hiding in plain sight on Shelter Island and occupied by the 11th generation of the family that settled it in 1651.