This two-storey Baroque mansion was built in the middle of the 18th century by combining several medieval dwelling houses. In 1805 the building became the property of Antal Almásy, councillor in the Council of the Governor General, who expanded the building by buying the neighbouring house, too. In 1868 the Zichy family became the owner of the mansion, and used it until the 1944 -1945 siege. The block that stands at the corner of Babits Lane formed an exception, as it was detached from the building in 1909. 

The mansion rises over the area that used to be the town's Jewish quarter in the late Middle Ages. In 1964-1965 László Zolnay excavated the remains of a two-nave, Gothic style synagogue, which was still in good condition. The synagouge is still awaiting to be "re-born", and while some of its remains have been uncovered, some are still buried in the soil.