The foundation stone was laid in July 1824, reportedly on a site which "used to be a favourite haunt of mushroom gatherers in the early days of the 19th century". The chapel — which was also known as Ramsden Street Independent — was opened in December 1825 and cost around £6,500, with £5 18s having been spent on "beer for workmen".
The noted Congregational minister Rev. Samuel Ringgold Ward, an African American who escaped from slavery and later settled in Canada, gave an address at the chapel on 31 May 1854. The Huddersfield Chronicle reported that "the novelty of hearing a black man deliver an address attracted a large audience" and "he appeared to rivet attention the whole of the time he spoke". Ward spoke of his early life and noted that he was "a Yorkshireman talking to the people of Yorkshire, for he was born in York, United States". A total of £6 was collected for the work of the Colonial Missionary Society.
After the church closed in October 1933 — "because of lack of support and financial difficulties" — Huddersfield Corporation purchased the building in order to secure a site for the new library. Demolition work was underway by February 1936 and the foundation stone of the Public Library & Art Gallery was laid on 30 October 1937.