He was born in 1846 in Huddersfield, the son of Irish parents Michael McCabe and his wife Mary. At least two newspaper sources state he had a physical disability, and reportedly used a crutch.
About two years after James was born, his father was accused of involvement in a vicious triple murder in Mirfield of James Wraith, his wife Ann and their servant Caroline Ellis on 12 May 1847 — all three were found with their throats cut and skulls smashed. Michael, a hawker of earthenware, was arrested having been seen near the house prior to the murder and was found to have a small amount of blood on his clothes. Another hawker, Patrick Reid, was also arrested and both were sent to trial. Despite Reid later making a full confession and stating McCabe was innocent, controversially the trial judge was of the opinion that McCabe was possibly involved so sentenced him to be transported for life.
The 1851 Census records Mary McCabe (aged 35) as a "pauper convict's wife" residing at 97 Upperhead Row with her widowed mother, pauper Julia Keasley (69) of Dublin, brother James Keasley (44), and her sons James (5) and John (4). Mary was still residing on Upperhead Row at the time of the 1861 Census.
The circumstances of Mary McCabe were discussed by the Huddersfield Board of Guardians on 3 December 1858, where she was described as a "big, strong, strapping Irishwoman" in receipt of 5 shillings per week who was refusing to seek work. It was stated that local magistrates had written to the board "to say it was necessary that the woman and her family be removed to Ireland" as she was refusing to enter the Workhouse. A motion was passed allowing the "relieving officer to take out removal orders for McCabe and her children." The Huddersfield Chronicle noted that:
Mr. Floyd said if there were any devils in the world, the two children belonging to McCabe were. They were perfect imps. One of them [James] used a crutch, but might be backed to run against any other boy in the town, for he could run like a race-horse.
According to a letter published in the Yorkshire Gazette (28/Feb/1863), Michael McCabe was released on 24 December 1862 into the care of the Prisoners' Aid Society. Seeking donations, the Rev. Robert D. Jackson wrote:
His health during his term of transportation was such that he could not perform any hard work. He has a wife and two sons ; one a cripple, aged 17, and another aged 16, who is now out of employ through scarcity of work in the mill. I shall be most happy to receive any donations on behalf of this much injured and innocent man, to enable him to make a fresh start in life.
Their father's ill-treatment by the judiciary appear to have led to James and his younger brother John swearing an oath to throw stones at the police.
James died aged 21 in 1867.