Over the next ten years James and Mary had at least five children. I think there might have been one daughter that died young. In the 1871 census James and Mary live on Brighton St. with her mother. The Lynches had lived on Brighton since the 1851 census. They had two children, John Thomas age 4 and Michael Joseph aged 11 months. James continued to work as a cotton spinner and Mary as a hawker.
The 1881 census reveals the terrible changes that have occurred in the family over the past few years. Mary died in 1879 leaving five children aged 13 to 3. Her own mother, Bridget, had died in 1873 so there was no female to look after the children. James and his oldest son John Thomas lived together in what was called the Chadwick Buildings. The rest of the children are living a pauper house also known as a work house called Crumpsall.
In 1886 James married for the second time, the new wife is Emily Dodson. They had four daughters in rapid succession, the last one Teresa Caroline did not survive her first year. The 1891 census is the last one in which I can find James, that doesn't mean he isn't there but it is possible that he died.
John Thomas Banks b. 1866 in Boston by 1894, last found in a Boston directory in 1904
Michael Joseph Banks b. 1870 in Boston by 1895, marries Mary L. Scott, but they divorce as she is an alcoholic. He had three children. He died sometime after 1940.
Margaret Ann Banks b. 1873. Married William Bowker. Two of their children emigrate to Boston. She died in 1914.
Catherine Banks b. 1875, I think she married William Frost but cannot locate her.
James Patrick Banks b. 1876, in Boston by 1896, never married. Died after the 1940 census.
James' children with Emily were:
Alice b. 1886
Bridget b. 1887
Mary Ellen b. 1889
Theresa Carolina b. 1891 d. 1892
I cannot find anything on the daughters of Emily.
I'm not sure that James found a better life in Manchester but I know that his children, at least those by his first wife Mary Lynch seemed to have done better for themselves.