It looks like James died in 1861 shortly after the census:
BROWN, JAMES 0
GRO Reference: 1861 S Quarter in SHEFFIELD Volume 09C Page 211
And William could have been perhaps a few months later:
BROWN, WILLIAM HENRY 3
GRO Reference: 1861 D Quarter in SHEFFIELD Volume 09C Page 233
or just possibly possibly a year later:
BROWN, WILLIAM 5
GRO Reference: 1862 S Quarter in SHEFFIELD Volume 09C Page 207
Francis definitely died in 1868:
BROWN, FRANCIS 32
GRO Reference: 1868 J Quarter in SHEFFIELD Volume 09C Page 274
Elizabeth survived because she married in 1879:
BROWN, Elizabeth (Widow, age 40, ~, residing at Brightside).
Married Charles WHELTON, on June 22, 1879, by Arthur J R Shaw (Banns) at
Holy Trinity, Wicker. Father's name is Henry Cowlishaw,deceased (~).
Married in the presence of Henry Hollows,Sarah Holland (mark).
Notes: Bride signed with a mark.
Dave
On 1881 Census Elizabeth Whelton, formerly Brown, nee Cowlishaw is with her husband Charles Whelton and they are living next door to her son Francis Brown and her mother Lucy Cowlishaw.
I have been looking at some of the weekly and quarterly reports on Births and Deaths which appeared in the Sheffield newspapers in 1866 and 1867. It seems that Sheffield largely escaped the 1866 epidemic. London, Liverpool and some Welsh towns were the worst affected.