Justice Michael Kirby, speaking at a book launch critiquing Australia's refugee policy, said "we should reach out" to farmers being displaced and attacked in South Africa.

"If they have claims for refugee status they should be granted refugee assistance," he said in Sydney on Wednesday.

"I'm sure some of them are being dealt with very cruelly and very unfairly in South Africa."

Justice Kirby said whether the farmers' claims amounted to a well-founded fear of persecution - a central aspect of the refugee definition - were another question.

Mr Dutton caused widespread controversy and diplomatic tensions after arguing last week the farmers needed help from a "civilised country" like Australia.

He wants the farmers fast-tracked through Australia's refugee program.

"Giving them special status because they have white skin is something too deeply embedded in Australian history and it's time we got over it," Justice Kirby said.

Mr Dutton said he would stare down the growing criticism from "crazy lefties", arguing his detractors in the media are "completely dead" to him.

Australian Associated Press