Does a tax deficiency have to be large to be criminal?

In most courts, yes , it must be substantial. But in some circuits, any deficiency can be enough. The Department of Justice will also move forward with a criminal prosecution for small dollars if the taxpayer was someone in a position of trust who violated that trust, like a government employee, member of Congress, etc.