Is just not filing a return enough for tax evasion?

No. Not filing alone is usually a misdemeanor under IRC 7203. For felony evasion, there must also be an intentional act to conceal or mislead. So when it comes to the failure to file a tax return there is generally some other affirmative act as part of the crime to make the failure to file a crime. For instance, failing to file returns and yet the taxpayer is dealing solely in cash and transfers away assets into the names of other people could be enough to create a tax evasion charge for a non-filer.