I think the term "crack corn" has to do with working in a mill and the process of producing corn flour.

In working on the farm, one of his duties was probably working at the mill as a miller and while working he probably made up this song to pass the time.

As you know, cracked corn is the corn that is discarded... so it doesn't matter. The flour that is kept and used is a fine powder and this is what is used for baking.