We give a structural description of the finite subsets A of an arbitrary group G which obey the polynomial growth condition for some bounded d and sufficiently large n, showing that such sets are controlled by (a bounded number of translates of) a coset nilprogression in a certain precise sense.