The new downtown bakery grew from a farmers market venture into a brick-and-mortar bakery seeing 20% weekly sales growth.