Depends on Your Definition on "Down"
Are you referring to the motion of the clubhead or the direction of your force?
One can argue that the club is always moving "down" until it reaches an in line relationship with the left arm. Given that there is (hopefully) some axis tilt at impact, the club can be moving "down" (in relation to the swing) but could be moving "up" (in relation to the ground).
Also, due to the foward bending of the shaft at the bottom of the swing, this will cause a slightly more "upward" movement of the club in relation to the direction of force.
A better question is "do we want to impact the ball before the club shaft is in line with the left arm (still moving "down")" or "do we want to impact the ball at or after the club shaft is in line with the left arm ("level" or "up")"?
I think most would want to impact at or before the club shaft is in line with the left arm (regardless of whether the club is moving "down" or "up" in relation to the ground.
Bruce