From normal fitness level to improved fitness level your greatest gains will be early on and will only continue to improve when you train harder.
The best way to describe this logically is by mentioning 'annual progression' of a running athlete. For example: Man of 20 yeras old starts training from a base level of fitness. After a few weeks regular running he will become stronger and more fit. If he were to take part in a 10km event he would acheive a time. To improve on that time he would need to continue to train, if so, he would improve on that time at his next 10km event. There comes a point where he will level out and require specific training to improve his time. Annual progression is where each season he improves. His greatest time improvements will be during the first season or two as he adapts to training and racing. Any improvement in his time will be fractions over the following years until he reaches an age where he will do well to hold his performance times.
This is relevant to field athletics and all other physical events. However, you may not be seeking specific time goals but the principle is exactly the same. Sorry I couldn't explain it more simple than that!