It depends on what you want! Annuals have to be planted each year because they are tender and don’t overwinter in VT, so you have to buy new ones each season. However, once they start blooming, they do so all summer, if you take good care of them. At the end of the summer just pull them out and throw them in the compost pile.
Perennials are long lived, often indefinitely, but only bloom for a part of the season. In order to have color all season, you need to plant a wide variety of plants. Some need to be cut down after blooming and you may have an empty spot in your garden. Most perennials need dividing in order to be at their best and not look scraggly or take over the garden. Often a mix of annuals and perennials works well to ensure constant color in the garden. Biennials are plants that live for two years but behave somewhat like perennials because they flower and self-seed in their second year, and the seedlings will return year after year. Hollyhocks are a good example.