This is probably a stupid question, but do the plants ever die after a certain number of harvests or is it all to do with how they're being looked after?
I'm looking for my sim's garden size to be reduced but I don't want to have to neglect any plants so that they die purposefully - even for plants, that seems mean!

I've had dedicated and obsessed gardeners whose plants have died; I've never timed it versus neglect.

On the wiki here, it lists how many times you can harvest each plant until it is barren (unable to harvest anymore and is now dead.)

Yes, it's to do with the number of harvests. There's a challenge Sims can do so plants no longer die of neglect, but IIRC that's it.

They can also die if you don't water them daily or weed them when necessary, unless you fulfill the challenge as BH already referenced.

Yes, garden plants will die (or more accurately, go barren) after being harvested enough, though this has less to do with the number of times harvested, and more to do with the amount of produce grown per plant. For example, a single tomato plant will produce 20 tomatoes (over however many harvests) before going barren.

Also of note: Sims with the Green Thumb trait and level 5 Gardening skill can revive plants that die due to neglect, but not plants that have gone barren.