You bring up a valid point, any change that is implemented should have exceptions to avoid screwing ordinary people out of the fruits of their labor that were legitimately invested under the prevailing system. The longer answer is that house itself is an improvement even if you didn’t build it and it should not be taxed, and you should be able to pass it on to your children without interference from anyone, but the land underneath it is a different matter. What if the value of the land has increased from $100,000 to $1,000,000 during the time someone owns it, even though they have made no improvements to it? Where does that increase in value come from? It comes from the work of the surrounding community. Is it right for one person to be able to capture a percentage of that? The goal of a land value tax is not to punish anyone, it is only to prevent landowners from either charging rent without adding value or realizing large capital gains, all at the expense of everyone else.

Because the earth is finite and you having that land means it’s denied to everyone else. You shouldn’t have to pay rent to a landlord just because his great great grandfather was a Marquis, but you should compensate everyone else for excluding them from something they have as much right to as you do. The idea of the land value tax is not to punish you, but to prevent you from profiting from the work of others that increases the value of your land without you having to do anything. Henry George explained all of this better than I can.