I’m 43, and I Borrowed Against My 401(k) to Buy the Connecticut House I’ll Retire in

Thinking about buying a home, but not sure how to afford it? Welcome to the Down Payment Diaries, where real people spill about how they saved and splurged on their path to homeownership. If you’d like to submit your own Down Payment Diary, please fill out the form here.

Today, a New York City couple shares their journey of getting outbid on homes in the Hudson Valley, giving Connecticut a chance, and falling in love with a home they hope to retire in.

The basics

Age: 43

Marital status: Partnered

Occupation: Finance and Crypto

What was your home experience when you were growing up?

I grew up in rural New England. My parents and everyone in my family owned modest houses. Back then, I envied the historic homes in the area, but they seemed too expensive. More than that, those homes weren’t my dream. My mom says that when I was a kid and people asked me what I wanted to be, I said, “I don’t know, but I want to live in New York City.” And I have—for decades. 

What changed? 

Now I’m middle aged, and my dream is to live in the woods. My partner and I have been together four years and we’ve both been wanting an escape from the city. 

When did you start thinking about buying?

We could sense that the COVID boom of bidding wars and grossly inflated home prices was coming to an end, so we started looking. 

What areas were you looking at?

We started by exploring a handful of towns in New York’s Hudson Valley. We made three offers and lost every time to people paying so much over asking price. We both love the Berkshires and wanted to look there, but that area is about 30 minutes too far for a weekend trip. Besides that, everything was pricey, and the area is very restrictive with Airbnb rules. We were discouraged but thought, “Let’s just go over the border to Connecticut and see what’s there.” I always thought Connecticut was polo, golf, country clubs, and uppity white people but was willing to give it a look.

Where did that take you?

We knew right away that we wanted to rent out our place half of the year to help us cover property taxes, so we limited our search to cities and counties that had relaxed ordinances regarding short-term rentals. With all this in mind, we started looking from Brewster, New York, over to New Millford, Connecticut. We looked at a few houses in the town of Sharon and a few in Kent. There wasn’t a huge inventory of affordable homes, but there were enough gems. Most of all, the hunt felt less competitive than it did in the Hudson, New York area.

How long were you looking?

We took our time and looked for a year. We searched town by town, getting an Airbnb for the weekend, then hitting three or four open houses in the area. We saw about 30 to 40 houses in person and too many to count online.

Did you have any non-negotiables?

Location was our first, second, and third priority. We wanted the area to be semi-pastoral. We had found a home in Kingston, New York that we liked, but it was on a busy street, and we heard traffic noises the whole time. We really wanted this home to be pin-drop quiet at night, and that was a challenge. We knew we wanted an experience that was the opposite of our New York City experience.

How did you find this home?

We saw this home online on a Thursday. I called out sick from work on Friday to see the house with our Realtor^®. We made an offer that same day, and they chose us.

How did you know this was the one?

The second we stepped in, we knew. We tried so hard not to be emotional buyers, but this one just hooked us. This house is part of a small hamlet of 20 historic homes and had such a great feel. We loved the rich history of the area and of the home, which was built in 1760. Our living room has hand-hewn beams. That kind of early American construction is rare unless you live in New England. This home also had doorways and ceilings that worked for me. I’m 6 feet 3 inches tall. We had seen another house we liked, but I had to duck a bit to pass through the doorways. 

What offer did you make?

The home was so cute that we knew it would go fast. Our Realtor helped us come up with an offer price. We came in at $16,000 over asking, and that was enough.

Where did the down payment come from?

We are both 43 and have been saving a long time. We also both have 401(k) plans. My partner took out some of his 401(k). My company has a 401(k) loan provision, which allows you to keep your money invested while still being able to borrow 50% of it. The best part of that program is that the interest you pay on the loan is paid back to yourself. You’re serving as your own bank.

Did you do any renovations?

We took on quite a few big projects, including converting the laundry room into a second bathroom. We also had to install a new septic tank, which cost $30,000. All in, the renovations totaled $200,000. A lot of the work was aesthetic, altering the home to our own style, which I’d say with this home was largely cottagecore.

What is your favorite thing about being in the house?

We’ve been coming here every weekend since we closed. No question, the best thing is the fact that we are just off the banks of the river. When the windows are open, you hear water. The first time I noticed it, I asked my partner if the TV was on. He just laughed. The nature sounds are a big part of the magic of this place. When we first bought, our goal was just to get out of the city more. Now, we’re finding that we love it here more than we had imagined and plan to retire here. We’re already making plans to see how quickly we can move full time to this home.

In the meantime, what’s the plan?

We are still paying expensive rent and property taxes in New York City. We’ve decided to rent it out half of the year on Airbnb to help defray costs.

What advice would you give other homebuyers?

I’m such an advocate for the area now. I’d tell anyone who’s looking in the Hudson, New York area: Don’t knock Connecticut. It’s not as trendy, and it’s a different crowd—and we like that. Plus, there are so many restaurants that we don’t feel like we are missing out on anything. Overall, Connecticut is not what we expected. People don’t peacock. Some residents live in $7 million homes, and drive Subarus. That’s a sensibility I relate to, and a big part of why we can see ourselves living here long term.