James Mortimer said "significant harm" would be caused to his Grade II-listed Prospect House as a result of the housing plans for land off Springfield Close.
He also criticised the designs from developers Newett Homes and said he believed there was an "error of law" in a council report which could lead to a judicial review.
Mr Mortimer told a council meeting on Tuesday: "We contest the officer's report which indicates that harm was mitigated to less than significant.
"We have also taken legal advice and believe there is an error of law on sufficient grounds in the officer's report to request a judicial review."
Prospect House is listed for its special architectural or historic interest, which includes its use as a officers' mess for soldiers en route to the Somme during the First World War.
Mr Mortimer complained that the housing development would be in "full view" from the rear of the five-bedroom property which dates back to 1835 and sits around 120 metres away from the site.
His complaints were backed up by Ripon councillor Sid Hawke who said trees between the two sites would provide little screening.
Councillor Hawke said: "When you go into Prospect House and stand in his bedroom you can see that site completely."
An agent for Newett Homes told councillors that several changes had been made to the plans since they were refused in June 2020.
This included a reduction in the number of homes from 38 to 30, bungalows and design changes such as imitation chimneys to try to make the development in keeping with the area.
The agent also said the homes were repositioned to reduce the impact on Prospect House "as much as feasibly possible".
The plans were approved by councillors with nine votes for and one against.
By Jacob Webster, Local Democracy Reporter