The Section 106 cash is agreed during planning talks to balance the impacts of new housing and is intended to be used for improvements such as village hall refurbishments and new sports facilities.
More than £613,000 was spent on projects during 2020/21 - and the council has been praised by some communities.
But with around £3,730,000 unspent at the end of that period, there are concerns that many residents are not feeling the benefits at a time when the construction of hundreds of new homes is piling pressure on the need for community facilities.
John Hansard, a member of the Kingsley Ward Action Group, said the area is facing the construction of around 700 new homes but residents have had "nothing at all" in terms of community improvements.
"Some of this money needs to come our way," he said.
"Harrogate can't cope with much more new housing and to think they want to build an extra 700 homes in our area is absolutely crazy."
Tim Ellis, who also lives in the Kingsley ward, added: "There is no community hall, no church halls and not even a pub any more, therefore nowhere local groups can have meetings.
"With all the new housing destroying the last of the fields, and new houses having tiny gardens, we will need a public park.
"The triangle of meadow and trees to the east of Kingsley Farmhouse bordering Kingsley Road would be ideal... but greedy developers are threatening to put houses even on this."
Most of the money spent by Harrogate Borough Council during 2020/21 was on the purchase of affordable housing.
Councillor Tim Myatt, cabinet member for planning at the council, described the funding as a tool to "support future communities across the Harrogate district" and said other projects included new play areas and open space.
He also said the authority helped North Yorkshire County Council secure around £1.2million in the last year for improvements to roads, sustainable travel and schools.
But when questioned by the Local Democracy Reporting Service, the county council did not say how much unspent money it has held onto.
A county council spokesperson said Section 106 money has a "significant lifespan" and that the funds will be used for their intended purpose "at the appropriate time".
Harrogate Borough Council also said it is "not unusual" for the authority to have large sums of unspent money and that there are several reasons why this is the case .
It said the money is sometimes paid in instalments and cannot be spent until it has all been received. It also said parish councils often ask for the money to be saved up for larger projects.
Despite this, councillor Pat Marsh, leader of the opposition Liberal Democrat party, said the almost £4million unspent by Harrogate Borough Council "does seem a lot" and has called for more regular reviews of how the money is spent.
The council also collects cash through Community Infrastructure Levy (CIL) contributions which are another type of charge on new housing.
These work on a pounds per square metre basis and because of recent changes to Harrogate Borough Council's charging schedules no contributions were collected during the last financial year.
It is now intended that a review will be carried out to develop a list of spending priorities and projects.
Councillor Myatt added: "Payment of CIL is due upon commencement of development, therefore there will be a time lag of up to two years before the first CIL contributions are received.
"In the future, the CIL will provide us with a pot of money so that we can work with local communities and partners to provide the right infrastructure in the right places to support development and the continued growth of the Harrogate district."
By Jacob Webster, Local Democracy Reporter