Ever since the election, the government has been promising new laws on surveillance powers. Finally the draft bill has been presented, and it has a truly staggering scope. It would allow the state to hack citizens and businesses here and abroad, intercept data in bulk, and force the keeping of all UK citizens’ communications information and web use.

While the powers asked for are striking, Theresa May is right in suggesting this is essentially nothing new. Most of these capabilities already existed in some form, it’s just only recently been admitted. For example, we saw the Draft Equipment Interference code published in February this year, acknowledging for the first time that the intelligence agencies can “locate and examine, remove, modify or substitute hardware and software which is capable of yielding information”.

What is the case is that we have never had these powers explicitly approved or debated democratically. It is an improvement on the coalition that the government is coming (more) clean on surveillance, it is true that safeguards will potentially be strengthened in a number of ways.

But the Investigatory Powers Bill is not just a simple upgrade, a new layer of protection for our rights or tidying up the status quo. Make no mistake, it would cement the UK government’s extraordinary abilities to intrude in to every part of our lives. If that’s not a Snoopers’ Charter, it’s hard to know what is.

In the run up to the bill, there had been a lot of speculation that the government intended to outlaw encryption. That was the conclusion the tech community drew from Cameron saying he did not intend to “leave a safe space … for terrorists to communicate with each other”.

However, there are no “safe spaces” either legally or practically. Legally, RIPA already covered encryption keys. Practically, the state was already cracking systems. The bill intends to continue and expand that. Section 62 requires “maintaining the ability to remove any encryption applied by the (communications service provider)”. Section 189 allows the Secretary of State to impose “obligations relating to the removal of electronic protection applied by a relevant operator to any communications or data”. Cameron’s statement was about protecting the practice that already exists.

The announcement of the bill also saw a significant new admission of a programme to collect vast amounts of data about UK phone calls. This was based on a broad interpretation of the 1984 Telecommunications Act, but its operation had never been made public. As David Anderson who reviewed the entire existing surveillance legislation put it: “it wasn’t outside the law, it’s just the law was so broad… that nobody knew it was happening”.

The avowal of these abilities led to a change in the spin to support the measures. Previously, any tragedy could be, and was, used to advance the argument that there was a lack of intercept powers which could have prevented the incident.

For example, in the wake of the brutal murder of Lee Rigby, three former home secretaries called for the reviving of the Communications Data Bill citing the need to increase capabilities. We must now see those calls in a new light. Bluntly, the death of Rigby did not prove the point that David Blunkett, Alan Johnson and others made at the time, and Johnson would have known that.

Now we are seeing a different version of linking calls for surveillance to the latest outrage in the news, claiming it’s vital to keep hold of bulk intercept to stop further versions of the atrocity.

For example Lord Carlile rather prematurely used the Sharm-el-Sheikh Russian plane crash to ask, “how many more planes might have been affected by this carry-on bomb butchery?” but for the intervention of the security agencies. The fact that it has been reported “chatter” was intercepted by the American intelligence services will be of little comfort to grieving relatives, or anyone planning a holiday in Egypt.

Conversely, the bulk communications data factsheet that accompanied the bill launch credited the use of the Telecommunications Act with thwarting a 2010 plot against the London Stock Exchange resulting in successful convictions. What we don’t get presented is all the times this capability was used and it led to nothing, and if and how its use had collateral impact on innocent people’s lives.

What is certain is that if the government and security agencies wish to take credit for successes, they must take responsibility for failures too. When pressed on the Today programme about how Lee Rigby was let down, MI5 boss Andrew Parker said: “There cannot be a guarantee that we will find and stop everything. That’s not possible.”

This is an acknowledgement that broad investigatory powers can fail us. But the problems go further than that.

One of the constantly frustrating aspects of the intercept powers debate has been how surveillance’s most ardent supporters dart from arguing about local policing matters to global security threats to justify the same means. Surely the aim of these most powerful tools should have been first and foremost to analyse the broad threat picture our nation faces.

This century has been dominated by the failure of western intelligence. This period of bulk collection and equipment interference has seen the rise of ISIS, events spiralling out of control in the Middle East and in Russia’s sphere of influence. As Christopher Lee writing for British Forces News wrote about UK policy on Syria “we are where we are because of miscalculation and shabby scholarship of intelligence analysis”. As so many of us have warned, information is swamping analysis, putting the nation at risk as a result.

We have become so frightened of our own shadow, that we have been unable to distinguish the real darkness that threatens us.

Theresa May is right to say that liberty versus security is not a “zero-sum game”. But this bill is dangerously close to reducing both too close to zero. We have already tried mass surveillance and it has failed us. Now it is time to get protecting both our liberty and our security right.