Owners of cars with push-button start may have to worry about a new threat. Automotive News reports that these vehicles may be vulnerable to being stolen, no thanks to a new device that allows thieves to break into them and drive away without leaving traditional evidence such as broken glass.

The United States’ National Insurance Crime Bureau (NICB) said that it has obtained and tested the so-called “mystery device,” which comes in two pieces and works by picking up a signal from the vehicle’s key fob from a distance of up to 10 feet. The signal is then transferred to a smaller “relay box,” which can be used to unlock and start the vehicle.

Spokesperson Roger Morris said that the NICB tested the system with used car retailer CarMax, on 35 makes and models in various locations – including new and used car dealerships – in the Chicago area over a period of two weeks. They found that they was able to unlock 19 of those vehicles, and drive away in 18.

Morris added that it is impossible to ascertain the amount of vehicles that may have been stolen using these devices, as no evidence is left behind. He also said that both owners and law enforcement often have no idea that such technology exists, although the bureau first noted the rise in its use in 2014.

“[The] scary part is that there’s no warning or explanation for the owner,” said NICB CEO Joe Wehrle in a statement. “Unless someone catches the crime on a security camera, there’s no way for the owner or the police to really know what happened. Many times, they think the vehicle has been towed.”

Morris also mentioned the need for carmakers to be diligent in making sure they adapt their technology to counter these devices, adding that thieves will certainly do the same in response. “It’s a matter of tug of war between manufacturers and thieves,” he said.

Meanwhile, Morris advised vehicle owners to keep valuable items out of their vehicles, keep their key fobs on their person at all times and park in secure or crowded areas whenever possible. He also said that police and vehicle owners should be on the look out for any suspicious activity near parked cars.

However, completely preventing thefts may be impossible, as long as the thief is within the radius to pick up the key fob’s signal, he said. “If these thieves know the device works on a certain make and model, I don’t know there’s a lot you can do about it right now.”

The bureau said that it obtained the device “via a third-party security expert from an overseas company” that provides “manufacturers and other anti-theft organizations the ability to test the vulnerability of various vehicles’ systems.”

Morris added that while thieves can purchase these devices from a variety of sources, one with computer technology skills could build one on their own, which makes cracking down on the makers all the more difficult. “The manufacturers have made tremendous strides with their technology, but now they have to adapt and develop countermeasures as threats like this surface,” said NICB COO Jim Schweitzer.

Alliance of Automobile Manufacturers spokesperson Wade Newton said that cybersecurity remains one of the “top priorities” for the automotive industry. “Some automakers may include a series of redundant systems and mechanisms as one approach to enhancing vehicle safety,” he said in a statement.

“Our initial understanding of this particular tool is that it is a high-technology device similar to the old-fashioned threat of a lock pick or ‘slim jim.’ Obviously, any of these devices in the wrong hands can be used for wrongdoing. The industry does not condone the release of information or the sale of equipment that would further facilitate those seeking to break into vehicles.”

As vehicles become more and more connected, concerns over hacking and other cybersecurity issues have become a hot topic in recent years, and combating these devices could be the latest battle between carmakers and criminals in the modern era.