Why I Worry About Chinese Chips & Hardware

One More Time… (The Chinese Hacked Hardware Singalong begins… again…)

In addition to USB thumbdrives pre-loaded with viruses, and “electronic picture frames” similarly infected, there’s an ongoing effort to bugger computer motherboards and systems. So far this has focused on Intel / AMD oriented hardware, but eventually I think the ARM IoT type SOC (System On Chip) and SBCs (Single Board Computers) will also be targets. That’s why I’ve stopped buying any Chinese SOC based products or any SBCs assembled in China (IF I can determine place of assembly…)

It is just no longer safe to have Chinese origin products in your computer or network. Realize other countries do this too, but they are less malicious in what they do with the buggery.

https://www.theregister.com/2021/02/12/supermicro_bloomberg_spying/

Supermicro spy chips, the sequel: It really, really happened, and with bad BIOS and more, insists Bloomberg

Server maker says latest article is ‘a mishmash of disparate allegations’
Thomas Claburn in San Francisco Fri 12 Feb 2021 // 23:28 UTC

Following up on a disputed 2018 claim in its BusinessWeek publication that tiny spy chips were found on Supermicro server motherboards in 2015, Bloomberg on Friday doubled down by asserting that Supermicro’s products were targeted by Chinese operatives for over a decade, that US intelligence officials have been aware of this, and that authorities kept this information quiet while crafting defenses in order to study the attack.

“China’s exploitation of products made by Supermicro, as the US company is known, has been under federal scrutiny for much of the past decade, according to 14 former law enforcement and intelligence officials familiar with the matter,” states Bloomberg in its report, said to rely on interviews with more than 50 sources, mostly unnamed, in government and the private sector.

That’s pretty much proforma how you do it. When you discover a new attack, you do not just immediately shut it down. An assessment is done to decide if it is a dramatic risk Right Now, or is the longer term persistent threat greater? If the “Now” threat is greatest, you shut it down. If the longer term persistent threat is judged the bigger one, you let it run and study it to get mastery.

I did that with an attack on Apple in the ’80s. A Russian source had hit our “Honey Pot” target but could not get inside the corporate network, then bounced off us to a Military site in Hawaii. We contacted the .mil site and informed them of the attack, but given that our important corporate data was not at risk, and only the “attractive appearing but useless false target” Honey Pot was at risk, we let the exploit run while we figured out everything we could about them and their methods.

BUT, doing that for a decade? Really? “Slow learner” comes to mind…

The article – a follow-on to BusinessWeek’s 2018 spy chip bombshell – cites three specific incidents: the 2010 discovery by the Defense Department that thousands of its computers were sending military network data to China due to code hidden in chips that handle the server startup process; Intel’s discovery in 2014 that a Chinese hacking group penetrated its network via a server that fetched malware from an unidentified supplier’s update site; and a 2015 warning issued by the FBI to multiple companies that Chinese agents had hidden an extra chip with backdoored code on one manufacturer’s servers.

In other words, Bloomberg has expanded its claim that chips containing malicious spyware were added to Supermicro server motherboards, to also include claims of malicious alterations to BIOS-level software to load and run surveillance code hidden in firmware, and to include alleged attacks on other vendors.

All we’d heard about before was the last of those three. Why ANY USA .MIL site allows ANY traffic with a Chinese IP address to exit the site is beyond me. Frankly, why are they not limited to only route to other sites in the .mil address range? That’s a first and most basic layer of the many layers of firewall that ought to exist. Only allow addresses of which you approve through your routers.

The buggering of the BIOS chips is very worrisome as that’s something that persists through shutdowns, software updates, etc. That kind of attack is why I’m not fond of the Intel Management Engine. The ME is a small complete computer where you have zero visibility or control of the software running on it. A prime target for all things Buggery and, IMHO, highly likely to have TLA code embedded from the manufacturer (see prior PRISM program in the USA, and CCP domination of all corporations in China, along with similar things in the UK and EU).

The scenario is not entirely implausible to some in the security industry. “In the hierarchy of cyber attack techniques preferred by intelligence agencies, the highest levels attacks are those that are persistent even when machines are turned off and software is reloaded,” said Alan Paller, director of research at the SANS Institute and president of the SANS Technology Institute, in an email to The Register. “A malicious chip is the simplest solution. Simple solutions work.”

After all, even the NSA ran its own chop shops, adding backdoors to IT gear as it was shipped across the world.

It isn’t paranoia when they ARE out to get you…

But at least now you know why I don’t run any Intel / AMD based hardware newer than a decade or two old, and have stopped buying Chinese source products even in the SBC tier as of a few years back. (My older Chinese based SBCs were likely clean, but I’d only bought them to assess function anyway so they have spent almost all their time either powered off or in isolated networks. I’m presently employing one in m “TV_Network” where at most it can discover I watch TV…)

Segmenting and isolating functions by network with firewalls between them and lack of routing facilities is a basic level of protection that folks ought to do. There is NO reason for your financial activities to be on the same network as your TV and your kids video games and chats. Put a second router / WiFi hot spot behind the one from your Telco or Cable provider and segment your network into functional parts, then do not route between them. Do not let Chinese sourced computers or network equipment share the network segment that has things of importance on it. (But realize that likely just means you are sharing with the NSA, GCHQ, or other TLAs instead…)

The Register spoke with a former executive at a major semiconductor company who asked not to be named, about the plausibility that the subverted silicon cited in the Bloomberg report might exist and we were surprised to find that he found it credible.

“I have physically held evidence in my hands,” he said with regard to the existence of compromised hardware. “I have seen it from multiple governments.”

China, Israel, and the UK have excelled at these operations, he said, with France, Germany, and Russia also involved but, in his view, somewhat less capable in terms of hardware subversion.

Such attacks absolutely do happen, our source said, adding that there are government contracts seeking to study subverted hardware attacks so they can be replicated and improved upon. However, they’re generally not directed at the public, he said. Rather they’re focused on obtaining access to critical systems, on developing durable national security assets.

If a Chinese chip has indeed been identified, he said, then those involved in the operation messed up by not spending the resources to hide it better.

So some expectation of un-buggery by obscurity from buying “uninteresting” hardware. That’s been my basic SBC strategy. There’s little likely gain from, for example, putting spying exploits in a board that’s likely to end up running a yard thermometer or toy robot and the risk of discovery is great when it is sitting on the developer lab bench hooked up to monitoring gear during system design / devo. So why expose your secret exploit that way? Instead reserve the high end exploit for high end targets and high end equipment, then put your simple stupid virus plant into random bits of low end gear like USB Thumbdrives that are unlikely to be tested and examined before use.

So I try to occupy the middle ground between those two. Uninteresting and likely to be discovered if used.

Another reference to the Bloomberg story:

https://www.datacenterdynamics.com/en/news/years-later-bloomberg-doubles-down-disputed-supermicro-supply-chain-hack-story/

It starts off with a lot of denials of evidence, then moves on to the same story.

Years later, Bloomberg doubles down on disputed Supermicro supply chain hack story
Providing more context and quotes to back up a huge claim – that China put secret chips on Supermicro servers

February 12, 2021 By Sebastian Moss
[…]
Supermicro hired a third-party investigations firm, Nardello & Co, to examine the veracity of the report, but it was unable to find any evidence supporting Bloomberg’s claims. The US Department of Homeland Security also disputed the claim, while the NSA said it was “befuddled” by the article.

Following the article, Supermicro moved manufacturing out of China, due to customer concerns from the reporting, but continued to deny it. Bloomberg said that it stood by the article, but did not provide any other updates. The two journalists named in the article did not have any more bylines for months.

Now the same journalists have returned with a new, deeper report. It looks at a wider history of alleged tampering of Supermicro motherboards.
[…]

But there’s 2 problems with that denial process. First off, the US TLAs were interested in “exploiting the exploit” and getting a good bit of study of it done. So they (along with the companies supposedly buying the equipment) would all have strong motivation to hide the fact of it. Second,the denials were in some cases of the form “it never went into production” and NOT of the form “it didn’t exist”…

Then there’s the Boomberg article itself:

https://www.bloomberg.com/features/2021-supermicro/

The Long Hack: How China Exploited a U.S. Tech Supplier
For years, U.S. investigators found tampering in products made by Super Micro Computer Inc. The company says it was never told. Neither was the public.

By Jordan Robertson and Michael Riley

February 12, 2021, 5:00 AM
[…]
Whether that probe continues is unknown, as is a full account of its findings. But as recently as 2018, the FBI enlisted private-sector help in analyzing Supermicro equipment that contained added chips, according to an adviser to two security firms that did the work.

The Supermicro saga demonstrates a widespread risk in global supply chains, said Jay Tabb, a former senior FBI official who agreed to speak generally about China’s interference with the company’s products.

“Supermicro is the perfect illustration of how susceptible American companies are to potential nefarious tampering of any products they choose to have manufactured in China,”
said Tabb, who was the executive assistant director of the FBI’s national security branch from 2018 until he retired in January 2020. “It’s an example of the worst-case scenario if you don’t have complete supervision over where your devices are manufactured.”
[…]
A spokesperson for the Chinese Foreign Ministry called accounts of these attacks “attempts to discredit China and Chinese enterprises” and accused U.S. officials of “making things up to hype up the ‘China threat.’”

“China has never and will never require enterprises or individuals to collect or provide data, information and intelligence from other countries for the Chinese government by installing ‘back doors,’” the spokesperson said in a written statement.

This story is drawn from interviews with more than 50 people from law enforcement, the military, Congress, intelligence agencies and the private sector. Most asked not to be named in order to share sensitive information. Some details were confirmed in corporate documents Bloomberg News reviewed.

Yeah, right, China claims it would never do what it has laws demanding companies do and what has been observed to be done in the past. Sure, I believe every word… NOT!

Subscribe to feed

About E.M.Smith

A technical managerial sort interested in things from Stonehenge to computer science. My present "hot buttons' are the mythology of Climate Change and ancient metrology; but things change...
This entry was posted in Tech Bits. Bookmark the permalink.

10 Responses to Why I Worry About Chinese Chips & Hardware

  1. michaelh says:

    Thanks for your points – a lot to think about.

    Have you played with setting up a RaspPi or other somewhat trustworthy SBC as a NIDS and netflow capture device? The horsepower is low but should be good for a small network.

    My biggest concern is the NIC hardware. If it’s been compromised in such as way as to hide certain layer 2 traffic it might be difficult to capture egress layer 3 packets of interest.

    I know that’s all assuming there isn’t a wireless based outbound connection. I would think a late-generation Pineapple would be good in that event. But again, that uses assumptions about communication via 802.?? standards. In fairness those are the ones I’m more immediately concerned with. If it really came down to it I could run a device inside a Faraday bag, but that wouldn’t be my first choice.

  2. E.M.Smith says:

    @Michaelh:

    I’ve done a very basic install, but not much beyond that. It’s on my “todo” list to get one into production.

    For most Network Intrusion & Detection stuff, you are network speed limited anyway, so for a home 100 Mb network, it just isn’t much load. (Especially given that I have the bulk of my traffic isolated to a TV Net).

    FWIW, I’d likely use one of the Odroid family if I wanted more horsepower. Ether the XU4 or the N2 is quite zippy and can do a lot more than the network can provide… Even the C2 ought to be too much and they are now making a C4 …

    Per wireless exits: Just don’t use gear with wireless hardware in it if you don’t need wireless… Only my Pi M3 boards have built in WiFi chips. The others, I have a nubby or dongle and it only goes in the USB port if I need it on WiFi (and then I set up the connection and can see what’s happening on my router…)

    FWIW, I’ve been known to make my own Access Point out of a R.Pi board and “see what it sees”… AirSnort, it’s a thing..
    https://sourceforge.net/projects/airsnort/

    At a “Major Entertainment Company” we had a project to put Raspberry Pi / Kali systems in backpacks and walk around the “facilities” of a few thousand acres to detect unapproved “hot spots” and inspect for “unusual” signals and traffic. So it can be done (I didn’t run that project but worked near the guy who did it. He had an interesting presentation in staff meeting…)
    https://www.kali.org/

    How much you choose to do depends on how much threat level you think you ought to defend against. At work that has been “a lot”, but at home it is “not so much”. The only wifi in my area are neighbors I know (and generally non-tech) while my network structure has a lot of defense already in it. So I’ve been lazy about IDS gear… Also my work habits come into it. Regularly “rotating systems” so any buggery gets flushed as systems get reflashed, always running “htop” so any unusual system load gets a system shutdown or inspection (and on a minimal SBC, any added load tends to show up…), etc.

    As a result the IDS Project has been mostly motivated to ‘show others’, and that’s not been quite enough to put it ahead of other projects… Sigh. But there’s lots of Pi based projects out there:
    https://raspberrypi.stackexchange.com/questions/71293/deploy-hybrid-intrusion-detection-on-r-pi

    https://dergipark.org.tr/en/download/article-file/1160762

    https://www.tripwire.com/state-of-security/security-data-protection/sweet-security-part-2-creating-a-defensible-raspberry-pi/

    https://security.stackexchange.com/questions/201417/how-to-connect-a-raspberry-pi-ids-to-the-home-router-to-detect-intrusions-networ

    And more…

    https://blog.cryptoaustralia.org.au/5-privacy-focused-raspberry-pi-projects/

    Oh, and one other trivial thing:

    I like “blinky lights”, so some systems are plugged into network hubs or switches with nice blinky lights before they go to the router. IF I see a lot of blinking going on with a system that is NOT being actively used, I take a look at “Why?”… Yeah, crude, but pretty effective. As I have NO “auto-update” stuff running, if I’ve not launched some program to run, there ought not be any significant network traffic (beyond occasional broadcast packets, time updates and similar minor blips).

    Between “blinky lights and “htop” I’m to some extent a human based IDS…

  3. michaelh says:

    Thanks for the links, I will check them out. One of the great things about the SBC community is that I’m never the first to have tried something.

    You are right about the on board wireless. Most of the time there’s a pair of mini-coax connectors for the antennae that just need to be disconnected. That’s not always the case though depending on how miniaturized or cheaped-out the components are.

    Like you I always have ‘htop’ running in a terminal somewhere ;)

  4. Pinroot says:

    I saw that Register story about Supermicro, not too long after you first posted about cheap Chinese chips being compromised. Of course, does it really matter after this:
    Press Secretary Dodges on Why Biden Revoked Rule Protecting U.S. Power Grid From China
    Stuff like that makes me wonder if Texas was able to FUBAR their grid all by themselves or have they had outside help?

  5. Taz says:

    “Put a second router / WiFi hot spot behind the one from your Telco or Cable provider and segment your network into functional parts, then do not route between them. Do not let Chinese sourced computers or network equipment share the network segment that has things of importance on it. (But realize that likely just means you are sharing with the NSA, GCHQ, or other TLAs instead…)”

    I’ve used this simple trick for 20 years, but am feeling more and more overwhelmed by the constant pressure manufacturers subject us to. Have a grim forboding over what I’ll find when we take delivery of a Smart TV. Am I up to defanging it? Will it even operate without an internet hookup?

    These are not the things a typical consumer should EVER be forced to deal with.

  6. Kneel says:

    EM: ” There’s little likely gain from, for example, putting spying exploits in a board that’s likely to end up running a yard thermometer or toy robot …”

    Indeed. And that’s why I prefer to “repurpose” consumer android boxes (TV boxes) – 99.99% of them are spending their time decoding mp4 streams, either locally sourced or from somewhere on the net, maybe the odd game and not much else – and that’s when they’re powered up, which most of the time they aren’t. And they tend to be in “uninteresting” places, like households – hardly likely to show up somewhere where you can get hold of nuclear launch codes or anything.
    As I noted before, the SoC systems are always pushing the limits of what is possible to actually fit on a single chip (and trying not to melt it :-) ), so little room for jiggery-pokery I think, especially given the market for these is somewhat price conscious too. And reflashing the bootloader as well adds a little bit extra – if you can’t bugger the hardware, the best place to bugger the software is in the bootloader, which is essentially “unseen” and remains “untouched” by most people.

    I guess it comes down to a couple of things:
    Are you paranoid enough?
    How much will a break-in cost you?
    Do you have sufficient diversity in equipment?
    Do you have sufficient depth in security/monitoring?

    If you are sufficiently paranoid and/or have a lot to lose, I may be worth investigating virtualisation too – I don’t imagine that it would be particularly easy to “extract” data from a VM virtual disk, especially an encrypted one, and getting “inside” it is likely harder if you only have access to the host OS/hardware. An encrypted VM disk on an encrypted real filesystem would be pretty hard to read nefariously direct from the hardware, I think. Sure, not impossible, but a lot of extra work. Security by obfuscation is hardly the best plan, but what you really need to do is make your systems at the next level of difficulty, so that they are focused on trying elsewhere, as yours is just “too hard” and likely not worth the effort. So they just give up and try elsewhere.

    Of course, the best security is the old air gap – you can’t break in if you don’t have any comms to the box! A little limiting though… So perhaps several “hops” via internal networks that NAT/masquerade traffic would help too – all on different hardware and different OS, and only accessible from your “core” side. Plus EM’s “hardware rotation” plan.

  7. E.M.Smith says:

    @Kneel:

    A man after my own heart!

    Yeah, it’s a constant tussle of the pulls… and you try to end up in the one place not interesting enough for the cost…

  8. Ossqss says:

    All I can say here is how long was it before you got access to GPS post implementation?

    Think about it, and apply the same delta.

  9. E.M.Smith says:

    @Another Ian:

    That comment would work better on the Texas thread:
    https://chiefio.wordpress.com/2021/02/15/texas-cold-dark-might-do-it/

Anything to say?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.