Revealed: ISPs Already Violating Net Neutrality To Block Encryption And Make Everyone Less Safe Online

from the scary-news dept

One of the most frequent refrains from the big broadband players and their friends who are fighting against net neutrality rules is that there’s no evidence that ISPs have been abusing a lack of net neutrality rules in the past, so why would they start now? That does ignore multiple instances of violations in the past, but in combing through the comments submitted to the FCC concerning net neutrality, we came across one very interesting one that actually makes some rather stunning revelations about the ways in which ISPs are currently violating net neutrality/open internet principles in a way designed to block encryption and thus make everyone a lot less secure. The filing comes from VPN company Golden Frog and discusses “two recent examples that show that users are not receiving the open, neutral, and uninterrupted service to which the Commission says they are entitled.”

The first example you may have actually heard about. It got some attention back in July, when entrepreneur Colin Nederkoorn released a video showing how Verizon was throttling his Netflix connection, which was made obvious when he logged into a VPN and suddenly his Netflix wasn’t stuttering and the throughput was much higher. That video got a lot of attention (over half a million views) and highlighted the nature of the interconnection fight in which Verizon is purposely allowing Netflix streams coming via Level 3 to clog. As most people recognize, in a normal scenario, using a VPN should actually slow down your connection somewhat thanks to the additional encryption. However, the fact that it massively sped up the Netflix connection shows just how much is being throttled when Verizon knows it’s Netflix traffic. Nederkoorn actually was using Golden Frog’s VyprVPN in that video, so it actually makes Golden Frog look good — but the company notes that it really shows one way in which “internet access providers are ‘mismanaging’ their networks to their own users’ detriment.”

But the second example Golden Frog provides is much scarier and much more pernicious, and it has received almost no attention.

In the second instance, Golden Frog shows that a wireless broadband Internet access provider is interfering with its users? ability to encrypt their SMTP email traffic. This broadband provider is overwriting the content of users? communications and actively blocking STARTTLS encryption. This is a man-in-the-middle attack that prevents customers from using the applications of their choosing and directly prevents users from protecting their privacy.

They demonstrate this with the following graphic:

This is scary. If ISPs are actively trying to block the use of encryption, it shows how they might seek to block the use of VPNs and other important security protection measures, leaving all of us less safe. Golden Frog provides more details of what’s happening in this case:

Golden Frog performed tests using one mobile wireless company?s data service, by manually typing the SMTP commands and requests, and monitoring the responses from the email server in issue. It appears that this particular mobile wireless provider is intercepting the server?s banner message and modifying it in-transit from something like ?220 [servername] ESMTP Postfix? to ?200 ********************.? The mobile wireless provider is further modifying the server?s response to a client command that lists the extended features supported by the server. The mobile wireless provider modifies the server?s ?250-STARTTLS? response (which informs the client of the server?s capacity to enable encryption). The Internet access provider changes it to ?250-XXXXXXXA.? Since the client does not receive the proper acknowledgement that STARTTLS is supported by the server, it does not attempt to turn on encryption. If the client nonetheless attempts to use the STARTTLS command, the mobile wireless provider intercepts the client?s commands to the server and changes it too. When it detects the STARTTLS command being sent from the client to the server, the mobile wireless provider modifies the command to ?XXXXXXXX.? The server does not understand this command and therefore sends an error message to the client.

As Golden Frog points out, this is “conceptually similar” to the way in which Comcast was throttling BitTorrent back in 2007 via packet reset headers, which kicked off much of the last round of net neutrality concerns. The differences here are that this isn’t about blocking BitTorrent, but encryption, and it’s a mobile internet access provider, rather than a wired one. This last point is important, since even the last net neutrality rules did not apply to wireless broadband, and the FCC is still debating if it should apply any new rules to wireless.

After reading the Golden Frog filing, the answer should be that it is absolutely necessary to apply the rules to wireless, because practices like these put us all at risk by undermining the encryption that keeps us all safe. As Golden Frog notes:

Absent enforceable Commission rules, broadband providers can (and at least one already does) block and discriminate against entirely acceptable Internet uses. In this case, users are not just losing their right to use the applications and services of their choosing, but also their privacy. It is not at clear that this type of encryption blocking would be forbidden for fixed broadband Internet access, under the proposed rules? exception for reasonable network management. This example involves mobile wireless broadband, however, and it is clear that the proposed rules would not prohibit the activity. STARTLLS encryption does not constitute ?a lawful website? or ?an application[] that compete[s] with the provider?s voice or video telephony services[.]?11 The proposed rules on their face do not prohibit mobile broadband Internet access providers from blocking user efforts to maintain privacy through encryption.

Furthermore, Golden Frog concludes:

The claim that rules banning blocking and unreasonable discrimination are solutions in search of a problem is flatly wrong. There have been problems in the past and there are problems now. The proposed rules do not resolve all of the problems identified in the NPRM. Further broadband Internet access providers are still interfering with beneficial and privacy-enhancing applications users want to employ.

This is incredibly important — just at a time when we need stronger encryption and privacy online, the FCC may undermine it with weak net neutrality rules that allow this type of behavior to continue.

A few months ago, I got into a conversation with a well-known internet entrepreneur/investor, who asked about possible “compromise” rules on net neutrality, suggesting that maybe it’s okay to throttle Netflix traffic because there’s so much of it. He argued that, perhaps there could be some threshold, and if your traffic was above that threshold it’s okay to throttle it. After some back and forth, I asked the hypothetical about encryption: what if, at a time when more and more encryption is important, such a rule was in place, and overall encrypted traffic passed that threshold, then suddenly access providers could throttle all encrypted traffic, doing tremendous damage to security and privacy. What I didn’t realize was that some access providers are effectively already attacking privacy and encryption in this manner.

Filed Under: , , , , , , ,
Companies: golden frog

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “Revealed: ISPs Already Violating Net Neutrality To Block Encryption And Make Everyone Less Safe Online”

Subscribe: RSS Leave a comment
136 Comments
Anonymous Coward says:

Re: Re:

If I could guess, they’re disabling encryption on mobile because it uses more bandwidth and doesn’t allow caching. That’s not a good enough reason to disable encryption if the user is specifically asking for encryption, of course.

One thing that seemed weird to me is that they didn’t name the guilty party anywhere in the filing. That pretty much makes it impossible to verify the claims.

Josh in CharlotteNC (profile) says:

Re: Re: Re:

No one under the guise of stupidity alters packets in transit

Happens all the time. Misconfigured mail servers or aggressive spam rules could both explain the responses given.

in attempts to disable security.

It remains to be seen if that was the intent. There’s just not enough data given to lead to a conclusion here. We don’t know the methodology of the tests. We don’t know what service this result was from. All we know for sure based on the graphic and description is that they did a telnet session over port 25 to apps.[redacted].com and got some admittedly odd results. Is the redacted server under Golden Frog’s control, or is it the ISP’s or someone else’s?

My point isn’t that there isn’t malice here, but before jumping to the ‘this mobile wireless service is disabling encryption on my email’ you should examine other possibilities and give more evidence than a single screenshot. If you think packets are being altered, then proof I would expect would be packet capture logs from both sides showing that. That’s not a new concept – its how the P2P forced reset packets Comcast was injecting were discovered.

Anonymous Coward says:

Re: Re: Re:2 Re:

Dear person you ARE STUPID!!!!

Modifying an in-transit packet as per article…
“It appears that this particular mobile wireless provider is intercepting the server’s banner message and modifying it in-transit from something like “220 [servername] ESMTP Postfix” to “200 ********************.””

This IS NOT ACCIDENTAL CONFIGURATION!

I work in IT, I have done this before, and based on your comment you do not work in IT and if you do you need to be FIRED so damn fast that your head spins!

Misconfiguration on email SERVER would be allowing unencrypted sessions to take place when it should be encrypted (which appears to be the case here as well) but DOES NOT MODIFY THE DAMN PACKET!!!! That is something else entirely!

Josh in CharlotteNC (profile) says:

Re: Re: Re:3 Re:

Calm down and read what I actually posted and not what you misinterpreted through that fog of anger.

I never said a misconfigured mail server would modify a packet in transit. I said that a misconfigured mail server or spam rules could explain the screenshot. I also said jumping to the conclusion of modified packets without evidence from both ends of the connection wasn’t warranted based on the extremely brief information presented in the article. I’ll stand by that judgement. Any claim requires evidence, and their claim of an ISP deliberatly breaking email encryption was a heavy one. And now since we have a plausible explanation of a default configuration on a router that could cause exactly what is shown, I’ll stand by my first comment: Stupidity, not Malice.

And I’ll be sure to let my boss know I should be fired because some anonymous person on the internet said so.

Anonymous Coward says:

Re: Re: Re:4 Re:

Part of your post…
“Happens all the time. Misconfigured mail servers or aggressive spam rules could both explain the responses given.”

This indicates to me that you are saying a misconfigurated mail server or spam rules could do this. which is not the case in either situation.

The server can respond to an encryption request badly due to misconfiguration, but it cannot alter the packet. SPAM filtering does not alter any packets either, but it does study them to determine if the message will be delivered or placed in quarantine or even deleted.

Shawn (profile) says:

Re: Re: Re:3 Re:

“I work in IT, I have done this before, and based on your comment you do not work in IT and if you do you need to be FIRED so damn fast that your head spins!”

You really out to understand what you are talking about before you make idiotic statements like that. While you may “work in IT” you obviously have never seen how a Cisco ASA (going all the way back to when they were Pix’s) treats SMTP by default. While arguably some of the things the default SMTP security server (SMTP fixup) protects against are nice it will absolutely clobber the starttls command and it will look just like the screenshot provided.

Shawn (profile) says:

Re: Re: Re:5 Re:

Mr “I work in IT” https://stomp.colorado.edu/blog/blog/2012/12/31/on-smtp-starttls-and-the-cisco-asa/
I do not even think anyone is arguing this is the exact cause as what is shown in the screen shot but it is not only possible it is the default behaviour for firewalls that are all over the freaking place and those of us who ‘Work in IT’ and have troubleshot SMTP and ESMTP have seen screens like the one posted for years.
Not going to try and prove anything to you. Your statements prove to the rest of us that you are talking out of your rectum.

PaulT (profile) says:

Re: Re: Re:6 Re:

I worked an end user helpdesk for a hosting company for a couple of years. The biggest, most reliable indication that someone didn’t really know what they were talking about was if they started the conversation with their credentials. Especially if their “credentials” were “I work with computers” rather than “I’m a CCIE qualified engineer” when trying to diagnose a Cisco fault, for example.

Yeah, AC, you “work in IT”. So does the guy who reads from a script in a helpdesk call centre for a living and the guy who came in to service our printers this week. That doesn’t mean they know anything about routing and security at the ISP level. By the look of things, neither do you. Unless you want to quote actual knowledge and credentials, in which case I think we’re all ears.

Paul L (profile) says:

Re: Re: Re:4 Re:

I don’t really like commenting here any more. I think TechDirt has gone off the deep end on a variety of topics. But it’s good to see your response (as well as others) putting some actual real info out there containing research that SHOULD have been done before this article was ever published.

But you’re dead on. I’ve been using Cisco ASA firewalls all the way back to the old PIX models as well, and this is something that I’ve had to deal with myself a few times over the years. SMTP fixup can be a bitch, and the moment I saw the screenshot with the telnet session, I knew what was up.

I’ve also seen configurations where they expect ANY TLS traffic to be submitted over 587, even though STARTTLS is just fine to use over port 25. But that doesn’t seem to be what’s going on here.

Where I live, my ISP forces me to use THEIR mail server for all outgoing mail. They will relay anything, from any source address, as long as it’s in their IP space. If I try to connect to port 25 outside of my ISP’s network, the connection is always blocked. They implemented this as an anti-spam measure a few years ago.

Anonymous Coward says:

Re: Re: Re:3 Re:

You. Are. An. Idiot.

Misconfigured/misdesigned/misdeployed devices in the data path do things like this all the time. The Cisco PIX firewalls were notorious for their non-compliance with standards (RFCs) and they’re not the only ones — just one of the more notable examples from a very long list.

Moreover, mail servers do EXACTLY what you say they don’t do. Oh, they’re not supposed to do it, but then again they’re also not supposed to do accept-then-bounce processing or retry when presented with a 5XX SMTP response, but some of them do those things too.

Anyone who has actually worked in the field, has read and understood the RFCs, actually reads their mail system logs, and periodically looks a traffic with tcpdump, knows all these things. They’re just part of the brokenness that crops up all day, every day. Nobody is saying that these devices and these servers should be doing this stuff, but due to broken code or administrator error or unanticipated interactions or some combination of those (or other things), they really do.

Of course, ignorant newbies like you don’t know any of this stuff. You might learn one day, if you STFU long enough to listen. But it seems more likely you’ll just rant and eventually turn out to be one of the incompetent assholes running sites that do wacky things like this, causing problems for the rest of us and triggered another set of rants from another generation of ignorant newbies.

Don Bowman (user link) says:

Re: Re: Re: Most ISP block port 25

Most consumer ISP block port 25. It’s a best practice. If you don’t. If you don’t, outbound spam comes from your network and you get blacklisted.

This ISP appears to be running a spam scrubber instead, which is much better.

I would rather have a port 25 enabled ISP w/o STARTTLS than a no-port-25 ISP.

I would guess if you sign in to the SMTP server you will see it not do this modification.

Aethera (profile) says:

Re: Re: Thank you

I remember years ago as a sysadmin being stupefied at a mail server that was not functioning correctly. Everything checked out yet mail was not getting through. Only after breaking out the telnet mail.host.com 25 as above did I query the network admin and found out that the Cisco firewall’s ESMTP security feature was messing with the application layer.

It does seem kind of unusual that a wireless company’s equipment would have such an option enabled or even that such an option would be present on their type of equipment.

kenichi tanaka (profile) says:

Here’s a novel idea, if you don’t want your internet connection tHrottled, STOP SHARING MOVIES, MUSIC. SOFTWARE, ANIME AND TELEVISION SHOWS.

ISPs throttled a user’s internet connection when they are downloading pirated content that they have not paid for. VPNs always get targeted because 99% of people who are using VPNs are doing so to watch streaming content that is not licensed for their region or they are sharing copyrighted content.

nobody says:

Re: Re: Re:

And Srpint, among others, also intercepts DNS, replacing answers to queries with their own, usually to redirect NXDOMAINqueries to a search-vertising site. And if you query some domains, such as ietf.org, Sprint will not return anything and the query times out.

This is not net neutrality. It is a clear and obvious effort to deny privacy and security to their users, to monitor, track, intercept, interdict and control their activity.

Spins Meats says:

Re: Re: Re:

There’s an awful lot of “security” devices out there that have protocol handlers for an awful lot of different applications. The fact that these ones are allowing ESMTP at all is an improvement on 10 years ago.

That said, these sorts of devices shouldn’t be used on customer-facing networks. They’re designed for corporates and other managed networks, for the most part.

Hopefully this is idiocy rather than malice, but you wouldn’t know either way.

Rikuo (profile) says:

Re: Re:

Your comment might have made the slightest bit of sense if you had read the article and come to the comprehension that this article is talking about disabling the use of encryption via EMAIL.
So in your insane world, if loads of people use torrent programs to get at unauthorized content, then their ISP should just disable encryption for those same users on their email? Last I checked, people aren’t using email to distribute content.
Not only that, but this is done without any sort of due process. You are in essence casting a wide net, saying that because a subset of the population infringes copyright, then it’s perfectly all right for an ISP to punish all of the population by disabling encryption.
And no, VPNs aren’t mostly used to infringe copyright. I myself used a VPN – mostly so I could access Crunchyroll’s USA catalog. Businesses and industries use VPNs all the time – ATMs use VPNs for one example.

Uriel-238 (profile) says:

Re: Company enforced tragedy-of-the-commons.

What if we like sharing movies, music, software, anime and television shows? Doing so shouldn’t be illegal and often isn’t. Frankly, regional licensing is way inappropriate.

Incidentally, VPNs are primarily used for commerce and the protection of companies accessing internet resources, not end-users pirating. So please stop tossing out bullshit statistics.

Just Another Anonymous Troll says:

Re: Re:

Get a clue, kenichi.
They aren’t making YOU slower, they’re making Netflix slower. Also, they’re disabling encryption, which is bad.
Also, there are many legit reasons for using VPNs.
Please return to your bridge and enjoy watching Netflix on your throttled connection.
Oh, and if you’d like to make another inane post in response to mine, use the reply button indicated by the arrow.
||
/

Michelle Sullivan (user link) says:

Re: Re: Re:

VPNs are generally GRE or IPSec which use different protocols (like 50 or 51 not TCP).. this is not the same. SSL VPNs are done over port 80 using STARTTLS or 443 (SSL) – this would not block TCP port 443.

Personally I’d call b**shit on this, normally when blocking commands you block the full command or part of it… IE the response should be XXXXXXXX or XXXXXXXS *not XXXXXXXA.

Michelle

Anonymous Coward says:

Re: Re:

This is pretty much the scorched earth stance on copyright enforcement. Screw collateral damage, as long as people stop doing things I don’t like. If it gets people to stop sharing my (insert copyright material) then I don’t care if the rest of the legitimate world burns.

And that’s why people hate copyright so much.

John Fenderson (profile) says:

Re: Re:

“ISPs throttled a user’s internet connection when they are downloading pirated content that they have not paid for”

Incorrect. ISPs can’t possibly know what downloads constitute a copyright infringement and what don’t.

“VPNs always get targeted because 99% of people who are using VPNs are doing so to watch streaming content that is not licensed for their region or they are sharing copyrighted content.”

Even more incorrect. The primary use of VPNs is business-related. The amount of VPN use that is devoted to piracy is rather difficult to determine, but in my experience about 99% of VPN use is totally unrelated to piracy.

But even if that number is lower, it still doesn’t justify ISPs blocking this traffic.

tqk (profile) says:

Re: Re:

VPNs always get targeted because 99% of people who are using VPNs are doing so to watch streaming content that is not licensed for their region or they are sharing copyrighted content.

You’re an ignorant fool, or an idiot.

How’s about a gun analogy? You can use a gun to shoot bottles on a fence, or targets on a shooting range, or rabbits (you beast!) or lots of other things. They can also be used to shoot attacking bears or pumas, and humans.

My last big client (I’m a geek) provided me with a laptop which was so locked down, all it could do was communicate with their servers via VPN. Well, until you booted from a LiveCD Linux disk, that is (but I digress).

Guns don’t shoot people. People do.

Tech is just complicated hammers (tools). The intent of those using it is what matters. VPN’s just a tool.

the threat to peace is the USA says:

@#2

and if i am doing a game purchase online i do it unencrypted right you fuckwit

if i play the game online i do it unencrypted right

NO YOU STUPID FUCKING TWIT

this is the stupidest person period , no justification for it.

THESE LAZY DO NOTHING GOOD FOR NOTHING ACTORS MUSICIANS NEED TO FUCK THE HELL OFF.

and the usa and it spuppet hollywood govt can kiss my fuckign ass

i dont download movies games or tv
YET i use encryption and i cant play the online game well cause its crashing due to your throttling all the fucking time.

FUCK FUCK FUCK YOU.

cerda (profile) says:

this is MITM done wrong.

Look no further than offerings to monitor HTTPS — the easy way to to is to intercept the user’s request, replace the web host certificate by your own (and the roots for it), and then act as a MITM between the user and the target server.

You end up with a user monitor, and a monitor target web server. Easy. Privacy? Gone. Security? Gone as well.

And yes, available from many of the “security” vendors.

you can minimise the risk with certificate pinning, I guess. Easily done if the user is running a program. But very difficult to do for web browsers.

Anonymous Coward says:

Re: Tyranny of the default

An ASA with SMTP fixup is usually for the end user. As you can see in the telnet test, it’s showing the banner so fixup isn’t activated between the user and the mail server. My guess would be a proxy server incorrectly intercepting SMTP traffic or purposely depending on the ISP, or a DPI box that is actually intercepting and mangling packets such that STARTTLS is given incorrectly to the MTA.

My guess would be a full transparent proxy server (IE User -> Proxy -> Mailserver), but I would need to wireshark a computer on the network and an mail server to see what’s sent/received on both ends to really drill down.

Anonymous Coward says:

Throttling

There was ZERO evidence that throttling occurred against Netflix. All claims of throttling have been debunked and Netflix has repeatedly confirmed Verizon did not throttle Netflix.

Service degradation due to congested peering ports is not the same as throttling.

Although at times controversially biased, Techdirt usually sticks to facts. Apparently not anymore.

andrew_duane (profile) says:

Re: Throttling

Technically Verizon wasn’t throttling Netflix, they were throttling Level 3, which carries Netflix traffic. And congested peering ports aren’t throttling, unless they are deliberately under-provisioned to the point where they cannot carry their designated load.

Link capacity of peering connections (as well as all other Network-with-a-capital-N connections) aren’t just pulled out of a hat. Extremely large amounts of throughput analysis is done to understand the requirements of the link. It is extremely important to ISPs to be able to handle the required bandwidth.

This issue would have been spotted in 5 minutes by a junior engineer. Add the fact that it was fixed only after weeks of press shaming and the “payments” from Netflix means it was deliberately designed to extort such payments.

Mike Masnick (profile) says:

Re: Throttling

There was ZERO evidence that throttling occurred against Netflix. All claims of throttling have been debunked and Netflix has repeatedly confirmed Verizon did not throttle Netflix.

Verizon purposely let its connection clog and didn’t do the most minor thing to connect another port to clear it up. To me, that’s throttling. End result is identical.

Cramer says:

Re: Re: Throttling

Verizon purposely let its connection clog and didn’t do the most minor thing to connect another port to clear it up.

And the fact that VPNs could “route around” congested peering links means Netflix doesn’t know jack about traffic engineering. In other words, Netflix has every opportunity to direct users to netblocks that are routed through different paths, thus avoiding the issue entirely. Instead they like prefer to whine to the media in a lame attempt to get free peering and/or free colo hosting of their “OpenConnect” servers.

(Don’t fall for the BS. OpenConnect isn’t “open”; it carries only Netflix traffic.)

JMT says:

Re: Throttling

“Service degradation due to congested peering ports is not the same as throttling.”

Deliberately allowing those ports to become and remain congested is exactly the same as throttling, particularly to the person at the end of the line.

“Techdirt usually sticks to facts.”

That they do, unlike the ‘truthiness’ from shills like yourself.

Anonymous Coward says:

Improper interception happening on Time Warner too

Though not quite as bad as shown here, Time Warner Cable recently began actively disrupting customers too. They decided they wanted to tell customers about their great new faster modems. Rather than do something sane like send an e-mail to the primary account holder e-mail address or include an insert in the account statement, their solution was to deploy a packet interceptor that dropped all traffic on tcp/443 (https) and intercepted all traffic on tcp/80 (http). All http traffic not pointed to the server advertising their new modems was answered with a redirect to that server. I was using only https when they turned it on, so everything mysteriously broke. When I happened to use http and saw what they’d done, it was not too hard to acknowledge their ad, but they caused a huge mess in the meantime, including losing all my work on the https site I had been using.

To top it all off, upon acknowledging the ad, they showed a page confirming the acknowledgement, then remotely reset the cable modem without telling me, which broke what few connections I had on protocols they had not broken with the idiotic deep packet inspection.

Vladlagg (profile) says:

VPN and Comcast

My internet download speeds are 5-6 times faster through Comcast using PIA VPN routing through their largest exit gateway (Toronto, CA). I pay for about 30 megs from Comcast and get over 150 mb download on average through a new docsis 3 modem. for some reason Comcast prioritizes encrypted traffic (at least on my network). I am not about to complain, but if i should be pissed off if they throttled it, should I instead praise their boosted speeds? Them giving me the maximum speed capable for a fraction of the price is very nice indeed. It always leaves me feeling conflicted.

Anonymous Coward says:

Re: Re:

When you are a big business then you get special protections… and besides… if you read the contract you signed for service it tells you that you have given all of your packets to them for whatever they want to do with them.

Either way this can be handled by having both client and server refuse anything other than encrypted connections, but that is not likely, even though it should be the default.

Kenpachi says:

In need of some basic clarification...

Could anyone explain, in laymen terms, the following aspect of this issue? (Maybe it’s already explained but I’m unable to see it, so I apologize in advance, I don’t have the technical background to understand most of this)

The fact that packets are being modified in transit to purposely disable encryption, that happens with no warning or error message to the end user, unless of course the user knows what he/she’s doing, and the connection takes place nonetheless, IN THE CLEAR, even tough the user wanted to use encryption technology?

So, the message (or wahtever) is sent, and the whole time the user thought was using an encrypted means of communication and never noticed that that wasn’t the case?!

Am I missing something? Please someone tell me that the user gets an error message and the connection is NOT established, or something, that the user does not send anything.

(Even if that’s the case, what about machine to machine communications? Big data sent back and forth automatically, without user intervention?)

If I’m correct, (and I hope I’m not), this should be a crime, it’s encryption disabling technology without the user’s consent and knowledge that it’s happening in the background, in real time. What about corporate secrets, R&D, scientific/academic research, medical records, financial records, etc, etc.

This is flat out malicious interference, wtf…

I really thank anyone and everyone who can comment on this and put my suspicion (or my worst fears) to rest.

Anonymous Coward says:

Re: In need of some basic clarification...

Looking at the logs of the transaction, the server is supposed to be advertizing STARTTLS, which is opportunistic encryption upon request and it’s been disabled.

JP probably has the best guess of what’s taking place, the problem is we don’t know where the ASA resides.

This could in fact be on GoldenFrog’s network and they simply forgot to turn off SMTP fixup, or it could be in bridge mode on the wireless carrier. Either way a simple call and one line in the config will fix the issue. I’d chalk this up to stupidity more than malice.

Kenpachi says:

Re: Re: In need of some basic clarification...

Thanks a lot. On the token of “stupidity trumps malice”, call me a tin foil, but it’s undeniable at this point, when every day the entire world is waking up on what has been and is happening behind the curtains, that malice is winning by a landslide (as far as government and corporate overreach goes).

Of course, that does not take anything away from stupidity, which is rampant. But if even with all the facts and evidence that are a click away, we still attribute to stupidity things like this, we are really making it exponentially easier for this things to keep on going.

In any case, ultimately it does not matter the nature of the beast, what matters is how to avoid it. And always be vigilant of things like this.

John Fenderson (profile) says:

Re: In need of some basic clarification...

If you’ve configured your mail reader to require an encrypted connection and that connection cannot be set up, then the mail should not be delivered and you should get an error message. If this is not happening, you’re using a terrible mail reader and should use a different one.

Shawn (profile) says:

Re: Re: In need of some basic clarification...

SMTP is as we all know by know a clear text protocol used for email for the last 32 years give or take. About 12 years ago the SMTP protocol was extended (ESMTP) to support among other things TLS (Transport Layer Security) which is basically SMTP over SSL. Since ‘plain old’ SMTP was (and probably still is) the most popular method of delivering email the Extended commands are optional. STARTTLS is called opportunistic because if it can use TLS it will, otherwise it will fall back to plain old SMTP and deliver the email in clear text. You can configure your server to only use TLS and it will refuse to send email ‘in the clear’ but if you do this on a border email gateway (one that sends and receives to the general Internet) You will quickly see that many many many email servers are not configured to accept TLS connections and you will have a lot of dead mail on your server that could not be delivered.

Kenpachi says:

Re: Re: In need of some basic clarification...

Thanks man. Now, what really bothering me is what’s happening at the other side of the pond. That the user adopts good practices when configuring software is a must. Been there, done that.

And then there’s this little tiny thing, where at the other side, a 3rd party is actively disabling that way of communication. Whether an encrypted communication is possible or not, that’s not an issue, the fact is that they are, arbitrarily and without user intervention, making it impossible to use a secure method.

If that’s not a dead giveaway of the intentions behind it, I frankly don’t know what is…

Taking into account the fact (well known to ISPs and the like), that 99.9 % of internet users do not have the first clue as to how to properly configure an email client.)

I don’t know man… I hope it’s all just a perfect blend of my level of paranoia and my lack of understanding of the subject matter.

Anonymous Coward says:

Re: Re: Re: In need of some basic clarification...

“aking into account the fact (well known to ISPs and the like), that 99.9 % of internet users do not have the first clue as to how to properly configure an email client.)”

This is correct. 99.9% of Internet users also do not have the first clue how to even select an email client, which is why — in 2014 — spam, phishing and malware distribution via email are still trivially easy.

But wait: it gets worse. A large fraction of those Internet users have their email client forced on them by organizations run by ignorant newbies…who specify things like Outlook, despite a continuous history of Fail dating back to its initial release.

So yes, ISPs and others are doing stupid dangerous unethical unprincipled things at all layers of the IP protocol stack, but a hell of a lot of users are undercutting themselves in far worse ways.

Dylan says:

Re: In need of some basic clarification...

Many servers (and clients) will attempt to relay a message over SSL/TLS, if possible. If SSL/TLS is not available, many servers/clients will relay the message in plain text.

Apparently encrypted email is a relatively new thing. Many servers are/were not equipped with SSL capability. Expecting encryption for all of your email was unrealistic and was treated as a “nice to have.”

By breaking the STARTTLS negotiation, but servers/clients will silently switch to plain text transmission.

I’ve personally set my server to REQUIRE secure connections (it’s no longer a “nice to have.”) I hope everyone else has seen enough and started to do the same.

mike says:

Re: Re: In need of some basic clarification...

Wondering if anyone that uses ssl/tls for email on their server is having the following issue.
When ssl tls is enabled on my server email from verizon users is not accepted. Logs show unrecognized command with uid as a command from verizon. Turn off secure delivery and email from verizon users comes in!

Anonymous Hero says:

?

I’d like more info about this.

First, there’s some cognitive dissonance going on here. On one hand, ISPs are blocking encryption, on the other hand, people are seeing better throughput if they proxy their network traffic through an encrypted channel (i.e., VPN). This would suggest that the ISPs are not targetting encryption as a whole, but rather are targetting ESMTP or STARTTLS specifically.

Golden Frog’s write-up, is confusing to me. I think there is a serious problem with downgrading encryption at the ISP level, but the Golden Frog report leaves me with many more questions than it answers. Is it SMTP over port 25 specifically? What other services are being downgraded? Etc.

Anonymous Coward says:

legitimate use?

So, as referenced in the above links, Cisco’s “ASA Series Firewall” apparently has a feature where it rewrites the mail server’s response to the EHLO command to hide that the server supports STARTTLS. It also intercepts and fails any client attempt to run a starttls command.

Why would it do this? Seriously – what is the supposedly legitimate use of this feature? This stinks badly of a some corporate-internal-spying trick similar to how they put their own “trusted CA” cert in PKI settings for company laptops, so they can run a Man-in-the-Middle attack on all HTTPS usage by routing traffic through a transparent proxy.

The last time I saw legitimate editing of in-band content like this was during IP Masquerading (NAT) to fixup old, pre-IPMasq protocols such as FTP that sent the local IP address in-band. If there is such a legitimate use for stripping encryption while still forwarding to the real SMTP server, I’d love to hear about it. For now, while this story may be “legitimately” be a case of someone using a Cisco ASA firewall, why they are using a firewall with such a compromising feature?

As a side note, Cisco’s (voluntary or involuntary) involvement with the NSA and the resulting “implanted” hardware has not gone unnoticed. How this relates (or not) to the issues mentioned in the story is an open question that is worth considering.

Kenpachi says:

Re: legitimate use?

Of course it has everything to do with the NSA’s takeover of the internet. Now that the rest of the world has a clue as to what to look for, these things are popping up left and right.

The problem with a lot of technical people having trouble in differentiating this from misconfigurations or simple stupidity, is that they are attacking the problem from an engineering point of view.

It is to be expected, after all, it’s really hard to reconcile the reality of this madness and not feel overwhelmed and/or powerless.

The first reaction is to tackle the issue with an “IT mentality”, to speculate on the nature of what’s being discovered, and to try to come up with a solution/workaround.

Anyway, the most important thing is, in my view, that good people around the globe, who work for a free and open internet, keep looking for things like this and bring them to light.

Nobody asked or wanted to be dragged into this, but we need to come together and defend our privacy, our freedom of communication/association, without monitoring or outside interference.

Anonymous Coward says:

Re: Re: legitimate use?

I generally agree with your points – especially with regard to the blindness common in engineering (I say this as an engineer myself). I just like to sanity-check that I’m not missing some obvious explanation that I might have missed. Compatibility workaround for legacy software (like the FTP+IPMasq/NAT issue I mentioned) would be a good example. Until I hear such an explanation, I tend to believe that this is a feature created with poor intentions.

What many engineers don’t seem to understand is that while the technical is certainly important, if you ignore the political side you surrender that front to those that are not ignoring politics.

While it is from a work of fiction, I think this quote from Susan Ivanova in “Sleeping In Light” (the last episode of Babylon 5) said it nicely: (emphasis mine)

It taught us that we had to create the future, or others will do it for us.
It showed us that we have to care for each other – because if we don’t, who will?
And that strength sometimes comes from the most unlikely of places.

Mostly, though, I think it gave us hope that there can always be new beginnings, even for people like us.

Shawn (profile) says:

Re: legitimate use?

The use at the time it was introduced (and ESMTP was less common) was to enforce the RFC 821 https://www.ietf.org/rfc/rfc821.txt This stance limits it to only accepting only the origional SMTP commands and blocking anything else. As to why it is still defaulting to that mode (you can configure it to allow ESMTP) I couldnt hazzard a guess. I have never been a fan of dealing with SMTP security on a firewall vs a good SMTP gateway but some people want the firewall to be the be-all end all in their border security. I am still leaning towards laziness vs malice

Anonymous Coward says:

Re: legitimate use?

“why they are using a firewall with such a compromising feature?”

Because they don’t know any better.
Because they’re not aware of this problem (and others).
Because nobody gets fired for buying Cisco.
Because they only know about Cisco and don’t want to learn anything else.
Because they think Cisco is the be-all end-all.
Because they don’t care if this is a problem.
Because it’s not a problem for them.
Because they have to justify their bullshit Cisco certifications. (They really are, you know.)
Because it’s what they’ve always done.
Because they don’t want to learn.

There are more excuses, of course; there will always be more. But this little list explains much of what’s wrong with the current Internet environment.

Anonymous Coward says:

Re: Re: legitimate use?

The question isn’t why somebody would buy a Cisco firewall that happened to have this feature. I’m sure “Nobody ever got fired for buying Cisco” is a common first and last thought when choosing vendors.

The issue is why a security-disabling MitM-attack feature exists at all.

It’s one thing if Cisco created it to work around, for example, some legacy compatibility issue. It is a very different thing if it is just some idiotic “pop corporate design” that is totally unnecessary and just happens to allow that corporation to log all the emails that went through their firewall.

Anonymous Coward says:

Re: Re: Re: legitimate use?

“The issue is why a security-disabling MitM-attack feature exists at all.”

Because security is not — to them — an important consideration. Minimizing the number of hours spent on support calls (and thus the cost) is an important consideration, so they’ve made a conscious decision to include this and enable it by default in order to save money.

That shouldn’t surprise anyone: Cisco, Oracle, Microsoft, Apple, all the major vendors have done this for years. Oh, it’s all (almost all) documented and anyone that knows what they’re doing can change/fix it, but they’ll likely find that if they need support post-change, the first thing that the help desk personnel will them is to revert that change or reset to factory defaults, because “it’s not a supported configuration” or “it’s not a recommended configuration” or whatever.

This is yet another of the many reasons why so many security breaches and dataloss incidents happen. Even moderately clueful mid-level engineers who are trying to do things right and who want to do things right are consistently undercut, e.g. CTO: “Just set it up like Cisco says and we’ll live with it”. Engineer: “But it’s wrong.” CTO: “Do it anyway.”

Until the people at the C-level are held personally accountable for their decisions, including purchasing, deployment, and maintenance, this won’t change. Whoever’s responsible for the fiasco at JP Morgan will keep their job, or at worst, be given a golden parachute and a reference that will let them land an equivalent one across the street. Meanwhile the poor schleps who have been working 90-hour weeks trying to do damage control will be saddle with their next boss, who will also be on the way out from somewhere else and will repeat the exercise.

And Cisco will happily sell them more product any time they want to write a purchase order. Thus the symbiotic relationship and race to the bottom continues.

Josh in CharlotteNC (profile) says:

Re: Re: Re: legitimate use?

The issue is why a security-disabling MitM-attack feature exists at all.

Because everyone’s needs are different, and that feature is just a tool that has legitimate uses.

and just happens to allow that corporation to log all the emails that went through their firewall.

That is a perfectly legitimate use – if you’ve never worked in a highly regulated industry – like banking and financial services – then you might not be aware that there are regulations that require just that in some situations.

Anonymous Coward says:

Re: I need to find someone to remove my tinfoil hat, but

Of course it makes it easier for the NSA. Getting people to not use encryption – by corrupting standards, bribing/convincing people that run infrastructure, and many other methods – is probably their primary tool. You don’t break encryption by brute force when you can go after a side channel.

I strongly recommend watching PHK’s talk that explores these ideas called “NSA Operation Orchestra“.

Brett Glass (profile) says:

That looks like an anti-spam proxy.

That behavior does not look like the ISP’s equipment is altering messages from the remote server, but rather as if it is redirecting the session to a proxy server — most likely for the purpose of stopping outbound spam.

Such a proxy can send the mail onward in encrypted form once it’s checked for spam. If this is the case, and there is encryption or even just physical security on the inbound connection, the privacy of the message isn’t compromised.

It’s sad that such measures are necessary, but they are — not only in public hotspots but in situations where a new user might sign up for new service from an ISP and then send boatloads of spam until the account is turned off. (This is happening a great deal nowadays.)

Anonymous Coward says:

TLS traffic...

Actually… I see this occasionally myself a lot… I work in a tech support group that works with many other email servers, and usually this is due to some router on a network that’s misconfigured to inspect SMTP traffic, which usually follows strict SMTP spec, as opposed to ESMTP which includes the usage of TLS.

gojomo (profile) says:

Confused filing & neturality definition

It’s very suspicious that Golden Frog is not naming this “one mobile wireless company’s data service”. Other than the context that this is a filing to the FCC, can we even be sure it’s in the US?

Such in-flight data-tampering is a very different animal than traffic prioritization, and shouldn’t be lumped together other. As a rare and egregious practice, it could be solved via name-and-shame in the market, or even basic laws against fraud. It doesn’t need a federal ‘neutrality’ bureacracy.

(It’d also be relatively easy to work-around via self-help… such as VPN-like tunnelling.)

Christopher Smith says:

This is spam prevention, not something malicous

Ummm… the specific case they are referring to doesn’t sound like anything nefarious.

The specific example is using STARTTLS with SMTP at port 25, which is a *very* important and specific case. Many, many ISP’s block or filter direct SMTP traffic that doesn’t go through their own SMTP gateways. This is because by talking at port 25 you can effectively impersonate a mail server, thereby creating…. SPAM! Particularly for consumer facing ISP’s, this is a very common problem because spammers exploit botnets on their customers to deliver their SPAM.

The work around has been to declare a different port, 587, as a “submission” port. It speaks the same protocol, but the server knows that it should only accept messages that it is a reasonable SOURCE for (basically they know it is one of their customers so they can require authorization).

Lots of ISP’s outright block port 25. These guys are being nice and leaving it open but filtering for spam activity. Once it switches to STARTTLS though, the ISP can’t see anything that is going on, and spammers exploit the heck out of this.

So in fact this ISP is getting ripped to shreds for being *more permissive* than a lot of other ISP’s.

If an ISP gets labeled as a source of SPAM, the consequences are dire, so this kind of intervention isn’t considered evil or deceitful by anyone I know.

If this actually causes a problem for a customer, they can report it to tech support. The usual protocol is that someone will either inform them of the 587 option, or in a lot of cases just open up the access between that machine and whatever servers they are trying to talk to (it might let some spam through, but if it is actually causing a problem, that’s worth allowing).

Techdirt should know better.

GEMont (profile) says:

Business as usual

Sadly, this mess is probably just another “sheep to the sheering pen” scam, where the players – Verizon and their ilk, and the FCC – bat the ball – Net Neutrality – back and forth, until the public demands a set of rules/laws be created to protect their interest.

Then the players sit down and manufacture another “for the children” law where the wording is just loose enough to allow the whole wish list of the players to be interpreted from the text, after the laws are in place.

The government can then claim that the Public “made’em do it”, so its not their fault that the rules turned out to be exactly what the Players wanted and nothing that the public wanted.

Methinks you will get your new Net Neutrality Rules.
But I don’t think anyone but the Players are gonna like’em.

Zehd says:

Um, guys.... calm down a moment

This can very well be simply an ANTI-SPAM measure implemented for the express purpose of blocking botnets from sending spam using these wireless connections.

End users should not be using port 25 to connect to their SMTP hosts to begin with–they should be using either 587 or 465. Port 25 is often blocked altogether from end users connecting to any mail server but the one hosted by the ISP–but it’s quite possible that the ISP may provide a firewalled mail service that is MITM to block spam from being relayed–but the middleware cannot work if the connection is encrypted–they cannot detect that the connection is from a botnet trying to encrypt the session or a regular computer (since the login request/response is encrypted). The way they are doing this means that they on the one hand can be assured that an infected machine cannot simply connect directly to thousands of mail servers to send spam–but on the other hand don’t have to field all the customer support complaints about not being able to send emails at all and having to walk them through changing from port 25 to a different port number (port 587 at least is usually configured to require authentication even if the recipient is local to that server).

J-P says:

throttling == connection speed

He argued that, perhaps there could be some threshold, and if your traffic was above that threshold it’s okay to throttle it.

That sounds quite fair to me.

Actually that sounds EXACTLY like it already IS currently.

If you get 100Mbps connection, you get throttled at 100Mbps. With 10, the limit is 10 etc.

If you get any less, you don’t get what you are paying for (or the problem is at the other end).

stoat (profile) says:

The real error

Is that they allow endusers access to world:25 AT ALL.

Most ISPs nullroute such traffic and with good reason – it’s the #1 vector for spam traffic.

Endusers (you and me) use MUAs, not MTAs. MUAs talk to MTAs on ports 993 (imaps) or 465/587 (smtp auth).

Allowing endusers to directly connect to an external MTA allows spammers to do their thing, which in turn means the IPs involved get blacklisted. If you have a few clues to rub together and can demonstrate you can run a MTA without annoying the world then most ISPs will punch a hole for you.

The real question is “Why on earth do people such as Golden Frog think they can run a MTA from a consumer connection, without proving they’re competent to do so?”

The fact that they can’t interpret a PIX firewall’s responses show that they lack that competence.

chilling farts says:

That's why FCC bribed peruvian regulator?

Inside peru at year 2012 i explained something similar with ISP America Movil (aka Claro). While they tried to trick the old Glasnost test (used against them in a claim at 2011) some encrypted sites doesn’t load correctly.

Months later, they blocked the swedish and criticized service VPNTunnel via DNS. Affected service was mobile internet.

The report was in spanish, essentially i discovered some differences on VPNTunnel launcher behavior filtering it’s data via MS Network Monitor.

https://es.scribd.com/doc/109732457/

Oh, and Why i talked about FCC? Our regulator OSIPTEL published a neutrality bill last year to intensify infringements against net neutrality, but it got lost after a meeting with FCC in mid 2013. Months later, other bill opened some exceptions to net neutrality instead.

John Thacker (profile) says:

At this point, some news outlets would breathlessly reprint an article claiming that blocking spam from an open mail relay was a violation of net neutrality– and I suppose by some definitions it technically is. I suppose John Gilmore supports his open mail relay for that very reason.

Network neutrality seems like a simple concept, but if you don’t take the strong Gilmore position that things like open mail relays should be allowed, actually writing regulations is pretty tricky (and having them be flexible enough to ban new bad things but also allow new good things.)

GEMont (profile) says:

Criminals don't break law when law protects criminals

Making monopolies illegal again might be the best way to slow this whole thing down long enough to get a good handle on it.

Monopolies used to be illegal, for a very good reason.
Does anyone remember what that reason is, or was?

Monopolies are now a standard business model.
Does anyone know why monopolies are now legal?
Did the reason for their illegality go away?

rtechie (profile) says:

The article is 100% wrong

I had to create an account here because this article is so misleading.

I did some research and this is actually just a bug in that particular Cisco ASA:

“Yes, if you upgrade to the newest firmware (version 8, my ASA is running 8.0(4)) then it support TLS in the esmtp inspection policy.”

It also may be that they’re using the latest firmware, but they haven’t explicitly enabled STARTTLS, which is turned off by default. https://stomp.colorado.edu/blog/blog/2012/12/31/on-smtp-starttls-and-the-cisco-asa/%5B1%5D

So Golden Frog could have just contacted the ISP and asked them to fix their ASA.

In any event, this is definitely not malicious on the part of the ISP. It is normal for consumer connections to interfere with SMTP server traffic from consumer hosts. This is to prevent spam.

By having this appliance in place this ISP shows that they were being good to customers. The alternative is just to block all SMTP traffic, and many ISPs do that.

Notafraid (profile) says:

This story is insane

First off, much of the info on this issue is static. There are a bunch of Red Herring issues that are brought up, while the guy is actually arguing that we need to worry about private corporations wanting to deny us encryption, and freedoms. His suggestion is that we put the Internet in America fully under government control, under a mountain of laws and then we will be save from violations….

Basically violations by the government will no longer be called violations. They will just be obeying the law. All kinds of internet traffic and piracy will be deemed illegal, and the packets will be stopped, and they will have the total ability to do that.

There will also be billions in new taxes applied to everyone’s internet services, so if you are worried about paying more without Net Neutrality, you can be assured that you will absolutely pay more with it. The internet will be far less free, and far more expensive to everyone in America.

It will also benefit big business, and harm small business. It will be harder for smaller ISP’s to comply with the mountains of government regulations. Big businesses with big money will also be able to buy favorable outcomes and decisions for themselves because, if you think the control Washington has now over the internet, it will be 10 times more power, and so only those with the means will be able to buy what they want.

Unfortunately there are a lot of very smart useful idiots who are pushing for this. They are naive, and they blindly trust government. They are like zombie drones going around making all the arguments for the big money players who want it to happen. They believe all the articles that big money gets pushed. They don’t get the fact that they are just being used.

kmcg3413 says:

The fix will require root certificates and authentication of the remote endpoint, and then negotiation of encryption. The SMTP protocol is very insecure and high susceptible to attacks, and that is why the ISP went that far to begin with.

I am not advocating root certificates as they exist today, but rather the design of that mechanism. It eliminates the ability for the endpoint to pretend to be someone else, and it eliminates your ability to accidentally confirm that endpoint is who they say they are.

This is a game that the ISP(s) can not play forever… trust me.

The game they can play is just completely banning encryption and forcing you to use their own protocols, but because of the complexity of applications out there right now and the differents ports and protocols (udp/tcp/ipv4/ipv6) it is a difficult road, and it will take a long time for that to happen.

mini mac says:

blocking torrents and encryption

My ISP, Casair Inc, is blocking the use of encryption and torrents. My service was suspended for 6 hours today because I filed a support ticket about it. Also under Net Neutrality, the ISP can still throttle when they are at 70% network utilization but they can NOT block it entirely. Just wanted to share some info that I had found. Also, if your ISP is violating the Internet Openness Policy or Net Neutrality Act, you CAN file a complaint with the FCC.

tommy says:

With regards to the *******

If you have a Cisco ASA hooked up to an ISP like comcast. It is not actually Comcast that is **** out the Starttls The issue here is SMTP fixup which is a setting you can turn off on the cisco asa. It messes with ESMTP commands including starttls. I dont know about other parts to this post, but I am very certain of that is going on here.

Kenny Hendrick (profile) says:

TWC IP PROVIDER COMPROMIZED BLACKLISTING DISSENT

I have a site which pretty much spells it out.

http://timewarner.computer-repair-springfield.com

I think what started this whole ordeal of finding myself on 1, then 3, then 13 permanent blacklists (even after changing modems, routers, op sys’, etc.) was the fact that I was posting rather heavily videos, comments, and pictures containing political content (not exactly mainstream monopolistic media form).

Life sure changes after being targeted by the “good”guys.

rtechie (profile) says:

Re: TWC IP PROVIDER COMPROMIZED BLACKLISTING DISSENT

Let me explain this to you AGAIN.

You, that is a customer using a consumer Time Warner cable modem, are NOT ALLOWED to run a private email server on your connection. Period. This is because countless spammers in the past have abused this to send spam.

Nowadays, if you want to send lots of email you are required to pay email providers like Yahoo and use security technologies like DNSSec and DKIM. IF YOU DON’T DO THIS YOU WILL BE BLACKLISTED BY ISPS FOR BEING A SPAMMER!

Did you even read this notice that you posted?
http://timewarner.computer-repair-springfield.com/Used%20Supporting%20Documents/Time%20Warner%20additional%20violations/Claim%20from%20Zen.Spamhaus.jpeg

This says Spamhaus blocked for running an unauthorized mail server.

You’ve also got ads for a 9/11 “truther” book in your little site.

You’re a paranoid nut who doesn’t understand how ISPs work.

Just stop.

Add Your Comment

Your email address will not be published. Required fields are marked *

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »