Verizon Gets Snarky, But Basically Admits That It's The One Clogging Its Networks On Purpose
from the snarky-admission dept
So the war of words over interconnection has continued. Last week, we wrote about the back and forth between Verizon and Level 3 on their corporate blogs concerning who was really to blame for congestion slowing down your Netflix video watching. As we noted, Level 3 used Verizon’s own information to show that Verizon was, in fact, the problem. Basically, in spite of it being easy and cheap, Verizon was refusing to do a trivial operation of connecting up a few more ports, which Level3 had been asking them to do so for a long time. In other words, Verizon was refusing to do some very, very basic maintenance to deliver to its users exactly what Verizon had sold them.
Earlier this week, Verizon went back to its blog with another blog post from David Young, this one even snarkier than the last. Snark can be fun, but if the underlying message is completely bogus, you’re going to run into trouble. In fact, Young’s underlying message is so weak, that he more or less admits to absolutely everything that Level 3 was claiming in its post — while pretending it’s Level 3 that actually admitted fault!
Last week, Level 3 decided to call attention to their congested links into Verizon?s network. Unlike other Content Delivery Networks (CDNs), which pay for connections into ISP networks to ensure they have adequate capacity to deliver the content they have been hired to deliver, Level 3 insists on only using its existing settlement-free peering links even though, as Level 3 surprisingly admits in their blog, these links are experiencing significant congestion. Level 3?s solution? Rather than buy the capacity they need, Level 3 insists that Verizon should add capacity to the existing peering link for additional downstream traffic even though the traffic is already wildly out of balance.
Except… no. Level 3 did not, in fact, call attention to its congested links. It showed that Verizon was the one making them congested by refusing to do the most basic thing that Level 3 had asked them to do: open up some more ports. The claim that Level 3 needs to “buy the capacity” it needs is simply wrong. As was quite clear, Level 3 has plenty of capacity. The problem is the bottleneck… and the bottleneck is Verizon. And Verizon is refusing to fix that bottleneck unless Level 3 pays up. And not the cost of the upgrade. Remember, Level 3 offered to pay the cost of the upgrade itself. Verizon, instead, is trying to change the nature of the deal, allowing its border routers to clog on purpose to force Level 3 to pay a totally new kind of fee to free up the bottleneck that Verizon itself created. It’s basically acting as a classic troll under the bridge — failing to deliver what it promises both sides of the internet market, unless it can squeeze a ton of extra cash from Level 3.
Most of the rest of Verizon’s snarky post takes a fight that Level 3 had with Cogent a decade ago concerning peering totally out of context. In that fight, it’s true that Level 3 cut off peering to Cogent, arguing that Cogent was using much more traffic than Level 3, but that was a true peering arrangement between two transit providers, rather than a connection between a transit provider and the monopoly provider of the end users (who has sold connectivity to those users with the promise that it will enable them to access content from any website). The traffic ratios argument between a downstream/last mile provider and a backbone/transit provider is ridiculous. The traffic ratios have always been way off in part because the broadband providers themselves have always offered more downstream bandwidth than upstream bandwidth.
So, Verizon sets up a world in which the traffic ratios are always going to be off… and then complains that the traffic ratios are off and thus it needs truckloads of extra cash just to connect up a few more open ports? Yikes. Verizon’s snarky post simply confirms what many of us have been saying from the beginning. The company is deliberately letting its border router clog up because it wants to wring a lot more money out of other companies, based on a plan to twist old peering disputes between transit providers into a dispute about transit-to-last mile connections… when the traffic ratio has always been way off, in part because of how Verizon itself designed its network! That takes incredible hubris… or incredible market power. Maybe both.
Filed Under: clogging, david young, interconnection, net neutrality, open internet, snark
Companies: level 3, level3, netflix, verizon
Comments on “Verizon Gets Snarky, But Basically Admits That It's The One Clogging Its Networks On Purpose”
This is like a company demanding UPS shove all the packages it ordered for it’s company headquarters be delivered through a 6″ mail slot and then blaming UPS for the congestion.
It’s Verizon’s customers who ordered the data, data Verizon promised to deliver them at high speed. Netflix sends it straight to Verizon at locations near where the data was requested, all paid by Neflix and delivered via via Level 3, and instead of saying “thanks for getting that to us” Verizon is saying “how dare you! You should pay us!” This is just all sorts of effed up.
Re: Re:
Its more like shipping a package with UPS overnight, it not getting there for three days and when you call them on it they complain the freeway was busy.
This is pure extortion; a content provider buys access to the internet via his ISP at a certain speed or range of speeds (minimums, burstable max’s, etc.) and rate per gb.
You buy access to the internet also at a speed and usage rate.
What the ISPs are trying to do is charge companies extra if consumers choose to use a larger percentage of their usage on them. This disregards the fact they those companies are already being governed by a set of contractual standards which determine their pricing.
I wonder how much this will change things…
http://www.usatoday.com/story/tech/personal/2014/07/23/fcc-broadband-speed-transparency/13043983/
Re: Re:
Ok then.
How about now… Can we call this anti-competitive monopolistic actions now?
Re: Ok then.
Or do you mean: Can you hear me now?
Re: Re: Ok then.
No, Verizon can’t because they made themselves deaf to customers yonks ago. Simples!
Re: Ok then.
Netflix can sue Verizon if they want.
Re: Re: Ok then.
Re: Ok then.
Are former Comcast Execs still running the FCC? Are congressmen whoring themselves to cable companies for tens of thousands of dollars in “contributions”?
If so, nobody is going to call it anti-competitive. It will be an emperor’s new clothes situation.
“The traffic ratios argument between a downstream/last mile provider and a backbone/transit provider is ridiculous. The traffic ratios have always been way off in part because the broadband providers themselves have always offered more downstream bandwidth than upstream bandwidth.”
Thanks for that. I almost fell for Verizon’s argument.
Verizon
Here’s the proof:
http://iamnotaprogrammer.com/Verizon-Fios-Netflix-Vyprvpn.html
Re: Verizon
It would appear that the blogger doesn’t understand how the internet works, and you didn’t bother to catch that fact.
A Willingness for Dialogue
It is telling that the Level3 blog post has 192 comments, and Verizon’s blog doesn’t allow for comments.
Re: A Willingness for Dialogue
reminds me of the lily tomlin routine as a telephone operator (humor grandpa, kidz) where she answers the switchboard (google it) “ATT, we don’t care, because we don’t have to!”
but, the sad state of affairs is, most sites of nearly any type don’t allow either posts, or critical posts, much less warts-and-all free speech…
besides, hasn’t it been shown we actually are foetal battery-blobs in the NSA’s matrix ? ? ?
(okay, spel czech, i might have to give you foetal… aha! no, i don’t! i retract the apology, it is a britishism, but it is a word, you POS…)
People pay Verizon to connect them to the internet. Verizon simply isnt fofilling their end of the bargin. No one pays to be on verizons network, they pay for their internet connection. How can they sell bandwidth if their own network cant support it? Its bullshit and its time for a class action.
All Verizon customers should cancel their service and take their business elsewhere.
Hit Verizon where it hurts…. their bottom line.
Verizon sucks.
Re: Re:
The problem with that, is there is usually no where else to go. For instance, I have comcast, even though I hate comcast. But they are the only service I can even get, not counting dsl, which is not really comparative, so I’m stuck with comcast. That’s how most places are, they have one option for high speed internet, and maybe they have a second option for non-comparable internet
Re: Re: Re:
I also don’t like Comcast. Like you say, they are the only player in the game.
I don’t know much about “Ma Bell” and “Baby Bells” but maybe we need some new players. Maybe economies of scale only work to a certain point. Maybe some of the biggest players need to be broken up so there is more competition in the market place.
Comcast is far too high priced. At half the current price they might be OK. There internet is great but there TV is crap.
Re: Re: Re: Re:
The problem is building the necessary infrastructure is cost prohibitive for there to be multiple providers under the current system where each provider owns the infrastructure. What needs to happen is to create a regulatory system similar to what was setup for electricity in Texas several years ago whereby a company cannot provide retail service and own the infrastructure at the same time. The owner and maintainer of the lines would have to provide access to customers of resellers at a government set rate. The retailers then compete for customers in the market. This would solve the problem.
Re: Re: Re: Re:
Electric companies getting into fiber ISP. G.fast coming will change the landscape as well. The competitive moat is changing for legacy telecom and they are scrambling to defend the castle as the moat evaporates. VOIP even threatens mobile revenues in populated areas….Google looking to use old NYC pay phones as WiFi hot spots. Massive transition in tech and pricing structures has legacy telecoms pulling out all the stops to stall the advance.
Re: Re: Re:
It seems like Cox is the best american internet provider, I know it from all my years of researchin’ (just kidding, just a graph i’ve shown regarding youtube speed).
I thought it was a widely available ISP after seeing so many people on it on irc.
Re: Re:
I only use Verizon for my mobile service, a service I need to be reliable to perform my job.
Verizon Wireless is reliable everywhere I go.
AT&T delivers my friends SMS messages hours late.
Sprint does not work at my house.
T Mobile sucks at my work.
I would love to drop Verizon, but I NEED their reliable wireless service.
I am sure there are lots of users in the same boat with their wired Internet service. They need it and Verizon is the only option.
Boycotts don’t work when there are no alternatives.
Re: Re:
And go to Comcast? Fuck that.
Re: Re:
Sure…they will just do without the internet until comcast goes out of business… solid plan.
Re: Re:
Here in America the internet company’s have a monolopy over most states so most of the time you only have one choice
Re: Hit the Bottom Line
Maybe take it one step further…
If Netflicks stops all traffic destined for Verizon until steps up with connections, or Customers leave Verizon..
Re: Re: Hit the Bottom Line
If Netflicks stops all traffic destined for Verizon until steps up with connections, or Customers leave Verizon..
If their customers could leave, Verizon would never have pulled this stunt in the first place.
Re: Re:
How can they? Who would they switch to? In most markets, while there are TECHNICALLY other options, the others are either WISPs or just as terrible, if not worse, than Verizon. Especially those on FiOS, a few people may be lucky enough to have Google Fiber, or some smaller company providing fiber to their door, but the majority are totally out of luck. So your option is to switch to a smaller ISP and get a smaller (and potentially slower) pipe, never mind any early termination fees for anyone under contract or installation fees for the new service, or hope that eventually this madness ends and Verizon actually provides the service they promised. In the US surely this game they’re playing can only go on for so long before someone comes to the aid of the customers…right?
Free market versus regulation
At what point does the balance of opinion change from a “competitive free market” to “government regulation” as a path that is for the betterment of everyone?
Personally, it’s stories like this that sway me toward the side of regulating cable companies like public utilities.
Re: Free market versus regulation
When the people get angry enough, and in enough numbers, that their collective ‘voice’ is loud enough to drown out the noise of all the ‘campaign donation offers’ the companies are offering.
Re: Re: Free market versus regulation
Which is a physical impossibility.
Re: Free market versus regulation
Radarmonkey, Spot on. They are going to become public utilities with VOIP if the telecoms get their way with agenda to abandon copper. Make them class II and put the hammer on them with net neutrality. Throttling is antitrust when the ISP is bundling its own video services.
Re: Free market versus regulation
At what point does the balance of opinion change from a “competitive free market” to “government regulation” as a path that is for the betterment of everyone?
If we had a competitive free market, we wouldn’t be in this situation.
Re: Free market versus regulation
I would prefer a competitive free market, but that seems like a goal that is unattainable within my lifetime. I’ll take Title 2 classification as second-best solution. That seems more possible than getting a competitive market.
Can ya post something uplifting once in a while, Mike?
Re: Re:
Wasn’t there a post about a balloon outfitted with a wifi gateway or something like that? That was uplifting, no?
As long as there is ANY sort of broadband monopoly allowed by the US government it means the US isn’t the land of free markets.
It’s only a free market if those paying bribes to government officials don’t want complete control of a service/product.
Bribes are basically how companies such as Apple can have a monopoly and get praised for it, but Microsoft gets an anti-trust investigation.
The funny thing is, Verizon sells it’s FIOS customers an asymmetric connection (I’m 25 down, 10 up). Verizon is very much aware that their customers will have an imbalance of traffic since that is how they sell it!
Re: ..asymmetric ..
A bit offtopic but VZ has gone symmetrical for new and current customers, just for info (announced 2 days ago):
http://arstechnica.com/business/2014/07/verizon-fios-finally-symmetrical-upload-speeds-boosted-to-match-download/
I’m not sure 25/10 has been available to new customers recently, you could call them to verify that you’ll get 25/25
retroactively prove it should be peering..
http://tech.slashdot.org/story/14/07/21/1545214/verizon-boosts-fios-uploads-to-match-downloads
Verizon preempted this already. Simple solution… boost upload speeds to match, which of course they don’t expect anyone to use (or they have hidden caps on)… tada, now it’s peering! Now onto making people believe the fact that they get paid by the customer isn’t enough. Oh, right…
Re: retroactively prove it should be peering..
That might matter if Verizon was rolling FIOS out to new places. They’re not.
Re: Re: retroactively prove it should be peering..
Josh, When did VZ stop rolling out FiOS? 2010. When did G.fast tech work launch? 2010 http://arstechnica.com/information-technology/2013/12/500mbps-internet-over-phone-lines-might-solve-fibers-last-mile-problem/
When did G.fast receive standardization? April 2014. When is commercial launch expected now that standards were finished early (was expected no sooner than EOY 2014)? 2015. Is this tech going to impact cost structures when 1 gbps and pricing? You can bet your backside. 80% cost of install reduction vs. full fiber-to-the-home. Upload/download can be customized but as long as the fiber node is 100m or less you have 1 gbps at your discretion of upload/download. 800/200 or 500/500 simultaneous. Last Fall announcements hit that G.fast has been adapted to coaxial in addition to the work on twisted copper telephone lines. If Google can turn a profit on $120 ISP/TV bundle with cost of full fiber install then what is pricing with 80% of install cost removed using G.fast? $80 for an ISP/TV bundle? G.fast is the introduction of internal combustion engine for the old horse carriage. Buggy whip makers are worried and the barrier for entry is falling for content bundlers and producers. DTV purchase by AT&T has me ROFLMAO. DTV/DISH are today’s version of AOL market moat.
Re: Re: Re: retroactively prove it should be peering..
Your post looks like an advertisement and doesn’t really relate to the topic.
I’m aware of those lab tests. And standardization doesn’t necessarily mean a lot. I’ll believe it when I see it operating in real world conditions in a large deployment. I also highly doubt Verizon will be rolling that out to replace their DSL lines.
So basically, if I’m reading it right, Level 3 offered to spend their own money to buy and set up more connections for Verizon. Then Verizon turns around and demands Level 3 spend their own money.
What. The. Fuck.
Re: Re:
NO NO NO they want them to pay for the equipment then also pay a fee for sending traffic that their customers have demanded. Sadly this will be twisted to make it look like Level 3 is wrong and some stupid government paid for politicians will support them…the only way out of this is if customers sue and netflix sues, especially if netflix have paid any money to them to resolve the congestion problem, then it would be clear fraud on behalf of the isp and they would be made to compensate Netflix and their customers for deliberately degrading their service..I wish they would be forced to pay at least 50% of their profits but I believe in the US it will probably be a few hours profit for the isp as a fine and nothing to anyone involved.
Re: Re:
Verizon is demanding that Level 3 pay them for the traffic exchanged, not pay to put more ports in place.
Re: Re:
that’s oversimplified to absurdity. level 3 offered to pay for the one time hookup, verizon wants a recurring fee for the “bandwidth imbalance”. but bandwidth imbalance seems like bullshit. it is not true that verizon would be receiving more data than level 3 is sending – the same amount of data has to travel over both their equipment stacks. i don’t think that the data transmission is going to cause significantly more wear and tear on verizon’s equipment. so in effect verizon wants level 3 to pay both maintenance fees. why can’t level 3 say “your customers are drawing more data from across our pipes, you should pay us”?
I maintain therefore that the “imbalance” is a fiction; level 3 is clearly offering to pay maintenance fees for its end of the bargain, verizon should be willing to do the same. increased bandwidth demands leading to more equipment in service leading to higher maintenance fees is an operational cost of engaging in the data delivery business. if verizon attempts to say that they cannot afford the maintenance fees the increased bandwidth would incur, then that just adds more proof to the assertion that verizon has sold customers bandwidth that it cannot deliver.
that is the bottom line, the crux of the whole dispute, the tl;dr: verizon has sold customers bandwidth that it cannot deliver.
Re: Re:
Level 3 offered only to pay for the initial work building out the network. Verizon wants them to sign new contracts that force them to pay more money every month.
“The company is deliberately letting its border router clog up because it wants to ring a lot more money out of other companies”
The word is “wring” not “ring.”
Re: Re:
Maybe it is in reference to an old fashioned cash register.
😐
Re: Re: Re:
or a token ring. Just sayin’….
Re: Re:
Let Freedom Wring.
I wonder if Level 3 could just sneak in and connect the lines without Verizon knowing…
Re: Response to: Anonymous Coward on Jul 23rd, 2014 @ 6:39pm
No they can’t. Even if they could physically connect it you still need to enable the ports and configure them
Re: Re: Response to: Anonymous Coward on Jul 23rd, 2014 @ 6:39pm
That’s where the elite ninja-hackers come in.
Re: Re:
I wonder if Level 3 could just sneak in and connect the lines without Verizon knowing…
I’d prefer Level 3 just cut the existing wiring and tell Verizon that if they want network connectivity into Level 3’s ISP, they are going to have to pay for it.
If they want to peer, they have to deal with the free-transit, not treat Level 3 as a customer. They can’t have their cake and eat it too.
Ah, that Internet Cake, how lovely it would be...
-if Verizon would stop telling Level 3 to pay it just to put one icing decoration on when it’s already paid to have them all over it.
Purchase more Verizon bandwidth and magically the problem will clear up. What a bunch of greedy creeps.
..asymmetric ..
Funny, just yesterday in Arizona with Cox, I was downloading a game through Steam and noticed my speed was really fast. At first I figured maybe it was the time of day when the network for Valve wasn’t congested, but when I was done downloading the game I decided to run a test at Speedtest.net. My download speed in the past was 65 down and 13 up. I am now getting 117 down and 10 up.
We need Virtual Network Operators to breakup the Mabell last-mile monopolies.
From Wikipedia:
“A Virtual Network Operator (VNO) is a provider of management services and a reseller of network services from other telecommunications suppliers that does not own the telecommunication infrastructure.
These network providers are categorized as virtual because they provide network services to customers without owning the underlying network. A VNO typically leases bandwidth at the wholesale rates from various telecom providers in order to provide solutions to their customers. The VNO concept is relatively new in the North American market when compared to the European and Asian markets.”
https://en.wikipedia.org/wiki/Virtual_Network_Operator
VNOs work with mobile networks. Surely they can work with cable networks too.
Re: Re:
Anonymous Coward, MVNO may be coming. Google is looking at an MVNO positioning. Search it.
Re: Re: Re:
Anonymous Coward, MVNO may be coming. Google is looking at an MVNO positioning.
That’s VNOs. The M is for Mobile, and we already have those (Net 10, Tracfone, etc). We need VNOs for fixed broadband.
Can Level 3 become an ISP?
Maybe Level 3 should try to get into the ISP market in some select Verizon-only areas. I’m not in the US, so there may be rules/issues I’m not familiar with, but if it’s possible and the threat is there, it could be used to keep Verizon and the others in line.
Re: Can Level 3 become an ISP?
Problem there is the same problem Google has with setting up Google Fiber. It’s expensive, time consuming, and legally complicated (due to right of way access) to lay cable to addresses on any kind of scale. The incumbents have already have this in place.
If this isn’t a compelling argument for converting the internet into a public resource, I don’t know what is. The internet is just a digital highway system– we’re all fine with highways being public province (for the most part). The highway system is the foundation for commerce. The internet should continue that line of reasoning into the digital realm.
As it stands, what we have is akin to total privatization of all roadways. This is bound to lead to situations where road-owner Verizon will blackmail burgeoning trucking company Netflix because it is doing too well, therefore having a large amount of trucks on the road. If Verizon wanted to close its roads to all Netflix’s trucks then Netflix is denied access to the crucial backbone of its business model. This is never a situation where a corporation will act honorably, understanding that their product drives further innovation, yet is not actually entitled to the credit of those that took it further. Easy solution– don’t allow any private, profit-seeking-above-all-else, venture control any such fundamental resource. It cripples further innovation, and in the case of the tech sector, that is the last place we need to stifle. We’d all be lost without our smartphones.
So screw Verizon and their blackmail scam. Get rid of Verizon. Talking to other people should be a fundamental human right. Change that text we find so holy, yet was written long before a digital realm had a chance to even be considered.
Lastly, change the attitude that these silly narcissists have where they feel that everything they have accomplished was done solely by themselves. Got news for you, if you have learned anything from anyone other than yourself in your entire life, then you have been stealing the knowledge of the 100 billion humans that came before you. You are just the latest to piggyback on someone else’s ideas. Few among us have ever had an original thought. Realize that your vision/product/whatever could not exist without the combined effort of so many humans before you, as well as those in existence today that ship your product, put roads there for your product to be shipped, drill for the oil to fuel the delivery vehicles as well as produce the asphalt it drives on, mine for the raw materials of your product, etc. Also, you did not invent language or numbers or computers or the internet or anything, really.
No one does anything by themselves.
Whatever
I find it curious that Whatever has not commented on this story…
Re: Whatever
Quite…
This is like going to the airport to catch a flight and buying a ticket for the flight and then having to sit around the airport waiting because my destination city won’t pay the airlines to bring me there.
Re: Re:
+1
Sure Verizon doesn't have to provide more out of balance peer connections
But it then deserves the blame for poor delivery of things like Netflix. Verizon is demonstrating disdain for their broadband customers by suggesting they aren’t actually paying for decent content delivery.
Verizon gets Snarky
And we are supposed to believe they will be fair with net neutrality…beware!
VerizoNet
Perhaps Verizon needs to be honest about its products, they do not sell internet access they sell Verizonet access that backbone providers must pay to also access.
What are the peering relationships between Comcast and Verizon because this smells like collusion.
Sorry… I’ve read both positions, and I agree with Verizon’s position. Level 3 should not be relying on the free peering.
Re: Re:
Level 3 should not be relying on the free peering.
It isn’t “free peering”. Free peering is equal exchange of backbone traffic. This is a tier 1 provider handing off data to a last mile provider.
Verizon’s Deliberate Bullshitting on Peering
On suing Verizon for anti-competitive behavior
I am the retired founding CEO of Level 3. As someone with bit experience in dealing with companies’ that have a long and colorful history of blatantly wielding monopoly control of bottleneck facilities, I sympathize with those who say “sue the bums”. Unfortunately, a Supreme Court ruling makes taking this action somewhere between impractical and hopeless. In Verizon vs Tinko,the court’s opinion is generally read to mean, among other things, that in hearing any anti-trust lawsuit against a company regulated under the various telecom related laws, courts will defer to relevant regulatory bodies (i.e., the FCC) before ruling. Given the FCC’s poor track record in dealing abuses by monopoly last mile provider’s this is unfortunate.
But at least for now, that’s the law.
It means that if you are not happy with Verizon playing shameless games with a service they provide and you pay good money for, bombard the congress and the FCC with your objections.
Jim
On suing Verizon
Please add the following observations to my previous post:
Arguing asymmetric traffic ratios or the nuances of interconnection is may be interesting but it does not address the central issue
This is fundamentally an anti-trust question. If Verizon operates in a competitive market, they can charge whatever they choose and whomever they chose. If the market is constrained, then Verizon cannot use a bottleneck facility to disadvantage competitors. This is the reason the Verizon vs Trinko ruling is central to the discussion.
Jim