Yes, the backend services for the MySubaru app (formerly STARLINK) are hosted on Amazon Web Services (AWS). While the app itself runs on your phone and your Subaru’s in-vehicle system, it communicates with Subaru’s servers, which are built on AWS, to perform functions like remote start, lock/unlock, and vehicle location tracking.
Wonder why they got away from the name “Starlink”?
They were never affiliated, but shared the StarLink name.
This is a tipping point moment. Increasingly, it seems that the talent who understood the deep failure modes is gone. The new, leaner, presumably less expensive teams lack the institutional knowledge needed to, if not prevent these outages in the first place, significantly reduce the time to detection and recovery.
We all know Amazon has no idea how Amazon works, Well, AWS was different. WAS. No more. AWS has let people go with the whole downsizing. They were spared in the beginning, no more.
So I mean how do we read this? H1B issues and Diwali? Not sure I can gleam anything outside of the left hand doesn’t know what the right hand is doing aka silos aka amazon. We know.
In the article it talks about the loss of institutional knowledge (due to lay offs and work climate that reduced experienced AWS staff, are my takeaways):
You can hire a bunch of very smart people who will explain how DNS works at a deep technical level (or you can hire me, who will incorrect you by explaining that it’s a database), but the one thing you can’t hire for is the person who remembers that when DNS starts getting wonky, check that seemingly unrelated system in the corner, because it has historically played a contributing role to some outages of yesteryear. When that tribal knowledge departs, you’re left having to reinvent an awful lot of in-house expertise…
AWS WAS the exception to this rule. I could seriously call or chat with someone tech to tech within AWS as of 2 years ago and things would get fixed INSTANTLY. That is no more.
All is fine until the (unmentioned) upward scan and the (unmentioned) speaker starts laughing (or crying) hysterically while sharing the images to the web after being hacked!
So a little more tech insight to the outage. Amazon says DNS error, and honestly the way it was “fixed” in 3 hours but took another day for everything to heal does point towards a DNS error.
DNS is the service that turns SellersAskSellers.com in your web browser into the server’s IP (or in our case CloudFlare’s IP’s). SAS’s DNS provider is CloudFlare. Others use AWS, or their own DNS servers on or off AWS.
To take a LOT of the tech out of this post, everyone’s DNS talks to everyone else and update each other. So an error within 30% of the Internet’s DNS servers would ground to a halt serves on both AWS and anyone who uses AWS as their domain’s DNS server, even if they are not hosted on AWS (example, we are NOT hosted on CloudFlare, but if CF’s DNS went down, we would too).
I will wait to make a final conclusion once AWS publishes the final report here: Post-Event Summaries
And YES, it appears from those reports that AWS has a lot of issues in US-EAST-1 and US-NORTHEAST region, but those are the heaviest use regions and also tend to be the default location when deploying services.
I’m still having server connection problems on a few sites. Most of the sites that I usually visit are coming up but they seem to be loading more slowly. There are one or two sites that apparently use AWS extensively. Those sites have been for the most part, unavailable. They are struggling to make the site available to users. (It’s not just me, DownDetector is showing a whopping 75%+ of users reporting server connection problems).
I still want to know why FBA sellers get these return notifications. It ain’t coming out of our pocket but they sure make it sound like it is “We will adjust your seller account accordingly”
And yet Amazon will still try to encourage/coerce/force those of us who have delivery promises set to handle such delays to shorten our promises, and remove any wiggle room for problems of any sort.
Yep and thus my inferred point. I think the hit amazon takes for not delivering as promised is valid. There are times I need stuff at the delivered time periodt. There are times that I am not even in town when the item arrives late. Not my problem, which is what amazon effectively does to sellers.
One that surprised me is that one of the on-line poker sites, which are legally required to have their own servers (generally located within the casino), went into maintenance shutdown because so many players were encountering problems. I guess somewhere the packet routing went to something on AWS.
Of course it’s valid. Granted, there are times where things happen that simply cannot be accounted for (USPS sitting on a package for 2 weeks, etc.), but I’ve received multiple feedbacks for “delivered even sooner than promised!” and not a single one for “arrived too early”.
I grew up watching Star Trek; big fan of Scottie’s practice of under-promising and over-delivering.
Worded that way for the same reason we now get “disbursement attempted” notices instead of former wording.
If they can tinker with the wording so as to alarm and hopefully frighten, then they can derive a bit of malicious pleasure without really looking bad to others.
My guess is that AWS handles the DNS for their sites. They still have their own servers hosting the site, but AWS handles getting people to those servers.