Consolidated Business & Enterprise Computing Rant Thread

Discussion in 'Business & Enterprise Computing' started by elvis, Jul 1, 2008.

  1. Jase

    Jase Member

    Joined:
    Jun 28, 2001
    Messages:
    196
    Location:
    Sydney 2081
    https://xkcd.com/350/
    [​IMG]
     
  2. FatBoyNotSoSlim

    FatBoyNotSoSlim Member

    Joined:
    Sep 9, 2002
    Messages:
    14,082
    Location:
    SE Melbourne
    Yeah, discussing it with out team today and whilst it's possible to redesign for redundancy, the ROI is potentially not worth it.

    It's definitely amazing for scaling, at the core our platform is the same as it was for 1 user as it is now for 250k+. Going serverless has some amazing advantages, but the downside of the vendor and relying on them to be online.

    The S3 outage a few months back was way more disruptive than today's outage. Then again, daytime AU is our off-peak. Peak outage in the US would have sucked 10x.

    I'm not a developer, so I don't know the full details on what we'd have to re-engineer, I just have to support the customers. *Shrugs*
     
    Copie likes this.
  3. NSanity

    NSanity Member

    Joined:
    Mar 11, 2002
    Messages:
    18,315
    Location:
    Canberra
    I'm basically $60hr+some other stuff, I get billed out at 175-225 depending on the client, terms of engagement, length.

    I don't go out for more than 3 weeks, nor do i have interstate travel other than training/shows/etc.
     
  4. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,471
    Location:
    qld.au
    Doesn't cover Lambda :) There's some limitations even when using load balancing and Route 53:
    https://forums.aws.amazon.com/message.jspa?messageID=728350

    Lots of hackery to make it work, which again defeats the purpose of easy deployment / hands off deployment.

    That's the great Lambda advantage, scale is trivial. As you've found however, there's a few corner cases which make it unfun at scale. This is where Kubernetes and Docker really come into their own, you get 90% of the serverless fun but with all the control. Multi-region and even multi-cloud is also trivial, so well worth the time to learn.
    S3 at least allows easy multi-region. No excuses there :)
    So you get to wear the pain of poor decisions, never a fun position!
     
  5. Luke212

    Luke212 Member

    Joined:
    Feb 26, 2003
    Messages:
    10,142
    Location:
    Sydney
    so Why did it break?
     
  6. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,471
    Location:
    qld.au
    It's written right there. Lambda spans multiple AZ's not regions. Regional failures can affect all zones within it.

    Failures happen, you either accept them (risk management 101) or mitigate with increased redundancy.
     
  7. Luke212

    Luke212 Member

    Joined:
    Feb 26, 2003
    Messages:
    10,142
    Location:
    Sydney
    so 3 separate datacentres do not protect against failure. what prevents all regions going down if all redundant AZs can go down just the same. there is a lot of danger in hidden dependencies with AWS.
     
  8. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,471
    Location:
    qld.au
    You're thinking too simplistically. 16 different DC's won't help if it's a system level issue.

    AZ's are to isolate basic geographic issues, they do not eliminate a system level issues. AWS have different regions for further isolation of system issues which can affect an entire region.

    None of this is "hidden" btw, it's just basic planning. AWS aren't unique here either, if you're using a 3rd party vendor then it's just part of a basic risk assessment and planning process.
     
  9. GumbyNoTalent

    GumbyNoTalent Member

    Joined:
    Jan 8, 2003
    Messages:
    9,944
    Location:
    Briz Vegas
    Code:
                             Route 53 (round Robin)
    
               Region 1                 Region 2                          Region 3
                    |                              |                                    |
               API-Gateway            API-Gateway                   API-Gateway
                    |                             |                                     |
               Lambda                   Lambda                           Lambda
                    |                             |                                     |
               Read                        Read                              Read
    
               Writre-master
    
    The only bit that needs to be accessible across regions is the data, app stacks can be independent.

    You can even have write-master on hot-standby in all regions.

    Think of it as 3 independent installs in 3 regions with 1 shared resource the Data Storage, and DNS Round Robin spreads the love. Lose a region edit the DNS entry to 2.
     
    Last edited: Jun 24, 2017
  10. wintermute000

    wintermute000 Member

    Joined:
    Jan 23, 2011
    Messages:
    2,540
    ~4* the resource's rate is a standard benchmark for professional services. When you factor in total cost of employment, sales cost, bench time, overheads etc it's not as blatantly ripoff as it sounds. Don't believe me, who's stopping you starting your own consultancy/MSP?
     
  11. GumbyNoTalent

    GumbyNoTalent Member

    Joined:
    Jan 8, 2003
    Messages:
    9,944
    Location:
    Briz Vegas
    they tried switching to a token ring ;)
     
  12. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,471
    Location:
    qld.au
    Have you actually implemented this in prod though? Last time I tried, there were limitations in regards to the naming (as per the AWS forum thread).
     
  13. Luke212

    Luke212 Member

    Joined:
    Feb 26, 2003
    Messages:
    10,142
    Location:
    Sydney
    they are hidden to your average IT guy. even for 250K user systems as happened in this thread. i guarantee most people don't provision past the multi AZ. we talked recently about multi provider systems. i found it necessary to do these days as a way to avoid system level risk in mission critical systems. however it is a lot of extra work and in a way negates the simplicity and convenience of cloud setups. basically there are no silver bullets in this space.
     
  14. Daemon

    Daemon Member

    Joined:
    Jun 27, 2001
    Messages:
    5,471
    Location:
    qld.au
    If your average IT guy is playing in the AWS space then he's a moron, plain and simple. AWS is designed for dynamic workloads, not boring old IT stuff.

    As per the failures mentioned in this thread, you can bet your bottom dollar that someone evaluated and accepted the risk rather than simply not knowing about it.

    Most don't provision beyond AZ's because the cost & complexity are beyond diminishing returns. Accepting risk doesn't mean it won't eventuate, or seeing an event where it occurs mean you've made the wrong decision. To quote Amazon's CTO:

    Plan for failure, hope for success. As simple as that.

    As I've mentioned twice already in the last few posts, multi-cloud is trivial once you hit scale. Kubernetes makes stacking and packing thousands of VM's to optimal capacity quite easily. ]It's a silver bullet that can shoot multiple clouds, doesn't get much better than that :)
     
  15. PabloEscobar

    PabloEscobar Member

    Joined:
    Jan 28, 2008
    Messages:
    14,538
    Me, and my damn integrity. :).

    I'd go broke because I refused to upsell tonnes of crap the customer doesn't need, or put in 1/2 arsed solutions so I can make phat stacks of cash *fixing* them after the fact.
     
  16. GumbyNoTalent

    GumbyNoTalent Member

    Joined:
    Jan 8, 2003
    Messages:
    9,944
    Location:
    Briz Vegas
    Apologies you are correct.

    I have it across AZ's here in OZ, where I use serverless with API Gateway.

    Multi region is using cloudflare round robin DNS to muti regions then ELB to EC2 Node instances with a shared RDS.
     
  17. Diode

    Diode Member

    Joined:
    Jun 17, 2011
    Messages:
    1,736
    Location:
    Melbourne
    Unless what you're doing is global then I don't think spanning multiple regions in AWS is the answer to greater resiliency. Depending on the business storing certain types of stateful data has certain legal implications. I do agree that one should be not reliant on just one cloud provider and build tool many tools that make you reliant on that particular provider. Just like building a network you don't have your secondary links with the same provider.
     
  18. Luke212

    Luke212 Member

    Joined:
    Feb 26, 2003
    Messages:
    10,142
    Location:
    Sydney
    right. im not talking multi region actually, im talking running Azure and AWS in parallel in Sydney for example. so you have web server, db and file-storage parallel in both. there is no shared 'firmware' so you are pooling independent risks. well, the only shared dependency is your code. so if you accidently delete all, youre the weakest link lol
     
  19. Diode

    Diode Member

    Joined:
    Jun 17, 2011
    Messages:
    1,736
    Location:
    Melbourne
    Obviously easier said than done though. You'll need tooling that will work in each environment as well as the costs of any data you're transitioning between those environments. So I'd say it might only be practical and affordable stateless compute.
     
  20. TehCamel

    TehCamel Member

    Joined:
    Oct 8, 2006
    Messages:
    4,183
    Location:
    Melbourne
    Yea.. I used to feel like that. Now, I have a much better idea of the costs involved in sending a tech out, or employing them on a full time salary.
     

Share This Page

Advertisement: