The Java Posse
Serving the Java Community since 2005
Tor Norbye (Google), Carl Quinn (Netflix), Dick Wall (Escalate), Chet Haase (Google)

Roundup ‘13 - Devops and Culture Change

Fully formatted shownotes can always be found at

Extra thanks to D.J. Hagberg for the excellent and complete show notes.

- Netflix "Freedom & Responsibility" - Every team can organize as they choose

- "Firemen" who get paged when system goes down.  Sets best practices

- "Site Reliability Team" - sets best practices: deployment, monitoring, ...

- "Cloud Performance Team" - sets best practices for cloud performance

- Groovy

- Python

- Standarization: set up practices/tools such that choosing the "Standard" makes life easier

- Regulatory constraints (Sarbanes-Oxley, PCI DSS)

- Communication with Devops: Build, Deploy, Release

- Documentation: in many cases, auditors are happy with scripts

- tl;dr

- Adrian Cockcroft - InfoQ, YouTube, @adrianco

- Opportunities for Ops to deploy systems, devs to assist & learn from Ops

- #monitoringsucks

- Uptime and Availabitility - metrics

- Wake up the developer at 3am when they break things. Devs need to understand Operational Risk of changes

- PCI Compliance mitigation ideas

 . Separate out the SOX/PCI regulated parts of application, pared down to most minimum possible

 . Put those parts under separate accounts, machines, audit trails, etc.

- Metrics: what do you use to measure? Customer Happiness, Money, Loyalty?  (Netflix: "movie start metric")

- Adobe Cloud Services

- Quicken cloud

- Twitter - measure sentiment (for consumer-facing/high-profile services)

- Automation: bite-size, recoverable pieces

- Eiffel design-by-contract concept: preconditions/postconditions

- Redefining the job of operations

- Executive sponsorship

- QA does the deployment through DevOps-provided tools

- Post-Mortem, military AAR, Incident Review, Agile Retrospective: have thresholds for Big Meetings vs small outages/rollbacks

- Metric: time to get new feature to the customer

- Self-directed improvement: Deming/Lean improvement cycles - Kaizen

- Visible Ops Handbook - Gene Kim

- The Phoenix Project (book) - Gene Kim

- business Case for improvements?  What was cost of the last outage?

- Chase Manhattan - 3 day outage in 2010

- Every business in the future will be a software business

- Move developers into Ops teams - Google engineers get extra "hazard" pay/leather jacks - DevOps Culture Hacks

- Practices brought back from Ops to Dev

 . JMX: Measure/Instrument *all* the things (esp. consumer facing)

 . Vagrant/Opscode Chef/Puppet stack into Dev environment - exercise deployment on every Dev change.

- "Foreign Exchange" (hostage exchange?) program

- "Don't make customers modify XML Files"

- Static Config to Files.  Stays the same for release.

- Dynamic Config - looked up from service - Archaius - Netflix

- Feature Switches/Feature Flags - Etsy

 . Enable/disable features & settings at runtime

 . Hierarchical/tagged/scoped configurations

 . Use config changes for rollback


    • Opening - "Java" the parody song Copyright 1997 Broken Records and Marjorie Music Publ. (BMI),

    • Closing - Juan Carlos Jimenez - In the House (Intro No. 1)

  • To contact us:

The Java Posse consists of Tor Norbye, Carl Quinn, Chet Haase and Dick Wall

Direct download: JavaPosse428.mp3
Category:general -- posted at: 11:24am PDT

To browse other episodes, use the Archives list on the right >>