Practices and infrastructure put in place to help developers collaborate and communicate, as well as build, test, and release
software quickly. In reality, Dev Ops is implemented using toolchains, which are sets of tools that take care of the following
components:
- Code - Code Development, Code Review, Version Control, etc
- Build - Continuous Integration, builds
- Test - QA, Regression Testing, Perf Testing, Compatability Testing, etc
- Package - Artifacts, pre-deploy staging environments
- Release - Approvals, Release automation, Fallbacks, Change Management
- Configure - Infra configurations and management such as Servers
- Monitor - Build Status, Perf Monitoring, UX
- Collaborate - Version Control, Confluence
- Communicate - Slack!
An example of a toolchain might look something like this:
- Code - Github Enterprise
- Build - Jenkins with CI integration
- Test - Automated testing built into Tox
- Package - in-house package and package management solutions like SPM
- Release - Release standards, Atlassian JIRA to manage Releases
- Configure - Docker instances on AWS
- Monitor - in-house monitoring systems with customized UI/UX
- Collaborate - Atlassian Confluence Docs to manage documentation
- Communicate - Slack and Private Email Server
- Regression
- Performance
- Stress
- Integration
- Build tools
- Testing Libraries
- Build tools
- Documentation
- Perf monitoring
- Analysis in production
- pre-production code profiling
- static code analysis
- unit tests
- code coverage based on bugs found by users
- conduct code reviews
- Code Review
- Emphasis on refactoring, replacing, and modernizing legacy code
- Avoid technical debt at all costs using architecture design and review?
- Strategically incur short-term technical debt to enable experimentation and fast feedback?
- Track and measure short-term technical debt and pay it off ASAP?
- Keep track of high-cost technical debts in backlogs
- Do you suffer from technical debt as a result of legacy application development or lack of quality control in the past?
- Distributed Version Control
- Shared network drives and backups
- open source sharing systems, server-based sharing systems
- Google Docs!
Build Automation scripts and automates activities that programers have to do often like:
- Downloading dependencies
- compiling source into binary
- Packaging binary
- Running Tests
- Automatically Deploy to prod
What build tools are you using? What testing frameworks?
- Do you have continuous integration?
- Does it automatically deploy to prod?
- What conditions does the build fail?
- In-house solutions or SaaS?
- on-demand: builds through a user running a script on the command line
- scheduled: CI server, nightly builds, regular CRON jobs
- triggered: Triggered on commit to VCS?
- In-house package manager (SPM)
- Public or open source package manager?
- Maintain artifact repository with the latest binaries which all teams use
- Work item tracking system
- Versioned interfaces between different services
- Track versions in google docs or a spreadsheet
- Alerts that are triggered when thresholds are crossed before visible disruption in service occurs
- Manual inspection by staff (who does the testing? ops, devs, testers?)
- How is your instrumentation? Do you have in-house tools to help you monitor your stack? Logging and Log analysis?
- Reported by end users?
- We don't find out until its too late :(
- In-house instrumentation generates automated alerts to all relevant teams
- SaaS issue tracking tools that are available to all relevant teams in real-time
- End-user requests and complaints via direct contact
seconds, minutes, hours, days?
- Is your system high availability?
- Six Nines - available 99.9999% of the time - unscheduled downtime of 31.5 seconds a year
- What happens during a catastrophic server or infrastructure error?
- How do catastrophic errors propogate? Who gets notified? How do you handle such changes?
- Do you have a safety mechanism? Rollbacks?



