https://DevOpsCloud.io -- Cloud Monk Losang Jinpa

Fair Use Source: B082P97LDW

Return to Python for DevOps by Noah Gift

• Chapter 1, Python Essentials for DevOps - Python for DevOps - Python Essentials for DevOps
• Chapter 2, Automating Files and the Filesystem
• Chapter 3, Working with the Command Line

• Chapter 4, Useful Linux Utilities
• Chapter 5, Package Management
• Chapter 6, Continuous Integration and Continuous Deployment

• Chapter 7, Monitoring and Logging
• Chapter 8, Pytest for DevOps

• Chapter 9, Cloud Computing
• Chapter 10, Infrastructure as Code
• Chapter 11, Container Technologies - Docker and Docker Compose
• Chapter 12, Container Orchestration - Kubernetes
• Chapter 13, Serverless Technologies

• Chapter 14, MLOps and Machine learning Engineering
• Chapter 15, Data Engineering

• Chapter 16, DevOps War Stories and Interviews

---

• Preface
• What Does DevOps Mean to the Authors?
• How to Use This Book
  • Conceptual Topics
• Conventions Used in This Book
• Using Code Examples
• O'Reilly Online Learning
• How to Contact Us
• Acknowledgments
  • Noah
  • Kennedy
  • Alfredo
  • Grig

1. Python Essentials for DevOps - Python Essentials for DevOps

• Installing Python and Running Python - Installing and Running Python
  • The Python Shell
  • Jupyter Notebooks - Jupyter Notebook

• Procedural Programming - Procedural - Procedural Programming in Python - Python Procedural Programming
  • Variable - Variables - Python Variables - Python Variable - Variables in Python
  • Basic Math - Python Basic Math - Basic Math in Python
  • Comment - Comments - Python Comments - Comments in Python
  • Built-in Function - Built-in Functions - Builtin Function - Builtin Functions - Python Built-in Functions - Built-in Functions in Python - Python Builtin Functions - Builtin Functions in Python
  • Print - Prints - Print Statement - Print Function- Python Print - Python Print Statement - Python Print Function - Print Python - Print Statement in Python - Print Function in Python
  • Range - Ranges - Range Statement - Range Function - Python Range - Range Python - Python Range Function - Range Function in Python

• Execution Control - Python Execution Control - Execution Control in Python - Controlling Execution - Python Controlling Execution - Controlling Execution in Python
  • if/elif/else
  • for Loops

• while Loops

• Handling Exceptions

• Built-in Objects
  • What Is an Object?
  • Object Methods and Attributes
  • Sequences

• Functions
  • Anatomy of a Function
  • Functions as Objects
  • Anonymous Functions

• Using Regular Expressions
  • Searching
  • Character Sets
  • Character Classes
  • Groups
  • Named Groups
  • Find All
  • Find Iterator
  • Substitution
  • Compiling

• Lazy Evaluation Generators

Generator Comprehensions

More IPython Features Using IPython to Run Unix Shell Commands

Exercises

---

2. Automating Files and the Filesystem - Automating Files and the Filesystem - Automating the Filesystem]] - File system

• Reading and Writing Files

Using Regular Expressions to Search Text

Dealing with Large Files

Encrypting Text Hashing with Hashlib

Encryption with Cryptography

The os Module

Managing Files and Directories Using os.path

Walking Directory Trees Using os.walk

Paths as Objects with Pathlib

3. Working with the Command Line Working with the Shell Talking to the Interpreter with the sys Module

Dealing with the Operating System Using the os Module

Spawn Processes with the subprocess Module

Creating Command-Line Tools Using sys.argv

Using argparse

Using click

fire

Implementing Plug-ins

Case Study: Turbocharging Python with Command-Line Tools Using the Numba Just-in-Time (JIT) Compiler

Using the GPU with CUDA Python

Running True Multicore Multithreaded Python Using Numba

KMeans Clustering

Exercises

4. Useful Linux Utilities Disk Utilities Measuring Performance

Partitions

Retrieving Specific Device Information

Network Utilities SSH Tunneling

Benchmarking HTTP with Apache Benchmark (ab)

Load Testing with molotov

CPU Utilities Viewing Processes with htop

Working with Bash and ZSH Customizing the Python Shell

Recursive Globbing

Searching and Replacing with Confirmation Prompts

Removing Temporary Python Files

Listing and Filtering Processes

Unix Timestamp

Mixing Python with Bash and ZSH Random Password Generator

Does My Module Exist?

Changing Directories to a Module’s Path

Converting a CSV File to JSON

Python One-Liners Debuggers

How Fast Is this Snippet?

strace

Exercises

Case Study Question

5. Package Management Why Is Packaging Important? When Packaging Might Not Be Needed

Packaging Guidelines Descriptive Versioning

The changelog

Choosing a Strategy

Packaging Solutions Native Python Packaging

Debian Packaging

RPM Packaging

Management with systemd Long-Running Processes

Setting It Up

The systemd Unit File

Installing the Unit Log Handling

Exercises

Case Study Question

6. Continuous Integration and Continuous Deployment Real-World Case Study: Converting a Poorly Maintained WordPress Site to Hugo Setting Up Hugo

Converting WordPress to Hugo Posts

Creating an Algolia Index and Updating It

Orchestrating with a Makefile

Deploying with AWS CodePipeline

Real-World Case Study: Deploying a Python App Engine Application with Google Cloud Build

Real-World Case Study: NFSOPS

7. Monitoring and Logging Key Concepts in Building Reliable Systems

Immutable DevOps Principles Centralized Logging

Case Study: Production Database Kills Hard [Drive]]s

Did You Build It or Buy It?

Fault Tolerance

Monitoring Graphite

StatsD

Prometheus

Instrumentation Naming Conventions

Logging Why Is It Hard?

The basicconfig

Deeper Configuration

Common Patterns

The ELK Stack Logstash

Elasticsearch and Kibana

Exercises

Case Study Question

8. Pytest for DevOps Testing Superpowers with pytest

Getting Started with pytest Testing with pytest

Differences with unittest

pytest Features conftest.py

The Amazing assert

Parametrization

Fixtures Getting Started

Built-in Fixtures

Infrastructure Testing What Is System Validation?

Introduction to Testinfra

Connecting to Remote Nodes

Features and Special Fixtures

Examples

Testing Jupyter Notebooks with pytest

Exercises

Case Study Question

9. Cloud Computing Cloud Computing Foundations

Types of Cloud Computing

Types of Cloud Services Infrastructure as a Service

Metal as a Service

Platform as a Service

Serverless Computing

Software as a Service

Infrastructure as Code

Continuous Delivery

Virtualization and Containers Hardware Virtualization

Software Defined Networks

Software Defined Storage

Containers

Challenges and Opportunities in Distributed Computing

Python Concurrency, Performance, and Process Management in the Cloud Era

Process Management Manage Processes with Subprocess

Using Multiprocessing to Solve Problems

Forking Processes with Pool()

Function as a Service and Serverless

High Performance Python with Numba

Using Numba Just in Time Compiler

Using High-Performance Servers

Conclusion

Exercises

Case Study Questions

10. Infrastructure as Code A Classification of Infrastructure Automation Tools

Manual Provisioning

Automated Infrastructure Provisioning with Terraform Provisioning an S3 Bucket

Provisioning an SSL Certificate with AWS ACM

Provisioning an Amazon CloudFront Distribution

Provisioning a Route 53 DNS Record

Copying Static Files to S3

Deleting All AWS Resources Provisioned with Terraform

Automated Infrastructure Provisioning with Pulumi Creating a New Pulumi Python Project for AWS

Creating Configuration Values for the Staging Stack

Provisioning an ACM SSL Certificate

Provisioning a Route 53 Zone and DNS Records

Provisioning a CloudFront Distribution

Provisioning a Route 53 DNS Record for the Site URL

Creating and Deploying a New Stack

Exercises

11. Container Technologies: Docker and Docker Compose What Is a Docker Container?

Creating, Building, Running, and Removing Docker Images and Containers

Publishing Docker Images to a Docker Registry

Running a Docker Container with the Same Image on a Different Host

Running Multiple Docker Containers with Docker Compose

Porting the docker-compose Services to a New Host and Operating System

Exercises

12. Container Orchestration: Kubernetes Short Overview of Kubernetes Concepts

Using Kompose to Create Kubernetes Manifests from docker-compose.yaml

Deploying Kubernetes Manifests to a Local Kubernetes Cluster Based on minikube

Launching a GKE Kubernetes Cluster in GCP with Pulumi

Deploying the Flask Example Application to GKE

Installing Prometheus and Grafana Helm Charts

Destroying the GKE Cluster

Exercises

13. Serverless Technologies Deploying the Same Python Function to the “Big Three” Cloud Providers Installing Serverless Framework

Deploying Python Function to AWS Lambda

Deploying Python Function to Google Cloud Functions

Deploying Python Function to Azure

Deploying a Python Function to Self-Hosted FaaS Platforms Deploying Python Function to OpenFaaS

Provisioning DynamoDB Table, Lambda Functions, and API Gateway Methods Using the AWS CDK

Exercises

14. MLOps and Machine learning Engineering What Is Machine Learning? Supervised Machine Learning

Modeling

Python Machine learning Ecosystem Deep Learning with PyTorch

Cloud Machine learning Platforms

Machine learning Maturity Model Machine Learning Key Terminology

Level 1: Framing, Scope Identification, and Problem Definition

Level 2: Continuous Delivery of Data

Level 3: Continuous Delivery of Clean Data

Level 4: Continuous Delivery of Exploratory Data Analysis

Level 5: Continuous Delivery of Traditional ML and AutoML

Level 6: ML Operational Feedback Loop

Sklearn Flask with Kubernetes and Docker

Sklearn Flask with Kubernetes and Docker EDA

Modeling

Tune Scaled GBM

Fit Model

Evaluate

adhoc_predict

JSON Workflow

Scale Input

adhoc_predict from Pickle

Scale Input

Exercises

Case Study Question

Learning Assessments

15. Data Engineering Small Data Dealing with Small Data Files

Write a File

Read a File

Generator Pipeline to Read and Process Lines

Using YAML

Big Data

Big Data Tools, Components, and Platforms Data Sources

Filesystems

Data Storage

Real-Time Streaming Ingestion]]

Case Study: Building a Homegrown Data Pipeline

Serverless Data Engineering Using AWS Lambda with CloudWatch Events

Using Amazon CloudWatch Logging with AWS Lambda

Using AWS Lambda to Populate Amazon Simple Queue Service

Wiring Up CloudWatch Event Trigger

Creating Event-Driven Lambdas

Reading Amazon SQS Events from AWS Lambda

Conclusion

Exercises

Case Study Question

16. DevOps War Stories and Interviews Film Studio Can’t Make Film

Game Studio Can’t Ship Game

Python Scripts Take 60 Seconds to Launch

Putting Out a Fire with a Cache and Intelligent Instrumentation

You’ll Automate Yourself Out of a Job!

DevOps Antipatterns No Automated Build Server Antipattern

Flying Blind

Difficulties in Coordination as an Ongoing Accomplishment

No Teamwork

Fair Use Source: B082P97LDW