Examples

Howto

• API quickstart
  • 1. Model creation
  • 2. Probability Distributions
    • Unobserved Random Variables
    • Observed Random Variables
    • Deterministic transforms
    • Automatic transforms of bounded RVs
    • Lists of RVs / higher-dimensional RVs
    • Initialization with test_values
  • 3. Inference
    • 3.1 Sampling
    • 3.2 Analyze sampling results
    • 3.3 Variational inference
  • 4. Posterior Predictive Sampling
  • 4.1 Predicting on hold-out data
• Sampler statistics
• Multiple samplers
• Diagnosing Biased Inference with Divergences
  • The Eight Schools Model
  • A Centered Eight Schools Implementation
    • A Dangerously-Short Markov Chain
    • A Safer, Longer Markov Chain
  • Mitigating Divergences by Adjusting PyMC3’s Adaptation Routine
    • Adjusting Adaptation Routine
  • A Non-Centered Eight Schools Implementation
• Posterior Predictive Checks
  • Comparison between PPC and other model evaluation methods.
  • Prediction
    • Mean predicted values plus error bars to give sense of uncertainty in prediction
• How to debug a model
• PyMC3 Modeling tips and heuristic
  • Background information
  • A WinBugs/PyMC2 implementation
  • PyMC3 implementation using theano.scan
  • PyMC3 implementation using some matrix “trick”
  • PyMC3 implementation using Matrix multiplication
  • PyMC3 implementation using Sparse Matrix
  • A few other warnings
  • Final remarks
• LKJ Prior for fitting a Multivariate Normal Model
• Live sample plots

Applied

• Bayesian Estimation Supersedes the T-Test
  • The Problem
    • Example: Drug trial evaluation
    • References
• A Primer on Bayesian Methods for Multilevel Modeling
  • Example: Radon contamination (Gelman and Hill 2006)
  • Data organization
    • Conventional approaches
    • Multilevel and hierarchical models
    • Partial pooling model
    • Varying intercept model
    • Varying slope model
    • Varying intercept and slope model
    • Adding group-level predictors
  • Correlations among levels
  • Prediction
    • Benefits of Multilevel Models
    • References
• Stochastic Volatility model
  • Build Model
  • Fit Model
  • References
• A Hierarchical model for Rugby prediction
  • Motivation
  • What do we want to infer?
  • What do we want?
  • What assumptions do we know for our ‘generative story’?
  • The model.
  • Building of the model
  • Results
  • Covariates
• Bayesian Survival Analysis
  • A crash course in survival analysis
  • Bayesian proportional hazards model
    • Time varying effects
• The Dawid-Skene model with priors
  • Introduction
  • Data
  • Model
  • Results

GLM

• GLM: Linear regression
  • Linear Regression
  • Probabilistic Reformulation
  • Bayesian GLMs in PyMC3
    • Generating data
    • Estimating the model
    • Analyzing the model
  • Summary
    • Further reading
• GLM: Robust Linear Regression
  • Robust Regression
  • Summary
• GLM: Robust Regression with Outlier Detection
  • Setup
    • Load and Prepare Data
  • Create Conventional OLS Model
    • Define model
    • Sample
    • View Traces
  • Create Robust Model: Student-T Method
    • Define Model
    • Sample
      • View Traces
  • Create Robust Model with Outliers: Hogg Method
    • Define model
    • Sample
    • View Traces
  • Declare Outliers and Compare Plots
    • View ranges for inliers / outlier predictions
    • Declare outliers
    • Posterior Prediction Plots for OLS vs StudentT vs SignalNoise
• GLM: Model Selection
  • Setup
    • Local Functions
  • Generate Toy Datasets
    • Interactively Draft Data
    • Create Datasets for Modelling
      • Scatterplot against model line
    • Standardize
      • Create ranges for later ylim xim
  • Demonstrate Simple Linear Model
    • Define model using ordinary pymc3 method
      • View Traces after burn-in
    • Define model using pymc3 GLM method
      • View Traces after burn-in
  • Create Higher-Order Linear Models
    • Create and run polynomial models
  • A really bad method for model selection: compare likelihoods
    • View posterior predictive fit
  • Compare Deviance Information Criterion [DIC]
  • Compare Watanabe - Akaike Information Criterion [WAIC]
  • TODO
    • K-Fold Cross Validation and/or Leave-One-Out (LOO)
      • Left for future development - should be easy enough
  • Bayes Factor
• Rolling Regression
  • Rolling regression
  • Analysis of results
• GLM: Hierarchical Linear Regression
  • The data set
  • The Models
    • Pooling of measurements
    • Unpooled measurements: separate regressions
    • Partial pooling: Hierarchical Regression aka, the best of both worlds
  • Probabilistic Programming
    • Unpooled/non-hierarchical model
    • Hierarchical Model
  • Posterior Predictive Check
    • The Root Mean Square Deviation
  • Shrinkage
    • Connections to Frequentist statistics
  • Summary
  • References
    • Acknowledgements
• GLM: Poisson Regression
  • A minimal reproducable example of poisson regression to predict counts using dummy data.
  • Contents
  • Package Requirements (shown as a conda-env YAML):
  • Setup
  • Local Functions
  • Generate Data
    • Assumptions:
      • View means of the various combinations (poisson mean values)
    • Briefly Describe Dataset
  • Poisson Regression
    • 1. Manual method, create design matrices and manually specify model
    • Transform coeffs and recover theta values
    • 2. Alternative method, using pymc.glm
    • Transform coeffs
• Hierarchical Partial Pooling
  • Data
  • Approach
• GLM: Negative Binomial Regression
  • Contents
    • Setup
      • Convenience Functions
      • Generate Data
  • Poisson Data
  • Negative Binomial Data
    • Negative Binomial Regression
      • Create GLM Model
      • View Results

Gaussian Processes

• Gaussian Process Regression
  • Example 1: Non-Linear Regression
  • Example 2: A periodic signal in non-white noise
• Kernels / Covariance functions
  • Exponentiated Quadratic
  • Two (and higher) Dimensional Inputs
  • Both dimensions active
  • One dimension active
  • Product of two covariances, active over each dimension
  • White Noise
  • Rational Quadratic
  • Exponential
  • Matern 5/2
  • Matern 3/2
  • Cosine
  • Linear
  • Polynomial
  • Multiplication with a precomputed covariance matrix
  • Applying an arbitary warping function on the inputs
    • Periodic
    • Locally Periodic (Gabor)
  • Gibbs
• Gaussian Process Regression and Classification with Elliptical Slice Sampling
  • Gaussian Process Regression
    • Generate some example data
    • Examine actual posterior distribution
    • Sample from posterior distribution
    • Evaluate posterior fit
  • Gaussian Process Classification
    • Generate some example data
    • Sample from posterior distribution
    • Evaluate posterior fit
• Gaussian Process (GP) smoothing
  • Let’s try a linear regression first
  • Linear regression model recap
  • Gaussian Process smoothing model
  • Let’s describe the above GP-smoothing model in PyMC3
  • Exploring different levels of smoothing
  • Smoothing “to the limits”
  • Interactive smoothing

Mixture Models

• Gaussian Mixture Model
  • Full trace
  • After convergence
  • Sampling of cluster for individual data point
• Marginalized Gaussian Mixture Model
• Gaussian Mixture Model with ADVI
• Dirichlet process mixtures for density estimation
  • Dirichlet processes
  • Dirichlet process mixtures
• Dependent density regression

Variational Inference

• GLM: Mini-batch ADVI on hierarchical regression model
• Automatic autoencoding variational Bayes for latent dirichlet allocation with PyMC3
  • Dataset
  • Log-likelihood of documents for LDA
  • LDA model
  • Encoder
  • AEVB with ADVI
  • Extraction of characteristic words of topics based on posterior samples
  • Predictive distribution
  • Summary
  • References
• Variational Inference: Bayesian Neural Networks
  • Current trends in Machine Learning
    • Probabilistic Programming at scale
    • Deep Learning
    • Bridging Deep Learning and Probabilistic Programming
  • Bayesian Neural Networks in PyMC3
    • Generating data
    • Model specification
    • Variational Inference: Scaling model complexity
  • Lets look at what the classifier has learned
    • Probability surface
    • Uncertainty in predicted value
  • Mini-batch ADVI: Scaling data size
  • Summary
  • Next steps
  • Acknowledgements
• Convolutional variational autoencoder with PyMC3 and Keras
  • Load images
  • Use Keras
  • Encoder and decoder
  • Generative model
  • Inference
  • Results
• Changes in API: Variational Inference: Bayesian Neural Networks
  • Bayesian Neural Networks in PyMC3
    • Generating data
    • Model specification
  • Lets look at what the classifier has learned
    • Probability surface
    • Uncertainty in predicted value
  • Mini-batch ADVI: Scaling data size
• Minibatch ADVI