Exercise 1 – Deriving the Log-Partition Function
The exponential family is a class of probability distributions with a probability density (or mass) function expressed as:
\[
p(y | \theta) = h(y) \exp\left(\eta(\theta) \cdot T(y) - A(\theta)\right)
\]
where:
- \( h(y) \): Base measure, a non-negative function.
- \( T(y) \): Sufficient statistic, capturing the structure of the data.
- \( \eta(\theta) \): Natural parameter, a function of the distribution’s parameters.
- \( A(\theta) \): Log-partition function, ensuring normalization.
By definition the distribution integrates (or sums) to 1: \[ \int p(y | \theta) \, dy = 1 \] For a continuous distribution, substitute the exponential family form: \[ \int h(y) \exp\left(\eta(\theta) \cdot T(y) - A(\theta)\right) \, dy = 1 \] Derivation Steps:
1. Because \( A(\theta) \) does not depend on \( y \), factor out terms independent of \( y \): \[ e^{-A(\theta)} \int h(y) \exp\left(\eta(\theta) \cdot T(y)\right) \, dy = 1 \] 2. Solve for the integral: \[ \int h(y) \exp\left(\eta(\theta) \cdot T(y)\right) \, dy = e^{A(\theta)} \] Question:
What is the expression for \( A(\theta) \)?
- \( h(y) \): Base measure, a non-negative function.
- \( T(y) \): Sufficient statistic, capturing the structure of the data.
- \( \eta(\theta) \): Natural parameter, a function of the distribution’s parameters.
- \( A(\theta) \): Log-partition function, ensuring normalization.
By definition the distribution integrates (or sums) to 1: \[ \int p(y | \theta) \, dy = 1 \] For a continuous distribution, substitute the exponential family form: \[ \int h(y) \exp\left(\eta(\theta) \cdot T(y) - A(\theta)\right) \, dy = 1 \] Derivation Steps:
1. Because \( A(\theta) \) does not depend on \( y \), factor out terms independent of \( y \): \[ e^{-A(\theta)} \int h(y) \exp\left(\eta(\theta) \cdot T(y)\right) \, dy = 1 \] 2. Solve for the integral: \[ \int h(y) \exp\left(\eta(\theta) \cdot T(y)\right) \, dy = e^{A(\theta)} \] Question:
What is the expression for \( A(\theta) \)?