How to Count Parameters in Artificial Neural Networks (ANNs)

When building neural networks, one of the first questions you should ask is: How many trainable parameters does my model have? The number of parameters determines: Model complexity Memory usage Training speed Risk of overfitting In this article, we'll learn how to calculate the number of parameters manually and verify the results using TensorFlow 2.x (Keras). What Are Parameters? Parameters are the values that the neural network learns during training. There are two types: Weights Biases Every neuron has: One weight for every input it receives One bias Therefore, for a layer with: Input features = n Neurons = h Example 1: Single Hidden Layer Suppose we have: Input features = 4 Hidden neurons = 5 Output neurons = 1 Architecture: Input(4) │ Hidden(5) │ Output(1) Hidden Layer Each of the 5 neurons receives 4 inputs. Weights: 4 × 5 = 20 Biases: 5 Total: 20 + 5 = 25 Output Layer Input = 5 Output neurons = 1 Weights: 5 × 1 = 5 Biases: 1 Total: 5 + 1 = 6 Total Parameters 25 + 6 = 31 TensorFlow Verification import tensorflow as tf model = tf.keras.Sequential([ tf.keras.layers.Input(shape=(4,)), tf.keras.layers.Dense(5, activation="relu"), tf.keras.layers.Dense(1) ]) model.summary() Output: Layer (type) Output Shape Param # dense (None, 5) 25 dense_1 (None, 1) 6 Total params: 31 Perfect match. Example 2: Two Hidden Layers Architecture: Input(8) │ Hidden(16) │ Hidden(10) │ Output(3) First Hidden Layer Input = 8 Neurons = 16 Weights = 8 × 16 = 128 Biases = 16 Total = 144 Second Hidden Layer Input = 16 Neurons = 10 Weights = 16 × 10 = 160 Biases = 10 Total = 170 Output Layer Input = 10 Output neurons = 3 Weights = 10 × 3 = 30 Biases = 3 Total = 33 Total Parameters 144 + 170 + 33 = 347 TensorFlow Verification import tensorflow as tf model = tf.keras.Sequential([ tf.keras.layers.Input(shape=(8,)), tf.keras.layers.Dense(16, activation="relu"), tf.keras.layers.Dense(10, activation="relu"), tf.keras.layers.Dense(3, activation="softmax") ]) model.summary() Output: Layer (type) Param # dense 144 dense_1 170 dense_2 33 Total params: 347 Again, the manual calculation matches TensorFlow exactly. Example 3: Deep Neural Network Architecture: Input(20) │ Hidden(64) │ Hidden(32) │ Hidden(16) │ Output(5) Layer 1 (20 × 64) + 64 = 1280 + 64 = 1344 Layer 2 (64 × 32) + 32 = 2048 + 32 = 2080 Layer 3 (32 × 16) + 16 = 512 + 16 = 528 Output Layer (16 × 5) + 5 = 80 + 5 = 85 Total Parameters 1344 +2080 +528 +85 ------ 4037 TensorFlow Verification import tensorflow as tf model = tf.keras.Sequential([ tf.keras.layers.Input(shape=(20,)), tf.keras.layers.Dense(64, activation="relu"), tf.keras.layers.Dense(32, activation="relu"), tf.keras.layers.Dense(16, activation="relu"), tf.keras.layers.Dense(5, activation="softmax") ]) model.summary() Expected output: Layer (type) Param # dense 1344 dense_1 2080 dense_2 528 dense_3 85 Total params: 4037 Why Don't Activation Functions Add Parameters? Layers such as: Dense(32, activation="relu") or Dense(10, activation="sigmoid") have exactly the same number of parameters. Activation functions like: ReLU Sigmoid Tanh Softmax perform mathematical operations but do not learn any weights or biases, so they contribute zero trainable parameters. Quick Reference Layer Formula Dense (inputs × neurons) + neurons Dense (alternative form) (inputs + 1) × neurons Biases One per neuron Total Model Parameters Sum of all layer parameters Key Takeaways Every Dense layer learns weights and biases. The parameter count for a Dense layer is: Parameters = (Input Units × Output Units) + Output Units Equivalently: Parameters = (Input Units + 1) × Output Units The output size of one layer becomes the input size of the next layer. The total number of trainable parameters is the sum of the parameters across all trainable layers. You can always verify your manual calculations using model.summary() in TensorFlow 2.x with Keras.