Phase 8 ยท Algorithm 39 of 50

Federated Learning

Train the model. Never move the data.
Federated Learning technical poster

๐ŸŽฏCore Concept

Normally you haul all the data to one machine and train there. Federated Learning inverts that: it ships the model to wherever the data already lives โ€” your phone, a hospital's server, a bank's vault โ€” trains locally, and sends back only the weight updates. The raw data never leaves the device. The server never sees it. What gets averaged is learning, not records.
wglobalt+1  =  ฮฃk (nk / n) ยท wkt+1 FedAvg: each client's update is weighted by how much data it holds (nk of n total). More data โ†’ more say in the average.

๐ŸงฉKey Components

Global Model (Server)
The single shared set of weights. It coordinates rounds but stores zero training data โ€” it is an aggregator, not a data lake.
Clients & Local Data
Thousands to millions of devices, each with a small, private, non-IID slice of the world. Your typing habits are not your neighbour's.
FedAvg Aggregation
Data-weighted average of returned weights (McMahan et al., 2017). Simple, and the workhorse baseline everything else is measured against.
Privacy Layer
Secure aggregation so the server sees only the sum, plus differential-privacy noise so no single client is recoverable from it.

โš™๏ธHow It Works โ€” One Round

  1. Broadcast. Server sends current global weights wt to a random sampled subset of available clients (often <1% of the fleet per round).
  2. Train locally. Each client runs E epochs of SGD on its own data, producing wkt+1. Nothing is uploaded yet.
  3. Upload updates only. Clients send the weight delta โ€” compressed, quantized, often encrypted โ€” back to the server. Raw examples stay home.
  4. Aggregate. Server computes the data-weighted average (FedAvg), forming wt+1. Stragglers that miss the deadline are simply dropped.
  5. Repeat for R rounds until convergence. Communication rounds โ€” not GPU hours โ€” are usually the scarce resource.

๐ŸŒReal-World Applications

โŒจ๏ธ
Mobile keyboards Gboard's next-word prediction learns from what millions type without any of it leaving the handset. The original production use case.
๐Ÿฅ
Multi-hospital medical AI Hospitals legally cannot pool patient scans. Federated training lets a tumour detector learn across all of them anyway.
๐Ÿฆ
Cross-bank fraud detection Rival institutions improve a shared fraud model without exposing customer transactions to each other.
๐Ÿš—
Fleet & edge devices Vehicles and IoT sensors refine perception models on-device, uploading gradients over expensive, intermittent links.

โš ๏ธWhere It Gets Hard

The honest failure modes:
  • Non-IID skew โ€” client distributions differ wildly, so local models drift apart and the average can be worse than any single one (client drift).
  • Stragglers & dropouts โ€” phones go offline mid-round; the system must converge with partial, biased participation.
  • Communication cost โ€” sending millions of parameters per round dominates; hence compression, quantization, fewer-but-bigger local steps.
  • Gradients leak โ€” updates alone can partially reconstruct inputs. "No raw data moved" is not automatic privacy; you need DP + secure aggregation.

โœ…Checkpoint โ€” answer before you move on

Three questions

  1. In FedAvg, why is each client's update weighted by nk/n rather than averaged uniformly? What breaks if you weight them equally? Think about a client with 5 examples versus one with 50,000.
  2. Federated Learning never moves raw data โ€” so why is differential privacy still needed? What can an attacker learn from gradients alone? Consider gradient-inversion / membership-inference attacks.
  3. Your clients have highly non-IID data and accuracy stalls below centralized training. Name two concrete levers you'd pull, and the cost of each. Local epochs, proximal terms (FedProx), client sampling, personalization layers.