Like a failed communist state traditional machine learning centralises training of a model on a single machine. Centralising data in a single central location is not always possible for a variety of reasons such as slow network connections, and legal constraints. These limitations have produced a series of techniques that allow the decentralised training of a model. This collection of techniques is referred to as Federated Machine Learning.
What is Federated Machine Learning?
Federated machine learning was developed by McMahan et al who principally developed it for mobile devices such as cell phones to train a local model. These local models are aggregated centrally into a final central model. The original paper produced by McMahan presented a number of experiments on image classification, and language models where they used 2000 individual local models each were generated with small amounts of data. The image classification task they undertook was CIFAR-10 image classification challenge, and the language model task produced an LM of Shakespear’s plays. The image classification task gained an accuracy of 99% using a Convolutional Neural Network in a Federated strategy. This experiment alone demonstrates that Federated Machine Learning can achieve near state of the art results.
The obvious advantages of Federated Machine Learning have produced a number of tools that can be used as part of the aforementioned strategy. Tensorflow Federated is an Open-Source framework through which developers can simulate Federated Learning experiments. The API contains two main layers: Federated Learning (API) and Federated Core . The API has a number of tutorial that allow the user to replicate the experiments of McMahan et al.
Private and Secure Machine Learning
Federated learning can be also used in the field of machine learning called Private and Secure Machine Learning. This field is discussed in detail by Andre Macedo Faria. In short, this field allows the learning of a model without knowing details about the local data sources, as well as not having direct access to the data. His article demonstrated a viable Private and Secure Machine Learning that used a combination of PySyft and PyTorch. This technique is suitable for tasks where the data is particularly sensitive or is protected by legislation.
The restriction of access to data can be an impediment to the advancement of science, and the use of federated machine learning can be used to placate privacy concerns. For example, wearables such as Fitbit have become popular. These devices record information that can be used to predict health outcomes. The transfer of information from these devices to a central location is likely to subject to consumer resistance which may impact sales. The use of Federated Machine Learning can be used to protect the privacy of the users of wearables, and still leverage the information generated by the information. This approach is discussed by Chen et al, who developed a framework for healthcare wearables which securely used information generated by these devices to generate a model.
Types of Federated Learning
The initial work by McMahan has been extended by Yang et al who introduced the notions of: “horizontal federated learning, vertical federated learning and federated transfer learning”. They defined horizontal federated learning as “data sets share the same feature space but different in samples”. They state that horizontal federated learning can be explained by an example of two regional banks. A federated algorithm is likely to have access to different data samples because the customers come from different regions, but the features used to generate the model are likely to be the same because the business of each bank is the same.
The definition of vertical federated learning is where the sample space is the same or similar, but the feature space is different. The authors gave an example of vertical federated learning being similar to two companies which are located in the same city. One of these hypothetical companies could be a bank and the other could be e-commerce organisation. Any federated learning algorithm would have access to data on a user’s mobile devices with similar data samples. However, the federated algorithm is likely to have access to different features spaces because the business of each company is different.
Transfer learning is the process where the experience of one model can be used by another learning working in a similar domain. Transfer learning has been used successfully in image processing, where large image classification networks such as GoogleLeNet which have been trained upon hundreds of thousands of images, has been adapted to classify images that GoogleLeNet has not been trained on. Transfer learning limits the amount of labelled data required for the new domain. Federated transfer learning as defined by Zang et al applies where there the data on the local devices differs not only in their samples but also in feature spaces. This technique allows the algorithm to generate a solution for the sample and feature space of the complete dataset.
Exploits of Federated Learning
Federated machine learning can be used in a secure manner, but as usual in sensitive domains, there are published exploits that can be used to compromise the privacy of the data stored on the client devices. The main attack vector in these exploits is that the client devices can leak information. Training information can be obtained through the use of Membership Inference Attacks, and misclassification can be invoked through model poisoning. The exploits of Federated Machine Learning are not limited to these attacks. A survey of these attacks is beyond the scope of this article, but the users and designers of federated machine learning techniques should be aware of its limitations.
Future of Federated Machine Learning
Despite privacy concerns the future of federated machine learning is bright. There are an estimated 2.3 billion cell phones in the world, and their power is increasing year on year. Smartphones, in particular, underuse their computing power. Harnessing the always-on cell phone may allow computer scientists to solve problems that cannot be solved by traditional means. Applications such as health-care, predictions of traffic volumes are now feasible because of the data collected by and processed by mobile devices. In the near future, this area of machine learning will grow and generate unique applications that hopefully will improve our day to day lives.
Original Post can be viewed here.
