High Density Region Estimator (HDRE)
HDRE Goals
1. Approximate any empirical distribution
2. Build a parameterized density estimator
3. Outlier detection and dataset noise reduction
My Approach
1. Particle Swarm\Genetic Optimizer (KernelML)
2. Multi-Agent Approximation using IID Kernels
3. Discounted Reward Based Learning, (Reinforcement Learning?)
Here are a list of potential use cases in the form of a few questions and answers:
Question 1.
Has your client ever ask you to research, or “look into,” a single data point with respect to the output of a model?
Answer 1:
With HDRE you can get multivariate density estimate, cluster assignment, and similar data points all in one shot.
Question 2:
Do you need an outlier detection model that can be scaled to big-data for a low-latency production model?
Answer 2:
After training, the HDRE model can assign a matrix of data points to clusters in linear time.
Question 3:
Have you ever wondered what the density of a particular cross-section of data is across multiple dimensions?
Answer 3:
Its possible to plot the density distribution with respect to a cross section of data\selection of data points.
Question 4:
Do you have a lot of available cpu cores?
Answer 4:
The algorithm is automatically parallelized with numba! Though the training algorithm requires many iterations, the time complexity depends on the number of dimensions and not on the number of data points.
If you are interested in this algorithm, please DM on LinkedIn. The code is free to use and open-source. I would really like to try this algorithm out on various real-world use cases.
HDRE Github:
https://github.com/freedomtowin/high-density-region-estimator
Hierarchical Density Factorization:
https://www.datasciencecentral.com/profiles/blogs/new-algorithm-for…
Density Factorization:
https://www.datasciencecentral.com/profiles/blogs/high-density-regi…
