Index of /BDVis

	Name	Last modified	Size	Description
	Parent Directory		-
	AdultBval0pt2.jpg	2013-09-17 20:12	17K
	AdultRatioMF.jpg	2013-09-17 20:12	36K
	BARUG.pdf	2013-11-16 10:03	945K
	BDGraphs_0.5.0.tar.gz	2014-01-24 21:10	41K
	BankMarried.jpg	2013-09-17 13:55	21K
	Baseball.jpg	2013-09-17 13:55	30K
	BaseballGGParcoord.jpg	2013-09-17 16:30	33K
	BaseballGGParcoordAlph0pt2.jpg	2013-09-17 16:30	45K
	BigNGraphs_0.5.0.tar.gz	2014-01-24 21:10	41K
	DataMining.pdf	2013-11-16 10:04	1.0M
	Examples.html	2013-09-21 10:04	8.9K
	IADIAHSFO.eps	2013-09-21 09:22	1.3M
	IADIAHSFO.jpg	2013-09-17 21:07	18K
	JSM.pdf	2013-08-05 06:52	1.2M
	Letters.jpg	2013-09-17 13:55	37K
	LettersExtr2.jpg	2013-09-17 14:11	37K
	LettersRand50.jpg	2013-09-17 14:11	53K
	NoteToNM	2013-11-04 21:05	63
	Proceeds.pdf	2013-09-27 23:12	706K
	README	2013-04-11 13:24	54
	README.html	2013-09-27 23:13	2.7K
	RFiles/	2014-03-16 20:27	-
	freqparcoord_1.0.1.tar.gz	2014-03-28 16:44	809K
	parcoordf_1.0.0.tar.gz	2014-03-16 15:22	808K
	pc.tar	2014-03-16 14:31	2.3M
	u	2013-11-19 00:02	183

Norm Matloff's Big Data Visualization Tools

(Here I define Big Data rather generally. Any data set that is large enough to fill major portions of the screen when the points are plotted counts as Big from my point of view.)

I've developed a new graphical package for R, BDGraphs, which you can download here, or in individual files form, here. (In order to avoid confusion with an unrelated package BDGraph on CRAN, I will soon change the name of my package to BigNGraphs.) It consists of some novel tools for visualization of large data sets. They are computationally intensive, but use parallel processing to greatly reduce the workload.

Here are the main tools (best understood by clicking on the Examples link):

• freqparcoord(): A novel approach to parallel coordinates. The big problem with parallel coordinates is screen clutter. The approach here addresses that problem by plotting only "typical" lines, defined as those that have the highest estimated multivariate density.
• boundary(): Group comparison via comparison of regression boundary curves. Nonparametrically estimates the regression function of Z on X and Y, and plots the boundary curve separating the points (X,Y) having higher-than-age condition mean Z, and the points having lower-than-average values. By plotting one curve for each defined subgroup, this function enables exploration of the interaction of the groups and X, Y and Z.
• ratioest() Comparison of 2 groups via the ratio of their nonparametrically estimated regression functions of Z on X and Y.
• resid(): Classical regression model residual analysis, but with a new twist. Comparing a parametric fit to a nonparametric estimate of the regression of Z on X and Y, identifies the most frequent positive and negative residuals at each (X,Y) point, identifying regions of over/underfitting.

All tools here use nonparametric curve estimation methods, which may be computationally intensive, so that the package offers parallel computation, on either multicore machines or clusters.

My JSM talk is here, and the full paper is here.

Note: No warranties made of any kind regarding the software or methodology.