（一）sklearn数据预处理算法

from sklearn import preprocessing import numpy as np X_train = np.array([[ 1., -1.,2.], [ 2.,0.,0.], [ 0.,1., -1.]])

1、使得均值0，方差1

X_scaled = preprocessing.scale(X_train)

X_scaled

array([[ 0., -1.22474487,1.33630621], [ 1.22474487,0., -0.26726124], [-1.22474487,1.22474487, -1.06904497]])

X_scaled.mean(axis=0),X_scaled.std(axis=0)

(array([ 0.,0.,0.]), array([ 1.,1.,1.]))

“axis=0”代表对arr[i][j]..等维度进行计算，假如arr[i][j]是3?33?3的数组，
计算后得到1?31?3的结果，即对每一列数据求平均值。
2、使得放缩在某个范围[0,1]

X_train = np.array([[ 1., -1.,2.], [ 2.,0.,0.], [ 0.,1., -1.]])

min_max_scaler = preprocessing.MinMaxScaler()

X_train_minmax = min_max_scaler.fit_transform(X_train)

X_train_minmax

array([[ 0.5,0.,1.], [ 1.,0.5,0.33333333], [ 0.,1.,0.]])

可以将相同的变换运用到test数据集

X_test = np.array([[ -3., -1.,4.]]) X_test_minmax = min_max_scaler.transform(X_test) X_test_minmax

array([[-1.5,0.,1.66666667]])

3、使得放缩在某个范围[-1,1]

X_train = np.array([[ 1., -1.,2.], [ 2.,0.,0.], [ 0.,1., -1.]])max_abs_scaler = preprocessing.MaxAbsScaler() X_train_maxabs = max_abs_scaler.fit_transform(X_train) X_train_maxabs# doctest +NORMALIZE_WHITESPACE^

array([[ 0.5, -1. ,1. ], [ 1. ,0. ,0. ], [ 0. ,1. , -0.5]])

类似scale，对应有minmax_scale 和 maxabs_scale
4、缩放稀疏数据
MaxAbsScaler和maxabs_scale专门用于扩展稀疏数据
5、二进制归一化

X = [[ 1., -1.,2.], [ 2.,0.,0.], [ 0.,1., -1.]]binarizer = preprocessing.Binarizer().fit(X)# fit does nothing binarizer

Binarizer(copy=True, threshold=0.0)

binarizer.transform(X)

array([[ 1.,0.,1.], [ 1.,0.,0.], [ 0.,1.,0.]])

#设定0,1划分值 binarizer = preprocessing.Binarizer(threshold=1.1) binarizer.transform(X)

array([[ 0.,0.,1.], [ 1.,0.,0.], [ 0.,0.,0.]])

6.编码分类功能
特征不是连续值而是分类
属性可取的离散值：[“male”, “female”][“from Europe”, “from US”, “from Asia”][“uses Firefox”, “uses Chrome”, “uses Safari”, “uses Internet Explorer”]
[“male”, “from US”, “uses Internet Explorer”]可以表示：[0, 1, 3]
[“female”, “from Asia”, “uses Chrome”]可以表示：[1, 2, 1]

#给定数据，可以自动推断属性的类别数 enc = preprocessing.OneHotEncoder() enc.fit([[0, 0, 3], [1, 1, 0], [0, 2, 1], [1, 0, 2]])

OneHotEncoder(categorical_features='all', dtype=, handle_unknown='error', n_values='auto', sparse=True)

#对一个数据进行重新编码 enc.transform([[0, 1, 3]]).toarray()

array([[ 1.,0.,0.,1.,0.,0.,0.,0.,1.]])

#编码长度是9，即2+3+4，对应上面每个类别属性长度

#显示给出每个类别属性个数 enc = preprocessing.OneHotEncoder(n_values=[2, 3, 4]) # Note that there are missing categorical values for the 2nd and 3rd # features enc.fit([[1, 2, 3], [0, 2, 0]])

OneHotEncoder(categorical_features='all', dtype=, handle_unknown='error', n_values=[2, 3, 4], sparse=True)

enc.transform([[1, 0, 0]]).toarray()

array([[ 0.,1.,1.,0.,0.,1.,0.,0.,0.]])

【（一）sklearn数据预处理】7.遗漏值的估算

from sklearn.preprocessing import Imputer imp = Imputer(missing_values='NaN', strategy='mean', axis=0)

data=https://www.it610.com/article/[[1, 2], [np.nan, 3], [7, 6]]

data

[[1, 2], [nan, 3], [7, 6]]

imp.fit(data)

Imputer(axis=0, copy=True, missing_values='NaN', strategy='mean', verbose=0)

X = [[np.nan, 2], [6, np.nan], [7, 6]]

#用平均值去填充nan print(imp.transform(X))

[[ 4.2.] [ 6.3.66666667] [ 7.6.]]

8、生成多项式特征

import numpy as np from sklearn.preprocessing import PolynomialFeatures X = np.arange(6).reshape(3, 2) X

array([[0, 1], [2, 3], [4, 5]])

特征从 (X1,X2)转化为(1,X1,X2,X21,X1X2,X22).(X1,X2)转化为(1,X1,X2,X12,X1X2,X22).

poly = PolynomialFeatures(2) poly.fit_transform(X)

array([[1.,0.,1.,0.,0.,1.], [1.,2.,3.,4.,6.,9.], [1.,4.,5.,16.,20.,25.]])

9、自定义转换器

#np.log()、np.log10()、np.log2()、np.log1p() #分别为自然对数（e）、底数为10、底数为2、log(1+x)

import numpy as np from sklearn.preprocessing import FunctionTransformer transformer = FunctionTransformer(np.log1p) X = np.array([[0, 1], [2, 3]]) transformer.transform(X)