Dimensionality Reduction in DataRobot Using t-SNE

t-SNE (t-Distributed Stochastic Neighbor Embedding) is a powerful technique for dimensionality reduction that can effectively visualize high-dimensional data in a lower-dimensional space.

Dimensionality reduction can improve machine learning results by reducing computational complexity of the algorithms, preventing overfitting, and focusing on the most relevant features in the dataset. Note that this technique should only be used when the number of features is low.

Import libraries

In [ ]:
import datarobot as dr
import pandas as pd
import seaborn as sns
from sklearn.manifold import TSNE

Connect to DataRobot

Instructions for obtaining your endpoint and token are located in the DataRobot API documentation here.

In [3]:

# either directly pass in your endpoint/token, use a config file, or connect using DataRobot notebooks
dr.Client()

Out [3]:

<datarobot.rest.RESTClientObject at 0x7f5f10312280>

Get dataset

This example uses data on the movement of a double pendulum which has already been loaded into DataRobot for this example, but can be found here.

In [40]:

# replace the dataset ID with your own data
ds_id = "62fbcdf583b30f0ef972dc31"

# get dataset from DataRobot
ds = dr.Dataset.get(ds_id)
df = ds.get_as_dataframe()
display(df)

Out[40]:

Reduce the number of features in the dataset

In [ ]:

# features to exclude from reduction
# can be target columns or ID columns or other
exclude_cols = ["t", "a2"]

model = TSNE(learning_rate=100, random_state=42)
transformed = model.fit_transform(df.drop(exclude_cols, axis=1))

In [25]:

transformed

Out [25]:

array([[  2.542573 , -80.301025 ],
       [  2.5057044, -80.29103  ],
       [  2.869162 , -80.113396 ],
       ...,
       [  9.5524645,  74.92201  ],
       [  9.630235 ,  74.90384  ],
       [  9.827253 ,  74.67084  ]], dtype=float32)

Create new dataframe with reduced columns and previously excluded columns

In [39]:

# get the tsne dataset
reduced_df = pd.DataFrame(transformed, columns=["tsne_x", "tsne_y"])

# join in target and time columns from original dataset
reduced_df = pd.concat([reduced_df, df[exclude_cols]], axis=1)

display(reduced_df)

Out[39]:

Upload back to DataRobot

In [42]:


ds = dr.Dataset.create_from_in_memory_data(
    data_frame=reduced_df, fname=f"{ds.name}.csv"
)
ds.modify(name=f"{ds.name} t-SNE Reduced")
ds
     

Out [42]:

Dataset(name='Double Pendulum.csv.csv t-SNE Reduced', id='65a970bc040d9a438cdfb9de')