Programming Language Classification
Solution for submission 171996
A detailed solution for submission 171996 submitted for challenge Programming Language Classification
youssef_nader3
Getting Started with Programming Language Classification
In this puzzle, we have to classify the programming language from code. For classifying programming language we will have code snippets from which we need to identify the programming language. As the code snippets are texts, at first we need to tokenize the code snippets. In this process, we will learn more about tokenization and classification algorithms.
In this starter notebook:
For tokenization: We will use CountVectorizer and TfidfTransformer.
For Classification: We will use Multinomial Naive Bayes Classifier.
AIcrowd code utilities for downloading data for Language Classification
In [25]:
!pip install aicrowd-cli
%load_ext aicrowd.magic
In [2]:
!pip install autogluon
In [3]:
!pip install mxnet
In [ ]:
!pip install guesslang
Login to AIcrowd ㊗¶¶
In [ ]:
%aicrowd login
Download Dataset¶¶
We will create a folder name data and download the files there.
In [5]:
!rm -rf data
!mkdir data
%aicrowd ds dl -c programming-language-classification -o data
Importing Libraries:¶
In [1]:
import pandas as pd
import numpy as np
import os
import matplotlib.pyplot as plt
import seaborn as sn
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.metrics import roc_auc_score,accuracy_score,f1_score
from sklearn import set_config
set_config(display="diagram")
plt.rcParams["figure.figsize"] = (15,6)
Diving in the dataset 🕵️♂️¶
In [2]:
train_df = pd.read_csv("data/train.csv")
In [3]:
test_df = pd.read_csv("data/test.csv")
In [4]:
from sklearn.preprocessing import LabelEncoder
LE = LabelEncoder().fit(train_df.language)
In [5]:
train_df["target"] = LE.transform(train_df.language)
Splitting the dataset¶
Here we will be splitting out dataset into training, validation and test set
In [23]:
X_train,X_comb,Y_train,Y_comb = train_test_split(train_df["code"],train_df["target"],stratify=train_df["target"],test_size=0.2,random_state=42 , shuffle = True)
X_validation,X_test,Y_validation,Y_test = train_test_split(X_comb,Y_comb,test_size=0.5,random_state=0 , shuffle = True)
In [ ]:
X_train.shape,X_validation.shape,X_test.shape,Y_train.shape,Y_validation.shape,Y_test.shape
Out[ ]:
In [35]:
!pip install flair
!pip install transformers
In [36]:
!pip install -U numpy
In [18]:
from flair.embeddings import TransformerDocumentEmbeddings
In [6]:
from flair.embeddings import TransformerDocumentEmbeddings
from flair.data import Sentence
from tqdm.notebook import tqdm
doc_e=[]
model_name='huggingface/CodeBERTa-small-v1'
model_name2='microsoft/codebert-base-mlm'
model_name3="huggingface/CodeBERTa-language-id"
model_name4='flax-community/gpt-neo-125M-code-clippy-dedup'
doc_embedding = TransformerDocumentEmbeddings(model_name3,pooling='cls',layer_mean=True)
for d in tqdm(train_df["code"].values):
sent=Sentence(d.strip())
doc_embedding.embed(sent)
doc_e.append(sent.embedding.detach().cpu().numpy())
In [7]:
test_doc_e=[]
for d in tqdm(test_df["code"].values):
sent=Sentence(d.strip())
doc_embedding.embed(sent)
test_doc_e.append(sent.embedding.detach().cpu().numpy())
In [20]:
np.save('./train.npy',doc_e)
np.save('./test.npy',test_doc_e)
In [ ]:
doc_e=np.load('./train.npy')
test_doc_e=np.load('./test.npy')
In [ ]:
!pip install catboost
In [ ]:
test_df.code.values[7075]
Out[ ]:
In [ ]:
from sklearn.ensemble import StackingClassifier
from catboost import CatBoostClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
estimators1 = [
# ('catboost', CatBoostClassifier(n_estimators=200)),
('LR', LogisticRegression()),
# ('KNN', make_pipeline(PCA(n_components=50),KNeighborsClassifier(n_neighbors=5))),
]
clf1= StackingClassifier(
estimators=estimators1, final_estimator=LogisticRegression()
)
estimators2 = [
# ('catboost', CatBoostClassifier(n_estimators=200)),
('LR', LogisticRegression()),
# ('KNN', make_pipeline(PCA(n_components=50),KNeighborsClassifier(n_neighbors=5))),
]
clf2= StackingClassifier(
estimators=estimators2, final_estimator=LogisticRegression()
)
In [ ]:
estimators = [
('pipe2', make_pipeline(CountVectorizer(analyzer='word',min_df=5,max_df=1500),TfidfTransformer(),clf1)),
('pipe1', make_pipeline(CountVectorizer(analyzer='char',ngram_range=(1,2),min_df=10),TfidfTransformer(),clf1))
]
clf3= StackingClassifier(
estimators=estimators, final_estimator=LogisticRegression(),verbose=3
)
clf3.fit(X_train,Y_train)
In [ ]:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.neural_network import MLPClassifier
from imblearn.pipeline import Pipeline
from imblearn.over_sampling import RandomOverSampler
from sklearn.ensemble import RandomForestClassifier
token_pattern = r"""([A-Za-z_]\w*\b|[!\#\$%\&\*\+:\-\./<=>\?@\\\^_\|\~]+|[ \t\(\),;\{\}\[\]"'`])"""
# vectorizer = TfidfVectorizer(token_pattern=token_pattern, max_features=3000)
classifier = Pipeline([('tfidf', TfidfVectorizer(vocabulary=test_vect.vocabulary_)), ('clf', RandomForestClassifier(random_state=0,max_depth=20))])
classifier = classifier.fit(train_df['code'],train_df['target'])
In [ ]:
lengths=np.array([len(c) for c in train_df.code])
In [ ]:
train_df.loc[lengths<10].code.str.replace(" ","").value_counts()
In [ ]:
train_df.loc[lengths<10].language.value_counts()
In [8]:
vals=[
'->','=>','++','+=','--','-=','<-','::',':=','&&',';','||','#','\\\\\\','\\\\','\\*','){','===','!==', '*','"""','\'\'\'','@param',
]
In [9]:
token_pattern = r"""([A-Za-z_]\w*\b|[!\#\$%\&\*\+:\-\./<=>\?@\\\^_\|\~]+|[ \t\(\),;\{\}\[\]"'`])"""
for language in train_df.language.unique():
sub_df=train_df[train_df.language==language].copy()
min_df=30 if sub_df.shape[0]>1000 else 5
min_df_chars=10 if sub_df.shape[0]>1000 else 5
max_features=300 if sub_df.shape[0]>1000 else 100
v1=CountVectorizer(token_pattern=token_pattern,min_df=min_df,max_features=max_features)
v1.fit(sub_df.code)
vals+=[t.strip() for t in list(v1.vocabulary_.keys())]
print(language,len(v1.vocabulary_))
In [11]:
vocab={k:v for v,k in enumerate(set(vals))}
In [ ]:
vocab
In [16]:
test_vect=CountVectorizer(token_pattern=token_pattern,min_df=3,max_features=1500)
test_vect.fit(test_df['code'])
Out[16]:
In [17]:
total_vocab=list(vocab.keys())+list(test_vect.vocabulary_.keys())
total_vocab={k:v for v,k in enumerate(set(total_vocab))}
In [48]:
len(total_vocab)
Out[48]:
In [12]:
from sklearn.feature_extraction.text import TfidfVectorizer,CountVectorizer
tfidf=CountVectorizer(vocabulary=vocab,token_pattern=token_pattern)
X=train_df.code.values
X=tfidf.fit_transform(X).toarray()
In [19]:
from guesslang import Guess
guess = Guess()
def to_prob(prob):
return np.array([p[1] for p in prob])
# Guess the language from code
language = guess.probabilities("""
% Quick sort
-module (recursion).
-export ([qsort/1]).
qsort([]) -> [];
qsort([Pivot|T]) ->
qsort([X || X <- T, X < Pivot])
++ [Pivot] ++
qsort([X || X <- T, X >= Pivot]).
""")
In [22]:
from tqdm.notebook import tqdm
probabilities=[]
for c in tqdm(train_df.code):
probabilities.append(to_prob(guess.probabilities(c)))
In [30]:
np.c_[np.array(doc_e),X]
Out[30]:
In [13]:
df=pd.DataFrame(np.c_[np.array(doc_e),X])
In [51]:
X.shape,
Out[51]:
In [14]:
df['target']=train_df['target']
In [15]:
df.to_csv("./train.csv",index=False)
In [15]:
train_df[['code','target']].to_csv('./train_df.csv',index=False)
In [ ]:
from autogluon.core.utils.loaders.load_pd import load
train_data=load('./train.csv')
In [ ]:
!pip install mxnet
In [ ]:
from autogluon.text import TextPredictor
predictor = TextPredictor(label='target', eval_metric='acc',)
predictor.fit(train_data, time_limit=7200)
In [16]:
from autogluon.tabular import TabularDataset, TabularPredictor
train_data = TabularDataset("./train.csv")
predictor = TabularPredictor(label="target").fit(train_data,time_limit=10800)
In [ ]:
predictor.leaderboard()
In [ ]:
v=[]
for val in list(test_vect.vocabulary_.keys()):
v.append(val.strip())
In [ ]:
len(set(v)),len(test_vect.vocabulary_)
Out[ ]:
Auto Keras¶
In [13]:
!pip install autokeras
In [14]:
import autokeras as ak
import tensorflow as tf
In [20]:
TRAIN_DATA_URL = "./train.csv"
# TEST_DATA_URL = "https://storage.googleapis.com/tf-datasets/titanic/eval.csv"
train_file_path = tf.keras.utils.get_file("./train.csv")
# test_file_path = tf.keras.utils.get_file("eval.csv", TEST_DATA_URL)
In [ ]:
# Initialize the structured data classifier.
clf = ak.StructuredDataClassifier(
overwrite=True, max_trials=20
) # It tries 3 different models.
# Feed the structured data classifier with training data.
clf.fit(
# The path to the train.csv file.
TRAIN_DATA_URL,
# The name of the label column.
"target",
epochs=20,
)
# Predict with the best model.
predicted_y = clf.predict(test_file_path)
In [16]:
x_train = np.array(train_df.code)
y_train = np.array(train_df.target)
clf = ak.TextClassifier(
overwrite=True, max_trials=2
) # It only tries 1 model as a quick demo.
# Feed the text classifier with training data.
clf.fit(x_train,y_train, epochs=5)
# Predict with the best model.
predicted_y = clf.predict(X_test)
In [ ]:
In [32]:
!nvidia-smi
Kmeans Repeated runs Model¶
In [ ]:
from tqdm.notebook import tqdm
from sklearn.cluster import KMeans,AgglomerativeClustering,SpectralClustering,MiniBatchKMeans
from sklearn.preprocessing import OneHotEncoder
from sklearn.metrics import adjusted_rand_score
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_selection import mutual_info_classif
n_runs=20
true_k = 15
run_results=[]
vectorizer=TfidfVectorizer(vocabulary=vocab)
X=vectorizer.fit_transform(train_df['code'])
# mf=mutual_info_classif(X,train_df.target)
# feature_importances=np.argsort(mf)[::-1]
# X=X[:,feature_importances[:500]]
rands=[]
for i in tqdm(range(2,n_runs)):
model = KMeans(n_clusters=i, init='k-means++',n_init=10, max_iter=1000)
X_pred=model.fit_predict(X)
# enc=OneHotEncoder()
# run_results.append(enc.fit_transform(X_pred.reshape(-1,1)).toarray())
print("here")
rands.append(adjusted_rand_score(train_df.target.values,X_pred))
# print(rands[i])
In [ ]:
rands
Out[ ]:
In [ ]:
from sklearn.neural_network import MLPClassifier
classifier=MLPClassifier(verbose=True,hidden_layer_sizes=(1000),learning_rate='adaptive',early_stopping=True, max_iter=100)
classifier.fit(np.array(doc_e),train_df['target'])
In [ ]:
import pickle
with open('model2.pkl','wb+') as f:
pickle.dump(classifier,f)
In [ ]:
from sklearn.decomposition import PCA
from sklearn.neighbors import KNeighborsClassifier
classifier = Pipeline([('vect', PCA(n_components=50)), ('clf', CatBoostClassifier(n_estimators=1000))])
classifier = classifier.fit(np.array(doc_e).astype(np.float32), train_df["target"])
In [ ]:
from sklearn.tree import DecisionTreeClassifier
classifier = DecisionTreeClassifier()
classifier.fit(X_train, Y_train)
Out[ ]:
In [ ]:
classifier['vect'].vocabulary_
In [ ]:
from skorch import NeuralNetClassifier
from torch import nn
# X = np.array(X_train).astype(np.float32)
# y = Y_train.astype(np.int64)
class MyModule(nn.Module):
def __init__(self, num_units=10, nonlin=nn.ReLU()):
super(MyModule, self).__init__()
self.output = nn.Linear(768, 256)
self.hidden=nn.Linear(256,256)
self.hidden2=nn.Linear(256,15)
self.nonlin=nn.Tanh()
self.softmax = nn.Softmax(dim=-1)
def forward(self, X, **kwargs):
X=self.nonlin(self.output(X))
X=self.nonlin(self.hidden(X))
X = self.softmax(self.hidden2(X))
return X
net = NeuralNetClassifier(
MyModule,
max_epochs=150,
lr=0.003,
# Shuffle training data on each epoch
iterator_train__shuffle=True,
)
net.fit(X_res,Y_res)
# net.fit(np.array(doc_e).astype(np.float32), train_df["target"])
# y_proba = net.predict_proba(X)
In [ ]:
from sklearn.linear_model import LogisticRegressionCV
net2=LogisticRegression()
net2.fit(X_train,Y_train)
Out[ ]:
In [ ]:
from sklearn.metrics import balanced_accuracy_score
In [ ]:
from sklearn.feature_extraction.text import TfidfVectorizer
vect=TfidfVectorizer(vocabulary=vocab)
X=vect.fit_transform(X_train)
Feature Engineering¶
In [ ]:
def line_starts(text):
processed=text.replace("\t",'').split("\\n")
for line in processed:
line=line.strip()
if line.startswith("#"):
pass
elif line.startswith('*'):
pass
elif line.startswith('\\*'):
pass
elif line.startswith('\'\'\''):
pass
elif line.startswith('\\\\'):
pass
elif line.startswith('\\\\\\'):
pass
def token_in_string(text):
processed=text.replace("\t",'').split("\\n")
for line in processed:
In [ ]:
from sklearn.metrics import balanced_accuracy_score
from sklearn.linear_model import LogisticRegression
from imblearn.pipeline import Pipeline
# X_v=vect.transform(X_validation)
print("F1:" ,f1_score(Y_validation,classifier.predict(X_validation),average='macro'))
print("Accuracy:" ,accuracy_score(Y_validation,classifier.predict(X_validation))*100)
print("Accuracy:" ,balanced_accuracy_score(Y_validation,classifier.predict(X_validation))*100)
In [ ]:
print("F1:" ,f1_score(Y_test,classifier.predict(X_test),average='macro'))
print("Accuracy:" ,accuracy_score(Y_test,classifier.predict(X_test))*100)
In [ ]:
from sklearn.metrics import plot_confusion_matrix
plot_confusion_matrix(classifier, X_train, Y_train)
Out[ ]:
In [ ]:
train_df[train_df.language=='python']
Out[ ]:
Prediction Phase ✈¶
In [27]:
from tqdm.notebook import tqdm
test_probs=[]
for c in tqdm(test_df.code):
probabilities.append(to_prob(guess.probabilities(c)))
In [28]:
test_df['target'] = net.predict(np.array(test_probs))
In [19]:
X_test=tfidf.fit_transform(test_df.code).toarray()
pd.DataFrame(np.c_[np.array(test_doc_e),X_test]).to_csv('./test.csv')
In [20]:
test_df['target'] = predictor.predict(TabularDataset('./test.csv'))
In [17]:
test_df['target']=clf.predict(np.array(test_df.code))
In [ ]:
test_df.target
Out[ ]:
In [27]:
test_df["prediction"] = LE.inverse_transform(test_df.target.astype(int))
In [20]:
test_df.prediction.value_counts()
Out[20]:
In [21]:
test_df.prediction.value_counts()/test_df.shape[0]
Out[21]:
In [21]:
test_df["prediction"] = LE.inverse_transform(test_df.target)
In [51]:
test_df.target.unique()
Out[51]:
Generating Prediction File¶
In [23]:
test_df = test_df.sample(frac=1)
test_df.head(20)
Out[23]:
In [ ]:
n=0
for c in test_df.code.values:
if len(c)<10:
print(c)
print('----------')
n+=1
In [ ]:
n
Out[ ]:
In [ ]:
for c in test_df[test_df.prediction=='python'].code.values:
# print(len(c))
# if len(c)>10 and len(c)<100:
# print(c)
# print('----------')
if "\t" in c:
print(c)
In [22]:
!rm -rf assets
!mkdir assets
test_df.to_csv(os.path.join("assets", "submission.csv"))
In [33]:
test_df['length']=test_df.code.apply(lambda x: len(x.strip()))
In [34]:
test_df[test_df.length<15]
Out[34]:
In [ ]:
test_df.shape
Out[ ]:
Submitting our Predictions¶
Note : Please save the notebook before submitting it (Ctrl + S)
In [ ]:
%aicrowd notebook submit -c programming-language-classification -a assets --no-verify
In [32]:
test_df.to_csv('./out1.csv')
In [32]:
probs=predictor.predict_proba(TabularDataset('./test.csv'))
In [46]:
probs.values[18]
Out[46]:
In [ ]:
classifier.predict_proba(test_df['code'][test_df.id==10489])
Out[ ]:
In [ ]:
test_df.loc[test_df.id==10489,'code']
In [ ]:
f=np.argsort(classifier['tfidf'].transform(test_df['code'][test_df.id==10489]).toarray())[::-1]
mylist=sorted(classifier['tfidf'].vocabulary_, key=classifier['tfidf'].vocabulary_.get)
print(np.array(mylist)[f[:50]])
for feat,imp in zip(np.array(mylist)[f[:50]],classifier['tfidf'].transform(test_df['code'][test_df.id==10489]).toarray()[0][f[:50]]):
print(feat,imp)
In [ ]:
clf_pred=classifier['tfidf'].transform(test_df['code'][test_df.id==10489]).toarray()[0]
mylist=sorted(classifier['tfidf'].vocabulary_, key=classifier['tfidf'].vocabulary_.get)
f=np.argsort(clf_pred)[::-1]
myslice=f[:50]
clf_pred[myslice]
Out[ ]:
In [ ]:
np.array(mylist)[myslice]
Out[ ]:
In [ ]:
from sklearn.feature_selection import mutual_info_classif
mf=mutual_info_classif(test_vect.transform(train_df['code']),train_df.target)
In [ ]:
test_vect.vocabulary_[np.argsort(mf)[::-1]]
In [ ]:
predictor.leaderboard()
Out[ ]:
Content
Comments
You must login before you can post a comment.