spark_practice

OS: Ubuntu 19.04, requirement software: spark 2.4.3, python3.7.3, jupyter notebook
required python packages: requirements.txt

$ pip install -r requirements.txt
edit $~/.bashrc file as below

export PATH=$SPARK_HOME/bin:$PATH

export PYSPARK_PYTHON=python3

export PYSPARK_DRIVER_PYTHON=jupyter

export PYSPARK_DRIVER_PYTHON_OPTS='notebook'
execute pyspark

$ pyspark

About Data directory

We use 'data' directory in spark distribution source file.

It needs to adjust actual 'data' directory path in example codes.

Name		Name	Last commit message	Last commit date
Latest commit History 27 Commits
LICENSE		LICENSE
MLlib_pactic_tf_idf.ipynb		MLlib_pactic_tf_idf.ipynb
MLlib_pactice_Collabrative_Filtering.ipynb		MLlib_pactice_Collabrative_Filtering.ipynb
MLlib_pactice_LDA.ipynb		MLlib_pactice_LDA.ipynb
MLlib_pactice_PCA.ipynb		MLlib_pactice_PCA.ipynb
MLlib_pactice_k_means.ipynb		MLlib_pactice_k_means.ipynb
MLlib_pactice_libsvm.ipynb		MLlib_pactice_libsvm.ipynb
MLlib_pactice_logistic_regression.ipynb		MLlib_pactice_logistic_regression.ipynb
MLlib_pactice_pipeline.ipynb		MLlib_pactice_pipeline.ipynb
MLlib_practice_svd.ipynb		MLlib_practice_svd.ipynb
RDD_Practice_01.ipynb		RDD_Practice_01.ipynb
RDD_Practice_ACCUMULATE_04.ipynb		RDD_Practice_ACCUMULATE_04.ipynb
RDD_Practice_FuncTest_03.ipynb		RDD_Practice_FuncTest_03.ipynb
RDD_Practice_REDUCEBYKEY_05.ipynb		RDD_Practice_REDUCEBYKEY_05.ipynb
RDD_Practice_SeqTest_02.ipynb		RDD_Practice_SeqTest_02.ipynb
README.md		README.md
SECURITY.md		SECURITY.md
SIMPLE_EXAMPLE.ipynb		SIMPLE_EXAMPLE.ipynb
SQL_Practice_01.ipynb		SQL_Practice_01.ipynb
SQL_Practice_InferSchema_02.ipynb		SQL_Practice_InferSchema_02.ipynb
STREAM_SQL_Practice_01.ipynb		STREAM_SQL_Practice_01.ipynb
STREAM_SQL_Practice_02.ipynb		STREAM_SQL_Practice_02.ipynb
STREAM_SQL_Practice_03.ipynb		STREAM_SQL_Practice_03.ipynb
graphx_pagerank.ipynb		graphx_pagerank.ipynb
requirements.txt		requirements.txt
wordcloud.ipynb		wordcloud.ipynb
wordcloud_answer.ipynb		wordcloud_answer.ipynb