Skip to content

Problem with load test #267

@MohammedElGharawy

Description

@MohammedElGharawy

After generating test data using PDGF tool and placing the .dat files under $DATA_DIR/SF
what should be the next step to run the benchmark?

running the load test generates .parquet files but when i try to run the benchmark's queries after
i get the following errors

Connected!
Pure Dask Queries
run 1: q01
Encountered Exception while running query
raise ValueError(
ValueError: The following columns were not found in the dataset {'i_category_id', 'i_item_sk'}
The following columns were found Index([], dtype='object') 

Output from load test (SF100)


Connected!
Converting warehouse of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting promotion of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting store of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting customer_demographics of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting product_reviews of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting web_clickstreams of 60 GB to 30 parquet files, chunksize: 128 MiB
Converting web_sales of 44 GB to 22 parquet files, chunksize: 128 MiB
Converting web_page of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting date_dim of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting reason of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting income_band of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting customer of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting item_marketprices of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting web_returns of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting web_site of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting store_returns of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting item of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting inventory of 34 GB to 17 parquet files, chunksize: 128 MiB
Converting household_demographics of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting time_dim of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting customer_address of 0 GB to 1 parquet files, chunksize: 128 MiB
Converting store_sales of 30 GB to 15 parquet files, chunksize: 128 MiB
Converting ship_mode of 0 GB to 1 parquet files, chunksize: 128 MiB
Load test with chunk size of 128 MiB took 537.89s 

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions